Propel: PopuLoRA Rocket-Launcher Scene Test

We introduce PopuLoRA, a population-based asymmetric self-play framework for reinforcement learning with verifiable rewards (RLVR) post-training of LLMs. Teachers and students are specialised LoRA adapters on a shared frozen base: teachers propose problems, matched students solve them under a programmatic verifier, and cross-evaluation between sub-populations replaces the self-calibration that limits single-agent self-play. A family of LoRA weight-space evolution operators (mutations and crossovers that produce same-rank population members in seconds) serves as the replacement step of a population-based training loop at 7B scale. We instantiate PopuLoRA on top of Absolute Zero Reasoner and compare it against a per-adapter compute-matched single-agent baseline. Where the single agent self-calibrates to generating easy problems it can reliably solve, the population enters a co-evolutionary arms race: teachers produce increasingly complex problems, student solve rates oscillate, and problem-space coverage keeps expanding throughout training. Despite lower training-time reward, the population outperforms the baseline on HumanEval+, MBPP+, LiveCodeBench, and seven math benchmarks including AIME, MATH-500, and OlympiadBench, with an average relative improvement of 15% across all ten benchmarks.

RL post-training has become a dominant regime for specialising large language models, from RLHF ¹ and DPO on human preference data through self-play fine-tuning ² and large-scale reinforcement learning with verifiable rewards (RLVR) ^{3 4}. The optimisation machinery has matured, but the supply of problems has not: most current systems depend on a hand-curated task distribution whose scope, difficulty, and coverage must be chosen in advance. The open question we study in this paper is how to generate the curriculum itself, the stream of problems the policy is trained on, without relying on human-authored datasets, using only a programmatic verifier as the external signal.

In short: population-based self-play is what we use to propel that curriculum, with no human-authored dataset in the loop.

The most direct approach lets a single model propose its own problems and grade itself through the verifier, as in Absolute Zero Reasoner ⁵ or, more broadly, single-model self-play ⁶. The shared flaw is that the same network generates problems and (implicitly, through its solve rate or judgement) estimates their difficulty. In this paper we provide empirical evidence that such single-agent self-generation self-calibrates: the proposer converges to generating problems it can consistently produce in valid format and consistently solve, and the training distribution collapses onto a narrow band long before the base model's capability is exhausted. The fix is structural: make the judge a different agent from the proposer. A growing line of work explores this asymmetry ^{7 8 9 10 11 12 13 14}, but all are scalar in agent count (3 agents or fewer total), whether the roles share parameters or train separate networks. We take the asymmetry further, building on the teacher-proposes / student-solves structure introduced by Sukhbaatar et al. ¹⁵, and replace the single agent with co-evolving populations of specialised teachers and students. Teachers are rewarded for producing problems that are hard for the particular student they face, students are rewarded by the verifier, and matchmaking across the two sub-populations turns difficulty into a population-level signal rather than a self-estimate. Population dynamics add a second layer of exploration on top of gradient updates: lineages diverge, members specialise, and a population-based training (PBT) ¹⁶ replacement step recombines what the gradient path has already discovered.

Running many independent full-parameter 7B models on a single node is expensive and memory-constrained, particularly when each member must support both rollout inference and gradient updates in a shared training loop. We therefore instantiate every population member as a LoRA adapter ¹⁷ over a shared frozen base, which collapses the population's memory footprint to the sum of its adapter weights rather than a full copy of the base per member. Classical PBT mutates members by copying one agent's full weights onto another and perturbing; at 7B scale that copy is itself expensive. Our second contribution is a set of LoRA weight-space evolution operators (SVD-structured, layer-selective, and component-masking mutations together with DARE ¹⁸, TIES-inspired ¹⁹, and task-arithmetic ²⁰ crossovers, in the spirit of evolutionary model merging ²¹) that produce same-rank children in seconds without any retraining. They serve as the replacement step of an online PBT loop, making a population-of-adapters regime that is inaccessible to prior LoRA composition work ^{22 23 24 25}. We evaluate in the Absolute Zero code-reasoning setting, with a sandboxed Python executor as the verifier.

1. Population-based asymmetric self-play for RLVR. Prior work is scalar in agent count (3 agents or fewer). PopuLoRA replaces these with populations of teacher and student LoRA adapters coupled by TrueSkill-weighted cross-evaluation.
2. LoRA weight-space evolution operators as the PBT replacement step. Mutations and crossovers produce rank-matched children in seconds; closest is evolutionary model merging , applied offline rather than inside an online PBT loop.
3. Empirical validation on held-out code and math benchmarks. The population outperforms a per-adapter compute-matched single-agent baseline; diagnostics confirm it avoids the mode-collapse we observe in the baseline.

RLVR and self-generated curricula. RLVR replaces learned preference models with programmatic checkers. AZR takes this to its logical extreme: a single code LLM both proposes and solves its own problems, rewarded only by a sandboxed executor, with no external dataset. Adjacent methods (STaR ²⁶, rStar-Math ²⁷, Self-Rewarding ²⁸) also train on self-generated data but rely on fixed problem sets or learned reward signals; only AZR treats the model as both proposer and verifier-checked solver. We evaluate PopuLoRA on the three code-reasoning task types AZR defines: code_i (infer-input), code_o (infer-output), and code_f (infer-function), all verified mechanically by the executor.

Self-play and asymmetric roles. Asymmetric self-play , where one agent proposes tasks and another solves them, is the structural ancestor of PopuLoRA's teacher-student loop. In the unsupervised environment design (UED) literature, PAIRED ²⁹ formalises regret-based adversarial curriculum generation, POET ³⁰ co-evolves environments and agents, and ACCEL ³¹ adds mutation operators on level structure; emergent autocurricula arise in multi-agent competition ³². In the LLM setting, a growing line of work separates proposer and solver: SPIN , Language Self-Play , SOAR , R-Zero , ALIVE , TriPlay-RL , and others. All are scalar in agent count (3 agents or fewer total). PopuLoRA differs in three ways: (i) we train populations of teachers and students rather than a single pair, (ii) updates are joint and on-policy rather than alternating, and (iii) the difficulty signal comes from cross-evaluation across the population rather than from a fixed target solve-rate band.

Population-based training and LoRA evolution. Classical PBT copies and perturbs full agent weights; at 7B scale the per-member footprint and the copy-and-perturb cost both become bottlenecks. Recent work lifts evolution into adapter space (GENOME ³³, EGGROLL ³⁴, ESSA ³⁵, ES-at-Scale ³⁶) but optimises against a fixed fitness function. PopuLoRA is orthogonal: evolution serves as the replacement step of an RLVR self-play loop, where the fitness signal is produced by the population itself through cross-evaluation. We embed LoRA merge operators (DARE , TIES-inspired , task arithmetic , plus SVD-structured mutations) inside this online loop, so children re-enter gradient training immediately after recombination, unlike offline merging .

PopuLoRA keeps AZR's core self-play loop but replaces the single agent with a population of specialised LoRA adapters and replaces self-calibrated difficulty with cross-evaluation between matched members. The following figure illustrates the architecture.

Each population member optimises its policy by RLVR: a verifier V (a sandboxed Python executor plus a format checker) emits a scalar reward on every rollout, with no learned reward model in the loop. For a student rollout on a problem p proposed by its matched teacher:

For a teacher-proposed problem p subsequently attempted by student s:

where rho(t, s, p) is the fraction of the student's rollout samples that solve p. The zero-reward case when no student solves the problem prevents teachers from being rewarded for generating impossible or degenerate problems. The key structural change relative to single-agent AZR is exactly in this equation: the teacher's reward depends on the matched student's failure rate, not on the proposer's own solve rate, so difficulty is an inter-population quantity rather than a self-estimate. Advantages are estimated with REINFORCE++ ³⁷-baseline (per-prompt centring followed by global whitening across the batch), in the critic-free GRPO ³⁸ family descended from PPO ³⁹, without a value network and without a KL penalty to a reference model. Every policy update pools all three AZR problem types (code_i, code_o, code_f) into a single mixed-type batch per member per step, matching the single-agent baseline's pooling.

The population consists of N_T teacher and N_S student LoRA adapters attached to a single shared frozen code-LLM base. Every adapter has the same rank r and attaches to the same set of projection matrices; only the adapter weights are updated, while the base model remains frozen. This layout has two immediate consequences. First, memory cost scales with the sum of adapter sizes rather than the sum of full-parameter base copies, which is what makes multi-adapter populations affordable on commodity hardware: a base that is tens of gigabytes serves arbitrarily many adapters, each costing only tens of megabytes. Second, the vLLM ⁴⁰ multi-LoRA scheduler ⁴¹ can dispatch each request to the correct adapter by tag inside a shared forward pass, so matched teachers and students generate and solve in the same batch without any per-request base-model swap, and swapping an adapter in or out of the rollout engine moves only the adapter delta, not the base.

Each training step proceeds in five phases.

(1) Matchmaking: each teacher is paired with one student via prioritised fictitious self-play (PFSP) ⁴² over TrueSkill ⁴³ ratings, concentrating pairings on informative near-balanced matchups. PopuLoRA adapts PFSP from game-playing agents to problem-proposing teachers: opponents are teachers generating problems rather than policies playing the same game.

(2) Teacher generation: each teacher proposes a batch of AZR code problems (split across code_i / code_o / code_f), validated by the sandboxed executor.

(3) Student solve: the matched student attempts all valid problems under rollout; the executor produces a per-prompt binary solve vector.

(4) Cross-evaluation and update: the teacher's reward on each valid prompt is 1 - rho (the matched student's solve rate), replacing AZR's self-estimation with an inter-population signal. All adapters are updated in a single mixed batch.

(5) Evolution: every k steps, the bottom fraction gamma of each sub-population (ranked by TrueSkill lower-confidence bound) is replaced by children produced via LoRA weight-space operators applied to top-ranked parents.

The design goal is fast child creation: each operator has to emit a same-rank LoRA adapter in seconds without any retraining, retain enough parent knowledge that the child can re-enter training immediately, and inject enough diversity to move the child away from its parents in weight space. We implement two families of operators: mutations (M1--M4) acting on a single parent and crossovers (X1--X4) combining two parents.

Mutations. Let delta-W = BA^T denote a single LoRA layer's effective update.

• M1 (SVD-structured) takes the singular-value decomposition, perturbs the singular spectrum by a small multiplicative noise, and applies a first-order Cayley rotation to U and V, preserving orthogonality while moving the child off the parent's subspace.
• M2 (layer-selective Gaussian) draws a random 33% subset of the layer-module slots and adds Gaussian noise to A, B only within that subset; the remaining two-thirds of the adapter are copied verbatim.
• M3 (component masking) zeroes a random subset of the SVD components, reducing effective rank and forcing the child to relearn masked directions.
• M4 (full Gaussian) adds per-tensor adaptive Gaussian noise to every A, B factor, with the noise scale set by the tensor's own running standard deviation.

Crossovers. Let delta-W(1) and delta-W(2) be two parents.

• X1 (DARE) applies the DARE drop-and-rescale recipe to each parent's delta then sums them.
• X2 (layer-wise) selects each layer-module slot independently from either parent, giving a layer-modular recombination reminiscent of TIES-style sign alignment .
• X3 (SVD subspace) takes the top-k singular components from one parent and fills the remaining r-k components from the other, mixing parents within a common rank budget.
• X4 (extrapolative) performs a task-arithmetic-style linear combination with a coefficient greater than one, extrapolating beyond the convex hull of the parents rather than interpolating between them.

All eight operators run in seconds on the adapter tensors alone, produce children at the same rank as the parents, and are validated as retention-preserving.

We evaluate PopuLoRA against a per-adapter compute-matched single-agent AZR baseline on held-out code and math benchmarks, plus training-dynamics and problem-complexity diagnostics, a population-dynamics analysis, a LoRA-operator retention test, and a population-size ablation. All comparisons share the same base model, reward scheme, and optimiser; the only difference is whether training runs one self-calibrating agent or a teacher-student population.

Instantiation. We use a frozen Qwen2.5-Coder-7B ⁴⁴ base, the same model used by AZR, which enables a direct comparison against their publicly released checkpoint. Every population member is a rank-32 LoRA adapter with scaling alpha = 64 attached to the attention projections (W_q, W_k, W_v, W_o). The main population is N_T = N_S = 4. Training runs for 200 update steps with AdamW at learning rate 5e-5 and REINFORCE++-baseline advantages (rollout n = 8, temperature T = 1, no KL penalty). Each matchup generates a batch of 72 prompts split equally across code_i, code_o, code_f; the student solves the same split. Evolution runs every k = 10 steps on the bottom fraction gamma = 0.25 of each sub-population ranked by TrueSkill lower-confidence bound.

Evaluation. At evaluation time we merge each final adapter into the frozen base with PEFT and report greedy pass@1 on (i) code benchmarks HumanEval+ ^{45 46}, MBPP+ ⁴⁷ , and LiveCodeBench v5 ⁴⁸, and (ii) math benchmarks AIME 24/25, AMC 23, MATH-500 ^{49 50}, Minerva ⁵¹, GSM8K ⁵², and OlympiadBench ⁵³. For the population we report both the per-benchmark best and the population mean for each role.

The following results report greedy pass@1 across three code and seven math benchmarks, comparing the population's mean and best teacher and best student against the per-adapter compute-matched baseline.

The population is at or above the per-adapter compute-matched baseline on every code benchmark: best student 78.7 vs baseline LoRA 76.8 on HumanEval+, 70.9 vs 69.3 on MBPP+, and 27.2 vs 17.3 on LiveCodeBench. The largest gap sits on LiveCodeBench, which contains recent competitive-programming problems from online contests and is therefore structurally different from the self-generated code triples AZR trains on.

Despite AZR providing no direct math supervision (its verifier is a Python executor on code triples, not a math grader), PopuLoRA shows consistent out-of-domain gains. The best population member improves on the baseline LoRA on every math benchmark: 13.3 vs 6.7 on AIME 24, 42.5 vs 37.5 on AMC 23, 70.0 vs 59.4 on MATH-500, 28.3 vs 19.1 on Minerva, and 33.9 vs 27.3 on OlympiadBench, with an average math gain of 40.3 vs 34.1. Both arms improve over the frozen base, but the population's margin is larger on every math benchmark, with the biggest absolute gaps on the competition-level benchmarks (AIME, OlympiadBench) where the baseline barely moves from the base model.

Notably, the mean teacher and student already outperform the baseline on most benchmarks; the best-of selection only adds a significant margin on math benchmarks (particularly AIME and AMC), where the population was never directly trained, suggesting that individual adapters specialise in complementary out-of-domain reasoning skills.

The 4T+4S Worst adapter (43.6% aggregate) still outperforms the baseline (42.3%), indicating that co-evolution lifts the entire population rather than concentrating gains in a few members.

The training dynamics of the baseline and the population tell two qualitatively different stories.

In the single-agent baseline, the solver's solve rate rises monotonically and plateaus within the first 200 steps. A near-perfect solve rate is not the right signal here: it means the teacher is unable to generate problems hard enough to make the student fail. The single agent simultaneously learns to produce valid problems in the correct format and to solve everything it generates. There is no increasing complexity, no co-adaptation, no pressure to improve further: the system has found a stable fixed point where both roles are satisfied with minimal effort. ⁵⁴

The population's dynamics are strikingly different. Student solve rates oscillate throughout training rather than monotonically rising. This pattern has a natural explanation: as teachers co-adapt and generate harder problems, students start failing; once students catch up, teachers are pushed to produce yet harder problems, and the cycle repeats. This phasic dynamic is the signature of a genuine co-evolutionary arms race rather than a self-calibrating fixed point.

Crucially, the population's seemingly lower training-time solve rate is a sign of strength. When we evaluate the end-of-training checkpoints on held-out benchmarks, they consistently outperform the baseline whose training curve looked near-perfect. The population's teachers kept pushing difficulty upward, which forced the students to develop capabilities the baseline never needed.

The per-type breakdown reveals that the oscillation runs at different frequencies across task types: code_o cycles rapidly between teacher- and student-dominated phases, while code_i and code_f oscillate more slowly, consistent with these tasks requiring deeper program understanding to sustain difficulty pressure. The baseline's policy entropy collapses to near zero while the population's teachers maintain non-trivial entropy throughout training, and the population's response length grows to approximately 1000 tokens versus the baseline's approximately 250, consistent with more elaborate reasoning .

The training-dynamics story is corroborated by direct measurements of problem complexity. We track four structural metrics of teacher-generated programs over training: AST depth, cyclomatic complexity, lines of code, and variable count.

The difference is clear. In every metric the baseline curves trend downward: the single-agent teacher learns to produce progressively simpler programs along every axis, converging on the simplest programs it can consistently generate in valid format and solve. The population teachers show the opposite trajectory: all four complexity metrics rise throughout training. Cross-evaluation rewards teachers for problems that are hard for the matched student, which creates sustained upward pressure on problem difficulty rather than the downward drift of self-calibration.

Problem-space coverage tells the same story: we tile the structural feature space with a CVT archive ⁵⁵ of 4,096 cells and track the fraction filled over training. Baseline coverage plateaus early; the population keeps expanding through 200 steps, generating increasingly diverse problems alongside increasingly complex ones. By step 100 the baseline has collapsed to programs as trivial as return number * 3.

The TrueSkill ratings provide a direct view of the co-evolutionary dynamics. Three features stand out.

Individual adapters differentiate from the population mean as training progresses. Early on, all members cluster near the prior mu = 25; by mid-training, distinct high and low performers have emerged in both sub-populations, indicating that the population dynamics produce genuine specialisation rather than homogeneous copies.

The arms race between the two roles shows an oscillating pattern. There are periods where teachers dominate --- their problems are too hard for the matched students --- but students eventually catch up, and the cycle repeats. This alternating lead is a characteristic of adversarial co-evolution: neither role is able to settle into a fixed strategy because the other keeps adapting. Per-student solve-rate profiles against different teachers are non-uniform, confirming that specialisation extends to the matchup level.

All eight operators that ship in the live population pass the retention test: every child recovers to parent-level reward within 10--20 update steps after being re-injected into training.

The mutation results confirm that perturbed children start close to their parent and resume gradient updates without resetting to the frozen base, validating the operators as a legitimate PBT replacement step. The crossover results combine parents trained on different AZR task types (e.g., one specialised on induction and the other on output prediction). The crossover child retains performance on both parents' tasks, demonstrating that weight-space recombination can compose complementary specialisations into a single adapter.

The population size ablation reveals that the co-evolutionary dynamics we observe are not simply a consequence of having two roles --- they require a population.

Even at the smallest population size, 1T+1S, decoupling the teacher and student into separate adapters is enough to avoid the baseline's mode collapse: the solver reward does not plateau at near-perfect levels, and downstream evaluation shows that 1T+1S already outperforms the single-agent baseline on most benchmarks. The oscillation pattern is less pronounced than in larger populations, but the structural separation of proposer and solver prevents the self-calibrating fixed point that limits single-agent training.

At 4T+4S the co-evolutionary oscillations become clearly visible and the reward dynamics show the phasic arms race described above. This is the configuration used for the main comparison. At 8T+8S the oscillations are more pronounced, though the downstream gains depend on the benchmark.

PopuLoRA replaces single-agent self-calibration with co-evolving teacher-student LoRA populations on a shared frozen base. Where the single-agent baseline converges to a fixed point of easy problems and near-perfect reward, the population sustains an arms race that produces increasingly complex problems and stronger downstream performance on every code and math benchmark we measured. Even the weakest member of the 4T+4S population outperforms the baseline on aggregate, indicating that co-evolution lifts the entire population rather than concentrating gains in a few specialists. Even a minimal 1T+1S configuration, which decouples proposer and solver without population dynamics, already improves over the baseline, confirming that structural asymmetry is the primary driver.

Compute cost. Per-adapter compute matches the baseline: same tokens, rollouts, and updates per step. Total work scales with adapter count, but wall-clock scales sub-linearly because vLLM multi-LoRA batching shares the frozen base forward pass . On the same 1x8xH100 node, 4T+4S trains 8x more adapters for only 1.31x the baseline wall-clock.

What LoRA evolution does not replace. Evolution operators recombine and perturb existing adapters; new behaviour still comes from policy-gradient updates after injection.

Limitations. We fix the LoRA rank to 32 across all population members; heterogeneous ranks could further diversify the population but we have not explored this axis. All experiments use a single base model (Qwen2.5-Coder-7B-Instruct) and a single verifier domain (sandboxed Python execution), so it remains open whether the co-evolutionary dynamics transfer to other base scales or to domains with weaker verifiers. Code will be released upon publication.

All 17 operators implemented in the benchmark: six mutations, nine crossovers, and two identity controls. Every operator consumes one or two rank-r LoRA state dicts and emits a rank-matched child in seconds, without any retraining.

• M1 (SVD-structured mutation): Preserve learned structure while perturbing it within the singular-value basis. Perturbs the singular spectrum multiplicatively with log-normal noise and applies first-order Cayley near-identity rotations to both U and V.
• M2 (layer-selective Gaussian): Uniformly samples a fraction f of LoRA module slots without replacement and adds per-tensor adaptive Gaussian noise to their A and B factors. The remaining 1-f of modules are copied verbatim. Defaults: epsilon = 0.1, f = 0.33.
• M3 (component masking): SVDs the effective delta, samples random singular indices, zeroes the corresponding singular values, and refactors. Default rho = 0.3. The dropped directions are uniformly random, not bottom-k.
• M4 (full Gaussian): Adds per-tensor adaptive Gaussian noise to every A and B in the adapter. Default epsilon = 0.15.
• M5 (NEFTune-style): Dimension-aware uniform perturbation on the input factor only, adapted from NEFTune's ⁵⁶ embedding-space noise.
• M6 (rank perturbation): Structured rank collapse plus fine-grained jitter on the survivors, inspired by DyLoRA ⁵⁷.

• X1 (DARE): Drop-and-rescale recipe applied independently to each parent's factors, then averaged.
• X2 (layer-wise crossover): For each LoRA module slot, a coin flip selects (A, B) entirely from parent 1 or parent 2. Preserves intra-module coherence but breaks inter-module coherence.
• X3 (SVD subspace crossover): Mixes principal and secondary singular subspaces from two parents within a common rank budget.
• X4 (extrapolative): Linear combination past parent 2 along the parent difference vector, analogous to task arithmetic with a coefficient beyond one.
• X5 (task-arithmetic linear merge): Tensor-wise convex combination. Default alpha = 0.5.
• X6 (TIES merge): Sign-aligned merge¹⁹: trim, elect consensus sign, disjoint merge.
• X7 (DELLA): Magnitude-proportional drop with DARE-style rescale, inspired by DELLA-Merging ⁵⁸.
• X8 (SLERP): Spherical linear interpolation on the flattened-tensor vector. Falls back to linear interpolation when norms are near-zero.
• X9 (Fisher-weighted merge): Data-free Fisher-weighted averaging ⁵⁹ using a squared-weight proxy for the diagonal Fisher.

Only the first subset (M1, M2, M3, M4, X1, X2, X3, X4) is active inside the live population. The remaining nine operators are characterised in isolation via the benchmark but did not enter the main self-play loop.

To illustrate the mode-collapse phenomenon, we pair one baseline-generated and one population-generated problem from matched training steps:

Step 5 -- Baseline AZR (mixed):

AST depth 9, cyclomatic 3.0, 6 LOC

Step 5 -- PopuLoRA Teacher (infer-input):

AST depth 9, cyclomatic 2.0, 8 LOC

Step 100 -- Baseline AZR (mixed):

AST depth 5, cyclomatic 1.0, 2 LOC

Step 100 -- PopuLoRA Teacher (infer-function):

AST depth 8, cyclomatic 3.0, 6 LOC

By step 100 the baseline has collapsed to return number * 3, while the population is still producing programs with branching and state.