PopuLoRA: Co-Evolutionary RL for LLMs

• PopuLoRA is a population-based, asymmetric self-play framework for reinforcement learning with verifiable rewards, enhancing out-of-distribution reasoning and code generation in LLMs.
• It co-evolves teacher and student LoRA adapters on a frozen LLM backbone using evolutionary operators and population-based training to maintain diversity and efficiency.
• Benchmark results show notable improvements over baseline LoRA on code and math tasks, demonstrating effective curriculum learning and model specialization.

PopuLoRA is a population-based, asymmetric self-play framework designed for reinforcement learning with verifiable rewards (RLVR) in the post-training phase of LLMs. It specializes in the co-evolution of “teachers” and “students”—each implemented as a LoRA (Low-Rank Adaptation) adapter on a shared, frozen LLM backbone (Qwen2.5-Coder-7B)—to expand out-of-distribution reasoning and code generation capabilities. Teachers propose problems for students, whose solutions are validated by a programmatic verifier, with population-level cross-evaluation and explicit role separation replacing the self-calibration that burdens single-agent self-play. PopuLoRA incorporates a family of weight-space evolution operators for adapter parameter mutation and crossover, underpinning a population-based training (PBT) regime that efficiently produces novel, diverse capabilities (Castanyer et al., 16 May 2026).

1. Asymmetric Self-Play and Framework Structure

PopuLoRA deploys two distinct populations: teachers and students, both realized as LoRA adapters of a shared, frozen base LLM. Each adapter contains only the updated LoRA matrices ($A,B$) with fixed rank ($r=32$), and all target the same projections (q_proj, k_proj, v_proj, o_proj), ensuring parameter isolation across population members while maintaining computational efficiency.

Teachers generate three types of “Absolute Zero Reasoner” (AZR) code problems: infer-input, infer-output, and infer-function. Each teacher-student pair is dynamically matched using population-based TrueSkill ratings, with teachers tasked to propose problems and students tasked to solve them under automated programmatic evaluation. Solutions are run through a sandboxed Python executor and format checker, which assigns outcome categories—correct, incorrect but well-formed, or format-error—serving as ground-truth verifiable signals rather than learned or reward-model-based proxies.

2. Reinforcement Learning with Verifiable Rewards

PopuLoRA formalizes both teacher and student incentives entirely in terms of results from the verifier $V$:

• Student reward for a rollout $\tau$ on prompt $p$:

$$R_\text{stu}(\tau,p) = \begin{cases} +1 & V(\tau,p)\text{ is correct} \ -0.5 & V(\tau,p)\text{ is incorrect but well-formed} \ -1 & V(\tau,p)\text{ is format error} \end{cases}$$

• Teacher reward for prompt $p$ attempted by a student $s$ with $n$ samples and solve rate $\rho(t,s,p)$:

$r=32$0

Teachers are incentivized to propose problems that are neither trivial nor unsolvable, as their reward is maximized when exactly some students fail.

Policy optimization is conducted with REINFORCE++ (a critic-free gradient-based policy optimization variant) with per-prompt centering, global whitening, and no KL penalty. Mixed-type batches (code_i/code_o/code_f) are used consistently for baseline and population variants.

3. Weight-Space Evolution and Population-Based Training

Evolutionary population management constitutes a central element of PopuLoRA. Each adapter’s LoRA layers are parameterized as $r=32$1, with a suite of specialized mutation and crossover operators for rapid generation of new population members.

Mutation operators:

• SVD-structured: Perturb singular values via reweighting and rotate singular vectors via Cayley transforms.
• Layer-selective Gaussian: Add Gaussian noise selectively to 33% of adapter modules.
• Component masking: Zero a random fraction of singular values, forcing partial function ablation.
• Full Gaussian: Apply Gaussian noise to all LoRA parameters.

Crossover operators:

• DARE drop-and-rescale: Apply masking and averaging across parent factors.
• Layer-wise recombination: Per-module selection from parents.
• SVD subspace mix: Pool top singular vectors from each parent.
• Extrapolative task arithmetic: Linear interpolation/extrapolation in parameter space.

Every $r=32$2 steps, adapters in each population are ranked by the TrueSkill lower-confidence bound ($r=32$3); the lowest-performing 25% are culled and replaced by new population members generated from the top-ranked via the above operators, efficiently maintaining diversity and exploratory pressure.

4. Training Loop and Co-Evolutionary Dynamics

PopuLoRA’s training proceeds in discrete steps:

1. Pairing: Teachers and students are matched using Prioritized Fictitious Self-Play (PFSP) optimizing for balanced TrueSkill gaps.
2. Problem generation: Teachers sample prompt batches; only valid, deterministic prompts are retained.
3. Solution and verification: Students perform multiple rollouts per problem, and the verifier determines correctness.
4. Policy update: REINFORCE++ gradient updates are applied per adapter over aggregate megabatch trajectories.
5. Population evolution: At regular intervals, bottom performers are replaced with children from the top using mutation/crossover in LoRA parameter space.

This co-evolution triggers an arms-race: as teacher difficulty increases, student solve rates oscillate, and teachers adapt to defeat the improving students, continually expanding coverage and complexity.

5. Experimental Outcomes

PopuLoRA was instantiated on a Qwen2.5-Coder-7B base with LoRA rank=32, $r=32$4, $r=32$5 by default and ablations at $r=32$6. Baselines used matched compute with a single LoRA adapter under single-agent AZR self-play. Pass@1 evaluation was performed on both code (HumanEval+, MBPP+, LiveCodeBench) and math (AIME24/25, AMC23, MATH-500, Minerva, GSM8K, OlympiadBench) benchmarks using greedy decoding after merging the adapter into the base LLM.

Benchmark Results

Benchmark	PopuLoRA (4T+4S)	Baseline LoRA
HumanEval+	78.7%	76.8%
MBPP+	70.8%	69.3%
LiveCodeBench	27.0%	17.3%
AIME24	11.7%	6.7%
AMC23	40.0%	37.5%
MATH-500	69.8%	59.4%
Minerva	27.0%	19.1%
OlympiadBench	33.9%	27.3%

PopuLoRA consistently outperformed the baseline on all code and math benchmarks except GSM8K (where both performed similarly). Aggregate math accuracy reached 38.2% versus 34.1% for the baseline, with all improvements exceeding $r=32$7 across adapters. Notably, even the lowest-performing population member exceeded the baseline aggregate (43.6% vs. 42.3%), and population size ablations confirmed that role separation alone (1T+1S) yields improvement, while larger populations (8T+8S) show incremental oscillatory gains.

6. Curriculum Dynamics and Population Specialization

PopuLoRA displays pronounced arms-race oscillations between teacher difficulty and student achievement, unlike the monotonic collapse to triviality observed in single-agent self-play. Student solve rates and teacher difficulty alternate out of phase; as teachers propose harder problems, student performance drops, generating gradients that push for improved capability, which then forces teachers to further escalate problem complexity.

Complexity and exploration diversity are quantitatively tracked across four code metrics (AST depth, cyclomatic complexity, line count, variable count); PopuLoRA populations trend upward in all measures. Problem-space coverage, tabulated as occupancy of a 4096-cell archive based on these metrics, surpassed 75% under PopuLoRA compared to less than 20% for the baseline.

Further, TrueSkill means and variances diverge among adapters (rather than collapsing to homogeneity), and cross-evaluation matrices indicate strong matchup-dependent specialization, with non-uniform solve rates for students against different teachers.

7. Insights, Limitations, and Extensions

PopuLoRA’s lower average training reward is not an indicator of underperformance; rather, it reflects a nontrivial, ever-evolving curriculum as opposed to the trivial self-calibrated tasks of single-agent self-play. The combination of structural role separation, weight-space recombination, and co-evolution continually forces population members to adapt, resulting in the emergence of previously unseen capabilities and broader problem coverage.

Open avenues include relaxing the fixed-rank constraint for LoRA adapters to promote further diversity, extending to less-capable verifier modules or non-code domains such as symbolic reasoning or math graders, and scaling to larger populations with distributed architectures or sharded validator pools. Scaling beyond 16 adapters currently increases communication and memory overhead; architectural improvements are required for efficient scaling beyond current hardware limits.

In summary, PopuLoRA demonstrates that population-based co-evolution with verifiable inter-agent rewards and efficient LoRA evolution operators yields robust automatic curricula and enhanced reasoning performance beyond what is accessible with single-agent self-play (Castanyer et al., 16 May 2026).