GASP: Guided Asymmetric Self-Play For Coding LLMs

Abstract: Asymmetric self-play has emerged as a promising paradigm for post-training LLMs, where a teacher continually generates questions for a student to solve at the edge of the student's learnability. Although these methods promise open-ended data generation bootstrapped from no human data, they suffer from one major problem: not all problems that are hard to solve are interesting or informative to improve the overall capabilities of the model. Current asymmetric self-play methods are goal-agnostic with no real grounding. We propose Guided Asymmetric Self-Play (GASP), where grounding is provided by real-data goalpost questions that are identified to pose a hard exploration challenge to the model. During self-play, the teacher first generates an easier variant of a hard question, and then a harder variant of that easier question, with the goal of gradually closing the gap to the goalpost throughout training. Doing so, we improve pass@20 on LiveCodeBench (LCB) by 2.5% over unguided asymmetric self-play, and through the curriculum constructed by the teacher, we manage to solve hard goalpost questions that remain out of reach for all baselines.

• The paper introduces GASP, a teacher-guided lemma-lift framework that aligns synthetic tasks with real unsolved coding challenges.
• It employs a dual-stage curriculum with rejection sampling to maintain task diversity and progressively enhance learning capability.
• Empirical results on LiveCodeBench demonstrate improved pass@1 and pass@20 metrics, outperforming traditional unguided self-play methods.

Guided Asymmetric Self-Play for Coding LLMs: Formal Summary and Technical Analysis

Motivation and Context

Advances in post-training strategies for code-oriented LLMs increasingly exploit self-play frameworks, where a teacher model generates problems intended to reside at the student’s learnability frontier. While asymmetric self-play can facilitate synthetic curriculum generation without ongoing human intervention, standard forms lack task grounding, making it difficult for the training process to focus on genuinely useful or generalizable capabilities. Many "hard" synthetic tasks may be trivialized, infeasible, or irrelevant for downstream code generation, leading to inefficient or unstable improvement.

GASP (Guided Asymmetric Self-Play) (2603.15957) addresses these issues by aligning synthetic question creation to a curated set of unsolved, real-world problems—termed "goalposts"—thereby grounding the construction of curricula in well-defined, high-value targets.

Figure 1: Overview of GASP. The teacher generates an easier variant (lemma) and then a harder variant (lift) for each real-data goalpost, incrementally pushing the model's knowledge boundary.

Algorithmic Framework and Curriculum Construction

In the GASP framework, the pipeline comprises three major stages per iteration:

1. Goalpost Definition: A rigorous filtering pipeline isolates a set of real-data coding questions that are persistently unsolvable by both standard RL with verifiable rewards (RLVR) and prior self-play baselines (AZR). On LiveCodeBench (LCB), this results in 146 "goalposts" from a 601-question training split.
2. Teacher-Guided Curriculum Generation:
   • Lemma Generation: The teacher conditions on a goalpost $h$ and is prompted to synthesize a simplified variant $\ell_0$ (lemma) encapsulating the core challenge of $h$ but with reduced complexity. Admission is subject to empirical learnability: pass rates must fall into a moderate, non-trivial interval.
   • Lift Generation: The teacher then generates a more challenging variant $\ell_1$ (lift), conditioned on $\ell_0$ but not directly on $h$, aiming to further bridge toward the original goalpost difficulty.
3. Solver Update: The student is trained on a mixed batch consisting of both lemma and lift tasks, using RLVR with explicit pass/fail evaluation.

To enforce continual novelty and prevent mode collapse, rejection sampling with embedding-based similarity metrics is applied, ensuring the diversity of generated tasks.

Figure 2: Dissimilarity of proposed lemma and lift questions. GASP with diversity-based rejection sampling avoids sample collapse, maintaining task diversity throughout training.

This structured two-step curriculum (lemma $\rightarrow$ lift) provides intermediate, pedagogically meaningful stepping stones, facilitating progressive capability expansion toward otherwise inaccessible problem classes.

Figure 1: Each training round iteratively brings generated tasks closer in difficulty to the true goalpost, as the model’s knowledge region expands.

Reward Structuring and Teacher Optimization

Teacher rewards are engineered around a parameterized learnability curve, peaking at pass rates optimal for learning signal:

• Lemma reward: Maximum at $p=0.5$ to target "just-learnable" tasks.
• Lift reward: Peaked at $p=0.1$, focusing on hard, low-success-rate tasks but still potentially informative.

For both, out-of-band candidates (too easy, too hard, invalid, or insufficiently novel) are strongly penalized. This framework is inspired by recent insights into regret-based and learnability-driven curriculum discovery.

Empirical Results: Benchmarking and Ablation

Performance on LiveCodeBench

GASP was evaluated on LiveCodeBench (LCB) v5, comparing against:

• Baseline Qwen2.5-Coder-7B (untuned)
• Real-data RL (standard RLVR on the whole real-data split)
• AZR (Absolute Zero, prior unguided asymmetric self-play)
• GASP + Real-data RL (joint synthetic and real task training)

Strong numerical claims:

• GASP improves pass@20 on LCB v5 by 2.5% over AZR
• On the target LCB v5 benchmark, GASP achieves a pass@1 of up to 19.1 and pass@20 of up to 33.7, consistently outperforming AZR and closely tracking real-data RL.
• GASP + Real-data RL further elevates pass@1 and pass@20 by 1-2 points (see Figure 3).

  Figure 3: Pass@k performance on LCB v5. GASP and GASP + Real-data RL consistently outperform unguided self-play (AZR) and the Qwen2.5-Coder-7B base model. GASP’s advantage is pronounced at higher sample counts (pass@20).

Ablation analyses highlighted the importance of rejection sampling and both curriculum axes (input/output complexity, function complexity). Disabling rejection sampling resulted in higher performance variance and task diversity collapse:

Figure 4: Pass@k for GASP without rejection sampling. Increased variance and diminished consistency demonstrate the necessity of diversity enforcement.

Figure 2: Without rejection sampling, generated tasks become increasingly similar, inhibiting progressive learning.

Finally, limiting the difficulty axis to function modifications (removing I/O scaling) led to reduced goalpost coverage and greater seed variance:

Figure 5: Restricting the teacher to only function-axis difficulty scaling increases variance and limits the breadth of solved goalposts.

Goalpost Solving and Curriculum Progress

A core metric is the set of originally unsolved "goalpost" tasks that GASP enables the model to solve as training progresses. AZR and RLVR baselines cannot solve any of these (by construction), whereas GASP achieves non-trivial success:

• GASP solved 11 of 146 hard goalposts across three seeds.
• GASP + Real-data RL solved a similarly-sized set with more temporal persistence.

Temporal analysis demonstrates that new goalposts are solved intermittently as training advances, corroborating the hypothesis that carefully constructed curricula can unlock genuinely new behaviors.

Figure 6: Goalpost coverage throughout GASP training. Green cells indicate previously unsolved tasks now solved at least once by the GASP-trained model.

Qualitative Analysis: Task Trajectories

Examples illustrate how GASP constructs lemma-lift curricula that preserve key problem motifs while spanning a graded complexity continuum. For instance, grid navigation with adjacency constraints is simplified to classic reachability, then extended back with new constraints, aligning the curriculum to the real-data problem’s abstract structure.

Theoretical and Practical Implications

The GASP methodology demonstrates several critical implications:

• Grounded Curriculum Quality: By anchoring curriculum generation to unsolved human data, the synthetic self-play loop remains targeted and efficient, avoiding drift into irrelevant or adversarial examples.
• Sample Efficiency and Model Robustness: Enforced task diversity, via hard rejection metrics, increases sample efficiency and robustness, limiting overfitting to narrow or redundant task subclasses.
• Complementarity: GASP is competitive with, and complementary to, direct real-data RL, indicating its deployment value both as a standalone post-training technique and as an augmentation to conventional pipeline elements.
• Frontier Advancement: The framework advances the capability boundary on genuinely unsolved tasks, rather than merely increasing routine benchmark scores.

Limitations and Open Questions

• Stepping Stone Quality: Teacher LLMs may lack sufficient abstraction for optimal intermediate problem synthesis, tending to select surface-level motifs or trivial constraint escalations.
• Static Goalposts: The fixed goalpost set may become misaligned as the model improves; online updating or dynamic selection could further enhance adaptation.
• Alignment Metrics: Absent explicit measures of lemma-lift alignment to goalpost semantics, there is no guarantee that stepping stones always provide optimal conceptual scaffolding.

Conclusion

GASP establishes a formal framework for guided curriculum generation in asymmetric self-play for coding LLMs, providing a structured mechanism for synthetic data creation tethered to real-data hard cases. The paradigm outperforms unguided self-play, rivals direct RLVR training, and actively unlocks previously inaccessible problem domains. Future directions should focus on dynamic goalpost maintenance, better reward shaping for abstract alignment, and scaling up to broader codebases or multimodal task spaces.