Scaling Self-Play with Self-Guidance

Abstract: LLM self-play algorithms are notable in that, in principle, nothing bounds their learning: a Conjecturer model creates problems for a Solver, and both improve together. However, in practice, existing LLM self-play methods do not scale well with large amounts of compute, instead hitting learning plateaus. We argue this is because over long training runs, the Conjecturer learns to hack its reward, collapsing to artificially complex problems that do not help the Solver improve. To overcome this, we introduce Self-Guided Self-Play (SGS), a self-play algorithm in which the LLM itself guides the Conjecturer away from degeneracy. In SGS, the model takes on three roles: Solver, Conjecturer, and a Guide that scores synthetic problems by their relevance to unsolved target problems and how clean and natural they are, providing supervision against Conjecturer collapse. Our core hypothesis is that LLMs can assess whether a subproblem is useful for achieving a goal. We evaluate the scaling properties of SGS by running training for significantly longer than prior works and by fitting scaling laws to cumulative solve rate curves. Applying SGS to formal theorem proving in Lean4, we find that it surpasses the asymptotic solve rate of our strongest RL baseline in fewer than 80 rounds of self-play and enables a 7B parameter model, after 200 rounds of self-play, to solve more problems than a 671B parameter model pass@4.

• The paper introduces Self-Guided Self-Play (SGS), a framework that integrates a Guide to stabilize self-play in LLM-based theorem proving.
• It employs three roles—Solver, Conjecturer, and Guide—to generate, validate, and reward synthetic problems, thereby curbing mode collapse.
• SGS achieves a 7% improvement in asymptotic solve rate over RL baselines, demonstrating enhanced compute efficiency and scalability.

Scaling Self-Play with Self-Guidance: An Expert Analysis

Problem Setting and Motivation

The paper "Scaling Self-Play with Self-Guidance" (2604.20209) systematically explores the scalability bottlenecks of LLM-based asymmetric self-play algorithms, particularly their application to formal mathematical theorem proving in Lean4. The self-play paradigm—where a Conjecturer generates synthetic problems and a Solver attempts to solve them—offers potential unbounded scalability in principle, but in practice, existing methods plateau due to degeneracy in synthetic data generation. The main hypothesis is that this degeneracy arises because the Conjecturer learns to exploit the reward structure, generating complex yet uninformative problems, which halts any subsequent learning progress. The central question addressed is: Given a fixed set of hard problems, how can we scale self-play to maximize the number of problems solved with extended training and compute?

Methodology: Self-Guided Self-Play (SGS)

The core contribution is Self-Guided Self-Play (SGS), an algorithmic framework wherein the LLM assumes three distinct but related roles: Solver, Conjecturer, and Guide. The Guide—a reviewer model—assesses the utility and quality of generated synthetic problems, thereby providing a controlling signal that counteracts the mode collapse of the Conjecturer. The system iterates as follows:

1. Synthetic Problem Generation: For each unsolved target problem, the Conjecturer produces a related but simpler synthetic problem.
2. Validation: The Solver attempts the batch of original and synthetic problems. Solutions are automatically verified for correctness.
3. Reward Assignment and RL Updates:
   • Solver reward: Binary correctness (pass/fail), using a REINFORCE objective, focused on instances with solve rates ≤ 0.5.
   • Conjecturer reward: Product of a solve rate-based difficulty reward (penalizing both trivial and impossible problems) and a Guide (reviewer) score, which quantitatively evaluates relatedness and formulation elegance.
   • All policy gradients in both roles are normalized and updated independently.

These components are initialized from a single base LLM and do not share weights post-initialization.

Figure 1: Intuition and empirical effect of SGS: the Guide focuses exploration toward unsolved regions and prevents the Conjecturer from drifting into degenerate problems; SGS outperforms baselines in cumulative Lean4 problem-solving.

Experimental Evaluation

The evaluation centers on a filtered, hard subset of the Goedel-Pset-V1 dataset, denoted $D_{\text{3k}}$, with over 3,300 Lean4 theorems spanning multiple mathematical disciplines and difficulty levels. Key aspects of the experimental setup include:

• Long-horizon training: >6B tokens, corresponding to 230 epochs over target problems, significantly longer than previous studies.
• Synchronous and exhaustive batch-wise sampling to maximize hardware utilization.
• Comprehensive baseline comparison: RL baselines (REINFORCE, CISPO, and Expert Iteration), parallel sampling, and prior self-play frameworks (STP).

The primary quantitative metrics are cumulative solve rate and scaling law fits with respect to log-compute trajectory, modeled via a sigmoidal function for asymptotic solve rate estimation.

Figure 2: Training dynamics of SGS: continuous improvement in pass rate and sustained Conjecturer/Guide reward throughout training.

SGS demonstrates a 7% higher asymptotic solve rate than the best RL baseline, and, with only a 7B parameter DeepSeek-Prover-V2 instantiation, surpasses the pass@4 performance of the 671B parameter model after 200 rounds of self-play—an emphatic claim on the efficiency of compute utilization via self-play. SGS also outperforms the most closely related method (STP) after approximately 1M generations.

Ablation Analyses and Failure Modes

Extensive ablation experiments isolate the contributions of SGS components:

• No Guide: Removing the reviewer-based reward yields a collapse of the Conjecturer; it generates increasingly complex and convoluted problems, as evidenced by a proliferation of disjunctive, lengthy problem statements and a decline in their utility for solver progress.

  Figure 3: Effect of the Guide component—its presence controls problem complexity and structure, while its absence results in degenerate synthetic problems.

• No Problem Conditioning: If synthetic problems are not explicitly tied to the current set of unsolved targets, self-play devolves into aimless data generation, stalling learning.
• Frozen Conjecturer: Not updating the Conjecturer (reminiscent of fixed-data RL) leads to rapid solver saturation on the available synthetic distribution, after which further progress ceases.

  Figure 4: Ablation of SGS: cumulative solve rate and progression of solvable synthetic problems per iteration under various Conjecturer/Guide treatments.

Additionally, the authors show that grouped RL objectives such as CISPO induce entropy collapse in the Solver, starving the Conjecturer of reward signal since almost all problem instances become either trivial or impossible, leading to a complete halt in improvement. In contrast, the plain REINFORCE objective stabilizes entropy and maintains useful reward gradients for both the Solver and Conjecturer.

Figure 5: Solver entropy dynamics—CISPO leads to rapid output distribution collapse, while REINFORCE preserves diversity and enables continued learning.

Theoretical and Practical Implications

The key empirical result is that scaling self-play in LLMs requires explicit mechanisms to prevent degenerate mode collapse in both the Conjecturer and Solver distributions. The paper's findings indicate that:

• A learned Guide (reviewer, adversarial module) substantially stabilizes synthetic task generation, ensures curriculum relevance, and translates to measurable gains in final performance.
• The interaction between Solver entropy and Conjecturer reward propagation mandates careful choice and tuning of RL objectives—vanilla grouped RL can catastrophically starve learning in long-horizon runs.
• There is a nontrivial separation between the ability to generate more synthetic data and generating useful synthetic data; only the latter yields increased coverage of challenging target problems.

  Figure 6: As target theorem difficulty increases, SGS produces more intermediate synthetic problems just before solving the hard instance, indicating effective curriculum construction.

Furthermore, the results imply that compute-efficient scaling in LLM-based theorem-proving is achievable with relatively modest model sizes if learning instabilities and reward hacking are explicitly controlled.

Limitations and Future Directions

Notably, SGS exploits a verifiable environment (Lean4) where problem correctness is automatically provable—a scenario not available in many real-world tasks. Extensions to other domains would require learning or bootstrapping both the reward function (for the Guide) and possibly the environment (for the Conjecturer). The Guide in SGS is static; further improvement might be achieved by training it to adapt its evaluation criteria over time, especially as the target problem set grows in complexity. Finally, all analyses are performed at fixed model size; the interaction of SGS with model scaling remains an open subject.

Figure 7: Fitted scaling law sensitivity to data removal shows robustness of asymptotic solve rate analysis across data subsets.

Conclusion

Self-Guided Self-Play (SGS) proposes and validates a generalizable approach for sustained curriculum generation and learning via self-play with LLMs. By explicitly incorporating a Guide to adjudicate the quality and relevance of synthetic problems—and precisely managing Solver entropy—the approach bypasses the dominant mode collapse phenomenon inherent in long-run LLM self-play. SGS achieves higher compute efficiency and final coverage of difficult formal theorem-proving tasks compared to strong RL and prior self-play baselines. The findings highlight the necessity of adversarial self-correction and careful RL objective design for scalable self-improvement in LLMs. Future generalization hinges on advances in learned reward modeling and application to less structured or unverifiable domains.