SPICE: Self-Play In Corpus Environments Improves Reasoning (2510.24684v1)

Abstract: Self-improving systems require environmental interaction for continuous adaptation. We introduce SPICE (Self-Play In Corpus Environments), a reinforcement learning framework where a single model acts in two roles: a Challenger that mines documents from a large corpus to generate diverse reasoning tasks, and a Reasoner that solves them. Through adversarial dynamics, the Challenger creates an automatic curriculum at the frontier of the Reasoner's capability, while corpus grounding provides the rich, near-inexhaustible external signal necessary for sustained improvement. Unlike existing ungrounded self-play methods that offer more limited benefits, SPICE achieves consistent gains across mathematical (+8.9%) and general reasoning (+9.8%) benchmarks on multiple model families. Our analysis reveals how document grounding is a key ingredient in SPICE to continuously generate its own increasingly challenging goals and achieve them, enabling sustained self-improvement.

• The paper introduces SPICE, a corpus-grounded self-play framework that addresses LLM hallucination and information symmetry by creating genuine information asymmetry.
• It employs a dual-role mechanism where the Challenger generates document-based questions and the Reasoner answers them, driving an adversarial curriculum with tailored reward functions.
• Experimental results show consistent improvements in both mathematical and general reasoning benchmarks across multiple model families, confirming SPICE’s scalability and transferability.

SPICE: Self-Play In Corpus Environments Improves Reasoning

Motivation and Problem Statement

The SPICE framework addresses the limitations of ungrounded self-play in LLMs, specifically hallucination amplification and information symmetry. In traditional self-play, both the problem generator and solver share the same internal knowledge, leading to trivial or repetitive tasks and compounding factual errors. SPICE introduces corpus grounding, where a large external document corpus serves as the environment for generating and verifying reasoning tasks. This approach creates genuine information asymmetry and ensures factual accuracy, enabling sustained self-improvement in LLMs.

Framework Overview

SPICE employs a single LLM, $\pi_\theta$, that alternates between two roles:

• Challenger (C): Samples a document $d \sim \mathcal{D}$ and generates a question-answer pair $(q, a^*)$ grounded in the document. The Challenger is rewarded for producing tasks with maximal variance in Reasoner success rates, targeting the frontier of model capability.
• Reasoner (R): Receives only the question $q$ and attempts to answer it without access to the document. The Reasoner is rewarded for correct answers, verified against the document-extracted gold answer.

This adversarial dynamic, coupled with corpus grounding, creates an automatic curriculum and prevents stagnation.

Figure 1: Reasoner pass rates when evaluating SPICE checkpoints at steps 200-640 against a fixed step-200 checkpoint on 128 documents. (a) Fixed Reasoner: Pass rate decreases from 55% to 35% as later Challenger checkpoints generate harder questions. (b) Fixed Challenger: Pass rate increases from 55% to 85% as later Reasoner checkpoints improve at solving questions.

Implementation Details

Task Generation and Verification

• Document Sampling: Uniformly sample from a large, diverse corpus (e.g., Nemotron-CC-Math, NaturalReasoning).
• Multi-Format Tasks: Challenger generates either MCQs (with plausible distractors) or free-form questions (integer, expression, string answers), guided by document analysis.
• Answer Verification: Use Math-Verify for mathematical equivalence and exact matching for other types.

Reward Functions

• Challenger Reward: Gaussian-shaped function of Reasoner pass rate variance, peaking at 50% success rate to maximize learning potential.
• Reasoner Reward: Binary correctness.

  Figure 2: Comparison of challenger reward functions as a function of reasoner pass rate $p$. The threshold reward (rollout size 8) assigns 0 for trivial (8/8) and impossible (0/8) tasks, 1 otherwise. Absolute Zero (1-p) linearly decreases with pass rate. R-Zero (1-2|p-0.5|) peaks at maximum uncertainty. SPICE's variance-based reward (dashed) uses $$\exp(-\frac{(\sigma^2 - \sigma^2_{\text{opt}})^2}{2 \cdot 0.01})$$.

Training Procedure

• Distributed Actor-Learner Architecture: Actors alternate between Challenger and Reasoner roles, generating trajectories for policy updates.
• Optimization: DrGRPO with role-specific advantages, centering returns per role to avoid difficulty-induced noise.
• Batching: 128 documents per iteration, 8 Reasoner samples per question, 640 training iterations.

Experimental Results

SPICE demonstrates consistent improvements over baselines (Base Model, Strong Challenger, R-Zero, Absolute Zero) across four model families (Qwen3-4B/8B, OctoThinker-3B/8B):

• Qwen3-4B-Base: +9.1% overall gain
• Qwen3-8B-Base: +5.7% overall gain
• OctoThinker-3B-Hybrid-Base: +10.5% overall gain
• OctoThinker-8B-Hybrid-Base: +11.9% overall gain

Improvements span both mathematical (+8.9%) and general reasoning (+9.8%) benchmarks.

Adversarial Dynamics and Curriculum Emergence

SPICE's adversarial co-evolution is evidenced by:

• Challenger: Generates increasingly difficult questions, reducing Reasoner pass rate from 55% to 35% (fixed Reasoner).
• Reasoner: Improves at solving questions, increasing pass rate from 55% to 85% (fixed Challenger).

This dynamic ensures the curriculum remains at the edge of the model's capability, driving sustained learning.

Figure 1: Reasoner pass rates illustrate the adversarial curriculum: Challenger increases difficulty, Reasoner adapts and improves.

Ablation Studies

Corpus Composition

Combining Nemotron-CC-Math and NaturalReasoning yields the best overall performance, with each corpus contributing domain-specific strengths.

Task Type

Mixing MCQ and free-form questions outperforms either format alone, balancing reliable verification and flexible reasoning.

Reward Strategies

Variance-based reward (SPICE) outperforms Absolute Zero, Threshold, and R-Zero, confirming that optimal learning occurs at balanced task difficulty.

Qualitative Analysis

SPICE's Challenger evolves from generating surface-level questions to multi-step, synthesis-based tasks. The Reasoner transitions from intuitive guessing to structured, step-by-step reasoning, demonstrating authentic capability growth.

Implementation Considerations

• Computational Requirements: Training requires substantial GPU resources (e.g., 8 H200 GPUs), efficient inference (vLLM), and distributed architecture.
• Scaling: Corpus size and diversity are critical; larger, more varied corpora yield broader reasoning improvements.
• Deployment: SPICE can be integrated into existing RLHF/RLVR pipelines, with corpus mining and answer verification modules as plug-ins.

Limitations and Future Directions

• Corpus Quality: Performance depends on the quality and diversity of the document corpus; noisy or narrow corpora may limit gains.
• Task Verification: While Math-Verify and exact matching suffice for current benchmarks, more sophisticated verifiers may be needed for open-ended domains.
• Generalization: Further work is needed to assess transfer to domains with less structured knowledge or ambiguous verification.

Theoretical and Practical Implications

SPICE demonstrates that external grounding via corpus mining is essential for sustained self-improvement in LLMs. The framework generalizes beyond mathematics and code, enabling broad reasoning capability development without human supervision. The adversarial curriculum mechanism provides a scalable path for autonomous capability growth, suggesting new directions for self-improving AI systems.

Conclusion

SPICE introduces a corpus-grounded self-play RL paradigm that overcomes the limitations of ungrounded self-play in LLMs. By leveraging external document corpora and adversarial co-evolution between Challenger and Reasoner roles, SPICE achieves consistent, transferable improvements in reasoning across diverse domains. This approach opens new avenues for autonomous, scalable self-improvement in AI, with implications for both theoretical understanding and practical deployment of advanced reasoning systems.