R-Zero: Autonomous Self-Evolving LLM

• R-Zero is a self-evolving large language model framework where a Challenger generates tasks and a Solver learns to answer them through reinforcement learning.
• It employs a model–model curriculum that produces pseudo-labels via majority voting and filters challenges with uncertainty rewards and repetition penalties.
• Empirical results demonstrate significant gains in mathematical and general-domain reasoning benchmarks using a scalable GRPO optimization strategy.

R-Zero is a fully autonomous self-evolving LLM framework that advances reasoning capabilities by generating its own training data from scratch—without any external task curation or human-provided labels. The core architecture is built around two co-evolving components, the Challenger and the Solver, which are independently optimized and interact in a closed-loop via reinforcement learning. The Challenger formulates tasks that lie at the Solver’s capability boundary, while the Solver is iteratively adapted to solve these tasks. This model–model curriculum produces continual, capability-targeted self-improvement, empirically yielding substantial gains in both mathematical and general-domain reasoning benchmarks. R-Zero thus provides a new paradigm for scalable, curriculum-driven LLM training with neither pre-existing datasets nor external supervision (Huang et al., 7 Aug 2025).

1. Autonomous Self-Evolving Framework

R-Zero is initialized from a single base LLM, producing two separate instances with different roles: the Challenger, which synthesizes challenging questions, and the Solver, which learns to answer them. Both models co-evolve—after the Challenger generates a batch of candidate questions, each is scored by the Solver, and pseudo-labels are formed via majority voting over multiple Solvers’ outputs. The central innovation is that all tasks, labels, and reward signals are derived entirely from within the system, establishing a data-generating and curriculum-sculpting loop uninfluenced by any pre-existing human-curated resources.

The high-level workflow is as follows:

Component	Role	Optimization
Challenger	Synthesizes new questions	RL (GRPO)
Solver	Answers and learns from them	RL/SL (GRPO, MC label)

This architecture allows via interaction (not teacher-forced or externally seeded) the emergence of tasks that cluster tightly around the Solver’s current capabilities—automatically calibrating curriculum difficulty.

2. Self-Improving Curriculum Generation and Filtering

The Challenger generates a pool of candidate questions, which are filtered and scored through a multi-stage process:

• Pseudo-labelling: For each task $x$, the Solver is sampled $m$ times, yielding output set $S_\phi(x) = \{y_1, ..., y_m\}$. The most frequent output is chosen as the pseudo-label $\hat{y}(x)$.
• Uncertainty Reward: To maximize learning signal, the reward for each question is designed to emphasize those near the decision boundary of the Solver:

$$\hat{p}(x; S_\phi) = \frac{1}{m} \sum_{j=1}^m \mathbf{1}[y_j = \hat{y}(x)] \ r_{\text{uncertainty}}(x; \phi) = 1 - 2 \left| \hat{p}(x; S_\phi) - \frac{1}{2} \right|$$

This reward assigns maximum value to questions for which the Solver is maximally uncertain ($\hat{p}=0.5$), incentivizing the Challenger to expose the Solver’s "learning edge."

• Repetition Penalty: To prevent collapsed or redundant queries, a penalty proportional to pairwise BLEU similarity within the batch is applied.
• Format Check: Tasks violating basic syntactic rules are filtered to ensure only valid, answerable data is used.

All surviving question–pseudo-label pairs are added to the evolving training set, enabling progressively harder and more diverse learning opportunities.

3. Co-Evolution and Optimization via Group Relative Policy Optimization

Both the Challenger and the Solver are optimized alternately via Group Relative Policy Optimization (GRPO), a variant of policy gradient RL that normalizes rewards within response groups for stability.

The core GRPO update for a group $\{x_i\}_{i=1}^G$ with reward vector $\{r_i\}_{i=1}^G$ uses the z-score normalized advantage

$$\hat{A}_i = \frac{r_i - \text{mean}(r_1,\ldots,r_G)}{\text{std}(r_1,\ldots,r_G) + \epsilon}$$

and adapts the policy using a clipped surrogate loss similar to Proximal Policy Optimization (PPO):

$$\mathcal{L}(\theta) = \mathbb{E}_i\left[ \min \left( \frac{\pi_\theta(x_i)}{\pi_{\text{old}}(x_i)} \hat{A}_i, \ \text{clip}(\frac{\pi_\theta(x_i)}{\pi_{\text{old}}(x_i)}, 1-\varepsilon, 1+\varepsilon)\hat{A}_i \right) \right] - \alpha \text{KL}( \pi_\theta || \pi_{\text{old}} )$$

The Challenger's composite reward is formulated as a weighted combination of uncertainty reward, repetition penalty, and a format check, while the Solver's reward is binary—whether its output matches the pseudo-label.

Critically, the Challenger and Solver are updated in alternation with the opposing model frozen, ensuring stable curriculum advancement and avoiding degenerate competitive cycles. The Challenger demands more skill, driving the Solver, which in turn becomes a moving target for the Challenger.

4. Empirical Results on Mathematical and General-Domain Reasoning

Extensive experiments demonstrate the empirical effectiveness of the R-Zero framework:

• On mathematical reasoning tasks (AMC, GSM8K, Olympiad), Qwen3-4B-Base improves by +6.49 points after three rounds of self-evolution. Larger models (Qwen3-8B-Base) exhibit similar trends.
• On general-domain benchmarks (MMLU-Pro, SuperGPQA), progressive gains are recorded: e.g., Qwen3-8B-Base increases from 49.18 to 54.69 average score after iterative self-evolving training.

Ablation analysis reveals that removing RL training for the Challenger, disabling the repetition penalty, or omitting the filter mechanisms each cause significant regressions in downstream performance.

Model	Math Gain (+)	General Gain (+)
Qwen3-4B-Base	6.49	7.54
Qwen3-8B-Base	Similar	5.51

Performance improvements are driven by the evolving, self-generated curriculum, as the Challenger persistently generates questions at the current capacity threshold of the Solver, leading to maximum learning efficiency.

5. Theoretical Principles and Mechanism Analysis

R-Zero’s reward formulation embodies an explicit curriculum learning principle by maintaining the average challenge level at the Solver’s "learning edge" (empirically, this is the point where the Solver’s success rate is near 50%). This property is grounded in information-theoretic analysis: tasks for which the Solver’s uncertainty (as measured by the empirical accuracy on a generated question) is maximized yield the greatest expected KL divergence between model output distributions before and after learning.

The use of repetition penalties combats question collapse and ensures curriculum diversity, while strict format checking prevents the accumulation of noisy or syntactically invalid tasks.

Importance is placed on majority voting for pseudo-labeling—since no ground truth is available, and self-generated answers may be noisy or ambiguous, only responses with a clear majority (neither too trivial nor too ambiguous) are retained, preserving a "Goldilocks" regime for effective curriculum construction.

6. Impact, Scalability, and Future Implications

By eschewing all external tasks and labels, R-Zero demonstrates that LLMs can drive their own, open-ended reasoning development—a property essential for the pursuit of scalable superintelligent systems. The self-evolving, model–model training loop guarantees perpetual curriculum advancement tailored to the system’s current ability, and may unlock progress beyond the ceiling of human-constructed benchmarks.

Potential scaling benefits include autonomous expansion into new problem areas without human curation, constant adaptation to previously unseen domains, and the possibility of self-discovery of currently unknown problem types. In practical terms, this could substantially reduce the dependence on costly and limited human annotation pipelines in the evolution of general reasoning AI.

7. Technical Challenges and Mitigations

Challenges inherent in a self-evolving curriculum paradigm include maintaining the reliability of pseudo-labels (addressed by majority voting and filtering), preventing degenerate learning cycles where Solver and Challenger "game" each other, and stabilizing RL optimization in non-stationary environments. The Group Relative Policy Optimization (GRPO) strategy and careful alternation of optimization mitigate instability, while composite reward shaping and filtering maintain curriculum validity and informativeness. Ablation studies confirm the necessity of each mechanism for robust performance gains.

---

R-Zero introduces a fully self-improving LLM training architecture in which an RL-optimized Challenger and an RL/SL-trained Solver co-evolve in a closed-loop, label-free paradigm, producing and mastering new reasoning tasks without external data. Empirical results demonstrate substantial improvements in both mathematical and general-domain reasoning capabilities across diverse backbone LLMs, suggesting this methodology as a robust path for scalable autonomous superintelligence development (Huang et al., 7 Aug 2025).