Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

Abstract: Reinforcement learning with verifiable rewards without human-annotated data, often referred to as zero RL, has emerged as a powerful paradigm for eliciting chain-of-thought reasoning. However, due to computational constraints, existing studies are largely restricted to small models, leaving the training dynamics and emergent capabilities at a large scale unexplored. To meaningfully explore this frontier, we aim to elicit high-quality reasoning behaviors from the model. However, we find that naive scaling often suffers from poor readability, token redundancy, and a lack of adaptive reasoning depth. To address these challenges, we present a stable and efficient training pipeline, incorporating algorithmic and system optimizations such as clipped importance sampling, training-inference ratio correction, and mixed-precision control. Our experiments offer three key findings that validate the "bitter lesson" of scaling: (1) scaling to 1T parameters significantly enhances sample efficiency and performance ceilings; (2) the training process progresses sequentially through an initial discovery phase followed by a sharpening phase; and (3) the model spontaneously develops advanced cognitive behaviors, including anthropomorphism, structured formatting, self-verification, parallel reasoning, and context anxiety, rendering hand-crafted heuristics redundant. Evaluated on seven mathematical benchmarks, Ring-2.5-1T-Zero achieves competitive performance. Additionally, to assess CoT quality beyond final-answer correctness, we propose a structured evaluation framework across three dimensions: comprehensibility, reproducibility, and efficiency, where our model demonstrates clear advantages in producing structured and concise reasoning traces. By sharing our observed emergent phenomena, we hope to provide the community with deeper insights into scaling behaviors, particularly at the 1-trillion scale.

• The paper demonstrates that scaling Zero RL to a trillion parameters triggers emergent cognitive behaviors, validating that sheer scale can replace handcrafted heuristics.
• It introduces a three-stage RL pipeline with interleaved self-distillation and adaptive reasoning depth to optimize chain-of-thought quality and inference efficiency.
• Empirical results show enhanced sample efficiency and improved comprehensibility on challenging math benchmarks, supporting the viability of large-scale RL approaches.

Scaling Zero-RL to Trillion-Parameter Models for Emergent Reasoning

Introduction

"Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning" (2607.12395) establishes a stable, minimalist pipeline for reinforcement learning with verifiable reward (RLVR) at unprecedented scale, applying pure Zero RL directly to a 1T-parameter Mixture-of-Experts (MoE) model. This work investigates the dynamics of RL-based chain-of-thought (CoT) reasoning at the trillion parameter frontier, substantiating the central hypothesis that scale alone is sufficient to elicit advanced cognitive behaviors and surpass human-engineered heuristics. The paper also introduces a structural evaluation framework for CoT quality beyond answer accuracy, proposing metrics for comprehensibility, reproducibility, and efficiency.

Stable Zero-RL Pipeline at Scale

The framework comprises three main RL stages interleaved with self-distillation, engineered for both algorithmic simplicity and infrastructural robustness.

1. First Stage RL – Reasoning Elicitation: Bootstraps reasoning chains from the base model using clipped importance sampling policy gradient, KL-regularization, and token-level loss, together with training-inference ratio correction to eliminate logit mismatch across distributed engines.
2. Self-Distillation – Compression and Stabilization: Shortens lengthy, redundant traces using self-evaluation and trimming, resets accumulated numerical errors, and fine-tunes the base model on the concise expert traces for stabilization.
3. Second Stage RL – Sustained Optimization: Continues RL on distilled foundations, switching to sample-level loss normalization and removing KL penalties for greater exploration without excessive length growth.
4. Third Stage RL – Adaptive Reasoning Depth: Introduces tier-based training, partitioning questions into difficulty buckets with corresponding prompt and window size for flexible compute allocation and adaptive response depths.

Infrastructure optimizations include mixed-precision control in attention and LM head (FP32) to address numerical instability from exponentiation in softmax operations, and context parallelism using all-to-all communication for Lightning/MLA attention modules, ensuring throughput and precision at long context lengths.

Figure 1: The Ring-2.5-1T-Zero pipeline: multi-stage RL, self-distillation, adaptive windowing, and emergent behaviors enabled by scale and minimal algorithmic changes.

Empirical Results and Scaling Phenomena

Ring-2.5-1T-Zero demonstrates strong performance on seven mathematical reasoning datasets, achieving competitive pass@1 accuracy across AIME, HMMT, and IMOAnswerBench benchmarks. Crucially, scaling from 104B to 1T parameters amplifies sample efficiency and unlocks higher capability ceilings, clearly evidenced by widened performance gaps on hard problems.

The training process divides into two sequential regimes:

• Discovery Phase: RL actively expands the reasoning boundary, evidenced by growth in pass@1024 as the model discovers new latent strategies.
• Sharpening Phase: Distribution refinement increases pass@1 without further boundary expansion, confirming coexistence of discovery and sharpening rather than a binary debate.

Zero RL further obviates the need for curated CoT traces or handcrafted heuristics, with the model autonomously developing advanced strategies such as anthropomorphic reasoning, structured phase segmentation, self-verification, parallel exploration, and context-aware truncation ("context anxiety"). These emerge solely from the RL objective, with no external supervision.

Figure 2: Pass@1 accuracy across benchmarks demonstrating competitive performance against state-of-the-art models at trillion-param scale.

Structural Evaluation of Reasoning Quality

The paper extends evaluation beyond answer correctness via a three-dimensional framework:

• Comprehensibility: LLM-as-a-Judge pairwise comparison reveals Ring-2.5-1T-Zero’s traces exhibit superior logical flow, explicit causal dependencies, and minimal hallucinations compared to baselines (GLM-5.1, Kimi-k2.6, MiniMax-M2.7, Qwen3.5).
• Reproducibility: Student models distilled from Ring-2.5-1T-Zero traces outperformed those distilled from DeepSeek-R1, achieving +5.8% and +4.5% gains for Qwen-32B and Llama-70B respectively with a fraction of the sample count, evidencing richer transferable skills in step-segmented traces.
• Efficiency: On mutually solved problems, Ring-2.5-1T-Zero produces reasoning traces with less than half the average token length of baselines, optimizing inference latency and token cost.

Figure 3: Comprehensibility comparison against strong baselines shows dominant win rates for Ring-2.5-1T-Zero.

Figure 4: Sequence length reduction and high efficiency of reasoning traces under the proposed evaluation framework.

Ablations and System Design Insights

Ablation studies validate several key design decisions:

• Algorithm Choice: CISPO and DAPO effectively stimulate reasoning from scratch by boosting low-probability tokens but require stabilization via ratio correction and KL regularization to prevent entropy collapse.
• Stabilization Strategies: Training-inference ratio correction addresses floating-point discrepancies across distributed engines, eliminating the need for expensive recomputation and threshold tuning. KL regularization prevents runaway divergence and preserves exploratory entropy.
• Format Reward, Length Inertia: Strict structural format constraints and sample-level loss normalization are critical to preventing uncontrolled token redundancy and length inertia, ensuring both correctness and efficiency.

Figure 5: All-failed group ratio comparison across RL algorithms, highlighting instability of gradient amplification methods without safeguards.

Figure 6: Learning rate, rollout group, and loss reduction ablation indicating robust convergence properties and efficiency tradeoffs.

Emergent Cognitive Behaviors

Scaling pure RL to 1T parameters unlocks sophisticated emergent behaviors:

• Anthropomorphic Reasoning: Traces simulate frustration, slack, and self-praise, capturing informal meta-commentary reminiscent of human forum data.
• Structured Formatting: The model autonomously organizes step-by-step reasoning with explicit segmentation, enhancing pedagogical clarity.
• Parallel Reasoning: Rollout traces evaluate competing solution strategies, resembling multi-agent tree-of-thought search.
• Context Anxiety: As token limits approach, the model strategically aborts deductive chains for heuristic guesses, demonstrating learned reward maximization under resource constraints.
• Self-Verification: Intermediate results are actively cross-checked and confirmed, replicating rigorous human sanity checks without template engineering.

Figure 7: Log-probability difference and stability metrics for ratio correction strategies.

Practical and Theoretical Implications

Practically, this pipeline democratizes trillion-scale RL training by lowering engineering barriers: minimal modifications suffice, provided ratio correction and targeted precision controls are implemented. Theoretically, it empirically validates the "bitter lesson"—computation and scale consistently outperform intricate human heuristics, even for advanced reasoning synthesis.

The model’s emergent behaviors imply that RL objectives at scale may be sufficient to induce not merely token-level policy changes but qualitatively novel action spaces and agentic strategies. However, improvements in curriculum design, context length scaling, and pretraining data diversity remain critical for further boundary expansion.

Speculation and Future Developments

Future advances are likely to hinge on:

• Maximizing sample-efficient, curriculum-driven RL for challenging reasoning distributions
• Joint optimization of reasoning quality and inference cost within unified RL objectives
• Scaling context windows to enable more complex, multi-staged derivations
• Investigating RL-induced emergent agentic behaviors in other domains beyond mathematics
• Integrating RL-native models into downstream agentic pipelines requiring structured, reproducible reasoning

Conclusion

Ring-2.5-1T-Zero demonstrates that pure RL with verifiable reward, applied at sufficient scale, is capable of eliciting complex reasoning and advanced cognitive behaviors in LLMs without human-annotated traces or handcrafted heuristics. The minimalist pipeline, enabled by careful numerical and algorithmic design, achieves both competitive quantitative performance and highly structured, efficient reasoning trajectories. These findings suggest that RL-based scalable reasoning will continue to benefit from parameter scaling, efficient system architectures, and adaptive agentic evaluation metrics, providing foundational capabilities for AI-driven scientific and mathematical workflows.