Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short

Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a leading paradigm for improving the reasoning ability of LLMs through outcome-based supervision. However, verifiable rewards frequently become uninformative at the group level: when all sampled traces of a given prompt receive identical rewards, group-relative advantage estimation provides no gradient signal, even though the traces may differ substantially in reasoning quality. We propose Reasoning Arena, an adaptive training framework that routes such non-diverse reward groups to a judge system instead of discarding them. Beyond examining the final answer, Reasoning Arena constructs trace tournaments, where reasoning traces are compared head-to-head to expose finer-grained preferences within the group, converting reasoning quality into rich relative reward signals. To make reward estimation efficient, rather than exhaustively comparing every pair, each new trace is evaluated against a small, dynamically updated pool of previously generated traces as anchors to efficiently establish a relative ranking. We then fit a Bradley-Terry model on the incomplete comparison graph, enabling scalable RL integration without quadratic pairwise comparisons. Empirical results demonstrate that Reasoning Arena consistently outperforms the RLVR baseline by 7.6% on average in competition mathematics and coding benchmarks. By converting otherwise wasted zero-advantage samples into useful gradient updates, our method accelerates training by 27% to 41%, saving nearly 50% of generation compute, and substantially improves overall reasoning performance.

• The paper introduces an adaptive hybrid-reward framework that routes non-diverse groups to LLM-judge trace tournaments, overcoming zero-gradient failures.
• It employs pairwise comparisons and fits a Bradley-Terry model to derive dense, relative rewards, significantly enhancing sample efficiency and reducing computation.
• Empirical evaluations show notable performance improvements across mathematics and code synthesis benchmarks, accelerating training speed by up to 41%.

Reasoning Arena: Trace Tournaments for Non-Diverse Reward Groups in RLVR

Motivation and Problem Formalization

Reinforcement learning with verifiable rewards (RLVR) has become standard for refining reasoning capabilities in LLMs on mathematical and program synthesis tasks. RLVR leverages group-relative policy optimization (e.g., GRPO/CISPO), where rewards are computed by verifiable, rule-based evaluators over batches of model-generated traces for a prompt, and policy updates rely on relative advantage. However, a critical limitation arises: when all traces in a group receive identical rewards, the group-relative estimator provides zero gradient, rendering the batch uninformative. These "non-diverse reward groups" (i.e., all-correct or all-incorrect rollouts) become prevalent both early in training (weak policy: all incorrect) and after improvement (strong policy: all correct). Existing approaches such as data curation or entropy-based shaping either discard these groups or inject noisy, policy-intrinsic signals, leading to substantial compute wastage and the loss of fine-grained credit assignment.

Adaptive Reward Routing and Trace Tournaments

To address this bottleneck, Reasoning Arena proposes an adaptive hybrid-reward scheme: rollout groups are routed, based on within-group reward variance, either to the standard verifiable reward or to an LLM-judge-based trace tournament. This preserves the strengths of deterministic verifiers where informative, and invokes the judge precisely when the verifier collapses to a constant (Figure 1).

Figure 1: Overview of Reasoning Arena’s adaptive routing, which sends reward-diverse groups to verifiable rewards and non-diverse groups to a trace tournament with trace-level comparison and Bradley-Terry scoring.

For non-diverse reward groups, Reasoning Arena eschews pointwise rubric scoring—known for calibration problems and vulnerability to artifact overfitting—in favor of pairwise comparison tournaments. For each sampled trace, the judge model performs head-to-head evaluations, outputting soft preference probabilities. Importantly, only a subset of all possible pairs is scored to maintain computational tractability.

To address the incomplete and dynamically constructed comparison graph, Reasoning Arena fits a Bradley-Terry (BT) model to the pairwise judge outputs. This approach infers latent trace strength scores efficiently, enabling robust reward estimation without exhaustive quadratic scaling. The BT-fitted scalar is used as a reward for policy optimization, and the per-group reward routing ensures that verifiable supervision is never replaced when informative.

Empirical Results

Comprehensive evaluation covers competition mathematics (AIME, BeyondAIME), graduate-level reasoning (GPQA-Diamond), and OOD code synthesis (LiveCodeBench). RL was performed with Ministral-3-8B-Instruct as the policy and DeepSeekMath-V2, Qwen3-235B, and Qwen3.5-122B as judges.

Key empirical findings:

• Performance gains: The adaptive approach, especially with trace tournament routing and live Bradley-Terry (Arena-Live), consistently and substantially outperforms RLVR-only, RLAIF (pointwise judge), and static tournament baselines with an average lift of +7.6% (e.g., +12.9 AIME’26), attaining the strongest results on all main benchmarks.
• Efficiency: Routing non-diverse groups to trace tournaments converts previously wasted compute into informative gradients, reducing average generation cost by nearly 50% and accelerating wall-clock training by 27%–41%.
• Robustness and generalization: Gains are robust to the choice of LLM judge backbone, and are observed both in-domain and on OOD code reasoning, indicating general improvements to reasoning and sample efficiency.

Tournament Rewards vs. Judge Baselines

Direct comparison with adaptive pointwise (same routing with pointwise rubric) and static tournament baselines establishes:

• Form of supervision is critical: Pairwise tournaments consistently outperform pointwise for non-diverse groups. Tournaments are less susceptible to surface-level artifacts and enable preference learning grounded in rigorous reasoning differences on intermediate steps.
• Adaptive routing dominates: Uniform application of tournaments or judge rewards to all groups (e.g., ArenaRL) fails to utilize the reliability of verifiable outcomes, leading to lower sample efficiency and suboptimal computation.
• Credit assignment: Arena-Live achieves higher fractions and larger magnitudes of nonzero advantage estimates on non-diverse groups, restoring dense, meaningful gradients that facilitate improved learning.

Judge Scalability and Model Family Effects

Ablation studies extending the judge to Qwen variants demonstrate that the method generalizes over judge families. Higher-capacity judges yield more consistent pairwise preferences on multi-step prompts, but robust improvements prevail even with smaller or alternate judges.

Figure 2: All judge models, when used for trace tournament rewards in Arena-Live, yield substantial improvements over RLVR, highlighting robustness with respect to judge selection.

Theoretical and Practical Implications

Reasoning Arena’s framework demonstrates that combining deterministic verification with adaptively-invoked model-based credit assignment unlocks previously wasted signal in RL training for reasoning. The approach is scalable due to:

• Per-group granularity in reward routing: Ensures minimal compute overhead beyond verifier, with live tournament mechanisms providing linear-scaling RL integration.
• Sample efficiency: Converts zero-advantage scenarios into learning opportunities, especially crucial for hard prompts, curriculum learning, and the long tail of data distributions.

Practically, this points toward more efficient and robust RL pipelines for LLM reasoning improvement. Theoretically, it emphasizes the limitations of outcome-level verification and the need for fine-grained, trajectory-level supervision in domains with sparse/degenerate rewards. As LLMs are extended to domains with complex tool-use, multistep planning, and modular reasoning, dynamic tournament-based routing is likely to be essential.

Conclusion

Reasoning Arena introduces a principled, scalable adaptive reward routing methodology for RLVR by routing non-diverse reward groups to LLM-judge-based trace tournaments. By fitting a Bradley-Terry model to pairwise judge preferences, it efficiently extracts dense, relative rewards from reasoning trajectories, circumventing zero-gradient failure modes and maximizing sample efficiency. The approach consistently outperforms static RLVR and judge-only baselines on both in-domain and OOD tasks, yielding significant gains in policy performance, training speed, and compute efficiency. The methodology is robust to both domain and judge model, positioning it as a generalizable solution for overcoming structural inefficiencies in outcome-based RL for LLM reasoning.