Tandem Reinforcement Learning with Verifiable Rewards

Abstract: Reinforcement learning with verifiable rewards (RLVR) has significantly improved the reasoning capability of LLMs, reaching expert or even superhuman performance in domains such as competition math. However, whether weaker agents and humans can actually harness this capability is far less certain, with RLVR documented to drift reasoning toward idiosyncratic patterns such as poor readability and language mixing. Tandem training is a recently introduced paradigm that targets this compatibility problem: a trained, stronger senior co-generates each rollout with a frozen, weaker junior, and the two are rewarded as a team, so the senior is pushed to reason in ways the junior can follow. Yet this paradigm has so far been demonstrated only in proof-of-concept settings, leaving open whether it scales to the long chains of thought of the modern RLVR pipeline. In this work, we propose Tandem Reinforcement Learning (TRL), which carries the tandem training paradigm into RLVR. In TRL, the senior and a frozen junior alternate stochastically to co-generate the reasoning, the resulting generation is rewarded, and the standard GRPO loss is applied to the senior. Training Qwen3-4B-Instruct on competition math, we find that TRL matches vanilla GRPO on solo reasoning capability while three properties emerge together from the same rollout structure: stronger handoff robustness with the junior, reduced distributional drift from the junior, and a chain-of-thought more legible to the junior. Our results demonstrate a promising route for RLVR with practical payoffs in multi-model communication and human compatibility.

• The paper introduces Tandem Reinforcement Learning (TRL) where a trainable senior and frozen junior co-generate responses to maintain base model compatibility.
• TRL achieves improved token legibility and multi-agent handoff robustness, outperforming conventional GRPO by up to +6.6 percentage points on benchmarks.
• Empirical results show TRL reduces distributional drift by 14% and lowers per-token cross-entropy, yielding more predictable and interpretable outputs.

Tandem Reinforcement Learning with Verifiable Rewards: An Authoritative Technical Summary

Motivation and Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for post-training LLMs in mathematical reasoning and competition settings. In RLVR, models are rewarded for producing correct final answers, leading to significant improvement in task accuracy and elicitation of self-correcting chains of thought. However, a central limitation of RLVR is its tendency to induce distributional drift from the pretrained base model—leading to idiosyncratic token patterns, reduced readability, language mixing, and outputs that are often inscrutable to weaker models or humans. Crucially, these shifts impair model compatibility in multi-agent settings and oversight scenarios, in which weaker entities need to follow or continue strong model outputs.

Conventional defenses against drift—such as KL penalties, supervised distillation, and process supervision—depend on explicit specifications of intelligibility. These methods are constrained by the need for advance definition and labeled traces, which are impractical outside narrow deployment contexts.

The paper proposes Tandem Reinforcement Learning (TRL), extending the recently introduced tandem training paradigm to RLVR settings. In TRL, a trainable senior and a frozen junior (initialized from the same pretrained base) co-generate responses by alternating authorship at word boundaries, and the team reward is assigned in the standard RLVR pipeline. The senior is updated only on tokens it emits, using the Group Relative Policy Optimization (GRPO) objective.

Figure 1: Tandem Reinforcement Learning (TRL) at a glance.

Methodology: Tandem Rollout and Policy Optimization

TRL's operational mechanism involves stochastic alternation between senior and junior models at word boundaries during each rollout. Both receive the same context; each token is sampled by the active model determined through coin flips at boundaries. The junior remains frozen throughout training, serving as a reference for compatibility. The co-generated response receives a binary verifier reward. The senior is updated via GRPO, but only on senior-emitted tokens.

Three design choices are instantiated:

• Self-pairing: Senior and junior are identical at initialization, maximizing baseline compatibility and minimizing unnecessary stylistic adaptation.
• Word-level alternation: Handoffs occur at every word boundary for maximal natural language granularity and resistance to reward-hacking through syntactic manipulation.
• GRPO optimization: Senior-only updates using standard GRPO logic, isolating the impact of rollout structure.

Empirical Results: Capability, Compatibility, and Distributional Anchoring

Solo Reasoning Capability

TRL is empirically validated on Qwen3-4B-Instruct trained for competition mathematics using DeepScaleR. Solo reasoning performance (pass@$k$) for both TRL and vanilla GRPO-trained seniors is indistinguishable; both significantly outperform the base model and retain RLVR’s capability lift. There is no measurable cost in solo accuracy associated with tandem rollouts.

Figure 2: Reasoning capabilities (measured by pass@$k$, $\uparrow$) on mathematical reasoning benchmarks for Qwen3-4B-Instruct and its GRPO- and TRL-trained seniors.

Handoff Robustness

TRL’s central claim is improved compatibility: the senior’s outputs are more intelligible to the junior, measured by reasoning-step handoff robustness. When paired at inference with the frozen junior under a step-wise alternation schedule, the TRL senior outperforms GRPO seniors by up to +6.6 percentage points in pass@$8$ on AIME, with macro-average gains across benchmarks.

Figure 3: Reasoning-step handoff robustness (measured by pass@$k$, $\uparrow$) on mathematical reasoning benchmarks of GRPO and TRL seniors paired with the junior, frozen Qwen3-4B-Instruct.

Distributional Drift and Legibility

TRL rollouts substantially curb distributional shift from the base model. In token-level analyses, TRL exhibits a 14% lower KL-divergence from the base than GRPO. Among the 500 tokens most displaced by GRPO, 87% shift back towards the base under TRL. These results demonstrate aggregate and targeted recovery of baseline distribution.

Figure 4: Distributional deviation from the base model for GRPO and TRL. Left: survival curves of absolute per-token log-ratio to the base over tokens. Right: for the top-500 most displaced tokens by GRPO, we show the drift versus TRL recovery.

Legibility is quantified via junior’s per-token cross-entropy and distribution overlap. TRL reduces per-token cross-entropy by up to 17% and increases overlap by approximately 30% relative to GRPO, denoting that the senior’s chain-of-thought is more predictable and aligned for the junior.

Training Dynamics and Computational Efficiency

TRL training is stable, matching GRPO in reward trajectory and response length. The dual-model rollout imposes a roughly 2$\times$ overhead in per-step latency but converges in fewer steps, partially offsetting the cost. With the vLLM backend, TRL is practical for long-context RLVR settings.

Figure 5: Training dynamics of TRL and GRPO. (a)~Mean reward. (b)~Average response length for rollouts. (c)~Per-step wallclock (left axis) and accumulated wallclock to best checkpoints (right axis, shaded). (d)~Macro-average benchmark accuracy under each run's evaluation protocol.

Qualitative Token Drift Analysis

Token-level drift analyses reveal that GRPO’s most extreme over-emitted tokens are structural formatting artifacts (e.g., enumeration suffixes, answer signalling) absent from the base. TRL eliminates these, concentrating residual drift in compact mathematical notation.

Figure 6: Tokens most over-emitted by Vanilla GRPO (left) and TRL (right) relative to the junior, sized by displacement magnitude.

KL Regularization Ablation

Ablation studies demonstrate that explicit per-token KL regularization towards the junior fails to recapitulate TRL’s gains. KL-Reg matches GRPO in solo capability and only partially improves distributional overlap but does not yield handoff robustness or conditional legibility. This is attributed to KL regularization’s inability to supply the team compatibility signal inherent in tandem rollouts.

Figure 7: Vocabulary drift from base for GRPO, KL-Reg, and TRL. Left: survival curve of the absolute per-token log-ratio to the base over tokens. Right: for the top-500 most displaced tokens by GRPO, the per-method drift relative to GRPO.

Implications and Forward Directions

TRL identifies rollout structure as an underexplored axis for RLVR pipeline design, offering practical and theoretical payoffs for multi-agent communication and compatibility with humans and weaker models. The results provide constructive evidence that it is possible to combine expert reasoning capability, compatibility, and distributional anchoring in a single intervention, without explicit losses or reward modifications.

Practical implications include improved models for oversight, distillation, and heterogeneous agent collaboration. Theoretically, TRL points toward rollouts as an operational device for enforcing legibility via outcomes rather than a priori distributional or labeling constraints.

Future research directions include:

• Generalizing tandem pairing to diverse juniors varying in language, style, or tool use, to regularize seniors toward broader intelligibility.
• Using juniors calibrated to human skill levels for direct applicability in human-AI handoff scenarios.
• Deepening mechanistic understanding of TRL’s pressure dynamics as the senior diverges from the frozen junior.

Conclusion

TRL, as operationalized in this work, enables RLVR-trained models to retain full reasoning capability, while measurably enhancing compatibility and legibility for weaker partners. The tandem rollout mechanism stands out as an efficient, architecture-agnostic and outcome-driven method for mitigating post-training distributional drift and facilitating transparent collaboration in multi-agent and human-in-the-loop systems.