When the Majority Votes Wrong, the Intervention Timing for Test-Time Reinforcement Learning Hides in the Extinction Window

Abstract: Test-time reinforcement learning (TTRL) reports substantial accuracy gains on mathematical reasoning benchmarks using majority vote as a pseudo-label signal. We argue these gains are systematically misinterpreted: most reflect sharpening of already-solvable problems rather than genuine learning, while problems corrupted from correct to incorrect outnumber truly learned ones, and this damage is irreversible once majority vote locks onto a wrong answer. Per-problem tracking reveals that correct-answer signals in low-ability problems are briefly active before being permanently suppressed, a phenomenon we term the \textit{Correct-Answer Extinction Window}, with Flip Rate (FR) as its leading indicator. We thus propose \textbf{TTRL-Guard}, a lightweight framework with three mechanisms targeting the extinction window: Flip-Rate-Aware Reward Scaling (FRS) down-weights at-risk updates as FR declines, Minority-Preserving Sampling (MPS) retains gradient signal from minority correct answers, and Risk-Conditioned Sparse Updatings (RCSU) suspends updates on polarized problems. Experiments across three models and four benchmarks show that TTRL-Guard achieves the best average pass@1 on Qwen2.5-7B-Instruct and Qwen3-4B, improves relatively over TTRL by +54\% on AIME 2025. \footnote{Our code and implementation details are available at https://github.com/linhxkkkk/TTRL-Guard.

• The paper reveals that standard majority voting in TTRL boosts performance only for already-solvable problems while irreversibly degrading many initially correct answers.
• It introduces the Correct-Answer Extinction Window and employs the Flip Rate as a dynamic, label-free marker to detect early instability in pseudo-labels.
• The TTRL-Guard framework, with its Flip-Rate-Aware Reward Scaling, Minority-Preserving Sampling, and Risk-Conditioned Sparse Updating, achieves a 54% improvement in pass@1 and reduces problem degradation.

Intervention Strategies for Test-Time Reinforcement Learning: Dynamics, Risks, and Mitigation via the Extinction Window

Motivation and Problem Setting

This work interrogates the mechanics of Test-Time Reinforcement Learning (TTRL) for LLMs, where self-improvement is achieved without supervision by leveraging majority voting over model rollouts as a pseudo-label reward signal. The paper identifies a critical misinterpretation in prior evaluations of TTRL: aggregate accuracy improvements mask a dynamic in which the majority of the actual "gains" derive almost entirely from sharpening the accuracy on already-solvable problems, rather than true capability expansion to previously unsolved instances. Conversely, many initially solvable problems are irreversibly degraded—corrupted from correct to incorrect answers—due to the myopic focus on majority consensus, a phenomenon that is systematically hidden in global accuracy curves.

Figure 1: Label migration demonstrates significant flow from correct to incorrect pseudo-labels (as well as modest gains from incorrect to correct), revealing the prevalence of degradation events in standard TTRL.

To dissect this pathology, the investigation introduces the concept of the Correct-Answer Extinction Window—a transient phase in early training when the ground-truth answer competes but has not yet been suppressed by an emergent majority consensus on the wrong answer. The leading dynamic statistic, Flip Rate (FR), serves as an online, label-free marker for this window.

Empirical Dissection of TTRL Pathologies

Experiments deploy Llama-3.2-3B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-4B across AIME 2024/2025, AMC, and MATH-500 benchmarks. Each problem is classified by its label trajectory during training, revealing that the fraction of truly "learned" problems (i.e., zero-to-one acquisition) is minuscule, while degraded problems (corrupted from correct to incorrect) dominate across all but the highest-capacity scenarios.

Figure 2: The post-TTRL distribution of fates (Stable AR, Degraded, Learned) for three models on four benchmarks; the Degraded category massively dominates, particularly in low-capacity or mid-difficulty regimes.

FR rises sharply early in training, indicating volatile consensus, before collapsing as the model locks onto a (correct or incorrect) pseudo-label. Critically, this lock-in is virtually irreversible—once the majority has stabilized on the wrong answer, the true signal is irrecoverably lost.

Figure 3: The correct-vote rate (probability ground-truth is predicted by the MV) on never/late solved problems shows a rapid, monotonic collapse—a quantitative signature of the extinction window phenomenon.

These dynamics are strongly modulated by the model-dataset capability gap: largest knowledge corruption occurs when a model is just strong enough to produce confident but wrong outputs, so initial competitive signal is extinguished by self-reinforced error majority. Pass@1 metrics systematically obscure this effect, as aggregate accuracy can rise or hold steady while a majority of problems silently degrade.

The TTRL-Guard Framework

To counteract these vulnerabilities, the TTRL-Guard framework introduces three synergistic interventions, all activated by per-problem monitoring of FR and match rate:

1. Flip-Rate-Aware Reward Scaling (FRS): Rewards associated with high-FR problems are attenuated, dampening the reinforcement of unstable or at-risk pseudo-labels.
2. Minority-Preserving Sampling (MPS): During periods of flip instability, minority (often correct) answers are preserved by upweighting their gradients, combating premature extinction under majority-wins-only RL.
3. Risk-Conditioned Sparse Updating (RCSU): Once consensus is reached (post-extinction window) and a problem is flagged as high-risk, updates are probabilistically skipped to avoid propagating damage.

   Figure 4: The architectural overview of TTRL-Guard details the three-path routing (FRS, MPS, RCSU), temporal intervention timing, and per-problem state-flow.

This architecture ensures that only at-risk problems are modulated, while progress on mastered problems is left untouched, resulting in targeted, minimal intervention only during the critical window.

Experimental Evidence

TTRL-Guard is extensively benchmarked against vanilla TTRL, SCOPE, CoVerRL, and ablations. On the most challenging settings (e.g., Qwen2.5-7B-Instruct, AIME 2025), TTRL-Guard yields a relative improvement over TTRL of 54% in pass@1, and, more importantly, reduces the fraction of degraded problems by over 30 points. Importantly, the Learned/Degraded (L/D) ratio is markedly increased (from 0.004 to 0.03 in extreme cases), indicating that TTRL-Guard transitions TTRL from a net knowledge-destroying to a net knowledge-conserving update scheme.

Figure 5: Training-end per-problem distributions (MATH-500) underscore the reduction in Degraded problem fraction and rise in true learning with TTRL-Guard; only TTRL-Guard closes the harm-to-benefit gap inherent in naive TTRL.

Per-problem ΔLabel Accuracy (ΔLA) further shows that degradation within the Degraded group is deep (mean shifts >0.6), not noise. Temporal curves of FR and pass@1 illustrate that only Guard-equipped methods prevent rapid FR collapse and subsequent irreversible accuracy loss.

Figure 6: Training dynamics for TTRL variants illustrate that accuracy gains exclusively occur post-FR collapse, directly tying intervention timing to outcome.

Microscopic Mechanisms: The Scissor Effect

Fine-grained tracking demonstrates a "scissor effect" in label survival: correct and incorrect majorities compete early, but once the crossover is achieved, error reinforcement dominates and recovery is impossible. The extinction window's narrow phase (analyzed quantitatively by duration and transition statistics) is thus the unique opportunity for beneficial intervention.

Figure 7: Sensitivity to FR threshold and corresponding effects on pass@1 and L/D point to optimal parameter tuning for FR-based interventions.

Practical and Theoretical Implications

This study reframes TTRL as a mechanism for high-confidence distribution sharpening, not for robust reasoning or genuine ability acquisition. The absence of label trust guarantees under consensus-based reward, and the sharp knowledge corrosion at the problem-level, highlight the fragility of self-improving adaptation via pseudo-labeling in LLMs.

Practically, TTRL-Guard’s architecture introduces negligible computational overhead when RCSU's skip mechanism is accounted for, and the approach generalizes across both model architectures and capability/difficulty regimes. The simplicity and label-free nature of FR as an online uncertainty measure make it broadly applicable in test-time, deployment, or unsupervised continual learning pipelines.

Theoretically, this work establishes that unsupervised RL-driven test-time adaptation cannot be blindly trusted: dynamic, per-sample interventions tied to early-phase pseudo-label instability are necessary to avoid structural knowledge damage. Future developments in label-free self-improving AI must deploy similar online uncertainty diagnostics and path-selective interventions, or else risk regressing in aggregate performance despite increasing “headline” accuracy on benchmarks.

Conclusion

This study provides a systematic, quantitative, and mechanistic critique of TTRL, exposing substantial yet hidden reliability failures. The Correct-Answer Extinction Window is established as the core dynamic, and FLip-Rate tracking as an actionable tool for unsupervised intervention. The TTRL-Guard framework effectively mitigates degradation, shifting TTRL toward a net-positive, reliable adaptation regime. Future research in label-free RL for LLMs should combine dynamic pseudo-label diagnostics with temporally- and per-problem-aware intervention policies, generalizing these results beyond the mathematical reasoning domain.