RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalization (2510.02172v1)

Abstract: Reinforcement learning with human-annotated data has boosted chain-of-thought reasoning in large reasoning models, but these gains come at high costs in labeled data while faltering on harder tasks. A natural next step is experience-driven learning, where models improve without curated labels by adapting to unlabeled data. We introduce RESTRAIN (REinforcement learning with Self-restraint), a self-penalizing RL framework that converts the absence of gold labels into a useful learning signal. Instead of overcommitting to spurious majority votes, RESTRAIN exploits signals from the model's entire answer distribution: penalizing overconfident rollouts and low-consistency examples while preserving promising reasoning chains. The self-penalization mechanism integrates seamlessly into policy optimization methods such as GRPO, enabling continual self-improvement without supervision. On challenging reasoning benchmarks, RESTRAIN delivers large gains using only unlabeled data. With Qwen3-4B-Base and OctoThinker Hybrid-8B-Base, it improves Pass@1 by up to +140.7 percent on AIME25, +36.2 percent on MMLU_STEM, and +19.6 percent on GPQA-Diamond, nearly matching gold-label training while using no gold labels. These results demonstrate that RESTRAIN establishes a scalable path toward stronger reasoning without gold labels.

• The paper introduces a self-driven RL paradigm using pseudo-label weighting, negative rollout penalization, and prompt-level weighting to improve reasoning without gold labels.
• It achieves significant Pass@1 improvements, nearly matching supervised GRPO baselines and surpassing them on science-heavy benchmarks.
• The framework mitigates mode collapse by penalizing unreliable outputs and leveraging full answer distributions for robust unsupervised training.

RESTRAIN: Self-Driven RL with Self-Penalization for Label-Free Reasoning

Motivation and Problem Setting

The RESTRAIN framework addresses a central challenge in reinforcement learning for LLMs: scaling chain-of-thought reasoning without reliance on gold labels. While RL with verifiable rewards (RLVR) has advanced LLM reasoning, its dependence on curated, high-quality labeled data limits scalability and generalization, especially for tasks where human annotation is infeasible or unreliable. RESTRAIN proposes a self-driven RL paradigm that leverages the model's own output distribution to generate robust learning signals, enabling continual self-improvement on unlabeled data.

RESTRAIN Framework: Core Components

RESTRAIN introduces three synergistic mechanisms to transform noisy, label-free training into effective learning signals:

1. Pseudo-Label Weighting: Instead of reinforcing only the majority-voted answer, RESTRAIN assigns soft weights to all unique model-predicted answers for each prompt, proportional to their empirical frequencies. This mitigates the brittleness of majority voting and prevents collapse to spurious modes, allowing minority correct solutions to contribute to learning.
2. Negative Rollout Penalization: For prompts with low self-consistency (i.e., low majority count), RESTRAIN applies a uniform negative offset to the advantage of all rollouts, effectively penalizing unreliable outputs and encouraging exploration of alternative reasoning paths.
3. Prompt-Level Weighting: Each prompt is assigned a fixed weight reflecting the model's confidence (computed offline from a frozen reference model). Prompts with low consensus are down-weighted, reducing the impact of noisy examples and stabilizing training.

The final RESTRAIN loss integrates these components into a GRPO-style policy optimization objective, enabling robust unsupervised RL.

Figure 1: RESTRAIN consists of pseudo-label weighting, negative rollout penalization, and prompt-level weighting, forming a unified self-penalizing RL framework.

Empirical Results and Analysis

RESTRAIN demonstrates strong empirical performance across multiple reasoning benchmarks and model architectures. On the DAPO-14k-MATH dataset, RESTRAIN achieves Pass@1 improvements of +140.7% on AIME25, +36.2% on MMLU_STEM, and +19.6% on GPQA-Diamond over the strongest unsupervised baselines. Notably, RESTRAIN nearly matches the gold-label GRPO upper bound, trailing by only 0.4 points on Qwen3-4B-Base, and even surpasses it on science-heavy benchmarks, indicating superior cross-domain generalization.

Figure 2: RESTRAIN outperforms TTRL and ETMR on label-free and test-time RL, nearly matching or exceeding gold-label GRPO on several benchmarks.

Ablation studies confirm the necessity of each RESTRAIN component. Removing pseudo-label weighting leads to rapid training collapse, while omitting negative rollout penalization or prompt-level weighting significantly degrades performance. Hyperparameter sweeps reveal that moderate values for the pseudo-label weight bias ($\sigma$) and negative advantage offset ($\delta$) are critical for balancing noise suppression and signal retention.

Figure 3: Pseudo-label weighting stabilizes training and prevents collapse, with optimal accuracy achieved at intermediate $\sigma$ values.

Figure 4: Model performance is sensitive to the negative advantage offset $\delta$; excessive penalization suppresses learning.

Figure 5: Majority vote statistics show that correct answers often diverge from the majority, motivating RESTRAIN's soft weighting approach.

Theoretical and Practical Implications

RESTRAIN's self-penalization strategy provides a scalable alternative to RLVR, enabling LLMs to improve reasoning capabilities without external supervision. By leveraging the full answer distribution and penalizing overconfident or low-consensus outputs, RESTRAIN mitigates mode collapse and overfitting, promoting generalization across domains. The framework is compatible with existing policy optimization algorithms and can be deployed for both large-scale training and test-time adaptation.

From a theoretical perspective, RESTRAIN advances the understanding of intrinsic reward estimation and self-consistency in unsupervised RL. It demonstrates that robust learning signals can be extracted from the model's own uncertainty and output diversity, challenging the reliance on majority heuristics and gold labels.

Implementation Considerations

RESTRAIN is implemented atop the GRPO algorithm, requiring only access to model rollouts and a reference policy. The pseudo-label weighting and prompt-level weighting modules are computationally efficient, involving simple frequency calculations and monotonic transformations. Negative rollout penalization is triggered by a tunable majority count threshold, allowing flexible control over penalization strength. RESTRAIN scales to large datasets and models, with experiments conducted on 32×A100 GPUs and rollout sizes of 16–64.

A representative pseudocode for the per-prompt RESTRAIN loss is as follows:

def restrain_loss(outputs, prompt_weight, threshold, neg_offset): answers = [extract_answer(output) for output in outputs] counts = Counter(answers) Mx = counts.most_common(1)[0][1] if Mx < threshold: rewards = [0.0] * len(outputs) adv = calculate_advantages(rewards) adv = [a - neg_offset for a in adv] loss = calculate_loss(adv) return prompt_weight * loss freqs = counts.values() / len(outputs) label_weights = calculate_label_weight(freqs) final_loss = 0.0 for i, label in enumerate(counts.keys()): rewards = [reward_fn(ans, label) for ans in answers] adv = calculate_advantages(rewards) loss = calculate_loss(adv) final_loss += label_weights[i] * loss return prompt_weight * final_loss

Future Directions

RESTRAIN opens several avenues for future research:

• Extension to Other Domains: The framework can be adapted to code generation, scientific reasoning, and open-ended tasks where gold labels are scarce.
• Integration with Intrinsic Reward Methods: Combining RESTRAIN with entropy-based or novelty-driven intrinsic rewards may further enhance exploration and robustness.
• Adaptive Penalization: Dynamic adjustment of penalization parameters based on training dynamics could improve stability and convergence.
• Theoretical Analysis: Formal characterization of the conditions under which self-penalization yields optimal learning signals remains an open question.

Conclusion

RESTRAIN establishes a robust, scalable approach for self-driven RL in LLMs, transforming spurious votes into actionable signals via self-penalization. By leveraging the full answer distribution and penalizing unreliable outputs, RESTRAIN achieves strong performance on challenging reasoning tasks without gold labels, nearly matching supervised RLVR. Its modular design and empirical stability make it a promising foundation for future advances in unsupervised LLM training and generalization.