Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning

Abstract: Reinforcement learning significantly enhances LLM capabilities but suffers from a critical issue: length inflation, where models adopt verbosity or inefficient reasoning to maximize rewards. Prior approaches struggle to address this challenge in a general and lossless manner, primarily because additive penalties introduce a compensatory effect that creates optimization shortcuts, while heuristic gating strategies lack generality beyond binary feedback. To bridge this gap, we present Group Relative Reward Rescaling (GR$^3$), which reframes length control as a multiplicative rescaling paradigm, effectively establishing a generalized, continuous, and reward-dependent gating mechanism. To further ensure lossless optimization, we incorporate group-relative regularization and advantage-aware calibration, which dynamically adapt length budgets to instance difficulty and preserve the advantage signal of high-quality trajectories. Empirically, across both RLHF and RLVR settings, GR$^3$~maintains training dynamics and downstream performance comparable to standard GRPO while significantly mitigating length inflation, outperforming state-of-the-art length-regularized baselines.

• The paper introduces a novel multiplicative reward rescaling method (GR³) that mitigates length inflation while preserving task accuracy.
• It leverages group-relative normalization and advantage-aware calibration to effectively balance response brevity with quality.
• Empirical results demonstrate over 40% token reduction with improved performance across reasoning and RLHF alignment tasks.

Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning

Motivation and Problem Statement

Reinforcement Learning (RL) has become the central protocol for post-training LLMs, particularly in RLHF and RLVR regimes. However, RL-trained models exhibit a pronounced tendency toward length inflation. Models maximize rewards by producing unnecessarily verbose, inefficient trajectories—either through extended reasoning chains or artificially long responses—resulting in increased inference costs and reward hacking phenomena. Prior solutions, such as additive length penalties or heuristic gating mechanisms, suffer from major limitations: they introduce optimization shortcuts, lack generality beyond binary reward landscapes, and generally induce undesirable trade-offs between performance and efficiency.

Group Relative Reward Rescaling (GR³): Paradigm and Core Contributions

The paper introduces Group Relative Reward Rescaling (GR³), which fundamentally reframes length regularization from additive/gating-based schemes to a multiplicative, group-relative, advantage-aware mechanism. GR³ rescales rewards by a length-dependent term, enforcing reward-aware efficiency throughout training and eliminating compensatory optimization shortcuts. The rescaling is dynamically calibrated via group-relative statistics and advantage-preserving constraints to guarantee that learning signals for high-quality trajectories remain intact.

Empirical results demonstrate that GR³ consistently resolves the performance–efficiency trade-off: it yields shorter generations while maintaining or even improving accuracy compared to standard GRPO, drastically outperforming length-regularized baselines across mathematical reasoning, code generation, and RLHF alignment tasks (Figure 1).

Figure 1: GR³ sustains stable performance gains under RL while mitigating length inflation versus baselines for efficient reasoning models.

Technical Framework

Multiplicative Reward Rescaling

Additive shaping introduces a decoupled length signal: models can maximize reward by compressing length independent of task correctness. GR³ adopts multiplicative shaping: $\,\hat{R}(x, y^{(i)}) = R(x, y^{(i)}) \cdot \frac{1}{1+\alpha \cdot \frac{\ell^{(i)}{\bar{\ell}}}\,$, where $\alpha$ is a calibrated penalty. This enforces that length regularization is strictly reward-conditioned; the policy cannot maximize shaped reward merely by shortening responses without simultaneously improving task outcomes. A theoretical decomposition reveals that length deviation is weighted by reward, ensuring optimization pressure remains robustly aligned to task capability.

Figure 2: Additive length regularization systematically degrades task reward, while GR³’s multiplicative rescaling preserves reward improvements.

Group-Relative Normalization and Advantage-Aware Calibration

GR³ leverages group-wise statistics (mean length $\bar{\ell}$) for normalization, sidestepping rigid global thresholds and dynamically adapting length budgets to instance difficulty. Advantage-aware calibration ensures representative high-quality responses (maximum reward, average length) retain positive advantages under group normalization, preventing suppression or dilution of learning signal. Calibration is empirically realized by selecting the largest $\alpha$ satisfying a near-perfect constraint satisfaction rate (CSR).

Figure 3: Reward gap relative to standard GRPO versus $\alpha$, with average CSR indicating advantage-preserving calibration.

Empirical Results and Training Dynamics

Efficient Reasoning and Code Generation

On mathematical reasoning (AIME-24/25, AMC23, MATH500) and code generation (LiveCodeBench, MultiPL-E), GR³ yields significant reductions in average token usage (>40%) while achieving strong accuracy improvements (e.g., +8 points), outperforming both length-oriented and performance-oriented RL baselines. Unlike threshold-based truncation, GR³ does not suppress reasoning on hard instances and consistently expands the efficiency–performance Pareto frontier.

RLHF Alignment and Length Bias Mitigation

In RLHF settings, standard GRPO exhibits reward hacking: artificially increasing rewards by generating verbose responses. GR³ decouples performance gains from length exploitation, matching alignment gains of GRPO while keeping response lengths stable and natural. Training dynamics reveal an adaptive length trajectory: initial expansion to secure alignment, followed by compression of redundant tokens as policy matures.

Figure 4: GR³ retains reward gains during training while reducing average tokens across base models.

Ablation and Structural Analysis

Strong numerical evidence supports the role of the penalty coefficient $\alpha$ as a regime controller: aggressive penalties suppress useful reasoning, while calibrated $\alpha$ achieves optimal length reduction without performance loss. The paper exposes strict impossibility results for advantage preservation in high reward density regimes, motivating the shift to average-case calibration and online filtering strategies.

Figure 5: Strict advantage-preservation constraint satisfaction decreases with reward density, even for small $\alpha$, illustrating structural limitations.

Comparative Analysis: Gating, Additive, and Multiplicative Schemes

GR³ unifies and generalizes heuristic gating in binary reward spaces, behaving equivalently in such regimes. In continuous reward landscapes (RLHF), multiplicative shaping acts as a soft, reward-aware gate, avoiding optimization discontinuities and unstable dynamics present in hard-gated additive shaping.

Figure 6: Multiplicative shaping achieves controlled length reduction with stable task reward, while gated additive shaping produces overly short, low-quality responses.

Practical and Theoretical Implications

GR³ delivers practical advances in computational efficiency and environmental sustainability (“Green AI”), substantially reducing token usage without sacrificing capability. By explicitly mitigating reward hacking and length inflation, it supports development of concise, interpretable, and user-aligned AI systems. Theoretically, the paper clarifies structural limitations of length regularization and group-normalized RL, setting the stage for future refinement of efficiency-oriented post-training protocols.

Conclusion

GR³ constitutes a general, principled framework for lossless length control in RL post-training of LLMs, grounded in multiplicative, group-relative, and advantage-aware reward shaping. It robustly mitigates length inflation and reward hacking, consistently shifting the performance–cost frontier outward across diverse tasks. The results empirically refute the notion that verbosity is necessary for intelligence, foregrounding GR³ as a practical paradigm for efficient, high-performing LLMs.

Future Directions

Extensions to fine-grained reasoning step attribution, hybrid length–structure regularization, and cross-modal RLHF regimes are logical next steps. Investigating policy generalization under variable reward density and integrating GR³ with causal reward modeling present promising avenues for robust, scalable alignment and efficiency optimization in AI.