Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling

Abstract: Token serves as the fundamental unit of computation in modern autoregressive models, and generation length directly influences both inference cost and reasoning performance. Despite its importance, existing approaches lack fine-grained length modeling, operating primarily at the coarse-grained sequence level. We introduce the Length Value Model (LenVM), a token-level framework that models the remaining generation length. By formulating length modeling as a value estimation problem and assigning a constant negative reward to each generated token, LenVM predicts a bounded, discounted return that serves as a monotone proxy for the remaining generation horizon. This formulation yields supervision that is annotation-free, dense, unbiased, and scalable. Experiments on LLMs and VLMs demonstrate LenVM provides a highly effective signal at inference time. On the LIFEBench exact length matching task, applying LenVM to a 7B model improves the length score from 30.9 to 64.8, significantly outperforming frontier closed-source models. Furthermore, LenVM enables continuous control over the trade off between performance and efficiency. On GSM8K at a budget of 200 tokens, LenVM maintains 63% accuracy compared to 6 percent for token budget baseline. It also accurately predicts total generation length from the prompt boundary. Finally, LenVM's token-level values offer an interpretable view of generation dynamics, revealing how specific tokens shift reasoning toward shorter or longer regimes. Results demonstrate that LenVM supports a broad range of applications and token length can be effectively modeled as a token-level value signal, highlighting the potential of LenVM as a general framework for length modeling and as a length-specific value signal that could support future RL training. Code is available at https://github.com/eric-ai-lab/Length-Value-Model.

• The paper introduces a token-level value estimation framework that formulates length modeling as a supervised regression problem using a discounted return with negative rewards.
• It attaches a scalar value head to each decoding state, achieving precise output control and improved credit assignment compared to sequence-level length penalties.
• The approach enhances generation quality and resource allocation, showing significant benchmark gains and scalability across diverse model sizes and prompts.

Detailed Summary of "Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling" (2604.27039)

Motivation and Problem Definition

Length modeling in autoregressive linguistics and multimodal models remains a challenge due to its coarse-grained handling in prior work. Most approaches regulate output length either at sequence-level via penalties or prompt-based heuristics, failing to capture fine-grained, per-token dynamics that underpin both computational cost and reasoning quality. The "Length Value Model" (LenVM) addresses this gap by formulating generation length modeling as a token-level value estimation problem, using a principled discounted return derived from assigning constant negative reward per token.

Model Formulation

LenVM attaches a scalar value head to each decoding state in an LLM or VLM, predicting the value of the remaining generation horizon. The approach leverages the RL value function formulation, where each generated token receives a constant negative reward, with future steps discounted by a factor $\gamma$. This yields a monotone, bounded proxy for the remaining sequence, transforming length modeling into a supervised value regression problem. Notably, this supervision is:

• Annotation-Free: Targets are computed from completed sequences, eliminating the need for human labeling.
• Dense: Every token position contributes a regression target, improving credit assignment and learning efficiency relative to sequence-level signals.
• Unbiased: As returns are deterministically calculated from generated completions under a fixed policy, no additional noise from reward modeling is introduced.
• Scalable: Supervision grows with model size, number of prompts, and samples per prompt, facilitating large-scale value pretraining.

A critical innovation is the bounded transformation of lengths into $(-1,0)$, which preserves monotonicity and Bellman consistency, avoiding the conditioning issues inherent in regressing raw or normalized lengths.

Experimental Evaluation

Length-Controlled Generation

LenVM enables precise inference-time control of output length. On LIFEBench, which provides exact, at-most, and at-least length constraint benchmarks, LenVM-guided decoding substantially outperforms frontier closed-source models. For instance, applying LenVM to a 7B Qwen2.5 model more than doubles the length score in the "Equal To" setting (from 30.9 to 64.8), with deviation reduced from 71% to 44%. This demonstrates that open models equipped with LenVM achieve precision superior to proprietary models that rely solely on prompt-based or sequence-level control.

Performance-Efficiency Trade-off

LenVM provides a continuous control knob for balancing quality and efficiency in token-wise generation. By exponentially tilting the next-token distribution toward shorter-horizon tokens, LenVM traces a smooth Pareto frontier between response quality and generation length. On GSM8K, at a fixed token budget of 200, LenVM maintains 63% accuracy, compared to only 6% for hard truncation baselines, demonstrating that many short, successful reasoning trajectories exist within the base generator and can be accessed via LenVM-guided decoding.

Generation Length Prediction

LenVM accurately predicts expected remaining generation length from the prompt boundary, facilitating scheduling and resource planning. Mean Relative Error (MRE) decreases consistently with model scale, reaching 9.8% on math tasks for a 32B model. This evidences its utility for both preprocessing and dynamic online settings, where precise length anticipation improves batching and memory allocation.

Scalability

Validation loss decreases monotonically with increased model size, more training prompts, and more sampled completions per prompt. This demonstrates LenVM's efficacy as a scalable value pretraining paradigm, unencumbered by annotation bottlenecks or diminishing returns at higher scales.

Interpretability: Length Tokens

LenVM's token-level values are interpretable markers of reasoning regime shifts. Tokens with high positive temporal-difference residuals (e.g., "ah", "but", "now", "wait", "let", "think") correspond to moments where generation pivots toward longer continuations, typically when the model corrects or expands reasoning. Negative tokens ("therefore", "clearly", finalization emojis) mark transitions toward closure and termination. This qualitative analysis reifies LenVM's utility for introspecting generation dynamics.

Ablation Studies

Multiple design choices are evaluated:

• Target Parameterization: Discounted return + sigmoid outperforms regression on raw or normalized length, aligning supervision with autoregressive Bellman recursion and avoiding the pitfalls of highly skewed length distributions.
• Batch Construction: Shuffling completions from different prompts within batches consistently yields lower evaluation loss than grouping, indicating improved generalization.
• Discount Factor $\gamma$: Higher $\gamma$ compresses long horizons but enhances early prediction; lower $\gamma$ allocates resolution near termination. Optimal $\gamma$ balances early-horizon compression with late-stage discrimination.
• Numerical Precision: LenVM exhibits stability across FP16, BF16, and FP32 formats, with negligible differences in convergence or final performance.

Reinforcement Learning Perspective

LenVM's value formulation is natively compatible with PPO-style RL. It serves either as a value baseline for length-specific rewards or as a potential function for token-level reward shaping, supporting dense, policy-invariant credit assignment. This decouples task quality from generation efficiency, enabling direct and interpretable trade-offs, as well as improved credit assignment for RL optimization. The token-level value signal addresses the limitations of sequence-level penalties, providing immediate feedback and variance reduction during training.

Theoretical and Practical Implications

LenVM demonstrates that generation length—previously treated as a sequence-level quantity—can be robustly modeled as a token-level value signal. Practically, this offers enhanced generation control, improved resource allocation, and more efficient utilization of compute and memory. The theoretically principled value function approach, Bellman-consistency, and scalability axes position LenVM as a general substrate for length-aware pretraining and potentially for RL fine-tuning. Moreover, the interpretable nature of token-level horizon shifts opens avenues for deeper analysis of generative reasoning traces.

Future Directions

The empirical results suggest several promising research trajectories:

• Integrating LenVM into RL fine-tuning for explicit quality-efficiency trade-offs.
• Extending LenVM to other modalities (vision, multimodal reasoning) and tasks requiring precise output regulation.
• Exploring potential-based reward shaping for more effective policy optimization in resource-constrained settings.
• Using LenVM-driven horizon analysis for debugging, interpretability, and generative auditing.

Conclusion

LenVM reframes length modeling as token-level value estimation within a discounted return framework, offering annotation-free, dense, and scalable supervision. The approach yields strong empirical gains in controlled generation, performance-cost balancing, horizon prediction, and interpretability. Its compatibility with RL frameworks and demonstrable scalability solidifies LenVM as a foundational method for length modeling in sequential generative systems.