How Much is Left? LLMs Linearly Encode Their Remaining Output Length

Abstract: LLMs generate one token at a time, yet their responses show remarkably consistent length structure: step-by-step solutions converge in predictable token counts, retrievals stop after a few sentences, retractions extend responses by measurable amounts. We ask whether the model carries an internal estimate of how much response remains. Training minimal-capacity linear probes on frozen hidden states of three open-weight 7-8B models across seven completion-style datasets, we find three converging pieces of evidence. First, total response length is linearly decodable from the prompt's last hidden state alone, before any output is emitted. Second, probe directions trained on natural-language datasets transfer broadly, including to controlled synthetic completions never seen in training, outperforming a statistical baseline; the converse direction generally fails, and this asymmetry is itself informative. Third, on curated high-loss completions, the probe's per-position estimate shifts upward at the moment the model retracts and restarts a partial solution, a directional behavior no position-only predictor can reproduce (qualitative, not aggregate). We frame this as approximate estimation of remaining generation length, distinct from exact-counting impossibility results for transformers, and interpret it as evidence that LLMs maintain a plan-like internal representation of output length (decodable, not necessarily used causally).

• The paper demonstrates that LLMs internally encode a linearly accessible estimate of their remaining output length using minimal-capacity linear probes.
• It employs linear regressors on hidden state activations and shows significant reductions in mean absolute error compared to constant baselines.
• The findings imply a flexible, plan-like internal representation in LLMs, with potential applications in efficiency, safety, and interpretability.

Linear Encoding of Remaining Output Length in LLMs: An Analysis

Introduction and Motivation

This paper investigates the hypothesis that LLMs maintain a linearly decodable internal variable indicative of the remaining output length during autoregressive generation (2607.05316). The empirical regularity of completion lengths—such as step-by-step solutions and retrievals terminating in predictable token counts—suggests the presence of a response-length estimation mechanism, despite the absence of explicit length tracking in transformer architectures. This study asks whether this regularity is purely a downstream artifact of decoding or reflects a plan-like, linearly encoded internal representation.

Methodology

The core of the analysis is based on minimal-capacity linear probes trained on frozen residual stream activations of three open-weight 7–8B parameter LLMs: Llama-3.1-8B-Instruct, Olmo-3-7B-Instruct, and Mistral-7B-Instruct-v0.3. The probes are trained to regress the number of tokens remaining to be generated ($r_t = T-t$) at each position $t$, given the hidden state $h_t$ at that position.

Three probe variants are evaluated:

1. Remaining Count Probe: Linear regressor trained on $h_t$ at each position to predict $r_t$.
2. Constant-Median Statistical Baseline: Always predicts the train-split median of $r_t$.
3. Completion Length Probe: Trained on the prompt-end hidden state to predict total length $T$, with per-timestep prediction $\hat{T}_0 - t$ during generation.

Seven datasets are used, spanning synthetic controlled-length tasks (Count, Countdown) and standard natural language completions (GSM8K, MATH, MMLU-Pro, OpenThoughts-1k, TriviaQA).

Key Experiments and Results

Prompt-End Length Estimation

The Completion Length Probe achieves substantially lower mean absolute error (MAE) in predicting total completion length from the prompt end, compared to constant baselines. Notably, on synthetic data (Countdown), Llama's prompt-end MAE drops from 150.18 to 5.27 tokens—indicating near-exact length estimation when length is strictly determined by the prompt.

Mid-Completion Length Re-Estimation

The Remaining Count Probe, which accesses per-token $h_t$, generally outperforms countdown baselines in per-token remaining length estimation, especially on longer natural language tasks. Crucially, its predictions are not constrained to be monotonic, in contrast to position-indexed baselines.

Figure 1: Predicting remaining tokens on a short example. The remaining count probe achieves the lowest MAE through non-monotonic updates, surpassing prompt-only and constant baselines.

Figure 2: Count prediction versus ground truth on a typical example with small MAE, illustrating accurate match.

Non-Monotonic Updates and Retractions

Qualitative analysis reveals spikes in the Remaining Count Probe's predictions upon token sequences indicating retraction or restart (e.g., "Wait — that's a contradiction"), while position-indexed baselines cannot exhibit such behavior.

Figure 3: Count prediction on a high-MAE example, with a clear upward spike in the estimated remaining count following a retraction signal.

Figure 4: Retraction example 1. The count probe's estimate jumps during the "Wait — that's a contradiction" retraction event.

Figure 5: Retraction example 2. Post-answer, a retraction phrase results in a marked bump in the predicted remaining count.

Figure 6: Retraction example 3. Entering a reconsideration phase, the probe's estimated remainder climbs sharply.

Figure 7: Retraction example 4. A sequence of retractions produces successive spikes in the probe's predicted output length.

This dynamic is repeatedly observed: the probe's per-token prediction can update upward in response to internal re-evaluation within the model, supporting the existence of a modifiable, linearly accessible plan-like variable.

Per-Layer Analysis

Per-layer probe sweeps demonstrate that length estimation ability increases in upper layers, consistent with the emergence of processed representations beyond mere token or position encoding.

Figure 8: Per-layer MAE reveals the residual stream's capacity for length estimation arises mostly in upper layers.

Accuracy as a Function of Completion Length

The probe performs best near the mode of the dataset’s completion lengths and significantly worse on the underrepresented long-output tail.

Figure 9: Probe MAE by output length bin: accuracy peaks at modal lengths and degrades for long, rare completions.

Cross-Dataset Generalization

A striking asymmetry is found in cross-dataset transfer. Probes trained on natural language data generalize robustly—sometimes even back to synthetic controlled tasks—while those trained on synthetic sets do not generalize to natural language. Probe directions arising from natural data appear to represent a more general output-length estimation signal.

Theoretical and Practical Implications

Distinction from In-Context Counting

The paper underscores that this phenomenon is unrelated to exact counting impossibility results for transformers; the probe recovers a coarse, continuous output-length estimation, not precise token or item counting. The mean absolute error for estimation ($\approx 30$ for $t$0) is well above the threshold for exact counting, affirming its approximate nature.

Mechanistic Interpretability and Planning

The results provide convergent evidence for the existence of a linearly decodable, flexible plan-like variable in LLM activations, controlling (or at least correlating with) predictions of future output length. This contributes to a growing body of work on linear representation and planning within transformer-based models.

Applications and Limitations

The existence of a decodable output-length variable can facilitate efficiency improvements (e.g., early termination), safety monitoring (e.g., flagging inconsistent retractions without length-adjustment), and the interpretability of LLM “plans” and self-evaluation. However, the study notes that probe decodability does not entail that the model causally relies on this variable during generation. Future causal interventions (e.g., activation patching) are needed to establish functional usage.

The dynamic re-estimation effect is documented only qualitatively on high-error examples; aggregation remains technical future work. Models are limited to the 7–8B parameter scale, so the degree to which the remaining-length variable sharpens or shifts at larger (frontier) scales remains unexplored.

Conclusion

This work provides robust evidence, via linear probing, that LLMs encode an internal, linearly accessible estimate of their own remaining output length. This length-tracking signal is decodable from the prompt, dynamically updated mid-generation (especially in cases of explicit self-correction), and is general enough to transfer across a wide array of natural language tasks. The findings reinforce a mechanistic narrative in which LLMs form high-level, plan-like representations—even when those variables are not used explicitly or causally in the forward computation graph. Future work should address direct intervention, aggregate quantification of dynamic updates, and scaling behaviors to further elucidate the theoretical and practical import of such internal representations.