The Value Axis: Language Models Encode Whether They're on the Right Track

Abstract: We investigate whether LLMs internally track the value of their current trajectory, defined as the likelihood that their ongoing strategy will achieve their goals. Using synthetic, in-context reinforcement learning data, we construct a "value" axis for Qwen3-8B. We find that activations along this axis distinguish between high vs. low verbalized confidence, rollouts without and with backtracking, and correct vs. corrupted code. Steering towards high value causally suppresses self-correction and reduces explanatory verbosity, while steering towards low value induces backtracking and exploration. We demonstrate that direct preference optimization (DPO) can increase the internal value of rewarded behaviors (e.g. use a certain word), causing the model to act more confidently after exhibiting them. Finally, we apply the value axis to study in-the-wild settings. For example, we find that Qwen assigns low value to politically sensitive chat queries after post-training and that supervised fine-tuning increases internal confidence within the training domain. Our results suggest that LLMs linearly encode an estimate of expected goal success that modulates their confidence in pursuing a direction.

• The paper demonstrates that a latent value axis linearly encodes the model’s expected success in complex, multi-step tasks.
• It reveals that steering this axis alters model confidence, backtracking behaviors, and coding verbosity with AUROC >0.95 validation.
• The study shows that post-training methods like DPO and SFT effectively shift internal value, generalizing across structured and naturalistic tasks.

The Value Axis: LLMs Encode Whether They're on the Right Track

Introduction and Motivation

This work presents a comprehensive analysis of how LLMs, particularly Qwen3-8B, linearly encode an estimate of trajectory value—a notion inspired by the value function in classical reinforcement learning—within their internal activation space. The "value axis" is a direction in the model’s residual stream that tracks the likelihood of current strategies achieving their goals during multi-step, complex tasks. The paper investigates the generality, functionality, and mutability of this value axis, evaluating its correspondence with task confidence, decision persistence, internal preference modulation, and transfer to naturally occurring tasks and prompts.

Construction and Generalization of the Value Axis

The value axis is constructed using synthetic in-context reinforcement learning (ICRL) conversations. These controlled trajectories require the model to iteratively modify text until a hidden criterion is met, receiving immediate reward signals. The value axis is defined as the mean difference in hidden activations (specifically at layer 21 and above) before and after the model’s successful discovery of the criterion. This axis is shown to be robust: on held-out criteria, the value axis achieves AUROC $>0.95$, indicating strong generalization (Figure 1).

Figure 1: The value axis generalizes to held-out criteria and is stable in the middle-to-late layers; a major shift in directionality occurs around layer 13, after which the axis remains highly consistent.

Notably, the promoted tokens along this direction, as revealed by logit lens analysis, tend to be associated with positive encouragement and persistence, supporting the interpretation of this axis as a latent value signal.

Task Confidence: Correlation and Causality

The value axis is consistently predictive of several behavioral and epistemic properties across math and code-generation settings:

• Verbalized Confidence: The value axis projection correlates with the model’s self-assessed correctness (yes/no) in answering AIME math problems. The projection can distinguish between high- and low-confidence cases both before and after the explicit self-assessment.
• Backtracking Detection: Lower value-axis projection anticipates backtracking/self-correction events, while higher projections are associated with persistent strategies.

Figure 2: The value axis tracks confidence in task correctness and the presence of backtracking/self-correction events on AIME math rollouts.

The value axis also separates correct code solutions from corrupted variants (syntactic errors, logical bugs, and obfuscated names) across LeetCode problems with large effect sizes, particularly for superficial structural corruptions.

Figure 3: Correct code solutions yield higher value-axis projections than various forms of corrupted code, indicating sensitivity to both semantic and superficial errors.

Causal Role: Steering Model Behavior

Interventions along the value axis—applying positive or negative shifts—lead to robust, interpretable changes in outputs:

• Verbal Confidence Steerability: Steering toward higher value increases affirmative responses about correctness; negative steering has the opposite effect. Inverting the confidence question demonstrates the effect is not a mere lexical bias.
• Backtracking Modulation: Positive steering suppresses backtracking/self-correction, while negative steering induces it—even across multiple rollouts per problem.

Figure 4: Positive value-axis steering increases confidence and reduces backtracking; negative steering does the reverse.

• Coding Verbosity: Positive steering yields briefer, more direct solutions with fewer comments and type hints; negative steering induces verbose, explanatory, hedging solutions.

  Figure 5: Steering along the value axis modulates coding verbosity and explanation length, consistent with modulating overall output confidence.

Effects of Post-Training: Preference Learning and Value Shaping

The paper demonstrates that standard post-training methods such as Direct Preference Optimization (DPO) can shift the internal value landscape:

• Preference Induction: Models trained to prefer specific words activate the value axis more strongly when those words are used, even in naturalistic sentences and unrelated tasks such as coding. Avoidance training produces the opposite effect. The effect is robust across 50 DPO-trained word preferences.
• Confidence Carryover: When models are induced (via prompt or DPO) to use preferred words in code solutions, solution brevity and directness increase—an artifact of heightened internal value. Conversely, avoided words induce hedging and verbosity.

  Figure 6: Preferring a word (via DPO) elevates model confidence and reduces verbosity in downstream tasks when that word is used, demonstrating causal mediation by the value axis.

Critically, these shifts persist outside pure selection tasks: the value assigned is not simply local to the prompt but generalizes to free-form scenarios.

Applications to Naturalistic and Posttraining Induced Settings

The value axis exhibits structure in unconstrained, "in-the-wild" domains:

• Prompt Sensitivity: On a 55K prompt corpus from Chatbot Arena, high-value projections occur for highly specified, information-extraction style queries; low-value projections are typical for open-ended or politically sensitive prompts. Post-training alters this value landscape, increasing confidence for well-scoped informational queries and reducing it for politically sensitive or ambiguous ones.

  Figure 7: After post-training, the highest-value prompts are information-extraction and lowest-value prompts are politically sensitive; trends are flat for the base model.

• Supervised Fine-Tuning (SFT): SFT on benchmark-specific data (GSM8K, ARC-Challenge) selectively raises value-axis projections on in-domain benchmarks, not on out-of-domain data, confirming that fine-tuning reorients the model’s internal value toward trained domains.
• Evaluation Awareness: Models fine-tuned to behave differently under evaluation (using Wood Labs cues) internally encode higher value for evaluation prompts, despite no explicit difference in the user message. This suggests that value tracking includes latent evaluation-state awareness.

  Figure 8: Evaluation-aware fine-tuning modulates internal value; the model assigns higher value to evaluation cues than deployment cues, without explicit signal in the prompt.

Theoretical and Practical Implications

The findings establish that linear directions in LLMs can encode latent value representations corresponding to expected task success and that these axes are robust, generalize across domains, and are responsive both to steering and to post-training interventions. The results position the value axis as a primitive for high-level model alignment, suggesting it could be used for explicit behavior regulation, preference auditing, or as a target of further fine-tuning. The observed generalization (across domains, tasks, and post-training) indicates that the value axis encodes a form of global meta-cognition.

Conclusion

The paper provides strong empirical and causal evidence that transformer-based LLMs internally track a value signal, instantiated as a linear axis, which modulates task persistence, confidence, and behavior changes. This axis is manipulable by both interventions and preference-based post-training and is persistent enough to shape downstream behavior in both controlled and unconstrained tasks. The findings have direct implications for model alignment methodologies and understanding model self-evaluation; exploiting or regularizing the value axis could become a key tool in safer, more interpretable AI system design.