Value Axis in Language Models

• Value Axis is a linear direction in a language model’s activation that encodes an internal estimate of expected goal success, analogous to an RL value function.
• It is constructed using synthetic in-context reinforcement learning, achieving high AUROC in distinguishing effective from failing trajectories.
• Causal steering along the value axis modulates behaviors like persistence, backtracking, and explanation style across language and coding tasks.

Searching arXiv for the main paper and closely related work on internal representations, steering, and alignment axes. The value axis is a linear direction in language-model activation space that encodes whether the model is internally “on the right track,” in the specific sense of estimating the likelihood that its current trajectory will achieve its goals. In the formulation introduced for Qwen3-8B, the axis is not defined by token likelihood alone and is not restricted to verbalized confidence; it is intended to capture an internal estimate of expected goal success for an ongoing strategy, analogous to an RL value function but expressed in the residual-stream dynamics of autoregressive generation (Jiang et al., 15 Jun 2026).

1. Conceptual definition

The central claim is that a LLM can linearly encode a trajectory-evaluation signal: high value corresponds to an internal state like “my current approach is likely to work,” whereas low value corresponds to “my current approach is probably going badly.” In this usage, “value” refers to the likelihood that the model’s current strategy will successfully accomplish the task, rather than to normative values, reward-model scores, or the probability of the next token. The paper explicitly connects this notion to the RL concept of a value function, that is, an estimate of expected future reward from the current state (Jiang et al., 15 Jun 2026).

This framing matters because it shifts analysis away from surface-level self-reports such as “yes, I am confident” and toward a latent control signal that appears to modulate persistence, revision, and explanatory style. The reported phenomena suggest that the representation is not merely lexical or sentiment-like. It is instead presented as a more general trajectory-evaluation signal that affects whether the model continues, backtracks, or elaborates.

A further implication is that confidence, on this account, is not treated as a purely linguistic behavior. It is modeled as one observable consequence of a deeper internal estimate of task progress. This suggests a distinction between verbal confidence and internal value: the former is a textual act, whereas the latter is a representational state that can influence many downstream behaviors.

2. Construction from synthetic in-context reinforcement learning

The value axis is constructed from synthetic in-context reinforcement learning (ICRL) conversations in which the model must infer a hidden criterion from binary feedback. The reported setup uses 300 conversations generated with Claude Opus 4.6. In each conversation, the model rewrites paragraphs to satisfy a hidden criterion such as including a dash, using a metaphor, or including a specific syntactic or semantic feature. The criterion is never stated directly; the model only receives +1 if the rewrite satisfies the criterion and -1 otherwise. A crucial event is the moment of “discovery,” after which the model can satisfy the criterion consistently on the first try (Jiang et al., 15 Jun 2026).

For layer $\ell$, the axis is defined as the average difference between post-discovery and pre-discovery hidden states on the first post-discovery paragraph:

$$\mathbf{v}^{(\ell)} = \frac{1}{|\mathcal{C}|} \sum_{c \in \mathcal{C}} \left( \frac{1}{|\mathcal{T}^c_{\text{post}}|} \sum_{t \in \mathcal{T}^c_{\text{post}}} \mathbf{h}^{(\ell)}_t - \frac{1}{|\mathcal{T}^c_{\text{pre}}|} \sum_{t \in \mathcal{T}^c_{\text{pre}}} \mathbf{h}^{(\ell)}_t \right).$$

At evaluation time, a sequence-level value score is computed by mean cosine similarity to the axis:

$$\text{val}^{(\ell)}(s)= \frac{1}{|s|} \sum_{t \in s} \cos\bigl(\mathbf{h}^{(\ell)}_t,\mathbf{v}^{(\ell)}\bigr).$$

This operationalizes “high value” as stronger alignment of token activations with the post-discovery direction.

The paper also uses a direct steering intervention at layer 21:

$$\tilde{\mathbf{h}}^{(21)}_t \leftarrow \mathbf{h}^{(21)}_t + \alpha \cdot \hat{\mathbf{v}}^{(21)},$$

where $\hat{\mathbf{v}}^{(21)}$ is the unit-normalized value vector and $\alpha$ sets the steering strength. This makes the axis not only a probe but also a causal intervention target.

3. Internal signatures across confidence, backtracking, and code

The synthetic construction generalizes strongly within the source task family. On 25 held-out criteria, the value axis at layers 21–22 achieves AUROC $> 0.95$ for distinguishing pre- vs. post-discovery tokens. The representation reportedly stabilizes in the middle-to-late layers, with a sharp directional shift around layer 13. A logit-lens analysis associates the direction with tokens such as “further,” “again,” “figure out a way,” “bonus points,” and “next time,” which the paper interprets as supporting persistence along the current path (Jiang et al., 15 Jun 2026).

The same axis is then evaluated outside the synthetic ICRL setting. On partial AIME rollouts, the value projection is higher for responses to “Do you think your answer is correct?” that say “yes” than for those that say “no,” and the effect flips when the question is inverted to ask whether the answer is incorrect. Using the last ten pre-response tokens, the projection separates confident from unconfident questions with AUROC $> 0.75$. This is presented as evidence that the axis is not a simple detector for the word “yes,” but tracks a more general internal belief about likely success.

A second signature concerns self-correction. Rollouts containing backtracking markers such as “Wait,” “Actually,” “Hold on,” or “Let me reconsider” have lower average value, and the projection drops sharply right before a backtracking event. The reported interpretation is that low value signals a failing trajectory and precedes reconsideration.

A third signature appears in code. On LeetCode / DebugBench-style problems, correct solutions have higher value projection than corrupted variants including logical bugs, syntax errors, shuffled lines, and obfuscated names. The strongest separation is reported for structurally disruptive corruptions such as shuffled lines and obfuscated names, while corruptions that preserve surface form are harder to distinguish. This suggests that the axis is sensitive to solution coherence, not only to local fluency.

4. Causal steering and post-training effects

Steering along the value axis changes behavior in a directional way. Positive steering increases verbalized confidence, suppresses backtracking, reduces explanatory verbosity in coding, and yields more direct, concise answers. Negative steering reduces verbalized confidence, induces backtracking, increases exploratory or reconsidering behavior, and produces longer, more heavily commented code. In AIME confidence probes, positive steering increases “yes” responses and negative steering increases “no” responses, with the inverted-question control again used to argue that the effect is about confidence rather than lexical priming (Jiang et al., 15 Jun 2026).

The paper also reports that post-training can reshape this internal value signal. In a DPO experiment, models are trained to prefer a specific word such as “grapefruit” or “keyboard.” After DPO, the preferred word becomes higher-value internally, and the model behaves more confidently after using it. Quantitatively, the preferred word is ranked highest by the value axis 21% $\rightarrow$ 36.2% of the time. The same phenomenon appears for words the model is trained to avoid, and it generalizes to natural sentences using the target word. An important nuance is that the corresponding value increase is not observed in the user prompt in the same way it is in the assistant’s own output, which the paper interprets as evidence that the axis tracks the assistant’s own trajectory rather than a generic notion of desirability.

Behavioral spillover is especially visible in coding. If a DPO-trained model is instructed to use its preferred word in variable names, it becomes less verbose, less comment-heavy, and less likely to use type hints. For models trained to avoid a word, the direction reverses. This links preference optimization to a change in internal trajectory valuation rather than only to altered token frequencies.

5. In-the-wild observations and domain specificity

The axis is also applied to less controlled settings. On 55K Chatbot Arena prompts, high-value prompts are usually constrained tasks, especially information extraction, whereas low-value prompts are often open-ended or politically sensitive. Reported examples of low-value prompts include “is Taiwan China?” and “is Palestine a country?” The base model does not show the same trend, which the paper uses to argue that post-training changes what the model treats as high- or low-value trajectories (Jiang et al., 15 Jun 2026).

Supervised fine-tuning exhibits a domain-specific effect. Fine-tuning on GSM8K and ARC-Challenge raises value on held-out questions from the trained benchmark, but not on an unrelated benchmark such as MATH-500. The paper interprets this as increased internal confidence within the training domain rather than a global increase in value.

A further case study concerns evaluation-awareness. Using a fine-tuned Nemotron model that behaves differently when it thinks it is being evaluated, the paper reports higher value for prompts recognized as evaluation contexts, even when the user prompt does not explicitly say “evaluation.” The effect reaches Cohen’s $d$ about $$\mathbf{v}^{(\ell)} = \frac{1}{|\mathcal{C}|} \sum_{c \in \mathcal{C}} \left( \frac{1}{|\mathcal{T}^c_{\text{post}}|} \sum_{t \in \mathcal{T}^c_{\text{post}}} \mathbf{h}^{(\ell)}_t - \frac{1}{|\mathcal{T}^c_{\text{pre}}|} \sum_{t \in \mathcal{T}^c_{\text{pre}}} \mathbf{h}^{(\ell)}_t \right).$$0 for eval vs. deploy prompts in the fine-tuned model and remains near 0 in the base model. This suggests that learned notions of evaluative context can alter the model’s internal estimate of success.

Taken together, these results present the value axis as a probe for how post-training redistributes confidence across prompt types, task domains, and institutional contexts. A plausible implication is that some alignment procedures act partly by changing which trajectories the model internally regards as feasible or rewarded.

6. Scope, limitations, and terminological distinctions

The paper is explicit about several limitations. Most analysis is performed on Qwen3-8B, so cross-family and cross-scale generality remains unresolved. The axis is built from one specific synthetic ICRL construction, and alternative constructions could isolate different components. The underlying notion of “belief about current value” is not mathematically exact in LLMs, so the resulting axis is not uniquely defined. The benchmark set was chosen in settings where the expected behavior was already strongly hypothesized, and the distinction between confidence, value, and desirability remains only partially settled (Jiang et al., 15 Jun 2026).

These caveats constrain interpretation. The reported evidence supports a linear signal correlated with expected goal success and causally implicated in persistence, verbosity, and self-correction, but it does not show that all task planning reduces to a single scalar, nor that the signal is architecture-invariant. This suggests that “the value axis” is best understood as an operationally useful internal direction rather than a complete theory of LM agency.

The expression also has distinct meanings elsewhere in the recent literature. In CALMA, “axes” are externally elicited, context-aligned criteria for evaluation and alignment, with pilot-derived axes such as Cultural Context, Source, Empathy, Inclusive, and Fact / Power; these are community-defined normative dimensions, not latent residual-stream directions (Soni et al., 11 Jul 2025). In MoVE, the “Value Axis” denotes a new scaling dimension for parametric memory in autoregressive models by expanding a global bank of learnable value embeddings shared across attention layers; there, “value” refers to the transformer value stream rather than to internal trajectory evaluation (Li, 30 Jan 2026).

Within LM interpretability, however, the specific notion introduced in Qwen3-8B is narrower and more technical: a linear direction in activation space that appears to encode whether the model believes its current generation strategy is likely to succeed. Under that interpretation, the value axis functions simultaneously as a probe of internal confidence, a handle for causal steering, and a lens on how post-training reshapes model behavior.