They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On It

Abstract: When a person shares something with a LLM, the model often answers the surface of the message rather than what the sender was doing by sending it: share a finished project and it critiques the code; share a raw late-night line and it runs a wellness check. We treat the sender's communicative intent, the Gricean what-was-meant, as a first-class interpretability object, and show the failure is one of readout on top of a robust representation. A linear probe decodes the sender's intent, whether they want a thing recognized or evaluated, from a model's default-pass hidden states, cleanly and surface-independently, across six models and four families and in the base checkpoints. The representation generalizes further, to intent that is only pragmatically inferred, and to a second, lexically clean intent (support versus help). The behavioral half of the story, and every causal test, is established on the recognize/evaluate contrast, where what varies is whether the default output acts on the intent. The readout lags the representation in depth within a model (the intent is decodable several layers before it drives the output); across models, which ones act on it by default is model-specific, an observed stratification (three of six show the failure) that we do not read as a scaling law. Where the gap is open, a direction closely tied to the representation, the discriminative direction at a searched-for layer, is a causal handle: steering it recovers the intended behavior, as well as an explicit instruction does and with no prompt at all. This direction is near-orthogonal to the feedback-offering axis, so it routes a represented intent rather than a generic feedback knob, though at the recovery dose the routed intent can override an explicit request. We support each link with controls against obvious deflations and report the nulls as plainly as the confirmations.

• The paper reveals a robust, surface-independent encoding of communicative intent across transformer models, achieving ~99% probe accuracy.
• It demonstrates a dissociation between intent representation and behavioral readout, requiring model- and layer-specific activation steering to elicit intended outputs.
• The study highlights implications for interpretability and alignment, advocating for advanced activation-based interventions over traditional prompt engineering.

Dissociation of Communicative Intent Representation and Behavioral Readout in LLMs

Overview

This paper investigates the internal representations of communicative intent in transformer-based LMs, focusing on the distinction between recognizing a sender's intent (e.g., seeking acknowledgment vs. critical evaluation) and acting on that intent in generated outputs. Linear probes reveal that intent is robustly and surface-independently encoded in model activations across multiple architectures and training checkpoints. However, behavioral alignment is partial: in a significant subset of models, default generations often ignore communicative intent, offering unrequested feedback or critique. Critically, the paper demonstrates that steering model activations along directions linked to inferred intent can recover intent-honoring behavior without prompt engineering, exposing a represent-readout dissociation that is both technically and theoretically consequential for interpretability and alignment research.

Figure 1: The represent-then-lagging-readout chain, exemplified on Qwen-3B: intent can be decoded from activations before it is routed into default behavior; steering a learned direction in activation space recovers the desired output.

Methodology

Stimuli Construction and Probing

The authors operationalize intent using binary contrasts—primarily "recognize" (desire for acknowledgment) versus "evaluate" (desire for critique)—in message stimuli with carefully matched linguistic surfaces. To ensure separation from lexical artifacts, they leverage exhaustive phrasing permutations and adopt leave-one-phrasing-out cross-validation. Linear $\ell_2$ logistic regression probes (with PCA pre-processing) are trained on last-token activations, and a bag-of-words classifier serves as a baseline for lexical leakage.

Controls and Generalization Tests

• Empirical chance is calculated using shuffled labels.
• The probe is validated on both instruction-tuned and base models to assess pretraining effects.
• A request-matched variant controls for directive presence.
• Inferred intent is tested with contextually implied goals (no explicit request) and cross-validated against confounds of valence (warmth).
• All analyses are conducted on both synthetic and compositional hand-written messages to evaluate ecological validity.

Behavioral Measurement and Causal Steering

Honoring of recognize intent (i.e., withholding unsolicited feedback) is scored via lexicon-based and embedding-based classifiers, with human rater validation. Activation steering is conducted by adding learned directions (probe weights) at optimized layers in the residual stream during generation, and dose-responses are characterized. Robust specificity controls, including random and label-permuted directions, assess whether observed behavioral changes are truly causal manipulations of the learned intent representation.

Main Findings

Universal Representation, Variable Readout

• Intent is robustly and surface-independently decoded from deep activations ($\geq$99% probe accuracy) even when linguistic surface is controlled and across multiple model families (Qwen, Mistral, Phi, Llama).
• This representation generalizes to inferred (implied) intent and other axes such as "support" vs. "help," as well as to non-templated, naturalistic prompts.

Lag Between Representation and Behavioral Routing

• Within a model, layers exist where the intent is decodable but not yet routed into output behavior; steering at later layers is required to elicit intended honoring (Figure 2).
• The activation-behavior gap is model-specific: three of six models (Qwen-3B/7B, Llama-8B) exhibit strong behavioral discard (default generations ignore intent), while others (Qwen-14B, Mistral-7B, Phi-3.5) align with intent by default, irrespective of model size.

  Figure 2: Layerwise probe accuracy (blue) and behavioral recovery from steering (red); intent becomes decodable several layers before behavioral honoring is recovered, indicating a represent-readout gap.

Causal Steering Recovers Intent-Honoring Behavior

• In models with readout lag, steering the residual stream along a discriminative direction derived from probe weights recovers honoring to near-ceiling levels ($\sim$98%), without prompt modification and with behavioral specificity.
• The required steering direction is both model- and layer-specific; application of norm-matched random or label-permuted directions fails to induce equivalent behavioral changes, confirming the necessity of the learned intent direction.

  Figure 3: The effect requires the learned direction—distribution of behavior separation for various random and shuffled directions; only the learned direction achieves maximal behavioral discrimination.

Specificity and Mechanistic Analysis

• The causal direction is near-orthogonal to both the generic feedback axis and obvious lexical or opener-token directions, arguing against trivial explanations like verbosity or opener bias.
• The steering intervention has the same impact as an explicit intent-disambiguating system prompt. Their effects are additive when combined, yet steering remains applicable in fixed-prompt settings (e.g., agent loops).

Theoretical and Practical Implications

Interpretability: This work demonstrates that communicative intent is linearly embedded in model activations independently of output behavior, suggesting a mechanistically accessible distinction between "knowing" and "acting." Such a dissociation aligns with prior theoretical work on lossy memory (Kwon, 24 Jun 2026) and theory-of-mind failures in LMs.

Alignment: The selective inaccessibility of communicative intent to default readout underscores a limitation of instruction-tuning as a method for robustly aligning outputs to user desires, calling for more advanced readout or routing mechanisms—potentially activation-based interventions within the residual stream.

Steerability: The result that steering the precise, model-learned intent direction—rather than generic behavioral axes or verbosity—recovers desired output resolves concerns about activation addition methods being prone to non-specific side-effects, bolstering their validity as practical (if interventionist) alignment tools.

Scaling and Maturity: The lack of a monotonic scaling law (smaller models can outperform larger ones in honoring) and the role of instruction-maturity highlight that mere parameter growth or more data does not guarantee improved readout. Model- and layer-specific diagnostics will continue to be essential in assessing model alignment and control.

Directions for Future Work:

• Extending the probe-routing paradigm to more nuanced or socio-contextual communicative intents, especially those without clean binary segmentation.
• Investigating the interface between activation-space alignment and reinforcement learning from human feedback.
• Mechanistic identification of components (heads, MLPs) responsible for routing, potentially exposing repair levers for persistent readout failure.

Conclusion

This paper establishes that a sender's communicative intent is robustly, surface-independently represented within LM activations across architectures and training regimens, often learned during pretraining. However, behaviorally realizing this internal knowledge is model- and depth-dependent: a lag between representation and readout is observed. Where this gap exists, intent-directed activation steering recovers behavior as effectively as explicit instructions, without requiring prompt modifications. The findings compel a mechanistic reframing of model "misalignment": LMs frequently "know" the user's intent, but an incomplete behavioral readout prevents them from acting on it by default. This calls for a deeper integration of interpretability, alignment diagnostics, and targeted interventions as models continue to scale and diversify in deployment contexts.