Context Steering in LLMs

• Context Steering is defined as directly modifying transformer hidden states with learned linear concept directions to guide semantic outputs.
• The framework employs attention-guided token selection and soft labeling to overcome brittle fixed token methods and manual block choices.
• Empirical results show nearly double the steerable concepts across 512 semantic targets, enabling scalable, efficient control in LLMs.

Steering in LLMs denotes direct manipulation of internal activations to guide responses toward specified semantic concepts. In the activation-steering formulation adopted here, concept vectors are learned as linear directions in hidden-state space that distinguish concept-active from concept-inactive contexts, and are then added to or subtracted from hidden states during inference. “Efficient and accurate steering of LLMs through attention-guided feature learning” introduces an attention-guided framework that targets three persistent failure modes of prior steering methods—token selection, heterogeneity of concept-related features, and layer selection—and reports substantial gains on a benchmark of 512 semantic concepts, nearly doubling the number of successfully steered concepts across architectures and model sizes up to 70 billion parameters (Davarmanesh et al., 30 Jan 2026).

1. Activation-level steering as contextual control

The paper formalizes an LLM as a transformer map

$$f : \mathbb{R}^{T \times d} \to \mathbb{R}^{1 \times d},$$

with hidden states $H^{(\ell)} \in \mathbb{R}^{T \times k}$ at block $\ell$. Steering consists of identifying, at each block, a concept vector $v^{(\ell)} \in \mathbb{R}^k$ and intervening on internal activations during inference via

$$\tilde{H}^{(\ell)}_{t,:} = H^{(\ell)}_{t,:} + \epsilon\, v^{(\ell)}, \quad t \in [T],$$

where $\epsilon \in [-1,1]$ controls steering strength and sign. Concept vectors are treated as linear directions that separate concept-active from concept-inactive contexts in activation space (Davarmanesh et al., 30 Jan 2026).

Within this formulation, steering is presented as a more surgical form of context steering than prompting. Rather than relying only on extra text tokens, it modifies the contextual representation itself. The paper therefore treats steering simultaneously as an interpretability problem—how semantic concepts are stored inside transformer activations—and as a control problem—how those concepts can be manipulated efficiently at inference time.

The framework is written in standard transformer notation. Inputs are embedded as

$H^{(1)} = X W_e + W_p,$

subsequent blocks are given by

$$H^{(\ell)} = B^{(\ell)} H^{(\ell-1)}, \quad \ell = 2,\dots,L,$$

and the output logits are

$f(X) = (H^{(L)} W_o)_{T,:}.$

Attention matrices $A^{(\ell)}$ are central because the proposed method uses them to identify where concept activity appears within a layer, how strongly a concept is activated in a prompt, and which layers are most relevant for intervention.

2. Brittleness of earlier concept-vector methods

A central claim of the paper is that existing steering methods are remarkably brittle. Small implementation choices can make the same concept appear either non-steerable or highly steerable. The reported causes are threefold: prior work often uses a fixed token position, typically the last token, for feature extraction; it uses hard labels that assign all prefixed prompts to “concept present” and all unprefixed prompts to “concept absent”; and it manually chooses steering blocks or performs a grid search over them (Davarmanesh et al., 30 Jan 2026).

The empirical examples on the Llama-3.1-8B model are particularly sharp. For fear concepts, using the token start_header_id yields about 75% successful steering, whereas using end_header_id yields about 9%. For mood concepts, start_header_id yields about 73% success, whereas end_header_id yields about 99%. The same underlying dataset can therefore support radically different conclusions about steerability depending only on token choice. Similar sensitivity is reported across PCA, linear regression, logistic regression, and Recursive Feature Machines, and across models.

Brittleness source	Prior practice	Attention-guided remedy
Token selection	Fixed token position	Attention-guided token selection
Concept activity heterogeneity	Hard labels	Attention-based soft labels
Block selection	Manual choice or grid search	Concept enrichment scores

These observations reframe “non-steerable concept” as, in many cases, an artifact of extraction and intervention choices rather than a property of the concept itself. A plausible implication is that a large part of the historical instability of steering arose from supervision mismatch rather than from the absence of linear concept structure.

3. Attention-guided token selection and concept-extraction data

For each concept $H^{(\ell)} \in \mathbb{R}^{T \times k}$0, the framework constructs paired prompt sets from a common base. The experiments use $H^{(\ell)} \in \mathbb{R}^{T \times k}$1 generic statements such as “Life is what you make it,” together with a base prefix such as “What do you make of the following statement?” Two prompt sets are then formed: a concept-activated set $H^{(\ell)} \in \mathbb{R}^{T \times k}$2, obtained by adding a concept-specific prefix, and a concept-inactive set $H^{(\ell)} \in \mathbb{R}^{T \times k}$3, obtained by omitting that prefix. All prompts are wrapped in chat-formatting tokens such as start_header_id, assistant, end_header_id, and newline, which become candidate positions for concept features (Davarmanesh et al., 30 Jan 2026).

At each block $H^{(\ell)} \in \mathbb{R}^{T \times k}$4, the method selects a single token position $H^{(\ell)} \in \mathbb{R}^{T \times k}$5 whose embedding is maximally enriched in concept activity. Concept activity is operationalized heuristically as attention to prefix tokens. If $H^{(\ell)} \in \mathbb{R}^{T \times k}$6 indexes prefix positions and $H^{(\ell)} \in \mathbb{R}^{T \times k}$7 denotes candidate tokens, the selected token is

$H^{(\ell)} \in \mathbb{R}^{T \times k}$8

The intuition is that tokens that attend strongly to the inserted prefix are likely integrating the concept-specific perturbation, and their embeddings should therefore contain the relevant feature (Davarmanesh et al., 30 Jan 2026).

This mechanism replaces the fixed-position heuristic with a layer-specific selection rule. The paper reports that attention-based token selection tracks the empirically best-performing tokens and significantly outperforms an alternative heuristic based on the largest change in embedding norm between prefixed and non-prefixed prompts. In effect, attention is used as an endogenous signal for locating concept-bearing representations, rather than assuming that concept features reside at a universal token position.

4. Feature learning under heterogeneous concept activity

Once token positions are chosen, the feature vector for prompt $H^{(\ell)} \in \mathbb{R}^{T \times k}$9 at block $\ell$0 is

$\ell$1

These features populate per-layer datasets

$\ell$2

from which concept vectors are learned (Davarmanesh et al., 30 Jan 2026).

The paper studies five feature-learning methods.

Difference in Means computes

$\ell$3

followed by normalization and orientation using Pearson correlation with labels.

PCA on paired differences forms a matrix of prefixed–unprefixed embedding differences and takes the top eigenvector as the concept vector.

Linear regression with $\ell$4 regularization fits $\ell$5 and uses the normalized, oriented $\ell$6.

Logistic regression fits $\ell$7 to binary labels and again uses normalized, oriented $\ell$8.

Recursive Feature Machines (RFM) use the Average Gradient Outer Product

$\ell$9

to learn features through kernel ridge regression updates, after which the top eigenvector of the learned matrix is used as the concept vector (Davarmanesh et al., 30 Jan 2026).

A key innovation is the replacement of hard labels with attention-based soft labels. The paper argues that prefixes do not uniformly activate the concept across prompts, layers, heads, or tokens: some prompts strongly evoke the concept, others only weakly, and hard binary supervision conflates these cases. For concept-activated prompts, the soft label at block $v^{(\ell)} \in \mathbb{R}^k$0 is defined as the total attention from the selected token to the prefix tokens,

$v^{(\ell)} \in \mathbb{R}^k$1

while concept-inactive prompts receive label $v^{(\ell)} \in \mathbb{R}^k$2 (Davarmanesh et al., 30 Jan 2026).

This soft-label construction makes the supervision signal sensitive to degree of activation rather than mere presence of a prefix. The reported best overall steering performance is obtained when RFM is paired with attention-guided token selection and attention-based soft labels, indicating that both representation choice and label structure matter for recoverable concept directions.

5. Layer selection and the distribution of concept features

The third problem addressed by the framework is block selection. The paper argues that concept-related features are not uniformly distributed across layers, so steering arbitrary blocks—or all blocks—can waste capacity and sometimes hurt performance. To address this, the framework introduces concept enrichment scores for identifying layers most relevant for steering (Davarmanesh et al., 30 Jan 2026).

Although the paper’s abstract emphasizes steering accuracy, it also presents the framework as a tool for studying where concepts live in LLMs. The same machinery used to improve steering is used to examine the distribution of concept-specific features across layers. At minimum, the reported findings establish that layer relevance is concept-dependent and non-uniform rather than concentrated at a single canonical depth.

This suggests a more structured view of semantic storage in transformers. Instead of treating a concept as a monolithic direction recoverable from any convenient block, the framework treats concept expression as distributed, with distinct layers differing in how strongly they encode, integrate, or expose concept-related information. That interpretation is consistent with the need for explicit layer selection rather than universal intervention rules.

6. Empirical scope, relation to adjacent steering methods, and implications

Across a benchmark of 512 semantic concepts, the framework substantially improves steering over the previous state of the art, nearly doubling the number of successfully steered concepts across model architectures and sizes up to 70 billion parameters (Davarmanesh et al., 30 Jan 2026). The paper presents this as both a robustness result and a scalability result: attention-guided extraction replaces several brittle choices that previously had to be fixed manually or tuned by exhaustive search.

Within the broader steering literature, the method occupies a distinct position. Context Steering (CoS) reweights next-token logits by comparing forward passes with and without context and linearly scaling the contextual influence (He et al., 2024). In-context Vectors construct a latent vector from demonstrations and shift hidden states by that vector during inference (Liu et al., 2023). Steering Vector Fields argue that static steering vectors are unreliable because the locally effective direction varies with the current activation, and therefore define steering directions through gradients of a learned concept score (Li et al., 2 Feb 2026). COLD-Steer approximates the activation change that would result from one gradient step on labeled in-context examples and applies that change directly at inference (Sharma et al., 6 Mar 2026). SKOP, by contrast, constrains attention rerouting so as to preserve focus-token attention while retaining over 95% of vanilla steering efficacy and reducing utility degradation by 5–7x (Luo et al., 7 May 2026).

Against this background, the contribution of attention-guided feature learning is specific: attention is used not merely as an object of intervention but as the supervisory signal that determines where concept features are read out, how strongly training examples are labeled, and which layers should be steered. A plausible implication is that steering performance depends as much on recovering the correct causal slice of the model’s internal computation as on the downstream intervention rule itself. The paper therefore links interpretability and control more tightly than methods that treat concept vectors as fixed global artifacts, and it frames attention-guided steering as a route toward efficient, highly scalable fine-tuning algorithms for industry-scale LLMs (Davarmanesh et al., 30 Jan 2026).