---
title: ICL Activation Alignment (IA²)
url: https://www.emergentmind.com/topics/icl-activation-alignment-ia
type: topic
---

# ICL Activation Alignment (IA²)

ICL Activation Alignment (IA²) denotes an activation-centric approach to relating in-context learning (ICL) to weight-based or activation-based control in large language models. In its primary formulation, IA² is a self-distillation technique that aligns a fine-tuned model’s hidden activations with activations produced under ICL, with the explicit aim of replicating ICL-like internal reasoning during supervised fine-tuning [2509.22621]. In a related but distinct usage, the same designation is used for a control perspective in which ICL contributes log-evidence and activation steering shifts log-prior odds, yielding a closed-form account of prompt length–steering trade-offs and phase boundaries in behavior control [2511.00617]. Taken together, these formulations place internal activations at the center of the relationship between prompt-based adaptation, supervised specialization, and inference-time steering.

## 1. Scope, motivation, and problem setting

IA² arises from the contrast between two established adaptation mechanisms. Supervised Fine-Tuning (SFT) updates model weights $\theta \to \theta'$ via backpropagation on a dataset of $(X_i,Y_i)$ pairs using a cross-entropy loss, after which the model no longer needs demonstrations at inference time. ICL leaves the base weights $\Theta$ intact and instead prepends a handful of input–output demonstrations $I=[X_1\Vert Y_1 \ldots X_n\Vert Y_n]$ to a new input $X_t$, letting the model adapt on the fly via its self-attention circuits [2509.22621].

The central empirical motivation is that these two forms of adaptation do not induce the same internal computations. In SFT, the model sees only $X$ and produces $Y$, so its internal activations $A_{\text{SFT}}$ reflect adaptation through weight changes that directly optimize token-level likelihoods. In ICL, the self-attention layers attend over both demonstration and query tokens, producing activations $A_{\text{ICL}}$ that embed the logic of processing examples in context. Across 100 held-out samples from text-classification benchmarks and multiple LLMs, the cosine similarity ($\text{asim}$) between ICL and SFT hidden activations at output token positions was reported as very low, especially in middle layers. The same experiments also found that ICL often yields better calibration, measured by lower expected calibration error (ECE), than SFT in few-shot settings, at comparable or slightly better accuracy.

These observations motivate IA² as a way to transfer not only outputs but also intermediate computation. Rather than treating ICL merely as a source of labels or demonstrations, IA² treats ICL activations as a target signal. A related motivation appears in the Bayesian control account of prompting and activation steering, where behavior is modeled through belief in latent concepts. That account does not define the same training pipeline, but it similarly treats activations as an intervention point and uses the label “IA²” for a predictable behavior-control perspective [2511.00617].

## 2. Formal definition as activation alignment

In the self-distillation formulation, the base model is a decoder-only transformer $M_\Theta$ with $L$ layers. Each self-attention block has weight matrices $W_Q$, $W_K$, $W_V$, and $W_O$, collectively $W_{QKVO}\subset\Theta$. For a token sequence $T$ of length $R$, self-attention at layer $\ell$ produces hidden activations $A[\ell]\in\mathbb{R}^{R\times d}$. If the model generates $G$ response tokens, the method focuses on activations at those $G$ output positions [2509.22621].

For sample $i$, demonstrations
$$
I_i=[X_1\Vert Y_1 \ldots X_n\Vert Y_n]
$$
using all but $X_i$ are prepended to $X_i$ to form
$$
T_i^{\text{ICL}}=[I_i\Vert X_i].
$$
Running the model on this prompt yields a response $\hat Y_i$ and activation tensor
$$
A^{\text{ICL}}_i\in\mathbb{R}^{L\times G\times d}.
$$
The same model is then run on the sequence $[X_i\Vert \hat Y_i]$ without demonstrations, producing activations
$$
A_i\in\mathbb{R}^{L\times G\times d}.
$$
IA² trains the model so that $A_i \approx A^{\text{ICL}}_i$.

The self-attention operator is written as
$$
SA(T;W_{QKVO})=Z
$$
with
$$
z_i=\operatorname{softmax}\!\left(\frac{q_iK^\top}{\sqrt d}\right)V\cdot W_O,\qquad
q_i=t_iW_Q,\qquad K=t_{:i}W_K,\qquad V=t_{:i}W_V.
$$
The activation alignment objective over $N$ training samples is
$$
L_{\text{IA}^2}
=\sum_{i=1}^N \|A_i-A_i^{\text{ICL}}\|_F^2,
$$
where $\|\cdot\|_F$ is the Frobenius norm. For reference, the standard SFT objective is
$$
L_{\text{SFT}}
=\sum_{i=1}^N\sum_{j=1}^G CE(\hat y_{ij},y_{ij}).
$$

This formalization is specific about what is being transferred: not the demonstration prompt itself, and not only the predicted response, but the layer-wise hidden state trajectory at generated output positions. The method is therefore distinct from output-only distillation.

## 3. Training pipeline and implementation pattern

The standard workflow is a two-stage pipeline, denoted “IA² $\to$ SFT” [2509.22621].

In **Step 1: IA² priming (self-distillation)**, for each training sample $(X_i,Y_i)$, demonstrations $I_i$ are formed from the other $N-1$ samples. The model runs in ICL mode on $[I_i\Vert X_i]$, generates $\hat Y_i$ up to $G$ tokens, and records $A_i^{\text{ICL}}$. The model is then run on $[X_i\Vert \hat Y_i]$ without demonstrations, records $A_i$, and updates only adapter parameters to minimize $L_{\text{IA}^2}$ until convergence. The detailed description emphasizes adapter updates such as LoRA blocks on $W_Q,K,O$.

In **Step 2: SFT on ground-truth (cross-entropy)**, the model initialized with the primed weights $\tilde \Theta$ is trained with the standard cross-entropy loss $L_{\text{SFT}}$ on $(X_i,Y_i)$ until convergence, yielding final $\Theta'$.

The reported implementation uses Qwen3-4B-Base for all tasks and Llama-3.2-1B, with Llama-3.2-3B for some tasks, using LoRA rank-8 adapters on $W_Q,K,O$. In Step 1, the model generates up to $G=200$ tokens to collect activation tensors $A\in\mathbb{R}^{L\times 200\times d}$. Learning rates were chosen from $\{1\text{e--}4,3\text{e--}4,1\text{e--}3\}$ by validation selection. For multi-token SFT, decoding uses $G$ equal to the ground-truth length.

The procedural point of the method is that alignment precedes token-level specialization. The priming stage is intended to bias the fine-tuning trajectory toward internal computations that resemble ICL rather than toward direct output alignment alone.

## 4. Benchmarks, regimes, and empirical performance

The empirical evaluation covers 12 popular benchmarks and 2 model families [2509.22621]. The task suite includes single-token classification and multi-token generation. Single-token classification includes SST2, FinS, PoemS (ID+OOD splits), StrategyQA (STF), AGNews$\to$AGNews/BBCNews, and SciQ$\to$SciQr/QASCr. Multi-token generation includes GSM8K$\to$GSM8Ks, HMathA, and generative SciQ answers. Few-shot regimes use $N\in\{2,4,8,16\ldots\}$, with five random subsets for each $N$, and OOD evaluation on held-out splits of size 500.

The reported results show a consistent pattern. In **Table 1 (Qwen-4B single-token, $N=4$)**, SFT only achieves accuracy $\sim 28$–$67\,\%$ with ECE $\sim 0.20$–$0.56$. IA² only reaches similar accuracy to ICL, for example $63\,\%$ vs $64\,\%$, with ECE as low as $0.10$. IA²$\to$SFT achieves consistently best accuracy, for example $78$–$92\,\%$, with moderate calibration, ECE $\sim 0.05$–$0.40$, often exceeding ICL accuracy.

In **Table 2 (Qwen-4B multi-token, $N=4$)**, the reported values are:
- GSM8K: SFT $70.9\,\%$, IA² $\approx 77.4\,\%$, IA²$\to$SFT $73.6\,\%$, versus ICL $76.4\,\%$.
- HMathA: SFT $50.4\,\%$, IA² $47.8\,\%$, IA²$\to$SFT $55.3\,\%$, closer to ICL $60.4\,\%$.
- SciQ gen: SFT $35.0\,\%$, IA² $6.9\,\%$, IA²$\to$SFT $40.8\,\%$, outperforming ICL $37.5\,\%$.

The abstract-level summary is that performing IA² as a priming step before SFT significantly improves the accuracy and calibration of model outputs. The detailed tables indicate that the gains are not uniform in the same way for “IA² only” and “IA²$\to$SFT”: the strongest and most stable improvements are associated with the two-stage pipeline rather than the alignment objective in isolation.

## 5. Mechanistic interpretation, calibration, and ablations

The method is accompanied by several mechanistic analyses [2509.22621]. Figure 3 reports a scatter of $\text{asim}(A,A_{\text{ICL}})$ versus accuracy and ECE, showing that higher activation alignment yields lower ECE linearly, and that the combination IA²$\to$SFT trades off some pure ICL alignment for improved accuracy. This links hidden-state similarity not merely to imitation of ICL traces but specifically to calibration behavior.

Figure 4 analyzes subspace overlap, using an SVD-based comparison between adapter weight updates. The reported result is that SFT-only spans a subspace almost orthogonal to IA² and IA²$\to$SFT, whereas IA²$\to$SFT shares $\sim 39\,\%$ overlap with IA² priming. The stated interpretation is that IA² discovers useful weight directions inaccessible to SFT alone. This is presented as evidence that ordinary SFT lacks a training signal present in ICL activations.

Additional ablations reinforce that the signal is not tied to one adapter family or one supervision format. A unified loss IA²+$\beta\cdot L_{\text{SFT}}$ on ICL-generated responses shows that the IA² signal is rich even without ground-truth tokens. Replacing LoRA with IA³ adapters yields the same qualitative gains.

The discussion section attributes the benefits of IA² to three closely related effects. First, ICL activations encode the model’s internal pattern-extracting circuits, so aligning SFT models to these activations forces them to replicate that internal “reasoning” rather than rely on shallow heuristics. Second, increased activation similarity strongly correlates with better calibration and improved generalization in low-data regimes. Third, the weight subspace analysis indicates that IA² identifies functional directions not reached by ordinary SFT.

## 6. Bayesian belief dynamics, steering trade-offs, and the broader IA² usage

A distinct but related formulation appears in the Bayesian account of prompt-based and activation-based control [2511.00617]. There, model behavior on a new query is mediated by a latent binary concept $c\in\{c,c'\}$:
$$
p(y\mid x)=\int p(y\mid c)\,p(c\mid x)\quad\propto\quad \int p(y\mid c)\,p(x\mid c)\,p(c).
$$
The posterior odds are
$$
o(c\mid x)=\frac{p(c\mid x)}{p(c'\mid x)}
=\frac{p(c)\,p(x\mid c)}{p(c')\,p(x\mid c')},
\qquad
p(c\mid x)=\sigma\bigl(\log o(c\mid x)\bigr),
$$
with prior odds
$$
\log\frac{p(c)}{p(c')}=:b.
$$
For $N$ in-context exemplars, the log-likelihood ratio is modeled as
$$
\mathcal E(N)=\log\frac{p(x\mid c)}{p(x\mid c')}
\approx \gamma N^{1-\alpha},
$$
so that
$$
p(c\mid x)=\sigma\bigl(b+\gamma N^{1-\alpha}\bigr).
$$

Activation steering is then modeled as a prior modification. Under the Linear Representation Hypothesis, there exists a steering direction $d$ in some hidden layer such that adding $m\cdot d$ to the representation $v$ shifts the model’s log-odds for $c$ by a constant amount proportional to $m$:
$$
\log\frac{p(c\mid v+md)}{p(c'\mid v+md)}
=
\log\frac{p(c\mid v)}{p(c'\mid v)}+am.
$$
Equivalently, the prior offset becomes $b+am$.

The combined update is therefore additive in log-space:
$$
\log o(c\mid x,m)=b+am+\gamma N^{1-\alpha},
\qquad
p(c\mid x,m)=\sigma\bigl(b+am+\gamma N^{1-\alpha}\bigr).
$$
Setting $\log o(c\mid x,m)=0$ yields the closed-form phase boundary
$$
N^*(m)=\Bigl[-\,\tfrac{b+am}{\gamma}\Bigr]^{1/(1-\alpha)}.
$$
The theoretical predictions are: **sigmoidal ICL learning curves**, **sigmoidal steering response**, and **phase transitions & additivity**. The empirical validation uses five persona-adoption tasks, Llama-3.1-8B-Instruct as the main model, and replications on Qwen-2.5-7B and Gemma-2-9B. The procedure varies $N\in\{0,1,2,\dots,128\}$ and steering magnitude $m\in[-10,+10]$, with fine steps around $[-1,1]$, and measures the Yes/No “persona-consistent” probability. Reported metrics include held-out cross-validated correlation $r\approx 0.98$ between predicted $p(c\mid x,m)$ and observed, and $r\approx 0.97$ for predicted $N^*$.

This usage of IA² is not identical to the activation-alignment training method. The former is an inference-time control framework in which ICL adds evidence and steering shifts the prior; the latter is a self-distillation method that aligns hidden activations during fine-tuning. The connection is that both treat ICL as a structured internal computation rather than as a black-box prompting effect. The practical tuning guidelines in the Bayesian formulation specify a **linear range of steering** in which steering is linear in $\log o$ only up to a magnitude threshold, roughly $\lvert m\rvert \lesssim 1$ in these models; recommend **balancing $m$ vs. $N$** through $\Delta N^*(m)=-(a/\gamma)m$ in the exponentiated boundary; propose **predictive control** by fitting $(\alpha,\gamma,a,b)$ once on a small grid of $(N,m)$ and then solving for a desired posterior target $p^*$; and emphasize **input invariance of steering**, since steering shifts the prior and applies equally to any query $x$, whereas ICL must be tailored per task distribution.

The broader significance of IA², across both usages, is therefore a shift from output-level adaptation to activation-level analysis and control. In one formulation, that shift improves supervised fine-tuning by internalizing ICL’s activation patterns. In the other, it provides a closed-form model for trading off prompt evidence and steering magnitude. Future work explicitly mentioned for the alignment method includes selective layer alignment, alternative distance metrics such as cosine versus MSE, rank-selection in adapters, and prompt optimization.

Source: https://www.emergentmind.com/topics/icl-activation-alignment-ia