---
title: Learned Task Vector
url: https://www.emergentmind.com/topics/learned-task-vector
type: topic
---

# Learned Task Vector

Searching arXiv for the cited papers to ground the article in current literature.
{"query":"id:2606.05367 OR id:2310.15916 OR id:2506.09048 OR id:2406.16007 OR id:2506.03426 OR id:2604.11129","max_results":10}
A learned task vector is a task-conditioned representation that encodes task-specific behavior as a direction in some model-associated space. In contemporary usage, the term does not denote a single formal object. In weight-space formulations, it is the parameter difference between a fine-tuned model and a pre-trained model. In in-context learning, it is a hidden-state representation distilled from demonstrations and reused by patching or injection. In other settings, it appears as a learned contextual input, a decoding-space logit shift, or a conditioning-space direction such as a speaker embedding offset for controllable text-to-speech. Across these formulations, the common idea is that a task can be summarized by a compact vector that supports addition, negation, scaling, or interpolation, thereby steering model behavior without reconstructing the full original training context [2310.15916; 2407.02880; 2604.11129; 2606.05367].

## 1. Conceptual scope and principal definitions

The most classical formulation defines a task vector in parameter space. Given a pretrained model with weights $\theta_{\mathrm{pre}}$ and a model fine-tuned on task $t$ with weights $\theta_{\mathrm{ft}}^{(t)}$, the task vector is
$$
\tau_t = \theta_{\mathrm{ft}}^{(t)} - \theta_{\mathrm{pre}},
$$
and the edited model is
$$
\theta_{\mathrm{new}} = \theta_{\mathrm{pre}} + \lambda \cdot \tau_t.
$$
This definition underlies task arithmetic in model editing and model merging [2606.05367; 2407.02880].

A distinct formulation arises in in-context learning. There, a task vector is a single reusable representation distilled from demonstrations. In one influential construction, if the input sequence is $[S, x', \to]$, the task vector is the hidden state at the arrow token after an intermediate layer $L$,
$$
\theta(S) := h_L(\to; [S, x', \to]),
$$
and it is later injected into a forward pass on $[x, \to]$ to approximate ordinary in-context learning [2310.15916]. Related work formulates task vectors in triplet-formatted prompts as hidden states at arrow tokens and analyzes them as linear combinations of demonstration-derived states [2506.09048].

Other literatures use the same term for compact task-conditioning variables rather than extracted hidden states. In learned-context multi-task networks, the learned task vector is the contextual input $c_t \in \mathbb{R}^k$ concatenated to the ordinary input and optimized jointly with shared parameters [2303.00788]. In Adaptive Task Vectors, the vector is query-conditioned: a small language model computes $v(x)=\phi(x;\theta_s)$, which is mapped into layer-wise modulators and additively injected into a frozen large language model [2506.03426]. In DeCoVec, the task vector lives in decoding space and is defined at each decoding step as a logit difference,
$$
v_T^t = z_t^{\mathrm{icl}} - z_t^{\mathrm{zs}},
$$
which is added to the decoding logits during generation [2604.11129].

A concise taxonomy is therefore useful.

| Space | Representative definition | Representative papers |
|---|---|---|
| Weight space | $\tau_t=\theta_{\mathrm{ft}}^{(t)}-\theta_{\mathrm{pre}}$ | [2407.02880], [2508.16082] |
| Activation space | Hidden state at a designated token/layer | [2310.15916], [2506.09048] |
| Distributed rule space | Per-demo rule vectors at answer tokens | [2406.16007] |
| Decoding space | Stepwise logit difference $z_t^{\mathrm{icl}}-z_t^{\mathrm{zs}}$ | [2604.11129] |
| Conditioning space | Speaker/style embedding direction | [2606.05367] |
| Task-conditioned input space | Learned contextual input $c_t$ or $\beta_j$ | [2303.00788] |

This plurality suggests that “learned task vector” is best understood as a family resemblance term rather than a single invariant formalism.

## 2. Weight-space task vectors and task arithmetic

In weight-space work, the learned task vector is the parameter displacement produced by fine-tuning. This interpretation supports addition, negation, and scaling of tasks via vector arithmetic in parameter space [2606.05367]. The aTLAS framework refines this picture by decomposing parameters into blocks $B=\{b_1,\dots,b_m\}$ and learning anisotropic block-wise coefficients,
$$
\Lambda_t = \mathrm{diag}(\lambda^{(t)}_{b_1} I^{(b_1)}, \dots, \lambda^{(t)}_{b_m} I^{(b_m)}),
$$
so that multiple task vectors can be composed as
$$
v_{\mathrm{comb}} = \sum_{t=1}^T \Lambda_t \Delta \theta^{(t)}.
$$
This blockwise parameterization is reported to reduce interference, improve task addition and negation, and leverage the low intrinsic dimensionality of fine-tuned solutions [2407.02880].

A complementary theoretical line links task vectors to gradients. Under full-batch gradient descent, the one-epoch task vector satisfies
$$
v_\tau^{(1)} = -\eta \nabla L_\tau(\theta_{\mathrm{base}}),
$$
so a one-epoch task vector is exactly a scaled negative gradient of the task loss. For multi-epoch training, the equivalence holds approximately up to a second-order curvature term, which leads to the interpretation of task arithmetic as approximate multitask learning [2508.16082]. This result is used to explain why merging models fine-tuned for only a single epoch can remain competitive with merging fully converged models [2508.16082].

Theoretical analysis has also addressed when task arithmetic is provably effective for nonlinear Transformers. In a conceptual binary-classification setting, task addition is shown to succeed for irrelevant or aligned tasks, task negation to succeed for irrelevant or contradictory tasks, and suitable linear coefficients to support out-of-domain generalization. The same guarantees are stated to hold for both dense-weight parameters and their low-rank approximations [2504.10957]. This suggests that weight-space learned task vectors are not merely heuristic editing directions, but can admit formal generalization guarantees under structured assumptions.

## 3. Activation-space task vectors in in-context learning

Activation-space formulations treat the task vector as an internal representation created during in-context learning rather than a parameter update. “In-Context Learning Creates Task Vectors” formalizes this view by positing
$$
T([S,x]) = f(x;A(S)),
$$
with the task vector extracted as
$$
\theta(S):=h_L(\to;[S,x',\to]),
$$
where a dummy query $x'$ is used to prevent dependence on the actual query $x$ [2310.15916]. Empirically, this hypothesis-based procedure retains about 80–90% of regular in-context learning accuracy across seven large language models and 18 tasks, far above a baseline without demonstrations [2310.15916].

Subsequent work has made the emergence mechanism more explicit. In linear transformers trained on triplet-formatted prompts, the Linear Combination Conjecture states that the injected task vector functions as a single in-context demonstration formed through a linear combination of the original demonstrations [2506.09048]. The triplet analysis gives an explicit form,
$$
z_{\mathrm{tv}} = [\alpha_1 X\beta;\; \alpha_2 Y\beta],
$$
showing that the arrow-token hidden state is a weighted sum of demonstration-derived states [2506.09048]. This theory also proves a rank limitation: a single task vector induces an effective predictor
$$
W' = Y\beta (X\beta)^\top,
$$
which is rank one, explaining failures on high-rank mappings such as general bijection tasks [2506.09048].

Large-scale analysis has challenged the strongest version of the single-vector hypothesis. On the QuiteAFew dataset of 3,096 few-shot tasks, performance peaks at an intermediate layer, specifically the 15th layer in Llama-3-8B, but effectiveness varies strongly by task type [2505.23911]. For simple tasks, a single hidden state at the final separator token can still be useful. For complex outputs such as structured JSON generation, multiple sub-task vectors extracted from specific output tokens outperform a single global vector, suggesting distributed task knowledge representation [2505.23911].

A further refinement appears in “Learning Task Representations from In-Context Learning,” where the task vector is represented as a weighted sum of attention head outputs,
$$
v_\ell^t = \sum_{j=1}^{J} \omega_{\ell,j}\,\bar{a}_{\ell,j},
$$
with the weights optimized causally while the base model is frozen and in-context learning is blocked [2502.05390]. This method is reported to generalize across both language and functional regression tasks and to work by aligning the distribution of the last hidden state with that of an optimally performing in-context-learned model [2502.05390].

## 4. From localized vectors to distributed rule representations

A major controversy in the literature concerns whether task information is localized in a single hidden-state vector or distributed across many positions. The distributed-rule-vector study argues that, for tasks whose rules require multiple demonstrations, a single task vector does not exist in the strong sense proposed by earlier work [2406.16007]. Instead, each demonstration contributes its own rule vector located at its answer token, and the final prediction depends on a distributed aggregation,
$$
h_t^{(l)} \approx b^{(l)} + \sum_i r_i^{(l)}.
$$
Patching donor hidden states at each demonstration answer token rescues performance in categorization tasks, whereas single task-vector patching remains near chance [2406.16007].

This conclusion does not eliminate localized task vectors altogether. The same study reports that in knowledge-retrieval tasks or tasks solvable from one demonstration, a single task vector at the final query’s “is” token can recover few-shot-like accuracy [2406.16007]. A plausible implication is that single-vector and distributed-vector accounts describe different operating regimes rather than mutually exclusive theories. The large-scale QuiteAFew analysis is consistent with this view: single task vectors are useful for some task families, whereas complex tasks require multiple subtask-specific vectors [2505.23911].

Recent synthetic work has provided a geometric framework for this coexistence. “Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers” defines task vectors as long-context limits of centered task-conditioned mean hidden states,
$$
v_k = \lim_{t\to\infty}\big(E[h_t\mid z=k]-\bar{h}_t\big),
$$
and shows that in-distribution behavior is governed by Bayesian task retrieval through convex combinations of learned task vectors, while out-of-distribution behavior arises through extrapolative task learning in a subspace nearly orthogonal to the task-vector subspace [2605.03780]. In that analysis, finite-context hidden states admit the approximation
$$
h_t \approx \bar{h}_t + \sum_{k\le K}\beta_{t,k} v_k + e_{s_t},
$$
with coefficients $\beta_{t,k}$ that track posterior task probabilities [2605.03780]. This provides a mathematically explicit account of how learned task-vector geometry can underwrite two distinct inference modes within one model.

## 5. Query-conditioned, decoding-space, and contextual-input variants

Not all learned task vectors are extracted post hoc from demonstrations. Some are trained as compact control variables. In learned-context neural networks, each task $t$ is associated with a learned context $c_t \in \mathbb{R}^k$, and the shared network computes
$$
y = f(x,c_t;\theta).
$$
The architecture concatenates $c_t$ to the input and is accompanied by a universal approximation result showing that, for a finite family of tasks, a scalar contextual input is sufficient in theory [2303.00788]. This formulation treats the task vector as a trainable contextual input rather than a hidden representation or a weight delta.

Adaptive Task Vectors introduce an explicitly query-conditioned version. A small language model computes
$$
v(x)=\phi(x;\theta_s)\in\mathbb{R}^{d_s},
$$
and a learned expansion maps it to layer-aligned modulators
$$
v_{\mathrm{ATV}}(x)=f_\theta(v(x))\in\mathbb{R}^{L\times d_\ell}.
$$
These are injected into the frozen large language model by
$$
h^{\ell\prime}=h^\ell+\lambda v_{\mathrm{ATV}}^\ell(x).
$$
The method is reported to outperform zero-shot, 16-shot in-context learning, BM25 retrieval ICL, ELICIT, I2CL, and LoRA on averages over 20 tasks, and to remain strong on unseen tasks [2506.03426]. The paper also proves that ATV is expressively equivalent to LoRA under equal rank budgets and more expressive than Prefix-Tuning under a linear attention approximation [2506.03426].

DeCoVec pushes the idea into decoding space. Rather than editing weights or hidden states, it computes a training-free logit-space task vector at each decoding step from the difference between few-shot and zero-shot output logits under the same generated prefix,
$$
v_T^t = z_t^{\mathrm{icl}} - z_t^{\mathrm{zs}},
$$
and injects it into the decoding process as
$$
\hat{z}_t = z_t^{\mathrm{de}} + \lambda \cdot v_T^t.
$$
Across seven models from 0.5B to 9B parameters on TruthfulQA, Math-500, and AQUA-RAT, this method is reported to outperform standard few-shot baselines with gains up to $+5.50$ average accuracy, while requiring no parameter updates or hidden-state interventions [2604.11129]. This broadens the learned-task-vector concept from latent representation control to token-level output-distribution steering.

## 6. Multimodal and speech-conditioned task vectors

The concept has also moved beyond text-only settings. In language-model-based text-to-speech, task-vector arithmetic was originally associated with weight-space deltas, but a systematic elimination study on Qwen3-TTS-12Hz-1.7B localizes the dominant carrier of emotional prosody to the ECAPA-TDNN speaker embedding, or x-vector, rather than to model weights or acoustic tokens [2606.05367]. The proposed training-free emotion direction is defined in x-vector space as
$$
\tau_{\mathrm{emo}}=\mathbb{E}_i[x(s_i,\mathrm{emo})]-\mathbb{E}_i[x(s_i,\mathrm{neutral})],
$$
and applied to a new target speaker by
$$
x_{\mathrm{new}} = x(\mathrm{target},\mathrm{neutral}) + \alpha \cdot \tau_{\mathrm{emo}}.
$$
Using English ESD as the source of $\tau$ and emoUERJ as a cross-lingual Brazilian Portuguese target, the method yields average gains of $+0.29$ in emotion2vec cosine over the ICL baseline on English held-out speakers and $+0.09$ on Brazilian Portuguese held-out speakers, while largely preserving identity and intelligibility [2606.05367].

This result is notable because prior work had argued that centroid-arithmetic style control is incompatible with token-based TTS architectures. The paper reports that the incompatibility can be circumvented when arithmetic operates on the speaker embedding rather than on discrete acoustic token embeddings, whose perturbation leads quickly from “no effect” to “degenerate noise” [2606.05367]. This suggests a broader design principle: in LM-conditioned generative stacks, useful task directions may reside in intermediate conditioning embeddings rather than in global parameter space.

Related multimodal evidence appears in plan conditioning and task embeddings. Compositional Plan Vectors represent trajectories as sums of subtasks and allow plan addition or subtraction in a one-shot imitation-learning setting, with policy conditioning of the form
$$
\pi_\theta(a_t\mid o_t,\; g_\phi(o_0,o_T)-g_\phi(o_0,o_t)).
$$
This enables generalization to longer multi-skill sequences and plan composition without retraining [1910.14033]. Task2Vec, by contrast, represents a visual classification task as a Fisher-information-based embedding of a probe network, yielding a fixed-dimensional vector independent of class cardinality and useful for meta-learning tasks such as feature-extractor selection [1902.03545]. These lines do not all use “task vector” in the same mechanistic sense, but they reinforce the general view that a task can often be encoded as a compact vectorial object with reusable geometry.

## 7. Limitations, controversies, and current research directions

Several limitations recur across the literature. In in-context learning, a single task vector can behave like a one-shot pseudo-example, which yields a rank-one effective predictor and fails on high-rank mappings such as general bijections [2506.09048]. Large-scale evidence further indicates that one task vector is not enough for many complex tasks, because task information may be distributed across multiple output positions or per-demonstration rule vectors [2406.16007; 2505.23911]. This directly challenges any universal claim that in-context learning always compresses the entire task into one localized hidden state.

In weight-space task arithmetic, effectiveness depends on gradient alignment, task compatibility, curvature, and fine-tuning regime. One-epoch vectors are exactly scaled negative gradients under full-batch gradient descent, but multi-epoch vectors deviate by a second-order term [2508.16082]. Provable guarantees for nonlinear Transformers exist only in conceptual settings with structured discriminative patterns, even though empirical validation on machine unlearning with Phi-1.5 and Phi-3-small supports the relevance of alignment-dependent addition and negation [2504.10957].

In training-free steering methods, failure modes differ by space. DeCoVec adds approximately $1.6\times$–$1.7\times$ wall-clock overhead versus few-shot decoding, and large $\lambda$ can degrade reasoning by increasing calculation or concept errors [2604.11129]. In x-vector TTS arithmetic, generalization beyond Qwen3-TTS-12Hz-1.7B-Base and beyond the ECAPA-TDNN conditioning regime remains to be tested, and emotion directions are not strictly orthogonal [2606.05367]. In Adaptive Task Vectors, mathematical reasoning can still favor retrieval-based ICL, and performance depends on the quality and domain alignment of the small language model that generates the task vector [2506.03426].

Current research directions accordingly move in several directions at once. One direction seeks better mechanistic grounding, as in the study of task-vector geometry and dual inference modes [2605.03780]. Another explores richer compositions, such as multi-vector injection in ICL, which empirically improves over single-vector injection on Knowledge, Algorithmic, Translation, Linguistic, and Bijection tasks [2506.09048]. A further direction shifts the search for task vectors away from monolithic parameter space and toward intermediate conditioning spaces, as illustrated by x-vector arithmetic in LM-TTS [2606.05367]. Taken together, these developments suggest that the learned task vector is no longer a single method but an organizing concept for task-specific structure across weights, activations, logits, and learned control embeddings.

Source: https://www.emergentmind.com/topics/learned-task-vector