Learned Task Vector

• Learned Task Vector is a compact, task-conditioned representation that encodes task-specific behavior across various model spaces.
• Different formulations, including weight-space deltas, activation states, and decoding-space logit shifts, enable arithmetic operations to control language models.
• Current research explores task arithmetic, distributed representations, and training-free steering methods to enhance in-context learning and fine-tuning.

Searching arXiv for the cited papers to ground the article in current literature. {"query":"id:(Brito et al., 3 Jun 2026) OR id:(Hendel et al., 2023) OR id:(Dong et al., 10 Jun 2025) OR id:(Zheng et al., 2024) OR id:(Kang et al., 3 Jun 2025) OR id:(Li et al., 13 Apr 2026)","max_results":10} A learned task vector is a task-conditioned representation that encodes task-specific behavior as a direction in some model-associated space. In contemporary usage, the term does not denote a single formal object. In weight-space formulations, it is the parameter difference between a fine-tuned model and a pre-trained model. In in-context learning, it is a hidden-state representation distilled from demonstrations and reused by patching or injection. In other settings, it appears as a learned contextual input, a decoding-space logit shift, or a conditioning-space direction such as a speaker embedding offset for controllable text-to-speech. Across these formulations, the common idea is that a task can be summarized by a compact vector that supports addition, negation, scaling, or interpolation, thereby steering model behavior without reconstructing the full original training context (Hendel et al., 2023, Zhang et al., 2024, Li et al., 13 Apr 2026, Brito et al., 3 Jun 2026).

1. Conceptual scope and principal definitions

The most classical formulation defines a task vector in parameter space. Given a pretrained model with weights $\theta_{\mathrm{pre}}$ and a model fine-tuned on task $t$ with weights $\theta_{\mathrm{ft}}^{(t)}$, the task vector is

$$\tau_t = \theta_{\mathrm{ft}}^{(t)} - \theta_{\mathrm{pre}},$$

and the edited model is

$$\theta_{\mathrm{new}} = \theta_{\mathrm{pre}} + \lambda \cdot \tau_t.$$

This definition underlies task arithmetic in model editing and model merging (Brito et al., 3 Jun 2026, Zhang et al., 2024).

A distinct formulation arises in in-context learning. There, a task vector is a single reusable representation distilled from demonstrations. In one influential construction, if the input sequence is $[S, x', \to]$, the task vector is the hidden state at the arrow token after an intermediate layer $L$,

$\theta(S) := h_L(\to; [S, x', \to]),$

and it is later injected into a forward pass on $[x, \to]$ to approximate ordinary in-context learning (Hendel et al., 2023). Related work formulates task vectors in triplet-formatted prompts as hidden states at arrow tokens and analyzes them as linear combinations of demonstration-derived states (Dong et al., 10 Jun 2025).

Other literatures use the same term for compact task-conditioning variables rather than extracted hidden states. In learned-context multi-task networks, the learned task vector is the contextual input $c_t \in \mathbb{R}^k$ concatenated to the ordinary input and optimized jointly with shared parameters (Sandnes et al., 2023). In Adaptive Task Vectors, the vector is query-conditioned: a small LLM computes $t$0, which is mapped into layer-wise modulators and additively injected into a frozen LLM (Kang et al., 3 Jun 2025). In DeCoVec, the task vector lives in decoding space and is defined at each decoding step as a logit difference,

$t$1

which is added to the decoding logits during generation (Li et al., 13 Apr 2026).

A concise taxonomy is therefore useful.

Space	Representative definition	Representative papers
Weight space	$t$2	(Zhang et al., 2024, Zhou et al., 22 Aug 2025)
Activation space	Hidden state at a designated token/layer	(Hendel et al., 2023, Dong et al., 10 Jun 2025)
Distributed rule space	Per-demo rule vectors at answer tokens	(Zheng et al., 2024)
Decoding space	Stepwise logit difference $t$3	(Li et al., 13 Apr 2026)
Conditioning space	Speaker/style embedding direction	(Brito et al., 3 Jun 2026)
Task-conditioned input space	Learned contextual input $t$4 or $t$5	(Sandnes et al., 2023)

This plurality suggests that “learned task vector” is best understood as a family resemblance term rather than a single invariant formalism.

2. Weight-space task vectors and task arithmetic

In weight-space work, the learned task vector is the parameter displacement produced by fine-tuning. This interpretation supports addition, negation, and scaling of tasks via vector arithmetic in parameter space (Brito et al., 3 Jun 2026). The aTLAS framework refines this picture by decomposing parameters into blocks $t$6 and learning anisotropic block-wise coefficients,

$t$7

so that multiple task vectors can be composed as

$t$8

This blockwise parameterization is reported to reduce interference, improve task addition and negation, and leverage the low intrinsic dimensionality of fine-tuned solutions (Zhang et al., 2024).

A complementary theoretical line links task vectors to gradients. Under full-batch gradient descent, the one-epoch task vector satisfies

$t$9

so a one-epoch task vector is exactly a scaled negative gradient of the task loss. For multi-epoch training, the equivalence holds approximately up to a second-order curvature term, which leads to the interpretation of task arithmetic as approximate multitask learning (Zhou et al., 22 Aug 2025). This result is used to explain why merging models fine-tuned for only a single epoch can remain competitive with merging fully converged models (Zhou et al., 22 Aug 2025).

Theoretical analysis has also addressed when task arithmetic is provably effective for nonlinear Transformers. In a conceptual binary-classification setting, task addition is shown to succeed for irrelevant or aligned tasks, task negation to succeed for irrelevant or contradictory tasks, and suitable linear coefficients to support out-of-domain generalization. The same guarantees are stated to hold for both dense-weight parameters and their low-rank approximations (Li et al., 15 Apr 2025). This suggests that weight-space learned task vectors are not merely heuristic editing directions, but can admit formal generalization guarantees under structured assumptions.

3. Activation-space task vectors in in-context learning

Activation-space formulations treat the task vector as an internal representation created during in-context learning rather than a parameter update. “In-Context Learning Creates Task Vectors” formalizes this view by positing

$\theta_{\mathrm{ft}}^{(t)}$0

with the task vector extracted as

$\theta_{\mathrm{ft}}^{(t)}$1

where a dummy query $\theta_{\mathrm{ft}}^{(t)}$2 is used to prevent dependence on the actual query $\theta_{\mathrm{ft}}^{(t)}$3 (Hendel et al., 2023). Empirically, this hypothesis-based procedure retains about 80–90% of regular in-context learning accuracy across seven LLMs and 18 tasks, far above a baseline without demonstrations (Hendel et al., 2023).

Subsequent work has made the emergence mechanism more explicit. In linear transformers trained on triplet-formatted prompts, the Linear Combination Conjecture states that the injected task vector functions as a single in-context demonstration formed through a linear combination of the original demonstrations (Dong et al., 10 Jun 2025). The triplet analysis gives an explicit form,

$\theta_{\mathrm{ft}}^{(t)}$4

showing that the arrow-token hidden state is a weighted sum of demonstration-derived states (Dong et al., 10 Jun 2025). This theory also proves a rank limitation: a single task vector induces an effective predictor

$\theta_{\mathrm{ft}}^{(t)}$5

which is rank one, explaining failures on high-rank mappings such as general bijection tasks (Dong et al., 10 Jun 2025).

Large-scale analysis has challenged the strongest version of the single-vector hypothesis. On the QuiteAFew dataset of 3,096 few-shot tasks, performance peaks at an intermediate layer, specifically the 15th layer in Llama-3-8B, but effectiveness varies strongly by task type (Tikhonov et al., 29 May 2025). For simple tasks, a single hidden state at the final separator token can still be useful. For complex outputs such as structured JSON generation, multiple sub-task vectors extracted from specific output tokens outperform a single global vector, suggesting distributed task knowledge representation (Tikhonov et al., 29 May 2025).

A further refinement appears in “Learning Task Representations from In-Context Learning,” where the task vector is represented as a weighted sum of attention head outputs,

$\theta_{\mathrm{ft}}^{(t)}$6

with the weights optimized causally while the base model is frozen and in-context learning is blocked (Saglam et al., 8 Feb 2025). This method is reported to generalize across both language and functional regression tasks and to work by aligning the distribution of the last hidden state with that of an optimally performing in-context-learned model (Saglam et al., 8 Feb 2025).

4. From localized vectors to distributed rule representations

A major controversy in the literature concerns whether task information is localized in a single hidden-state vector or distributed across many positions. The distributed-rule-vector study argues that, for tasks whose rules require multiple demonstrations, a single task vector does not exist in the strong sense proposed by earlier work (Zheng et al., 2024). Instead, each demonstration contributes its own rule vector located at its answer token, and the final prediction depends on a distributed aggregation,

$\theta_{\mathrm{ft}}^{(t)}$7

Patching donor hidden states at each demonstration answer token rescues performance in categorization tasks, whereas single task-vector patching remains near chance (Zheng et al., 2024).

This conclusion does not eliminate localized task vectors altogether. The same study reports that in knowledge-retrieval tasks or tasks solvable from one demonstration, a single task vector at the final query’s “is” token can recover few-shot-like accuracy (Zheng et al., 2024). A plausible implication is that single-vector and distributed-vector accounts describe different operating regimes rather than mutually exclusive theories. The large-scale QuiteAFew analysis is consistent with this view: single task vectors are useful for some task families, whereas complex tasks require multiple subtask-specific vectors (Tikhonov et al., 29 May 2025).

Recent synthetic work has provided a geometric framework for this coexistence. “Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers” defines task vectors as long-context limits of centered task-conditioned mean hidden states,

$\theta_{\mathrm{ft}}^{(t)}$8

and shows that in-distribution behavior is governed by Bayesian task retrieval through convex combinations of learned task vectors, while out-of-distribution behavior arises through extrapolative task learning in a subspace nearly orthogonal to the task-vector subspace (Yan et al., 5 May 2026). In that analysis, finite-context hidden states admit the approximation

$\theta_{\mathrm{ft}}^{(t)}$9

with coefficients $$\tau_t = \theta_{\mathrm{ft}}^{(t)} - \theta_{\mathrm{pre}},$$0 that track posterior task probabilities (Yan et al., 5 May 2026). This provides a mathematically explicit account of how learned task-vector geometry can underwrite two distinct inference modes within one model.

5. Query-conditioned, decoding-space, and contextual-input variants

Not all learned task vectors are extracted post hoc from demonstrations. Some are trained as compact control variables. In learned-context neural networks, each task $$\tau_t = \theta_{\mathrm{ft}}^{(t)} - \theta_{\mathrm{pre}},$$1 is associated with a learned context $$\tau_t = \theta_{\mathrm{ft}}^{(t)} - \theta_{\mathrm{pre}},$$2, and the shared network computes

$$\tau_t = \theta_{\mathrm{ft}}^{(t)} - \theta_{\mathrm{pre}},$$3

The architecture concatenates $$\tau_t = \theta_{\mathrm{ft}}^{(t)} - \theta_{\mathrm{pre}},$$4 to the input and is accompanied by a universal approximation result showing that, for a finite family of tasks, a scalar contextual input is sufficient in theory (Sandnes et al., 2023). This formulation treats the task vector as a trainable contextual input rather than a hidden representation or a weight delta.

Adaptive Task Vectors introduce an explicitly query-conditioned version. A small LLM computes

$$\tau_t = \theta_{\mathrm{ft}}^{(t)} - \theta_{\mathrm{pre}},$$5

and a learned expansion maps it to layer-aligned modulators

$$\tau_t = \theta_{\mathrm{ft}}^{(t)} - \theta_{\mathrm{pre}},$$6

These are injected into the frozen LLM by

$$\tau_t = \theta_{\mathrm{ft}}^{(t)} - \theta_{\mathrm{pre}},$$7

The method is reported to outperform zero-shot, 16-shot in-context learning, BM25 retrieval ICL, ELICIT, I2CL, and LoRA on averages over 20 tasks, and to remain strong on unseen tasks (Kang et al., 3 Jun 2025). The paper also proves that ATV is expressively equivalent to LoRA under equal rank budgets and more expressive than Prefix-Tuning under a linear attention approximation (Kang et al., 3 Jun 2025).

DeCoVec pushes the idea into decoding space. Rather than editing weights or hidden states, it computes a training-free logit-space task vector at each decoding step from the difference between few-shot and zero-shot output logits under the same generated prefix,

$$\tau_t = \theta_{\mathrm{ft}}^{(t)} - \theta_{\mathrm{pre}},$$8

and injects it into the decoding process as

$$\tau_t = \theta_{\mathrm{ft}}^{(t)} - \theta_{\mathrm{pre}},$$9

Across seven models from 0.5B to 9B parameters on TruthfulQA, Math-500, and AQUA-RAT, this method is reported to outperform standard few-shot baselines with gains up to $$\theta_{\mathrm{new}} = \theta_{\mathrm{pre}} + \lambda \cdot \tau_t.$$0 average accuracy, while requiring no parameter updates or hidden-state interventions (Li et al., 13 Apr 2026). This broadens the learned-task-vector concept from latent representation control to token-level output-distribution steering.

6. Multimodal and speech-conditioned task vectors

The concept has also moved beyond text-only settings. In language-model-based text-to-speech, task-vector arithmetic was originally associated with weight-space deltas, but a systematic elimination study on Qwen3-TTS-12Hz-1.7B localizes the dominant carrier of emotional prosody to the ECAPA-TDNN speaker embedding, or x-vector, rather than to model weights or acoustic tokens (Brito et al., 3 Jun 2026). The proposed training-free emotion direction is defined in x-vector space as

$$\theta_{\mathrm{new}} = \theta_{\mathrm{pre}} + \lambda \cdot \tau_t.$$1

and applied to a new target speaker by

$$\theta_{\mathrm{new}} = \theta_{\mathrm{pre}} + \lambda \cdot \tau_t.$$2

Using English ESD as the source of $$\theta_{\mathrm{new}} = \theta_{\mathrm{pre}} + \lambda \cdot \tau_t.$$3 and emoUERJ as a cross-lingual Brazilian Portuguese target, the method yields average gains of $$\theta_{\mathrm{new}} = \theta_{\mathrm{pre}} + \lambda \cdot \tau_t.$$4 in emotion2vec cosine over the ICL baseline on English held-out speakers and $$\theta_{\mathrm{new}} = \theta_{\mathrm{pre}} + \lambda \cdot \tau_t.$$5 on Brazilian Portuguese held-out speakers, while largely preserving identity and intelligibility (Brito et al., 3 Jun 2026).

This result is notable because prior work had argued that centroid-arithmetic style control is incompatible with token-based TTS architectures. The paper reports that the incompatibility can be circumvented when arithmetic operates on the speaker embedding rather than on discrete acoustic token embeddings, whose perturbation leads quickly from “no effect” to “degenerate noise” (Brito et al., 3 Jun 2026). This suggests a broader design principle: in LM-conditioned generative stacks, useful task directions may reside in intermediate conditioning embeddings rather than in global parameter space.

Related multimodal evidence appears in plan conditioning and task embeddings. Compositional Plan Vectors represent trajectories as sums of subtasks and allow plan addition or subtraction in a one-shot imitation-learning setting, with policy conditioning of the form

$$\theta_{\mathrm{new}} = \theta_{\mathrm{pre}} + \lambda \cdot \tau_t.$$6

This enables generalization to longer multi-skill sequences and plan composition without retraining (Devin et al., 2019). Task2Vec, by contrast, represents a visual classification task as a Fisher-information-based embedding of a probe network, yielding a fixed-dimensional vector independent of class cardinality and useful for meta-learning tasks such as feature-extractor selection (Achille et al., 2019). These lines do not all use “task vector” in the same mechanistic sense, but they reinforce the general view that a task can often be encoded as a compact vectorial object with reusable geometry.

7. Limitations, controversies, and current research directions

Several limitations recur across the literature. In in-context learning, a single task vector can behave like a one-shot pseudo-example, which yields a rank-one effective predictor and fails on high-rank mappings such as general bijections (Dong et al., 10 Jun 2025). Large-scale evidence further indicates that one task vector is not enough for many complex tasks, because task information may be distributed across multiple output positions or per-demonstration rule vectors (Zheng et al., 2024, Tikhonov et al., 29 May 2025). This directly challenges any universal claim that in-context learning always compresses the entire task into one localized hidden state.

In weight-space task arithmetic, effectiveness depends on gradient alignment, task compatibility, curvature, and fine-tuning regime. One-epoch vectors are exactly scaled negative gradients under full-batch gradient descent, but multi-epoch vectors deviate by a second-order term (Zhou et al., 22 Aug 2025). Provable guarantees for nonlinear Transformers exist only in conceptual settings with structured discriminative patterns, even though empirical validation on machine unlearning with Phi-1.5 and Phi-3-small supports the relevance of alignment-dependent addition and negation (Li et al., 15 Apr 2025).

In training-free steering methods, failure modes differ by space. DeCoVec adds approximately $$\theta_{\mathrm{new}} = \theta_{\mathrm{pre}} + \lambda \cdot \tau_t.$$7–$$\theta_{\mathrm{new}} = \theta_{\mathrm{pre}} + \lambda \cdot \tau_t.$$8 wall-clock overhead versus few-shot decoding, and large $$\theta_{\mathrm{new}} = \theta_{\mathrm{pre}} + \lambda \cdot \tau_t.$$9 can degrade reasoning by increasing calculation or concept errors (Li et al., 13 Apr 2026). In x-vector TTS arithmetic, generalization beyond Qwen3-TTS-12Hz-1.7B-Base and beyond the ECAPA-TDNN conditioning regime remains to be tested, and emotion directions are not strictly orthogonal (Brito et al., 3 Jun 2026). In Adaptive Task Vectors, mathematical reasoning can still favor retrieval-based ICL, and performance depends on the quality and domain alignment of the small LLM that generates the task vector (Kang et al., 3 Jun 2025).

Current research directions accordingly move in several directions at once. One direction seeks better mechanistic grounding, as in the study of task-vector geometry and dual inference modes (Yan et al., 5 May 2026). Another explores richer compositions, such as multi-vector injection in ICL, which empirically improves over single-vector injection on Knowledge, Algorithmic, Translation, Linguistic, and Bijection tasks (Dong et al., 10 Jun 2025). A further direction shifts the search for task vectors away from monolithic parameter space and toward intermediate conditioning spaces, as illustrated by x-vector arithmetic in LM-TTS (Brito et al., 3 Jun 2026). Taken together, these developments suggest that the learned task vector is no longer a single method but an organizing concept for task-specific structure across weights, activations, logits, and learned control embeddings.