Learnable Task Vectors (LTV) in In-Context Learning

• Learnable Task Vectors (LTV) are compact representations that encode task-specific information via optimization or extraction from activation, parameter, or decoding spaces.
• They are constructed using techniques like TVP-loss, direct vector learning, and closed-form adaptations to enable in-context learning, reasoning transfer, and model editing.
• Empirical findings show that LTV performance depends on layer choice and task complexity, fostering efficient cross-model transfer and dynamic merging strategies.

Learnable Task Vectors (LTV) are compact task representations that are optimized, adapted, or explicitly constructed so that a pretrained model can express task-specific behavior through vectors rather than full task-specific weights or long in-context prompts. In recent work, closely related objects appear in activation space, parameter space, decoding space, and residual-stream space, and they are used for in-context learning, reasoning transfer, model editing, cross-model transfer, federated fine-tuning, and dynamic model merging. The common premise is that task adaptation can often be mediated by a reusable vector or structured family of vectors, but the literature also shows that the viability of this premise depends strongly on layer, task type, representation space, and whether a single vector is sufficient (Hendel et al., 2023, Tikhonov et al., 29 May 2025, Yang et al., 29 Sep 2025).

1. Conceptual foundations and scope

The modern task-vector view of in-context learning was articulated by “In-Context Learning Creates Task Vectors” (Hendel et al., 2023). There, a prompt with demonstrations $S$ is modeled not merely as raw concatenated context but as a computation of the form

$T([S,x]) \approx f(x;\boldsymbol{\theta}(S)),$

where the demonstrations are compressed into a task vector $\boldsymbol{\theta}(S)$, and the query is then processed conditional on that vector. In this formulation, task vectors play the role of task parameters in a hypothesis class, but they live in activation space rather than in updated model weights.

Subsequent work expanded this view in two directions. First, “Task Vectors in In-Context Learning: Emergence, Formation, and Benefit” (Yang et al., 16 Jan 2025) showed in controlled synthetic training that task vectors can emerge naturally, but may be weakly and/or non-locally encoded; it then introduced task vector prompting loss (TVP-loss) to make the task representation stronger, more local, and easier to reuse. Second, “Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight” (Yang et al., 29 Sep 2025) replaced extraction with direct optimization of the vector itself, treating the task vector as a trainable parameter injected into the model’s hidden state.

The initials LTV are also used in a narrower sense by “Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learning” (Kwon et al., 20 May 2026), where Linear Task Vector denotes a specific regression-based method that estimates the in-context-learning-induced hidden-state shift by a closed-form linear map. Taken together, these works suggest that “learnable task vectors” are best understood as a design family rather than a single algorithmic template.

2. Representation spaces and construction mechanisms

Task vectors are not confined to one representational locus. In activation-space ICL methods, a task vector is typically a hidden-state activation at a specific token and layer. In the formulation used by (Hendel et al., 2023) and re-evaluated at scale by (Tikhonov et al., 29 May 2025), the canonical construction is

$$v_l = h_l(\text{final token } \texttt{->} \text{ of the few-shot prompt}),$$

followed by a causal intervention that replaces the corresponding hidden state at layer $l$ during inference on a new query. This makes the vector a reusable latent summary of the demonstrations rather than a parameter-space update.

In parameter-space methods, a task vector is a weight delta. “Cross-Model Transfer of Task Vectors via Few-Shot Orthogonal Alignment” (Kawamoto et al., 17 May 2025) defines the source task vector for task $i$ as

$$\Delta \mathbf{W}_i = \mathbf{W}^{(S)}_i - \mathbf{W}^{(S)},$$

and then studies how such deltas can be aligned across differently pretrained models. “Knowledge Composition using Task Vectors with Learned Anisotropic Scaling” (Zhang et al., 2024) uses the same task-arithmetic basis, $\btau_i = \btheta_i - \btheta_0$, but learns blockwise coefficients so that different parameter blocks of each task vector contribute anisotropically rather than through a single isotropic scalar.

A third construction moves the task vector into decoding space. “DeCoVec: Building Decoding Space based Task Vector for LLMs via In-Context Learning” (Li et al., 13 Apr 2026) defines a tokenwise steering vector directly in logit space: $$\mathbf{v}_{\mathcal{T}}^{t} = \mathbf{z}_{\rm icl}^{t} - \mathbf{z}_{\rm zs}^{t},$$ where the few-shot and zero-shot logits are computed with the same generated prefix. The decoding-time intervention is then

$$\tilde{\mathbf{z}}^{t} = \mathbf{z}_{\rm de}^{t} + \lambda \cdot \mathbf{v}_{\mathcal{T}}^{t}.$$

This formulation is explicitly training-free and non-invasive.

These constructions clarify a frequent misconception: task vectors are not only weight deltas. The literature now includes activation-space vectors extracted from hidden states, parameter-space deltas from fine-tuning, and logit-space vectors formed from output-distribution differences (Tikhonov et al., 29 May 2025, Kawamoto et al., 17 May 2025, Li et al., 13 Apr 2026).

3. Learning objectives and optimization regimes

One line of work makes task vectors learnable by shaping the model during pretraining or task training. In (Yang et al., 16 Jan 2025), TVP-loss augments ordinary ICL training with an auxiliary zero-shot objective in which a hidden state extracted from the demonstration is reinserted into the query path. This creates direct gradient pressure for a prescribed layer to contain a task representation sufficient for task prediction on its own, thereby turning task-vector localization into an explicit training target rather than a post hoc discovery problem.

A second line learns the vector directly while keeping the backbone fixed. In (Yang et al., 29 Sep 2025), the Learned Task Vector is optimized as a trainable parameter under the objective

$T([S,x]) \approx f(x;\boldsymbol{\theta}(S)),$0

The reported implementation uses AdamW, learning rate $T([S,x]) \approx f(x;\boldsymbol{\theta}(S)),$1, weight decay $T([S,x]) \approx f(x;\boldsymbol{\theta}(S)),$2, up to $T([S,x]) \approx f(x;\boldsymbol{\theta}(S)),$3 epochs, $T([S,x]) \approx f(x;\boldsymbol{\theta}(S)),$4 sampled examples per epoch or the full split if smaller, and early stopping with patience $T([S,x]) \approx f(x;\boldsymbol{\theta}(S)),$5. Unlike extracted vectors, the learned vector is not tied to one extraction token or one preferred layer; the same study reports that it can be injected at arbitrary layers and positions, and even into ICL prompts themselves.

Reasoning-specific variants follow the same pattern. “CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs” (Li et al., 1 Oct 2025) distinguishes extracted CoT vectors from Learnable CoT Vectors. The learnable version is trained under a teacher–student framework in which the teacher sees $T([S,x]) \approx f(x;\boldsymbol{\theta}(S)),$6, the student sees only $T([S,x]) \approx f(x;\boldsymbol{\theta}(S)),$7 plus the injected vector, and the loss is

$T([S,x]) \approx f(x;\boldsymbol{\theta}(S)),$8

Only the vector is trained; all LLM parameters remain frozen.

A third regime is closed-form adaptation rather than gradient descent over a vector. In (Kwon et al., 20 May 2026), Linear Task Vector models the query-dependent hidden-state shift as

$T([S,x]) \approx f(x;\boldsymbol{\theta}(S)),$9

and fits $\boldsymbol{\theta}(S)$0 by ridge regression: $\boldsymbol{\theta}(S)$1 The method is motivated by a distributional-alignment criterion,

$\boldsymbol{\theta}(S)$2

and by the bound

$\boldsymbol{\theta}(S)$3

This gives a principled bridge from hidden-state matching to next-token distribution matching.

4. Empirical behavior, expressivity, and failure modes

Large-scale empirical work has significantly narrowed the domain in which the one-vector hypothesis is plausible. “One Task Vector is not Enough: A Large-Scale Study for In-Context Learning” (Tikhonov et al., 29 May 2025) introduced QuiteAFew, a dataset of $\boldsymbol{\theta}(S)$4 diverse few-shot tasks, each with $\boldsymbol{\theta}(S)$5 unique input-output pairs derived from Alpaca, and evaluated task vectors in Llama-3-8B. The main findings are that task-vector performance consistently peaks around an intermediate layer, especially layer $\boldsymbol{\theta}(S)$6; that performance varies strongly by task type; and that for many tasks a single task vector does not match full few-shot prompting. The paper also reports that categories such as convert, classify, and analyze show clear correctness improvements at the best intermediate layer, whereas edit and describe are flatter and less amenable to a single injected vector. A further qualitative result is that format scores are often higher than correctness scores, indicating that output structure can be captured without fully capturing the task mapping.

The same study gives a concrete compositional failure case through a structured-generation task that converts free-form car descriptions into JSON with fields such as color, city, and model (Tikhonov et al., 29 May 2025). In that setting, the first useful “real” task vector is not at -> but at the token {", and different target tokens are restored by injecting hidden states from different token positions. The comparison among Natural Generation, Classic Task Vector, and Sub-Task Vectors shows that subtask-specific injections substantially improve next-token probability for both structure tokens and context-dependent values. This directly supports the view that complex tasks may require multiple vectors aligned to output components.

A complementary limitation is formalized in “Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitations” (Dong et al., 10 Jun 2025). That paper proposes the Linear Combination Conjecture: injected task vectors behave like single in-context demonstrations formed through linear combinations of the original demonstrations. Under that view, a single injected task vector behaves like a 1-shot predictor and is therefore effectively rank-one. The paper predicts failure on high-rank mappings and confirms this on practical LLMs, especially on bidirectional bijection tasks such as uppercase $\boldsymbol{\theta}(S)$7 lowercase and English $\boldsymbol{\theta}(S)$8 French, while noting the expected exceptions of copy and antonym. It also reports that injecting multiple task vectors into all arrow positions, in the TaskV-M variant, yields modest but consistent gains; on Llama-13B, the 1-shot average moves from $\boldsymbol{\theta}(S)$9 for the baseline to $$v_l = h_l(\text{final token } \texttt{->} \text{ of the few-shot prompt}),$$0 for TaskV and $$v_l = h_l(\text{final token } \texttt{->} \text{ of the few-shot prompt}),$$1 for TaskV-M, and the 4-shot average moves from $$v_l = h_l(\text{final token } \texttt{->} \text{ of the few-shot prompt}),$$2 to $$v_l = h_l(\text{final token } \texttt{->} \text{ of the few-shot prompt}),$$3 to $$v_l = h_l(\text{final token } \texttt{->} \text{ of the few-shot prompt}),$$4.

Reasoning vectors exhibit a distinct layerwise pathology. In (Li et al., 1 Oct 2025), Extracted CoT Vectors display a U-shaped performance curve across depth—good in shallow layers, poor in middle layers, good again in deep layers—while Learnable CoT Vectors are more stable and generally perform best in the earliest layers. This indicates that multi-step reasoning is not organized like simpler task-vector settings in which mid-layer injection is dominant.

5. Mechanistic and theoretical accounts

Recent work has moved beyond performance measurements toward explicit mechanistic models. At the circuit level, (Yang et al., 29 Sep 2025) argues that injected task vectors act primarily through attention-head OV circuits. If a task vector $$v_l = h_l(\text{final token } \texttt{->} \text{ of the few-shot prompt}),$$5 is injected before a later head $$v_l = h_l(\text{final token } \texttt{->} \text{ of the few-shot prompt}),$$6, the relevant transformed term is

$$v_l = h_l(\text{final token } \texttt{->} \text{ of the few-shot prompt}),$$7

and the aggregate downstream effect is the sum of such OV-transformed contributions over later heads. The same paper identifies a small subset of key heads whose ablation harms performance far more than random-head ablation, and reports a higher-level regularity: task-vector propagation through later layers is largely linear. Fitting a linear operator from the injected vector to the final hidden-state shift preserves most of the original effect, and polar decomposition of that operator yields a rotation–stretch picture in which early vectors are rotated toward task-relevant subspaces, whereas late vectors are mainly scaled.

A complementary theoretical account appears in “Provable In-Context Vector Arithmetic via Retrieving Task Concepts” (Bu et al., 13 Aug 2025). There, the latent task vector is a high-level concept vector $$v_l = h_l(\text{final token } \texttt{->} \text{ of the few-shot prompt}),$$8, and factual-recall ICL is modeled through Word2Vec-style arithmetic: $$v_l = h_l(\text{final token } \texttt{->} \text{ of the few-shot prompt}),$$9 The analysis uses a nonlinear residual transformer with softmax attention, layer normalization, residual connections, and cross-entropy training, and shows that QA training can recover the task vector cleanly, whereas word-label ICL or QA-ICL training leaves the learned state as a hybrid vector containing both high- and low-level components. The paper also proves strong generalization under QA training, including robustness to concept recombination and distribution shifts.

A third account concerns geometry across in-distribution and out-of-distribution inference. “Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers” (Yan et al., 5 May 2026) defines learned task vectors as asymptotic task-dependent offsets in hidden-state space,

$l$0

and shows that in-distribution inference can be represented by convex combinations

$l$1

with coefficients that approximately track the Bayesian posterior over latent tasks. Out-of-distribution behavior, by contrast, is supported by a representation that lies in a nearly orthogonal subspace. This yields a dual-mode account: Bayesian task retrieval inside the learned task-vector simplex for known tasks, and extrapolative task learning in a separate representational scaffold for novel tasks.

These studies converge on a nontrivial point. Task vectors are not merely descriptive embeddings attached to prompts after the fact. They appear as causally active internal variables, but their efficacy depends on circuit usage, latent geometry, and the statistical structure of the training distribution (Yang et al., 29 Sep 2025, Bu et al., 13 Aug 2025, Yan et al., 5 May 2026).

6. Composition, transfer, and systems-level extensions

Learnable task-vector methods have been extended well beyond single-model ICL interventions. In cross-model transfer, (Kawamoto et al., 17 May 2025) addresses the case where source and target models are pretrained independently and direct task-vector transfer fails because the parameter spaces are misaligned. The proposed few-shot orthogonal alignment learns layerwise orthogonal matrices

$l$2

from only $l$3 images per target dataset, with $l$4 augmentation and $l$5 mini-batches. On two Vision Transformer Base/32 models pretrained on YFCC100M and LAION400M and evaluated on eight classification datasets, the unadapted target model averages $l$6 accuracy, direct embedding-based transfer gives $l$7, direct LoRA transfer gives $l$8, orthogonally aligned embedding-based transfer reaches $l$9, orthogonally aligned LoRA transfer reaches $i$0, and few-shot fine-tuning reaches $i$1. This shows that a compact task representation can remain modular across model lineages if its coordinate system is learned.

Composition inside a single model has also become more structured. In aTLAS, (Zhang et al., 2024) splits each task vector into parameter blocks and learns one scalar per block, so that

$i$2

The number of learnable parameters is precisely $i$3. Empirically, the method reports the lowest average disentanglement error, $i$4, compared with $i$5 for standard task vectors with prior search and $i$6 for linear task vectors; in offline test-time adaptation with UFM it improves CLIP by $i$7 absolute points on average over $i$8 datasets while using only $i$9k learnable parameters versus $$\Delta \mathbf{W}_i = \mathbf{W}^{(S)}_i - \mathbf{W}^{(S)},$$0k for LayerNorm tuning. The paper interprets this as evidence that blockwise learnable scaling makes task vectors more disentangled and less interference-prone.

Federated and many-task settings motivate another form of learned composition. “Many-Task Federated Fine-Tuning via Unified Task Vectors” (Tsouvalas et al., 10 Feb 2025) defines per-task vectors in the standard way,

$$\Delta \mathbf{W}_i = \mathbf{W}^{(S)}_i - \mathbf{W}^{(S)},$$1

but merges multiple task vectors on each client into a unified vector

$$\Delta \mathbf{W}_i = \mathbf{W}^{(S)}_i - \mathbf{W}^{(S)},$$2

then reconstructs task-specific behavior through binary masks and scalar rescalers. The server aggregates same-task information and cross-task information using sign-based task similarity, avoiding separate full models or per-task adapters at the server. The method is therefore best described as computed plus merged plus modulated, rather than as a free latent vector directly optimized end to end.

Storage-efficient dynamic merging pushes the same logic into compression. “Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compression” (Gao et al., 30 Apr 2026) begins from the empirical observation that task vectors show an impulse-like activation pattern and high robustness to low-bit representations. T-Switch decomposes a task vector into a binary sparse mask, a sign vector, and a scalar scaling factor; FlexSwitch then makes the sparsity pattern and bit-width learnable through Learnable Gating Sparsification and Bit-width Adaptive Selection; Auto-FlexSwitch adds KNN retrieval with a learnable low-rank metric for dynamic composition. Reported results include $$\Delta \mathbf{W}_i = \mathbf{W}^{(S)}_i - \mathbf{W}^{(S)},$$3 storage reduction for the basic T-Switch representation, around $$\Delta \mathbf{W}_i = \mathbf{W}^{(S)}_i - \mathbf{W}^{(S)},$$4 sparsity while maintaining or exceeding fine-tuned performance under LGS, and on ViT-B/32 an average accuracy of $$\Delta \mathbf{W}_i = \mathbf{W}^{(S)}_i - \mathbf{W}^{(S)},$$5 for Auto-FlexSwitch+CKA with $$\Delta \mathbf{W}_i = \mathbf{W}^{(S)}_i - \mathbf{W}^{(S)},$$6 MB storage, compared with $$\Delta \mathbf{W}_i = \mathbf{W}^{(S)}_i - \mathbf{W}^{(S)},$$7 and $$\Delta \mathbf{W}_i = \mathbf{W}^{(S)}_i - \mathbf{W}^{(S)},$$8 MB for Auto-Switch.

The specific Linear Task Vector method in (Kwon et al., 20 May 2026) also belongs in this systems-oriented picture. Across eight classification benchmarks and five LLMs, it is reported to achieve the best average accuracy on all five models, to win on $$\Delta \mathbf{W}_i = \mathbf{W}^{(S)}_i - \mathbf{W}^{(S)},$$9 of $\btau_i = \btheta_i - \btheta_0$0 benchmark–model cases, to improve a highlighted LLaMA-2-13B case from $\btau_i = \btheta_i - \btheta_0$1 to $\btau_i = \btheta_i - \btheta_0$2, and to reduce per-sample latency to $\btau_i = \btheta_i - \btheta_0$3 s, compared with $\btau_i = \btheta_i - \btheta_0$4 s for ICL. The same paper reports a $\btau_i = \btheta_i - \btheta_0$5 average accuracy gain when task vectors extracted from Qwen2.5-72B are transferred to Qwen2.5-7B through logit-space regression. This suggests that learnable or adaptive task-vector systems can serve not only as interpretability objects but also as low-latency infrastructure for deployment.

Across these variants, the literature does not support a single canonical LTV object. A plausible synthesis is that robust task-vector systems are most likely to be layer-aware, query-dependent when necessary, compositional rather than monolithic, and sometimes distributed across multiple positions or modules rather than concentrated in one universal vector.