Task Vector Theory

• Task Vector Theory is a formal framework that defines and manipulates task-specific vectors in neural models, capturing the transformation from a base to a specialized model.
• The theory supports parameter-efficient multi-tasking and model merging through linear arithmetic, enabling techniques such as quantization and in-context task control.
• Empirical research validates that task vectors in latent and activation spaces optimize performance, enhance safety, and allow compositional generalization across various architectures.

A task vector is a mathematical object—often a difference between model checkpoints or a specific direction in latent space—that encodes the transformation required to specialize a model or its internal activations for a given task. Task Vector Theory formalizes the extraction, manipulation, and function of these vectors, providing unified perspectives across weight-space model merging, in-context learning in transformers, and activation/inference-time representation steering. Modern research demonstrates both practical benefits (parameter-efficient multi-tasking, model editing, memory reduction) and new limits, while theoretical analyses clarify their emergence, composition rules, and expressive boundaries.

1. Formal Definitions and Mathematical Foundations

Parameter-Space Task Vectors

Given a pretrained model with parameter vector $θ_{\mathrm{pre}}$ and a fine-tuned model $θ^t_{ft}$ for task $t$, the task vector is: $\tau_t = θ^t_{ft} - θ_{\mathrm{pre}}$ This vector represents the trajectory in parameter space that adapts the base model to a specific task. In model merging or editing, new multi-task models can be formed as

$$θ_{\mathrm{MTL}} = θ_{\mathrm{pre}} + \sum_{t=1}^T λ_t \widehat{\tau}_t$$

where $\widehat{\tau}_t$ may be a quantized or otherwise transformed version (Kim et al., 10 Mar 2025).

Latent/Activation-Space Task Vectors (ICL)

In transformer models (text or vision), task vectors in latent space are typically defined as hidden-state activations at specific layers and token positions after in-context demonstrations. For a prompt $P$, at layer $l$, and position $i_*$: $v_{\mathrm{task}}^{(l)} = h^{(l)}_{i_*}(P)$ For in-context learning, this vector can be injected into the same position on a new query to steer predictions (Tikhonov et al., 29 May 2025, Yang et al., 16 Jan 2025).

Compositional Plan Vectors

In control and RL settings, task vectors can be learned as plan or trajectory encodings that support arithmetic: $$\phi(\tau_A) + \phi(\tau_B) \simeq \phi(\tau_{A ∥ B})$$ where concatenation of demonstrations is mapped to vector addition (Devin et al., 2019).

2. Theoretical Insights and Guarantees

Gradient Correspondence and Model Merging

With full-batch gradient descent, the task vector for one epoch is mathematically equivalent to the negative gradient of the loss, scaled by the learning rate: $$\tau_t^{(1)} = -\eta \nabla_θ L_t(θ_{\mathrm{pre}})$$ Task arithmetic—adding up these vectors—implements approximate multitask learning, matching a single epoch of joint gradient descent up to curvature corrections. The first-epoch gradient dominates the subsequent fine-tuning trajectory, explaining why one-epoch model merging performs comparably to merging fully converged models (Zhou et al., 22 Aug 2025).

Error Bounds and Quantization

Task vectors typically occupy a much narrower dynamic range in parameter space than full fine-tuned weights. Quantizing task vectors with low-precision (as low as 2–4 bits) introduces only minor errors: $$|\varepsilon_i| \leq \frac{Δ}{2} = \frac{(\theta_{\max} - \theta_{\min})}{2 (2^b - 1)}$$ Residual quantization techniques further reduce memory with negligible impact on model performance (Kim et al., 10 Mar 2025).

Composition, Transport, and Interference

Linear task-vector arithmetic (addition/subtraction) provably enables multi-task learning and unlearning, provided tasks are not adversarial in feature space (correlation α≥0). For model editing, the correct choice of coefficients ensures generalization to out-of-domain tasks. All results hold for both dense and low-rank approximations (Li et al., 15 Apr 2025).

Transporting task vectors between non-identical pretrainings requires local geometric alignment. Gradient-sign masking (GradFix) retains update directions that are descent-aligned in the target loss landscape, offering a theoretically guaranteed first-order loss decrease (Rinaldi et al., 7 Oct 2025).

3. Task Vectors in In-Context Learning: Formation, Functionality, and Limitations

Task vectors in activation space naturally emerge in transformer models performing in-context learning. When demonstrations are tightly formatted and models are of moderate depth, distinct task-specific clusters appear in mid-layer activations. Augmenting training with a task-vector prompting loss (TVP-loss)—which encourages representation of task information at a prescribed location—yields robust, localized task vectors and allows zero-shot vector injection to match few-shot performance (Yang et al., 16 Jan 2025).

Empirical studies on large-scale benchmarks confirm that:

• Task vector efficacy peaks at intermediate transformer layers (e.g., layer 15 in Llama-3-8B).
• Homogeneous, single-rule tasks are well summarized by one vector, but complex/multi-component tasks require multiple, distributed “subtask vectors” or “rule vectors.”
• Subtask vectors can be composed as

$v_{\mathrm{comp}} = \sum_{m=1}^M α_m v_{t,m}$

with adaptive weighting (α_m) to match the compositional structure of the task (Tikhonov et al., 29 May 2025, Zheng et al., 23 Jun 2024).

The “Linear Combination Conjecture” posits that task vectors correspond to (learned) linear combinations of individual demonstration embeddings. Injecting multiple task vectors overcomes the inherent rank-one limitation of single-vector approaches, critical for representing high-rank or bijective mappings (Dong et al., 10 Jun 2025).

4. Extensions: Cross-Modal, Adaptive, and Dynamic Task Vectors

Task vectors naturally generalize across modalities and architectures. In vision-LLMs, equivalent task vectors can be derived from text examples, image examples, or instructions, all occupying the same low-dimensional subspace and enabling cross-modal transfer via patching (Luo et al., 29 Oct 2024). Transferring task vectors between architectures is possible when layer shapes and parameter indices match and when sensitive submodules are excluded from arithmetic (e.g., embeddings or layer norms) (Lee et al., 27 Sep 2025).

Recent methods such as Adaptive Task Vectors (ATV) generate input-conditioned task vectors via a small neural generator and expand them to match the target model’s layers—this offers expressivity at least as great as LoRA and greater than prefix-tuning, and allows per-query adaptation and efficient control over frozen LLMs (Kang et al., 3 Jun 2025).

Dynamic vector construction methods segment and optimize task vectors via REINFORCE or similar techniques, learning not just which latent subspaces to inject but also their optimal locations within the network (Cai et al., 23 May 2025, Hojel et al., 8 Apr 2024).

5. Distributed and Hierarchical Representations

In multi-demonstration in-context learning, “distributed rule vectors” emerge: each demonstration leaves an abstracted rule embedding at its answer position, and the final output aggregates information from all individual vectors in a distributed fashion. This aggregation is essential for tasks requiring combinatorial reasoning or multi-step rule extraction (Zheng et al., 23 Jun 2024). Empirical patching and saliency analysis confirm that compositional or distributed structures, rather than global single vectors, underpin LLMs’ success in complex settings.

Hierarchical concept models explain how transformers learn factual recall via retrieval and arithmetic over latent task vectors. The dominant task direction in the residual space is provably retrieved during inference and guarantees robust 0–1 loss convergence, including under concept recombination and distribution shifts (Bu et al., 13 Aug 2025).

6. Practical Implications, Applications, and Limitations

Applications

• Model merging and editing: Linear arithmetic over task vectors enables fast, scalable incorporation of new tasks, unlearning (by subtraction), and controlled multi-task compositions (Kim et al., 10 Mar 2025, Li et al., 15 Apr 2025, Zhou et al., 22 Aug 2025).
• Parameter-efficient multi-tasking: Quantized task vectors drastically reduce storage and computation costs, with empirical performance matching or exceeding full-precision baselines (Kim et al., 10 Mar 2025).
• Safety and guardrails: Composing “guard vectors” into LLMs yields instant safety filters across new languages or domains, without retraining, and prefix-aware SFT closes the gap between offline and streaming evaluations (Lee et al., 27 Sep 2025).
• Zero/few-shot control in vision and language: Dynamic or patched task vectors enable few-shot or zero-shot task switching and compositional generalization (Luo et al., 29 Oct 2024, Hojel et al., 8 Apr 2024, Devin et al., 2019).

Limitations and Open Directions

• Expressivity: Single-vector methods are limited to low-rank function classes; high-rank or highly compositional tasks require multiple or distributed representations (Dong et al., 10 Jun 2025).
• Transferability: Transporting task vectors across substantially different model geometries or architectures remains fragile; gradient-sign masking is one partial remedy (Rinaldi et al., 7 Oct 2025).
• Interference: Linear addition of multiple task vectors can induce interference, especially across tasks with non-orthogonal or correlated updates (Li et al., 15 Apr 2025).
• Dynamic/Compositional Expansion: Research continues on mechanisms for dynamic retrieval, injection, and hierarchical composition of task vectors, with reinforcement learning and attention-based gating as promising techniques (Cai et al., 23 May 2025, Tikhonov et al., 29 May 2025).

7. Summary Table: Task Vector Instantiations

Domain	Task Vector Definition	Key Mechanism / Operation
Model merging	$θ_{ft}-θ_{pre}$ in parameter space	Addition, subtraction, quantization, merging
ICL (language/vision)	Hidden-state at task token or output position	Vector patching, subtask vectors, dynamic injection
RL/control	Trajectory encoding $\phi(\tau)$	Arithmetic over plans or partial trajectories
Cross-modal models	Layer-i activation after demonstration/instruction	Cross-modal patching, transfer

• Task vector quantization and memory-efficient merging (Kim et al., 10 Mar 2025)
• Distributed and compositional task vectors in ICL (Tikhonov et al., 29 May 2025, Zheng et al., 23 Jun 2024, Dong et al., 10 Jun 2025)
• Theoretical guarantees for model editing and arithmetic (Li et al., 15 Apr 2025, Zhou et al., 22 Aug 2025)
• Latent and activation-space task vectors: formation and limitations (Yang et al., 16 Jan 2025, Cai et al., 23 May 2025, Hojel et al., 8 Apr 2024)
• Guard vectors and streaming-safe model safety (Lee et al., 27 Sep 2025)
• Hierarchical and compositional approaches in RL/Control (Devin et al., 2019)
• Formal construction of skill/task vectors in economic modeling (Xie et al., 2023)
• Provable vector arithmetic in transformers (Bu et al., 13 Aug 2025)
• Adaptive and dynamic vector generation frameworks (Kang et al., 3 Jun 2025, Cai et al., 23 May 2025)
• Cross-modal alignment and task vector transfer in VLMs (Luo et al., 29 Oct 2024)
• Gradient-sign masking and task vector transport (Rinaldi et al., 7 Oct 2025)

Task Vector Theory thus provides a rigorous, unifying geometric and algorithmic foundation for representing and manipulating tasks in neural models across modalities and application domains.