Task Vectors in Neural Models

• Task vectors are compact internal neural representations that encapsulate a demonstrated task’s input–output mapping through aggregated activations.
• They are extracted by averaging specific attention head outputs, showing empirical utility in function modeling, model merging, domain transfer, and robustness assessments.
• Their algebraic compositionality allows for modular editing through linear combinations, enabling rapid adaptation, concept erasure, and secure multi-task integration.

A task vector is a compact, internal neural representation that summarizes the input–output mapping (or “function”) underlying a demonstrated task within a large neural model. Task vectors emerge in a variety of contexts – from language and vision transformer architectures to multimodal and federated settings – and serve as a foundational abstraction for efficient task encoding, rapid adaptation, and modular model editing. Extensive research on task vectors has established their empirical utility in function modeling, model merging, domain transfer, interpretability, and robustness. Below, the key aspects of task vectors are surveyed using the primary methodological and theoretical developments from recent literature.

1. Emergence and Internal Mechanism

Task vectors arise when neural architectures process in-context demonstrations. In autoregressive transformers, exposure to prompt examples induces internal activations that encode the input–output rule as a small number of attention head outputs aggregated across layers (Todd et al., 2023). For a task $t$, the mean activation over the identified attention heads $𝒜$, $v_t = \sum_{(l,\,j)\in 𝒜} \bar{a}_{lj}^t$, acts as a compressed function representation. When directly “patched” into the hidden state at a selected layer, $h^\prime_\ell = h_\ell + v_t$, the model is causally compelled to execute the associated function, irrespective of the sameness of the context to the original demonstration template.

In controlled settings, such as transformers trained from scratch on synthetic functions, task vectors manifest as internal representations (e.g., hidden states at special token positions) that summarize a task. Format, depth, and prompt arrangement significantly impact the clarity and locality of these encodings (Yang et al., 16 Jan 2025, Dong et al., 10 Jun 2025). For linear attention models on triplet-formatted prompts, careful loss landscape analysis shows that task vectors are formed through linear combinations of demonstration representations at critical positions, embodying the essential mechanics of function learning in ICL (Dong et al., 10 Jun 2025).

2. Extraction and Identification

Extracting task vectors typically involves averaging activations from a subset of attention heads or specific network positions in response to multiple prompts from the same task (Todd et al., 2023, Hojel et al., 8 Apr 2024). Causal mediation analysis quantitatively identifies which heads act as conduits for task information by measuring the causal indirect effect (CIE) and average indirect effect (AIE), pinpointing “courier” heads whose activations, when patched, trigger the intended task execution (Todd et al., 2023).

In visual transformers (e.g., MAE-VQGAN), scoring schemes based on the ratio of overall to within-task activation variance systematically locate positions that are highly discriminative for task encoding. The REINFORCE algorithm can then select an optimal subset of these task vectors, which, when introduced into both encoder and decoder attention heads, steer the model toward zero-shot execution of specific downstream tasks (Hojel et al., 8 Apr 2024). In multimodal LMMs, analogous procedures aggregate mean activations at jointly chosen attention head locations, thus extending task vector computation to scenarios mixing natural language and images (Huang et al., 21 Jun 2024).

Table 1: Task Vector Extraction Methods

Setting	Extraction Procedure	Main Measurement
LLMs, ICL (Todd et al., 2023)	Mean over selected head activations	CIE/AIE (causal mediation)
Vision models (Hojel et al., 8 Apr 2024)	Activation variance scoring + search	$\rho_{token}(i)$
Multimodal (Huang et al., 21 Jun 2024)	Avg. activation, attention-head policy	Downstream loss
Federated FL (Tsouvalas et al., 10 Feb 2025)	Difference of local and base weights	Sign aggregation

3. Functionality, Robustness, and Limitations

Task vectors operate as functional “triggers”: when inserted into an appropriate hidden state location, they induce the LLM or vision model to apply the encoded function—even in zero-shot or contextually unfamiliar scenarios (Todd et al., 2023, Hendel et al., 2023).

Robustness: Task vectors are empirically robust to changes in prompt style, context structure, and – critically – to data distribution shift. In both ICL and cross-modal VLM scenarios, injecting a task vector trained under one format or modality can reliably steer the model on queries specified in another format or modality. In VLMs, task vectors derived from either text or image demonstrations – or even brief instructions – lead to functionally similar latent representations, with orthonormal clustering in latent space (Luo et al., 29 Oct 2024).

Limitations: Task vectors, especially those formed via linear aggregation of triplet tokens, have inherent representational constraints. In linear transformers, the injection of a task vector is functionally equivalent to a rank-one mapping. Thus, domains requiring high-rank transformations or nontrivial bijections cannot be faithfully encoded by a single task vector. Empirically, such tasks degrade to random performance under the task vector regime (Dong et al., 10 Jun 2025). Injecting multiple task vectors in few-shot prompts can partially overcome this, but a fundamental limitation remains for highly compositional functions.

4. Algebraic Composition and Modularity

A key property of task vectors is their algebraic compositionality. Linear combinations—addition, subtraction, or scaling—of multiple task vectors enable construction of composite or negated functions. For instance, the parallelogram law is used to build FVs for composite tasks (e.g., "Last-Capitalize") via $v_{BD}^* = v_{AD} + v_{BC} - v_{AC}$ (Todd et al., 2023).

In parameter-space task arithmetic, the learned difference vector between fine-tuned and base model, $\tau_t = \theta_t - \theta_{base}$, can be added or subtracted to edit model capabilities modularly. This property enables model merging, rapid domain adaptation, concept erasure, and even fairness-control via selective vector composition (Zhang et al., 3 Jul 2024, Pham et al., 4 Apr 2024, Naganuma et al., 30 May 2025).

Knowledge composition is further refined in methods like aTLAS and TSV-Merge, where anisotropic scaling across parameter blocks and singular vector orthogonalization, respectively, permit modular merging with minimization of cross-task interference (Zhang et al., 3 Jul 2024, Gargiulo et al., 26 Nov 2024). This yields improved multitask and few-shot learning, with compact parameterizations and scalable transferability.

5. Theoretical Foundations and Gradient Connections

The theoretical underpinning of task vectors in fine-tuning is formalized by their relationship to gradients. Given standard gradient descent, a task vector computed after one epoch is exactly equivalent to the negative gradient of the task loss (scaled by the learning rate), $$\tau_t = -\eta \nabla_\theta \bar{L}_t(\theta_{base})$$ (Zhou et al., 22 Aug 2025). In multi-epoch regimes, the equivalence persists up to a bounded, curvature-controlled second-order error term, with explicit error bounds derived for feed-forward architectures.

Consequently, task arithmetic—adding task vectors to merge models—is a concrete realization of approximate multi-task learning, closely aligned with joint gradient descent on the aggregate loss surface.

6. Applications: Model Merging, Concept Erasure, Fairness, and Beyond

Task vectors underpin a range of practical and emerging applications:

• Model Merging: Addition of task vectors, with techniques for interference minimization (e.g., orthogonal alignment (Kawamoto et al., 17 May 2025), task singular vector whitening (Gargiulo et al., 26 Nov 2024), data-free merging (Cheng et al., 11 Mar 2025)), achieves multi-task models with performance rivaling full retraining.
• Concept Erasure: Subtracting a scaled task vector associated with an unwanted concept erases that capability globally in a text-to-image model, outperforming prompt-dependent methods in robustness to adversarial reactivation (Pham et al., 4 Apr 2024).
• Federated Learning: Unified task vectors, aggregated via sign and magnitude or tailored by lightweight modulators, enable efficient communication and collaborative training under extreme task heterogeneity (Tsouvalas et al., 10 Feb 2025).
• Fairness Control: Task arithmetic, combined with subgroup-specific vectors and scalar scaling, allows explicit trade-off management between accuracy, demographic parity, and equalized odds, providing a pathway toward responsible model editing (Naganuma et al., 30 May 2025).
• Adaptive Inference: Query-conditioned adaptive task vectors (ATV) flexibly generate inputs for downstream LLMs, outperforming both static task vectors and fixed demonstration ICL, while offering theoretical equivalence to or greater expressivity than LoRA and Prefix-Tuning (Kang et al., 3 Jun 2025).
• Security: Task vector frameworks are susceptible to “BadTV” backdoor attacks, where malicious vectors embedded via addition or subtraction can trigger adversarial behaviors while preserving benign task accuracy (Hsu et al., 4 Jan 2025).

7. Future Research Directions

Current research suggests several open areas for expansion:

• Richer and Higher-Rank Representations: Exploring multi-vector composition and nonlinear aggregation to break fundamental rank limitations observed in single-vector approaches (Dong et al., 10 Jun 2025).
• Cross-Model Transfer: Developing orthogonal alignment procedures for application of task vectors across models with different pre-training regimes (Kawamoto et al., 17 May 2025).
• Dynamic and Hierarchical Injection: Investigating mechanisms for dynamic, context-dependent vector generation and hierarchical adaptive networks that conditionally propagate task updates to suit changing domain shifts (Ambekar et al., 11 Aug 2025).
• Robust Pre-Conditioning: Furthering methods (e.g., knowledge distillation pre-conditioning (Yoshida et al., 2 Aug 2025)) to guarantee reliable, secure merging even when source models vary in confidence or norm.
• Interpretability and Auditing: Leveraging the explicit task vector representation for interpretability, transparency, debugging, and modular control of large-scale neural systems.

Task vectors function as the substrate for modular, efficient, and interpretable encoding of task-specific behavior in modern neural models, with rigorous connections to optimization theory and substantial empirical justification across domains. Research continues to probe their compositionality, theoretical limits, robustness, and practical integration into the broader landscape of foundation model engineering.