---
title: Task Vector Bases in Multi-Task Models
url: https://www.emergentmind.com/topics/task-vector-bases
type: topic
---

# Task Vector Bases in Multi-Task Models

A task vector basis is a collection of vectors—typically derived from parameter, activation, or output spaces of models—such that any task-specific direction or transformation of interest can be represented as a structured (often sparse or low-rank) linear combination of these basis atoms. Task vector bases serve to compress, interpret, and efficiently manipulate the “atoms” of adaptation in modern multi-task, multi-modal, and in-context model architectures, from deep neural networks to large language and vision-language models. The construction, properties, and utilization of such bases has become central to recent advances in model merging, scalable in-context learning, task arithmetic, and interpretability, with strong theoretical and empirical foundations across modalities.

## 1. Mathematical Formulation and Expressivity of Task Vector Bases

Formally, task vectors are defined as directions in a vector space encoding a task's “effect”—e.g., in parameter space $\tau_t = \theta_{\rm ft}^t - \theta_{\rm pre}$, in hidden space as difference or mean activations, or in output/decoding space as logit differences [2502.01015][1206.6417][2604.11129]. A task vector basis is a set $\{b_j\}_{j=1}^M$ (with $M < T$ tasks) such that each task vector $\tau_i$ is well-approximated by a linear (often sparse or low-rank) combination:
\[
\tau_i \approx \sum_{j=1}^M c_{i,j} b_j
\]
or, in matrix notation, $T \approx B C^\top$ with $B\in\mathbb{R}^{d\times M}$ and $C\in\mathbb{R}^{T\times M}$ [2502.01015].

Task vector bases generalize the notion of "atomic" functions in multitask learning [1206.6417], where each task parameter vector is decomposed as $w_t = B s_t$ with $B$ the basis matrix and $s_t$ a sparse coefficient vector. Task vector bases unify heterogeneous forms—parameter, hidden, or logit space—enabling their use in compressed arithmetic, efficient model merging, and steerable representations [2604.11129][2410.22330].

The dimension $M$ of the basis captures the intrinsic rank of the family of task transformations: $M$ equals the number of linearly independent directions required to reconstruct all task effects up to an acceptable error. In compression settings, $M\ll T$ attests to substantial redundancy or shared structure among tasks [2502.01015]. In vision-language models or in-context learning, the span of task vectors is low-dimensional relative to the vast ambient space [2410.22330][2506.09048].

## 2. Basis Construction Algorithms and Structural Constraints

Multiple algorithms and principled criteria have emerged for constructing task vector bases:

- **Greedy selection:** Iteratively select task vectors maximizing explained variance, updating the basis and weights [2502.01015].
- **SVD/PCA:** Compute a low-rank approximation by projecting onto the top singular/vectors (principal components) of the task matrix. Yields optimal mean-squared error but may violate desiderata like nonnegativity [2502.01015].
- **Autoencoding (structural bases):** Use a softmax-activated encoder and linear decoder trained to minimize reconstruction error on the Gram matrix $G = T^\top T$, imposing constraints such as nonnegativity or block-sparsity for interpretability and efficiency [2502.01015].
- **Block-sparsity and low-rank per layer:** In neural networks, especially for parameter-space task vectors, imposing sparsity within layers or factorizing updates per layer further reduces memory and computational footprint [2502.01015].
- **REINFORCE-based selection in activation space:** In vision models, mean activations per task position are pruned via REINFORCE to yield a sparse, task-specific sub-basis guiding inference [2404.05729].
- **Linear regression for distributional alignment:** In in-context learning, optimal task vector offsets in hidden or logit space are found by ridge regression minimizing the distributional discrepancy to full ICL [2605.20730].

The construction algorithm directly determines recovery accuracy, basis interpretability, and the ability to support advanced operations such as unlearning (negation), OOD generalization, or extrapolation in compressed multitask settings [2502.01015][2605.03780]. Theoretical analyses show that error bounds for downstream risk or unlearning depend on the (M+1)-th eigenvalue of the task Gramian, offering guarantees as $M$ increases [2502.01015].

## 3. Functional Roles and Operations Enabled by Task Vector Bases

Task vector bases are central to several functional paradigms across learning and inference:

- **Scalable model merging:** Summing or weighting basis atoms yields new, merged models supporting multiple tasks with drastically reduced storage and computational cost [2502.01015][2503.06921].
- **Task arithmetic:** Vector arithmetic—addition, negation, or extrapolation—on task representations is preserved in the basis, with operations such as
  \[
  \theta_{\rm Add}^M = \theta_0 + \sum_{j=1}^M \alpha_j b_j
  \]
  for addition, and analogously for subtraction (negation/unlearning) [2502.01015].
- **Automatic grouping and overlap detection:** Structures in the coefficient matrix (sparsity, shared supports) reveal latent task clusters and overlaps, formalizing group structure in multi-task learning [1206.6417].
- **Efficient in-context learning:** In transformer ICL, task vectors distilled from demonstrations can be stored and rapidly injected, saving context length and supporting cross-modal or cross-scale transfer [2410.22330][2506.09048][2605.20730].
- **Steering and control in decoding space:** Task vectors in logit (decoding) space provide a universal, non-invasive steering mechanism for LLMs, achieving accuracy gains at zero training or parameter updates [2604.11129].
- **Principled unlearning:** For bases constructed via autoencoding or greedy selection, "forgetting" a task corresponds to subtracting its reconstructed vector, with precise error guarantees [2502.01015].

A summary table contrasts leading task vector basis methodologies:

| Method                 | Basis Construction      | Span Type            | Key Operations Supported            |
|------------------------|------------------------|----------------------|-------------------------------------|
| SVD/PCA                | Top singular vectors   | Orthogonal/Low-rank  | Addition, but not nonnegativity     |
| Greedy selection       | Maximize coverage      | Sparse/dense         | Addition, negation, interpretability|
| Autoencoding           | Softmax-activated AE   | Block-sparse         | Addition, negation, compression     |
| REINFORCE selection    | Stochastic subsetting  | Highly sparse        | Zero-shot guidance for visual tasks |
| Ridge regression (LTV) | Closed-form linear     | Hidden/logit space   | Distribution-aligned inference      |

## 4. Underlying Geometry and Rank Constraints

The geometry of task vector bases tightly links to the expressivity limits of the underlying model class and sampling procedure:

- **Linear independence and rank limitations:** In linear models or in-context learning, demonstration activations span a space of dimension at most $n$ (the number of demonstrations). Injecting a single vector yields a rank-one predictor; multiple vectors are needed for higher-rank mappings [2506.09048].
- **Convex polytopes and subspace partitioning:** In synthetic mixtures, task vectors for $K$ tasks span a $(K-1)$-dimensional subspace $V$, with in-distribution inference operating as convex combinations in $V$ and OOD generalization occurring in the near-orthogonal complement $V^\perp$ [2605.03780].
- **Cross-modal alignment:** In VLMs, task vectors derived from image and text inputs for the same task are nearly colinear (cosine $\approx 0.9$–$0.97$), showing that common functional axes are present across modalities [2410.22330].
- **Pseudoclosure and algebraic structure:** In the context of square matrices, special bases such as the Weyl–Heisenberg/Pauli and Hadamard bases are characterized by pseudo-closure (e.g., n-pseudo-closure for the Fourier basis, 2-pseudo-closure for Hadamard) under multiplication, supporting group-theoretic and transform applications [1205.3673].

## 5. Empirical Evidence and Benchmark Performance

Empirical studies comprehensively validate the utility of task vector bases:

- **Model merging and multitask addition:** In vision (ViT) and language (RoBERTa, Llama) settings, using $M = T/2$ bases via autoencoding or greedy selection recovers $97$–$100\%$ of full task arithmetic performance; even $M = T/4$ retains $>95\%$ [2502.01015].
- **Memory and latency:** Task vector quantization and residual decomposition maintain accuracy within $1$ point of FP32 merging at $2.375$ effective bits per task (a $92\%$ reduction in storage) [2503.06921]. Latency for LTV matches zero-shot inference [2605.20730].
- **Out-of-distribution generalization:** Task vector prompting loss and basis geometry interventions yield marked improvements in OOD/robustness [2501.09240][2605.03780].
- **Cross-modal and cross-model transfer:** In VLMs, patching in a task vector from one modality or even a different base model yields equivalent or superior performance; in LLMs, extracting LTV from a larger model and transferring to a smaller model provides a mean accuracy boost of $6.43$ points [2410.22330][2605.20730].
- **Logit-space steering:** Decoding-space task vectors (DeCoVec) outperform few-shot ICL on TruthfulQA, Math-500, and AQUA-RAT by up to $5.5$ points across open-source LLMs [2604.11129].

## 6. Theoretical Guarantees and Limitations

Task vector bases are governed by clear theoretical constraints and tradeoffs:

- **Generalization bounds:** As long as basis atoms approximate the full set of task vectors to within a small residual (controlled by the top $(M+1)$ singular value), addition and negation error bounds match those derived for the full (uncompressed) arithmetic [2502.01015].
- **Identifiability:** Under incoherence and sparsity assumptions, sparse-basis MTL (GO-MTL) recovers both support patterns and subspaces representing task groups, generalizing trace-norm and disjoint-group MTL [1206.6417].
- **Rank-one bottleneck:** In transformer ICL, a single task vector can only encode rank-one maps; multi-vector injection is required for complex (bijective) tasks [2506.09048].
- **Distributional alignment:** Minimizing the KL-divergence or hidden-state MSE between task-vector and ICL distributions (as in LTV) offers a principled extraction criterion, explaining accuracy gains and providing actionable alignment metrics ($d_{\rm NTP}$) [2605.20730].
- **Limitations:** Full orthonormal basis construction, basis arithmetic (addition/multiplication), and span analyses for large task families remain ongoing research areas, especially in the context of activation/logit subspaces and multi-modal integration [2410.22330][2604.11129].

## 7. Applications Across Modalities and Model Architectures

Task vector bases are a unifying abstraction deployed in varied contexts:

- **Multitask and continual learning:** GO-MTL and task arithmetic with compressed bases enable scalable learning and unlearning over hundreds of tasks, with controlled memory and interference [2502.01015][1206.6417].
- **Vision-language and multimodal models:** Shared subspaces of task vectors facilitate robust cross-modal transfer, patching, and cross-architecture adaptation without additional fine-tuning [2410.22330].
- **In-context learning acceleration:** Task vectors distilled from context demonstrations support amortized, minimal-latency inference, robust to modal shifts and model size changes [2501.09240][2605.20730].
- **Visual prompting and image manipulation:** Activation-space bases, pruned for sparsity and high “taskness,” allow efficient task steering in MAE–VQGAN and related architectures [2404.05729].
- **Quantum computation and algebraic signal processing:** Orthogonal bases with pseudo-closure generalize Pauli and Hadamard matrices, supporting matrix Fourier transforms, group theory, and stabilizer code construction in $M_n(\mathbb{C})$ [1205.3673].

In sum, task vector bases constitute the mathematical and algorithmic infrastructure for scalable, interpretable, and efficient manipulation of task-induced transformations across modern machine learning systems. Their principled construction, theoretical guarantees, and demonstrated empirical impact make them foundational in multitask model design, compression, and analysis [2502.01015][1206.6417][2503.06921][2605.20730][2410.22330][2506.09048][2604.11129][2605.03780][2404.05729][1205.3673].

Source: https://www.emergentmind.com/topics/task-vector-bases