Task Singular Vectors (TSV)

• Task Singular Vectors are structured singular vectors that extend classical SVD to multi-task and tensor settings, capturing low-dimensional task-specific perturbations.
• They offer robust theoretical guarantees with tight error bounds and stability under noise, leveraging advanced perturbation theory.
• Algorithmic procedures like TSV-Compress and TSV-Merge effectively reduce model parameters and interference while preserving high performance.

Task Singular Vectors (TSV) are structured singular vectors, typically arising in the context of multi-task learning, statistical inference under noise, variational regularization, and multiway tensor models. The concept serves as a unifying framework for extracting orthogonal task-relevant directions in high-dimensional data and model parameter spaces, providing both theoretical guarantees (e.g., stability under noise or perturbation) and practical algorithms (e.g., for model compression and merge in neural networks).

1. Foundational Definition and Formalism

Task Singular Vectors generalize the notion of singular vectors from matrix analysis to settings where tasks, data modalities, or constraint structures induce a block or tensor organization. Given a model with $L$ layers, and for each task $t$ a per-layer task matrix $M_t \in \mathbb{R}^{d \times m}$ (typically, the parameter difference between a fine-tuned and pre-trained model), the SVD

$$M_t = U_t \Sigma_t V_t^\top = \sum_{i=1}^{k} \sigma_{t,i} u_{t,i} v_{t,i}^\top$$

yields the left/right singular vectors. For each task and layer, the collection of top-$r$ left and right singular vectors

$$\{u_{t,1},\dots,u_{t,r}\},\quad \{v_{t,1},\dots,v_{t,r}\}$$

is termed the Task Singular Vectors of rank $r$ at that layer. These TSVs parameterize a low-dimensional subspace capturing task-specific perturbations or features (Gargiulo et al., 2024). For tensor data (e.g., multi-task, multi-modal settings), TSVs generalize to singular vector tuples satisfying multilinear critical point equations on the tensor (Robeva et al., 2016).

2. Stability and Error Bounds under Noise and Approximation

Estimating TSVs from noisy or approximate data is a central concern, especially in high-dimensional statistics and randomized linear algebra. Classical operator perturbation theory, notably the Davis–Kahan and Wedin $\sin\Theta$ theorems, provide worst-case bounds

$$\sin\Theta(V, \widetilde V) \leq \frac{\|E\|}{\delta}$$

where $E$ is the error/perturbation and $t$0 is the spectral gap. However, for random noise ($t$1 i.i.d., mean-zero, unit variance) and when the true data is low-rank, significantly tighter probabilistic bounds hold (Vu, 2010):

$t$2

where $t$3 and $t$4. This allows accurate computation of TSVs when the gap $t$5 is only $t$6, dramatically improving over the deterministic regime which requires $t$7. Recursive extensions provide similar guarantees for higher-dimensional singular subspaces.

For approximate singular vectors arising from projection or Ritz–Galerkin methods, tight error bounds are available in terms of block-residuals and the "big gap" (distance from wanted spectral values to the rest of the spectrum):

$t$8

where $t$9, $M_t \in \mathbb{R}^{d \times m}$0 are residuals and $M_t \in \mathbb{R}^{d \times m}$1 is the spectral separation (Nakatsukasa, 2018). These are often far sharper than what is predicted by classical sin$M_t \in \mathbb{R}^{d \times m}$2 theory.

3. Algorithmic Procedures: Compression, Merging, and Interference Reduction

Layer-wise SVD of task matrices exposes low-rank structure and enables compression. TSV-Compress (TSV-C) retains only the top-$M_t \in \mathbb{R}^{d \times m}$3 singular components of $M_t \in \mathbb{R}^{d \times m}$4, achieving up to 90\% parameter reduction while maintaining $M_t \in \mathbb{R}^{d \times m}$5 performance (Gargiulo et al., 2024). The steps are:

1. For each $M_t \in \mathbb{R}^{d \times m}$6, compute SVD: $M_t \in \mathbb{R}^{d \times m}$7.
2. Truncate to rank $M_t \in \mathbb{R}^{d \times m}$8.
3. Store only the top-$M_t \in \mathbb{R}^{d \times m}$9 singular vectors and values.

Model merge via TSV-Merge (TSV-M) leverages concatenation of TSVs across tasks, followed by a whitening/orthogonalization procedure (e.g., Procrustes or SVD-based) on these bases to minimize subspace overlap and interference. Final task-composed weights are constructed as

$$M_t = U_t \Sigma_t V_t^\top = \sum_{i=1}^{k} \sigma_{t,i} u_{t,i} v_{t,i}^\top$$0

with

$$M_t = U_t \Sigma_t V_t^\top = \sum_{i=1}^{k} \sigma_{t,i} u_{t,i} v_{t,i}^\top$$1

and $$M_t = U_t \Sigma_t V_t^\top = \sum_{i=1}^{k} \sigma_{t,i} u_{t,i} v_{t,i}^\top$$2, $$M_t = U_t \Sigma_t V_t^\top = \sum_{i=1}^{k} \sigma_{t,i} u_{t,i} v_{t,i}^\top$$3 obtained from concatenated and orthogonalized TSVs. This methodology significantly outperforms flat average parameter merge in both empirical accuracy and interference metrics.

4. Theoretical Frameworks: Variational and Tensorial Extensions

Nonlinear generalizations of singular vectors exist for convex one-homogeneous regularization, such as total variation or $$M_t = U_t \Sigma_t V_t^\top = \sum_{i=1}^{k} \sigma_{t,i} u_{t,i} v_{t,i}^\top$$4-norm frameworks (Benning et al., 2012). The ground state (minimal singular value/vector) is defined by the constrained minimization

$$M_t = U_t \Sigma_t V_t^\top = \sum_{i=1}^{k} \sigma_{t,i} u_{t,i} v_{t,i}^\top$$5

with optimality characterized by the subdifferential condition

$$M_t = U_t \Sigma_t V_t^\top = \sum_{i=1}^{k} \sigma_{t,i} u_{t,i} v_{t,i}^\top$$6

Higher singular vectors lack mutual orthogonality, but share important properties such as scale-localization and exact recovery (or unbiasedness) under Tikhonov and inverse scale space flows for suitable data and noise (Benning et al., 2012).

In higher-order tensor settings, particularly orthogonally decomposable (odeco) tensors, TSVs correspond to tuples $$M_t = U_t \Sigma_t V_t^\top = \sum_{i=1}^{k} \sigma_{t,i} u_{t,i} v_{t,i}^\top$$7 critical for the multilinear form. The algebraic variety of singular vector tuples decomposes into discrete (Type I) closed forms and positive-dimensional (Type II) base-point strata, reflecting the presence of symmetries or indeterminacies in multi-way task structure (Robeva et al., 2016).

5. TSVs in Multi-Task and Model Merging Applications

TSVs are instrumental for both compression and interference management in multi-task model merging, especially in deep learning. For a suite of $$M_t = U_t \Sigma_t V_t^\top = \sum_{i=1}^{k} \sigma_{t,i} u_{t,i} v_{t,i}^\top$$8 tasks on a shared backbone, per-layer TSVs are extracted, compressed via TSV-C, and merged via TSV-M. A key innovation is the definition of interference measures based on the overlap of TSV subspaces:

$$M_t = U_t \Sigma_t V_t^\top = \sum_{i=1}^{k} \sigma_{t,i} u_{t,i} v_{t,i}^\top$$9

Zero interference (orthogonal task subspaces) ensures non-destructive merging; higher scores indicate collinearities and potential destructive interference. A global Singular Task Interference (STI) measure aggregates this across all tasks via matrix trace and block-diagonal structures (Gargiulo et al., 2024).

Benchmarks on image classification tasks (e.g., CLIP ViT backbones with 8–20 tasks) demonstrate TSV-M achieving up to $r$0 gain over consensus Task Arithmetic baselines, and retaining $r$1 of individual fine-tune accuracy with as little as $r$2 of parameters stored per task.

6. Algebraic and Spectral Identities for Singular Vectors

Component-wise algebraic identities for singular vectors recover vector entries from singular values and singular values of submatrices. For $r$3 with nonzero singular values $r$4 and $r$5, the squared modulus of any component satisfies (Xu et al., 2020):

$r$6

where $r$7 is $r$8 with the $r$9-th row deleted. Analogously for right singular vectors and column deletions. This identity generalizes the classical Hermitian eigenvector-eigenvalue formula, and can be used to recover singular vector entries based solely on the spectrum and minor spectra.

7. Practical Implementation Guidance and Limitations

The practical procedure for TSV computation under random noise entails:

1. Estimating the operator norm of the noise (e.g., for sub-Gaussian noise, $$\{u_{t,1},\dots,u_{t,r}\},\quad \{v_{t,1},\dots,v_{t,r}\}$$0).
2. Computing the leading SVD of the observed data or model layer, selecting the top $$\{u_{t,1},\dots,u_{t,r}\},\quad \{v_{t,1},\dots,v_{t,r}\}$$1 singular vectors.
3. Verifying empirical spectral gap $$\{u_{t,1},\dots,u_{t,r}\},\quad \{v_{t,1},\dots,v_{t,r}\}$$2 exceeds $$\{u_{t,1},\dots,u_{t,r}\},\quad \{v_{t,1},\dots,v_{t,r}\}$$3 for parameter $$\{u_{t,1},\dots,u_{t,r}\},\quad \{v_{t,1},\dots,v_{t,r}\}$$4 (to guarantee stability).
4. Forming projectors and subspaces for downstream use, with explicit operator-norm error bounds scaling as $$\{u_{t,1},\dots,u_{t,r}\},\quad \{v_{t,1},\dots,v_{t,r}\}$$5 (Vu, 2010).

Algorithmic overhead is modest and implementation is training-free post-fine-tuning; main computational effort is in SVD and small-block orthogonalizations. Limitations include applicability to non-matrix layers (which default to flat SGD arithmetic), dependence on the assumed low-rank approximation, and tuning of per-layer compression ranks. Future work may address adaptive rank selection, higher-order decompositions, and extension to broader model classes (Gargiulo et al., 2024).

---

References:

• "Task Singular Vectors: Reducing Task Interference in Model Merging" (Gargiulo et al., 2024)
• "Singular vectors under random perturbation" (Vu, 2010)
• "Singular Vectors of Orthogonally Decomposable Tensors" (Robeva et al., 2016)
• "Ground States and Singular Vectors of Convex Variational Regularization Methods" (Benning et al., 2012)
• "Singular Vectors From Singular Values" (Xu et al., 2020)
• "Sharp error bounds for Ritz vectors and approximate singular vectors" (Nakatsukasa, 2018)