TSV-Merge: Low-Rank Model Merging

• TSV-Merge is a framework that exploits the low-rank structure of per-layer weight differences to merge fine-tuned models with reduced task interference.
• The method applies singular value decomposition, truncation, and whitening to isolate core task subspaces for robust multi-task aggregation.
• Empirical results show TSV-Merge significantly outperforms flat task arithmetic, improving accuracy by up to 15 percentage points while achieving high compression.

Task-Singular-Vectors Model Merging (TSV-Merge) is a model-merging framework that leverages the low-rank structure of per-layer weight differences between fine-tuned neural networks to suppress task interference and achieve superior multi-task model performance. TSV-Merge was introduced as a response to the limitations of flat task-vector approaches such as Task Arithmetic, establishing a rigorous, geometry-informed multi-model aggregation method applicable to large-scale vision and speech models (Gargiulo et al., 2024, Carvalho et al., 5 Mar 2026, Wang et al., 10 Feb 2026).

1. Motivation and Conceptual Foundation

Traditional model-merging paradigms, notably Task Arithmetic (TA), treat network parameters as flat vectors and aggregate fine-tuned "task vectors" (model deltas) via simple averaging. This flattening discards the intrinsic matrix structure of neural network layers, thereby failing to capture low-rank patterns and permitting significant overlap ("interference") between task-induced parameter perturbations. As a result, merged models often suffer from degraded performance on individual tasks—particularly as the number or heterogeneity of tasks increases.

TSV-Merge is motivated by the empirical observation that, for each layer, the difference matrix between the fine-tuned and base model weights is low-rank and encodes informative principal directions. By harnessing singular value decomposition (SVD) to identify and operate in the core task subspaces, TSV-Merge achieves higher multi-task accuracy and compression, and it precisely quantifies and suppresses destructive task interference at the layer level (Gargiulo et al., 2024, Carvalho et al., 5 Mar 2026).

2. Mathematical Formulation and Algorithmic Pipeline

Let $\theta_0$ denote pretrained model weights and $\theta_{t}$ for $t=1, \dots, T$ be the weights of domain- or task-adapted models. For each layer $i$, TSV-Merge operates as follows:

1. Layer-wise Task Delta Extraction: Compute $\tau_{i,t} = \theta_{i,t} - \theta_{0,i}$.
2. Low-Rank SVD Decomposition: Compute the skinny SVD of $\tau_{i,t} \in \mathbb{R}^{m\times n}$:

$\tau_{i,t} = U_{i,t}\Sigma_{i,t}V_{i,t}^\top$

with $$\Sigma_{i,t} = \operatorname{diag}(\sigma^{(1)}_{i,t},\ldots,\sigma^{(r)}_{i,t}),\ r=\min(m,n)$$.

1. Truncation (TSV-Compress): Select the top-$k$ singular components, with $k$ chosen as a fraction of $\theta_{t}$0 (e.g., $\theta_{t}$1 or to capture $\theta_{t}$299% of energy).
2. Task Subspace Concatenation: Assemble block matrices over tasks:

$\theta_{t}$3

$\theta_{t}$4

1. Orthogonalization (Whitening): As $\theta_{t}$5 and $\theta_{t}$6 from different tasks can overlap, apply an orthogonalization procedure (Newton–Schulz preferred for numerical stability) to obtain $\theta_{t}$7 with orthonormal columns.
2. Merged Layer Reconstruction:

$\theta_{t}$8

1. Final Model Assembly:

$\theta_{t}$9

where $t=1, \dots, T$0 is a merge-strength parameter (typically $t=1, \dots, T$1).

For non-matrix layers (e.g., LayerNorm), TSV-Merge defaults to simple elementwise averaging (Gargiulo et al., 2024, Carvalho et al., 5 Mar 2026).

3. Interference Quantification and Reduction

A central innovation of TSV-Merge is its fine-grained interference measure—Singular Task Interference (STI)—defined for each layer as

$t=1, \dots, T$2

where $t=1, \dots, T$3 are the concatenated singular-vector blocks for all tasks. Off-diagonal terms measure overlap between task subspaces; larger values indicate higher interference and empirically correlate with merged model error.

TSV-Merge reduces STI via whitening, i.e., orthogonalization of the concatenated singular vectors, which decorrelates task directions and preserves distinctiveness, yielding robust aggregation even as $t=1, \dots, T$4 increases (Gargiulo et al., 2024, Wang et al., 10 Feb 2026, Carvalho et al., 5 Mar 2026).

4. Compression and Singular-Value Boosting

TSV-Merge naturally yields high parameter efficiency: storing only $t=1, \dots, T$5 singular vectors per task and layer achieves compression to ≈10% of the original weight size, with $t=1, \dots, T$61.5 percentage points drop in mean accuracy across 20-task suites. This is enabled by stringent energy-based truncation of the SVD.

In high-rank regimes, severe decay of singular values can induce rank collapse, discarding weak but crucial directions. The BoostedTSV-M variant explicitly raises all small singular values below an energy threshold $t=1, \dots, T$7 to the threshold level, renormalizing per layer and task. This ensures retention of meaningful directions and improves numerical stability, particularly for automatic speech recognition (ASR) and highly multi-domain settings (Carvalho et al., 5 Mar 2026).

5. Comparative Results and Empirical Validation

TSV-Merge has been evaluated extensively on large-scale vision (CLIP ViT-B-32, ViT-B-16, ViT-L-14) and multi-domain ASR models. On vision tasks fused over 8/14/20 domains, TSV-Merge achieves average absolute accuracy gains of 10–15 percentage points over Task Arithmetic and related baselines. For ViT-B-32, absolute accuracy improvements are summarized below:

Method	8 tasks	14 tasks	20 tasks
Task Arithmetic	70.8	65.3	60.5
Consensus TA	75.0	70.4	65.4
TSV-Merge	85.9	80.1	77.1

In ASR, TSV-Merge and BoostedTSV-M outperform full fine-tuning in out-of-distribution settings while substantially closing the in-domain performance gap. For instance, BoostedTSV-M achieves in-domain WER of 9.27% vs. 8.54% for full fine-tuning and outperforms it in non-primary domains (Carvalho et al., 5 Mar 2026).

Ablation studies indicate that interference reduction alone yields +5 percentage points over TA, while the full TSV-Merge pipeline (compression plus whitening) delivers the largest gains (+15–17 percentage points). Lower STI values consistently align with higher normalized accuracy (Gargiulo et al., 2024).

6. Extensions: Layer-Adaptive Scaling (LARV) and Integration

TSV-Merge treats each layer uniformly, though transformer and deep network architectures manifest strong layerwise heterogeneity. The Layer-wise Adaptive Rescaling Veneer (LARV) extends TSV-Merge by applying deterministic, data-free per-layer scaling factors derived from diagnostics such as effective-rank contrast and commutator conflict coefficients. Shallow layers, prone to interference, are adaptively down-scaled; deeper, task-stable layers are up-scaled. This yields further gains: on FusionBench, TSV-Merge+LARV improves top-1 accuracy by up to 6.6 percentage points over TSV-Merge alone for ViT-B/32 (8 tasks), and similar improvements are observed across other architectures (Wang et al., 10 Feb 2026).

LARV operates as an orthogonal, merger-agnostic “wrapper,” and its tiered or continuous scaling rules require no retraining or hyperparameter tuning.

7. Practical Considerations, Limitations, and Future Research

TSV-Merge is training-free, requiring only access to fine-tuned models and matrix decompositions (SVDs and orthogonalizations), making it widely applicable to large models without additional data. The storage-accuracy trade-off is controlled via the per-layer rank $t=1, \dots, T$8; $t=1, \dots, T$9 or a 10% energy threshold is empirically effective (Gargiulo et al., 2024, Carvalho et al., 5 Mar 2026).

Limitations include:

• When two tasks have highly aligned singular vectors, whitening or low-rank truncation may fail to fully separate discriminative information or cause attenuation.
• Small numbers of tasks $i$0 provide insufficient basis for robust orthogonalization benefits.
• Orthogonal Procrustes is numerically unstable for high ranks; Newton–Schulz orthogonalization is preferred in practice.
• Fixed merge weights $i$1 may not be globally optimal; adaptive or per-layer weighting schemes are open areas of investigation.
• Task interference persists in the face of strongly conflicting domains; additional innovations such as per-layer weighting or sparse subspace alignment are being considered.

A plausible implication is that future model-merging frameworks will integrate vector-level boosting, automatic rank/energy allocation, and adaptive merge weighting, extending TSV-Merge's principled structure to further reduce interference and maximize multi-domain generalization (Wang et al., 10 Feb 2026, Carvalho et al., 5 Mar 2026).