TSV-Merge: Multi-Domain Neural Merging

• TSV-Merge is a training-free method that extracts and concatenates low-rank singular vector subspaces from fine-tuned neural models to build a unified multi-domain checkpoint.
• It employs SVD truncation followed by orthonormalization to minimize cross-task interference, with BoostedTSV-M addressing rank collapse through singular-value boosting.
• Quantized TSV-Merge enhances memory efficiency by reducing storage requirements to 5–8% while preserving near full-precision accuracy across vision and ASR tasks.

Task-Singular-Vectors Merging (TSV-M), commonly abbreviated as TSV-Merge, is a training-free, methodologically principled strategy for merging multiple independently fine-tuned neural models into a single multi-domain checkpoint. TSV-M operates by extracting and concatenating low-rank singular vector subspaces from task-specific weight updates, followed by orthogonalization to minimize cross-task interference. The methodology is motivated by the empirical observation that most fine-tuning-induced parameter changes are low-rank and largely orthogonal in their singular vector structure, particularly in models like transformers for computer vision and automatic speech recognition (ASR). TSV-Merge, its boosted variant BoostedTSV-M, and specialized quantized implementations have demonstrated superior multi-task performance and efficiency across vision and ASR domains (Carvalho et al., 5 Mar 2026, Gargiulo et al., 2024, Kim et al., 10 Mar 2025).

1. Formal Foundation and Algorithmic Structure

TSV-M builds on the decomposition of task-specific updates relative to a shared foundation model. For each of $T$ downstream tasks and at each matrix-weighted layer $i$:

• The task vector is defined as $\tau_{i,t} = W_{i}^{(t)} - W_{i}^0$, where $W_{i}^{(t)}$ is the fine-tuned weight and $W_{i}^0$ is the base model weight.
• A thin singular value decomposition (SVD) is performed: $\tau_{i,t} = U_{i,t} \Sigma_{i,t} V_{i,t}^\top$.
• Each task's update is truncated to the top $k$ singular values/vectors: $U_{i,t}^{(k)}$, $\Sigma_{i,t}^{(k)}$, $V_{i,t}^{(k)}$.
• These truncated bases are concatenated across tasks: $i$0, similarly for $i$1 and block-diagonal $i$2.
• To reduce singular task interference (STI), the concatenated bases are orthonormalized, typically via the Newton–Schulz or orthogonal Procrustes methods, yielding $i$3 and $i$4.
• The merged update for layer $i$5 is reconstructed as $i$6, and merged model weights are assembled as $i$7 (typically $i$8).

Closed-form pipeline pseudocode is given in (Carvalho et al., 5 Mar 2026, Gargiulo et al., 2024), requiring only SVDs, concatenations, and matrix orthogonalizations, with no additional gradient descent steps.

2. Motivations: Task Interference and Low-Rank Structure

Empirical studies indicate that per-layer task updates after fine-tuning are highly structured and typically low-rank with rapidly decaying singular spectra (Gargiulo et al., 2024). Direct arithmetic merging of full task vectors leads to significant cross-task interference, as subspaces corresponding to different tasks overlap. TSV-M addresses this by isolating the principal subspaces per task (by truncation) and enforcing cross-task subspace orthogonality (by whitening/orthonormalization before reconstruction), drastically reducing interference. The singular task interference (STI) score quantifies interaction among singular subspaces and is sharply reduced by TSV-M compared to task arithmetic.

Ablation experiments confirm that both low-rank truncation and subsequent orthogonalization are necessary: truncation alone retains subspace overlap, while orthogonalization on full-rank updates incurs high reconstruction error. TSV-M’s combination ensures minimal task interference and maximal subspace diversity within memory and compute constraints (Gargiulo et al., 2024).

3. Rank Collapse Pathology and BoostedTSV-M

A notable pathology of vanilla TSV-M is "rank collapse," which arises when, after SVD truncation, many singular values are vanishingly small. Upon concatenation and orthonormalization, these near-zero directions may collapse onto a lower-dimensional subspace, causing effective subspace degeneracy and loss of cross-domain robustness. Numerically, this is visible as the conditioning $i$9, which destabilizes orthogonalization algorithms and reduces the diversity of the merged subspace.

BoostedTSV-M resolves rank collapse by implementing "singular-value boosting": for each task and layer, singular values below a data-dependent threshold (set by a cumulative energy fraction $\tau_{i,t} = W_{i}^{(t)} - W_{i}^0$0) are clamped up to the threshold value before truncation. This process ensures that a substantial fraction of the energy is preserved in the subspace, even for small singular values, and mitigates numerical instability in orthonormalization. Empirically, $\tau_{i,t} = W_{i}^{(t)} - W_{i}^0$1 achieves an optimal trade-off between in-domain (ID) and out-of-distribution (OOD) performance in multi-domain ASR (Carvalho et al., 5 Mar 2026).

4. Memory Efficiency: Quantized TSV-Merge

TSV-Merge can be further adapted for memory efficiency via task vector quantization (TVQ) (Kim et al., 10 Mar 2025):

• Standard uniform quantization is applied to task vectors, leveraging their low dynamic range for bit-widths as low as 2–4 bits with minimal error.
• Residual Task Vector Quantization (RTVQ) decomposes each task vector into a shared high-precision base and per-task low-precision residuals, distributing bits according to quantization sensitivity.
• This approach supports scalable storage, reducing memory cost to 5–8% of full precision with negligible (<1%) loss in downstream accuracy.
• Sensitivity-driven bit allocation can be formalized via per-layer error-sensitivity metrics and Lagrangian optimization for bit allocation under a total budget.

This quantized variant supports merging over arbitrarily many tasks while preserving near-full accuracy and drastically reducing storage requirements (Kim et al., 10 Mar 2025).

5. Empirical Performance and Benchmarks

TSV-Merge and BoostedTSV-M routinely outperform previous gradient-free model merging methods:

• On 10-domain European Portuguese ASR (in-domain WER: zero-shot 15.62%, TSV-M 9.41%, BoostedTSV-M 9.27%, Full-FT 8.54%; OOD WER: TSV-M 16.07%, BoostedTSV-M 16.11%, Full-FT 17.65%) (Carvalho et al., 5 Mar 2026).
• In multilingual scenarios, TSV-M maintains competitive OOD performance and preserves cross-lingual generalization, e.g., African Portuguese WER (TSV-M 21.61%, BoostedTSV-M 21.58%, Full-FT 23.96%) and for English OpenASR-HF (TSV-M 7.24%, BoostedTSV-M 7.60%, Full-FT 8.83%).
• In vision tasks (CLIP ViT-B-32, 8–20 tasks), TSV-Merge shows absolute accuracy gains of 15–17% over task arithmetic and retains >94% performance compared to individually fine-tuned models (Gargiulo et al., 2024).
• Quantized TSV-Merge matches or slightly exceeds full-precision accuracy in both image classification and dense prediction within 0.3% given 4-bit quantization (Kim et al., 10 Mar 2025).

6. Implementation and Practical Recommendations

Hyperparameter and implementation guidelines supported by empirical evidence include:

• Set per-layer retention $\tau_{i,t} = W_{i}^{(t)} - W_{i}^0$2 (where $\tau_{i,t} = W_{i}^{(t)} - W_{i}^0$3 is the full SVD rank and $\tau_{i,t} = W_{i}^{(t)} - W_{i}^0$4 is the task count).
• Choose boosting threshold $\tau_{i,t} = W_{i}^{(t)} - W_{i}^0$5 for BoostedTSV-M for optimal ID/OOD balance.
• Scaling parameter $\tau_{i,t} = W_{i}^{(t)} - W_{i}^0$6 is typically optimal, with possible OOD preservation at $\tau_{i,t} = W_{i}^{(t)} - W_{i}^0$7.
• Prefer Newton–Schulz orthonormalization (10–20 iterations) over Procrustes for numerical stability in large models and low-rank truncations.
• Use truncated (power-method) SVD on GPUs and fuse concatenate/orthonormalize routines for efficiency.
• Store only compressed U/V bases and boosted singular values after merging; discard original task updates.

Compared to full fine-tuning, TSV-M and its variants reduce the need for repeated multi-epoch retraining and avoid the inference overhead of checkpoint juggling, offering a streamlined, one-shot merging solution for large-scale multi-domain adaptation (Carvalho et al., 5 Mar 2026).

7. Summary and Research Impact

TSV-Merge and its enhancements address core limitations of earlier training-free model merging methodologies by formalizing low-rank update compression and minimizing inter-task annihilation in shared representation subspaces. BoostedTSV-M corrects rank degeneracy that can emerge from aggressive SVD truncation. TSV-Merge quantization supports scalability for large $\tau_{i,t} = W_{i}^{(t)} - W_{i}^0$8 in bandwidth-constrained environments. These innovations substantially narrow the empirical gap to full independent fine-tuning, both in speech and vision applications, without introducing additional task-specific parameters or costly retraining (Carvalho et al., 5 Mar 2026, Gargiulo et al., 2024, Kim et al., 10 Mar 2025).

Ongoing research extends these techniques to other modalities and investigates automated subspace selection, adaptive boosting, and decomposed merging strategies under severe resource limitations. A plausible implication is that the singular vector paradigm, particularly when combined with subspace interference metrics and quantization-aware design, will remain central to future advances in multi-domain model merging.