Task Singular Vectors: Reducing Task Interference in Model Merging

Abstract: Task Arithmetic has emerged as a simple yet effective method to merge models without additional training. However, by treating entire networks as flat parameter vectors, it overlooks key structural information and is susceptible to task interference. In this paper, we study task vectors at the layer level, focusing on task layer matrices and their singular value decomposition. In particular, we concentrate on the resulting singular vectors, which we refer to as Task Singular Vectors (TSV). Recognizing that layer task matrices are often low-rank, we propose TSV-Compress (TSV-C), a simple procedure that compresses them to 10% of their original size while retaining 99% of accuracy. We further leverage this low-rank space to define a new measure of task interference based on the interaction of singular vectors from different tasks. Building on these findings, we introduce TSV-Merge (TSV-M), a novel model merging approach that combines compression with interference reduction, significantly outperforming existing methods.

• The paper introduces Task Singular Vectors (TSV) that decompose weight matrices via SVD to effectively reduce task interference in model merging.
• The TSV framework compresses models by retaining key singular vectors, preserving 99% of accuracy with only 10% of storage and decorrelating features to minimize interference.
• Empirical results show that TSV-M outperforms traditional methods by an average of 15% in accuracy across multiple tasks, highlighting its practical impact in multi-task learning.

Analysis of "Task Singular Vectors: Reducing Task Interference in Model Merging"

Introduction

The paper "Task Singular Vectors: Reducing Task Interference in Model Merging" (2412.00081) introduces a novel approach to address the challenges associated with merging machine learning models—specifically, the task interference that arises when multiple models are combined into a single framework. Task interference refers to the degradation in performance that occurs when models, originally trained on different tasks, are merged without regard for how their internal structures interact.

Methodology

Task Singular Vectors (TSV)

The paper's primary innovation is the introduction of Task Singular Vectors (TSV), derived from the Singular Value Decomposition (SVD) of layer-specific weight matrices. By decomposing these matrices, the authors capture key structural information neglected by previous methods that treat models as flat parameter vectors. This step not only reduces dimensionality but highlights significant directions of variation within each task, encapsulated in the singular vectors.

Compression and Merging with TSV

Two main approaches are proposed:

1. TSV-Compress (TSV-C): This method compresses the model by retaining only a small, significant portion of the task singular vectors. The authors demonstrate that this retains 99% of the original model accuracy while using just 10% of the storage space. This efficiency is crucial for scaling models across different tasks without exponential growth in resource requirements.
2. TSV-Merge (TSV-M): TSV-M further leverages the compressed representation, combining it with a task interference reduction technique. By decorrelating singular vectors through a whitening transformation, task interference is minimized, improving the reliability of the merged model across diverse tasks.

Empirical Validation

The TSV framework's efficacy is validated through extensive experiments across various computer vision benchmarks. The authors emphasize the robustness of their approach, as their merged models outperform traditional methods by an average margin of 15% in accuracy. Such a significant improvement underscores the ability of TSVs to finely manage the delicate balance between task specificity and model generalization.

Figure 1: Mean accuracy of a ViT-L-14 merged over 8, 14, and 20 tasks respectively, demonstrating the state-of-the-art results achieved by TSV-M.

Implications and Future Work

The introduction of TSVs offers several implications:

• Theoretical Impact: TSVs apply advanced linear algebra to elucidate and enhance model merging, providing a novel perspective that could extend to other areas of AI, such as model distillation or continual learning.
• Practical Applications: The capability to merge models with minimal performance loss opens the door to integrating various pre-trained models into a singular, cohesive system without prohibitive computational costs or retraining requirements.

  Figure 2: Visualization of task interference among 8 tasks computed on the first attention layer of a ViT-B-32.

Future developments may involve exploring alternative metrics for interference, applying TSV concepts to other neural architectures like RNNs or CNNs, or integrating task singular vectors directly into the training regimes to mitigate interference even before the merging stage.

Conclusion

This work lays out a compelling case for reconsidering how models are merged by focusing on the intrinsic structure captured by task singular vectors. The authors demonstrate that a strategic decomposition and compression approach yields significant enhancements in both model storage efficiency and predictive accuracy when merged models operate across multiple tasks. This research not only challenges conventional paradigms of model merging but also offers a scalable solution to integrate neural networks trained across varied domains into a single, highly functional outcome. The innovative methodologies discussed in this paper are poised to contribute significantly to the fields of multi-task learning and model optimization, serving as a cornerstone for future research endeavors and applications.