Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compression

Abstract: Model merging has attracted attention as an effective path toward multi-task adaptation by integrating knowledge from multiple task-specific models. Among existing approaches, dynamic merging mitigates performance degradation caused by conflicting parameter updates across tasks by flexibly combining task-specific parameters at inference time, thereby maintaining high performance. However, these methods require storing independent parameters for each task, resulting in prohibitive storage overhead. To address this issue, we first experimentally demonstrate that the fine-tuned weight increments (referred to as task vectors) exhibit an impulse-like activation pattern and high robustness to low-bit representations. Driven by this insight, we propose T-Switch, which decomposes task vectors into three compact components: a binary sparse mask, a sign vector, and a scalar scaling factor, achieving high-fidelity approximation at high compression ratios. We then introduce Auto-Switch, a training-free merging scheme that automatically composes task vectors via feature similarity retrieval. Building on this, we develop Auto-Switch, a training-free merging scheme that automatically assembles task vectors through feature similarity retrieval. Furthermore, to transform task vector sparsification and quantization from static rules to adaptive learning, we propose FlexSwitch, a learnable framework which jointly optimizes the compression strategy for each model unit via Learnable Gating Sparsification (LGS) and Bit-width Adaptive Selection (BAS), while employing the Sparsity-Aware Storage Strategy (SASS) to select the optimal storage encoding structure. Finally, by incorporating a K-Nearest Neighbor (KNN) inference scheme with a learnable low-rank metric, we present Auto-FlexSwitch, a dynamic model merging approach that supports highly efficient task vector compression.

• The paper introduces a framework that compresses task vectors via impulse-like activations and quantization robustness, achieving up to 16× storage reduction.
• The methodology combines learnable gating sparsification, bit-width adaptive selection, and a KNN-based dynamic routing mechanism to optimize multi-task adaptation.
• Experimental results demonstrate that Auto-FlexSwitch maintains high accuracy across vision, detection, and language tasks with only 1–17% of conventional storage overhead.

Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compression

Introduction

Auto-FlexSwitch advances dynamic model merging by introducing a highly parameter-efficient framework that adaptively compresses task-specific weight increments (task vectors) for multi-task adaptation. Previous dynamic merging methods mitigate performance degradation from conflicting task-specific weights but at the cost of substantial storage overhead. This paper systematically investigates the structural properties of task vectors—specifically their pulse-activation patterns and robustness to low-bit quantization—and leverages these findings to engineer a learnable pipeline for efficient compression and storage, enabling dynamic task routing with minimal performance loss and substantial reductions in storage requirements.

Task Vector Structural Properties

Empirical analyses reveal two crucial properties of task vectors:

1. Impulse-like activation: Only a small subset of high-magnitude parameters contributes significantly to task adaptation. The remainder can be pruned without performance loss; in some settings, pruning even improves task accuracy.
2. Quantization robustness: After high-magnitude pruning, sign-based binarization (retaining only polarity) coupled with L2-norm scaling preserves task performance, especially as sparsity increases.

These two findings collectively motivate decomposing task vectors into three compact components—a binary sparse mask, a sign vector, and a scaling factor—which achieves high-fidelity approximation with at least a 16× storage reduction.

Figure 1: Accuracy (%) trends of three pruning strategies C1, C2, C3 on eight vision tasks as pruning rate $\alpha$ varies; performance improvements for C1 are visible at moderate sparsity.

Figure 2: Task vector accuracy with P-Spar and B-Approx across pruning rates; binarization reduces storage with negligible loss, especially at high sparsity.

Lightweight Compression and Dynamic Routing

T-Switch and Auto-Switch

Building upon the structural insights, T-Switch decomposes each task vector as follows:

• Activation Switch: Sparse binary mask selecting activated parameters.
• Polarity Switch: Binary sign vector for update directionality.
• Switch Knob: Scalar scaling coefficient for approximate magnitude alignment.

Storage of task vectors becomes highly efficient, requiring only two bits per parameter plus a scalar. For dynamic merging, Auto-Switch uses a training-free KNN retrieval scheme, determining routing weights based on feature similarity between the input and small exemplars from each task—eliminating the need for explicit routers.

Figure 3: Compression pipelines for T-Switch and FlexSwitch; inference pipelines for Auto-Switch and Auto-FlexSwitch.

Learnable Adaptive Compression: FlexSwitch Framework

Analysis of module/task sensitivity shows vanilla T-Switch's fixed sparsity and quantization is suboptimal; layers and modules have heterogeneous tolerance to sparsity/bitwidth and scaling. Three bottlenecks are directly addressed:

• Non-uniform sparsity requirements across modules/layers.
• Bitwidth (quantization precision) sensitivity at the module/layer level.
• Static scaling norms are suboptimal for task adaptation.

FlexSwitch enables end-to-end learning for all these factors. Key components:

• Learnable Gating Sparsification (LGS): Each module's task vector is equipped with learnable thresholding, generating soft binary masks via temperature-controlled sigmoids; magnitude scaling is also learned.
• Bit-width Adaptive Selection (BAS): Quantization precision per module is learned (from $\{1,2,4,8\}$ bits), optimizing storage/performance.
• Sparsity-Aware Storage Strategy (SASS): Task vectors are stored in a grouped COO sparse format, adaptively selecting group sizes to minimize actual storage based on achieved sparsity.

  Figure 4: Heatmaps of module/layer sensitivity to sparsification and quantization; degradation is not uniform, motivating adaptivity.

  

  Figure 5: Performance drop as magnitude scaling tuning factor $\eta$ varies; distinct optima per task, demonstrating the need for learnable calibration.

  

  Figure 6: LGS effectiveness vs. P-Spar; LGS maintains accuracy at extreme sparsity, up to 98%.

  

  Figure 7: SASS vs. Indep storage overhead; SASS achieves pronounced storage gains as sparsity increases, especially for large vectors.

Dynamic Routing and Efficient Inference

Auto-FlexSwitch integrates FlexSwitch’s compressed task vectors with a KNN-based routing mechanism using a learned low-rank metric, improving accuracy and efficiency for task assignment in scenarios with overlapping feature distributions. Representative centers from each task query set are extracted via K-means, and a low-rank projection is trained to maximize retrieval discriminability.

Figure 8: Pareto comparison (performance vs. storage) between FlexSwitch and T-Switch; FlexSwitch achieves comparable accuracy with significantly lower storage.

Figure 9: Model performance sensitivity to sparsity ratio $\alpha$ and preservation coefficient $\lambda$; Auto-FlexSwitch shows greater robustness and performance controllability.

Figure 10: Performance under varying numbers of exemplars and neighbors; Auto-FlexSwitch achieves stable accuracy even with reduced exemplars.

Figure 11: Ablation analysis of center number and low-rank dimension; moderate values suffice for stable, high performance.

Experimental Results

Image Classification

Auto-FlexSwitch demonstrates competitive or superior accuracy and substantial storage savings across ViT-B/32, ViT-L/14, and ConvNeXt on eight vision tasks. For ViT-L/14, Auto-FlexSwitch achieves comparable or better accuracy than MTL and fine-tuned models with only $\sim$1% of their storage overhead.

Object Detection

On RoboFlow-100 detection tasks, static merging is largely ineffective (average mAP$<$4%), confirming that parameter conflicts are highly sensitive. Auto-FlexSwitch yields top accuracy (42.4% average mAP) with only 15% storage compared to the next-best dynamic method.

Language Understanding

On GLUE tasks using RoBERTa-base and Mamba-130M, Auto-FlexSwitch achieves best or second-best average performance with as little as 3--17% storage overhead relative to dynamic baselines, even exceeding individually fine-tuned performance on specific tasks.

Ablations

Component analysis confirms LGS and BAS synergistically optimize sparsity and bitwidth, reducing storage further while preserving accuracy. SASS's empirical storage closely tracks theory, achieving $40\times$--$200\times$ compression depending on sparsity.

Figure 12: SASS storage overhead scaling with sparsity $\alpha$; exceeds $\{1,2,4,8\}$0 compression at extreme sparsity.

FlexSwitch for Fine-tuned LLM Compression

On Llama-3.2-3B and Gemma-2-9b lit, FlexSwitch compresses fine-tuned weights to $\{1,2,4,8\}$10.1–2% of original storage with negligible or positive impact on task performance across several reasoning and coding benchmarks.

Implications and Future Directions

Auto-FlexSwitch proposes a practical paradigm for dynamic multi-task model merging, balancing accuracy and storage cost through learnable task vector compression and adaptive routing. Its empirical robustness, efficiency, and competitive performance across diverse architectures (transformers, convolutions, state space models, LLMs) marks a clear advancement in the domain of parameter-efficient multi-task adaptation.

Future work will aim to encourage sparsity/quantizability during fine-tuning, perform fine-grained architectural analyses for further reduction, and extend decision-making over task vectors in embodied and online learning settings.

Conclusion

Auto-FlexSwitch systematically addresses the dynamic merging/storage efficiency trade-off by proposing a learnable framework for task vector compression, adaptive quantization, and discriminative dynamic routing. Its demonstrated ability to deliver high performance with minimal storage overhead across heterogeneous downstream tasks and models establishes its utility for scalable, parameter-efficient multi-task adaptation and deployment (2604.28109).