Closed-Form Spectral Regularization for Multi-Task Model Merging

Abstract: Model merging combines several independently fine-tuned experts into a single multi-task model without any training data, reducing the storage, serving, and decentralized-development costs of large foundation models. State-of-the-art merging methods formulate merging as a layer-wise quadratic interference minimization problem. Although this problem admits an exact closed-form pseudoinverse solution, that solution underperforms hundreds of iterations of gradient descent in practice. The iterative loop dominates the cost of the pipeline, yet its effectiveness has remained unexplained. We revisit this regime and show that the iterative solver does not primarily act as an optimizer; rather, it serves as an implicit spectral regularizer for an ill-posed normal equation, where small-eigenvalue directions of the per-layer interference operator amplify proxy noise. Building on this finding, we formalize multi-task model merging as a noisy linear inverse problem and propose a spectral filtering estimator parameterized by a per-direction filter. We instantiate this estimator with SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent, with a hard top-K truncation that suppresses noise-amplifying small-eigenvalue directions. Furthermore, we propose SWUDI-A, an adaptive variant that replaces the global rank hyperparameter with per-layer rank rules, further improving robustness across architectures. Both variants share a single symmetric eigendecomposition per linear layer and require no training data or optimizer state. Across four general benchmarks and a multimodal merging benchmark spanning VQA, Geometry, Chart, OCR, Grounding, and modality merging, our proposed spectral solvers match or outperform state-of-the-art merging methods. Crucially, they reduce wall-clock time by 28-72x and peak GPU memory by up to 50%.

• The paper demonstrates that iterative solvers implicitly regularize merging via spectral filtering to reduce noise amplification.
• It presents closed-form methods (SWUDI/ASWUDI) that efficiently merge task-specific models, matching or exceeding iterative baselines while reducing computation.
• Results across vision, language, and multimodal benchmarks reveal significant accuracy improvements combined with dramatic speed and memory gains.

Closed-Form Spectral Regularization for Multi-Task Model Merging

Introduction and Motivation

Multi-task model merging addresses the integration of multiple independently fine-tuned experts—each trained for distinct tasks—into a single multi-task model without access to any of the original training data. This approach is critical for reducing the deployment, storage, and collaborative development costs associated with large-scale foundation models, especially as the open-source community continues to proliferate domain-, task-, and modality-specialized checkpoints. Established methods such as WUDI and OptMerge formulate this as a data-free, layer-wise quadratic minimization problem. While the objective admits a closed-form pseudoinverse, empirical results indicate that hundreds of steps of iterative optimization (e.g., Adam) surprisingly outperform the analytical solution.

Figure 1: Illustration of model merging methods, with WUDI/OptMerge iteratively solving a quadratic proxy loss and the proposed SWUDI/ASWUDI replacing the loop with closed-form eigendecomposition and spectral filtering.

This paper systematically investigates the spectral and noise properties underlying these observations, framing the merging objective as a noisy linear inverse problem. The central contribution is the identification and formalization of implicit spectral regularization as the true regularizing factor in iterative solutions, and the derivation of explicit, closed-form solvers that match or exceed baseline methods while delivering dramatic improvements in computational efficiency.

Theoretical Framework and Regularization

Problem Formulation

Given $N$ task experts (parameter differences $\tau_i$ per layer), previous merging methods minimize the layer-wise interference loss: $$\mathcal{L}(\tau) = \sum_{i=1}^N \frac{1}{\|\tau_i\|_F^2} \|(\tau - \tau_i)\tau_i^\top\|_F^2$$ This quadratic has a unique minimum-norm solution parameterized as $\tau^{\rm cf} = DC^\dagger$, where $C$, the aggregate covariance proxy, is often highly ill-conditioned. The ill-posedness arises from directions in parameter space weakly supported across task vectors, leading the pseudoinverse to amplify noise in those subspaces.

Figure 2: The majority of proxy reduction is explained by leading eigendirections; closed-form pseudoinverse minimizes the proxy but increases real interference in tail directions.

Spectral Regularization Analysis

The critical finding is that iterative solvers (e.g., Adam, SGD) implicitly act as spectral regularizers. Early stopped iterative descent attenuates updates along small-eigenvalue directions, avoiding the amplification of proxy-induced noise.

Figure 3: Iterative merging realizes implicit spectral filtering—large-eigenvalue directions are updated more quickly, and the Adam trajectory matches a soft exponential spectral filter.

This motivates the spectral-filtering estimator framework, parameterized by per-direction filters $h_k \in [0,1]$: $$\widehat{\tau}_h = \tau_\mathrm{init} + (D - \tau_\mathrm{init}C) Q \operatorname{diag}(h_k/\lambda_k) Q^\top$$ where $h_k$ can be chosen as an exponential filter (matching gradient flow's trajectory) or hard truncation to eliminate noise-amplifying tail directions.

Failure Modes of Unregularized Solutions

Applying unregularized pseudoinverse solutions in the presence of proxy noise leads to two intertwined issues: noise amplification in low-eigenvalue directions and norm inflation (parameter updates much larger than any actual single-task update), which destabilize the merged model.

Figure 4: Unregularized iterative optimization induces norm-shortcut failures, with merged vectors growing overly large to minimize the proxy, misaligning with true task-related subspaces.

Methodology: Closed-Form Spectral Solvers

SWUDI: Spectrally Regularized WUDI

SWUDI implements a hybrid spectral filter for each layer, multiplying an exponential (soft, early-stopping) and a hard (rank truncation) filter: $h_k = \mathbf{1}[k \leq K] (1 - e^{-t\lambda_k})$ Optimally tuned hyperparameters $t$ (stopping time) and $\tau_i$0 (retained rank) allow SWUDI to closely match or surpass the best empirical results of iterative baselines, while avoiding the explicit iteration over hundreds of steps.

ASWUDI: Adaptive, Hyperparameter-Free Variant

ASWUDI eliminates global rank tuning by using layer-wise adaptive rank selection rules:

• Participation-Ratio (psqrt) Rule: For heavy-tailed spectra, retains the effective number of subspace directions carrying significant signal.
• Marchenko–Pastur Gavish–Donoho Rule: For spiked-noise or noise-contaminated low-rank spectra, uses rigorous statistical thresholds informed by random matrix theory.

Both variants require a single symmetric eigendecomposition per layer, fully data-free, with no reliance on optimizer trajectories or external test-time data.

Figure 5: Noise amplification in small-eigenvalue directions necessitates adaptive rank truncation; participation and Gavish–Donoho rules yield conservative, spectrum-matched cutoffs.

Benchmarks and Numerical Results

The solvers were evaluated extensively on both standard and novel benchmarks spanning vision (CLIP-ViT, Flan-T5) and multimodal LLMs (MLLMs), with both LoRA and full-parameter fine-tuning regimes.

Key numerical results include:

• On CLIP-ViT, SWUDI and ASWUDI achieve 85.55–92.52% accuracy (across backbones), consistently exceeding WUDI/OptMerge baselines and outperforming other closed-form and iterative methods.
• On the MLLM merging benchmark, ASWUDI matches or surpasses OptMerge, especially in low-rank LoRA settings, where regularization is critical to prevent norm inflation and instability.
• For capability and modality merging on InternVL2.5 and Qwen2-VL, SWUDI/ASWUDI not only match or exceed mixture-trained reference models (requiring full data access), but also deliver over 28–72× speedup in wall-clock time and up to 50% reduction in peak GPU memory.

  Figure 6: Accuracy-cost Pareto frontier shows SWUDI/ASWUDI achieving higher or comparable accuracy with dramatically reduced computation, Pareto-dominating the iterative baseline.

• The merged multimodal models display stronger integrated QA performance than individual experts, indicating preservation of composite abilities without interfering with specialist capabilities.

  Figure 7: MLLM merging: capability merging (left) integrates multiple specialized MLLMs; modality merging (right) fuses vision/audio/video-language experts with shared LLM backbone, all data-free.

Spectral Diagnostics and Ablations

Analysis confirms that the spectral tail of ill-conditioned directions carries little transferable signal but comprises the principal risk for noise amplification and norm inflation.

• Diagnostics show the appropriateness of adaptive layer-wise rank rules and the inutility of global cutoffs.
• Additional ablations (soft-only, hard-only) demonstrate the necessity of combining soft exponential filtering with hard truncation for both accuracy and stability.

Figure 8: Task vector magnitude distributions differ sharply between full fine-tuning and LoRA, supporting the need for layer-wise rank tuning as implemented in ASWUDI.

Implications and Future Developments

This work clarifies the underlying mechanism by which iterative, data-free merging achieves regularization: not through direct optimization per se, but via implicit spectral filtering. The presented closed-form spectral solvers eliminate the need for expensive optimizer loops, reducing both engineering complexity and hardware requirements—critical for large-scale, decentralized, or federated MLLM development.

Practical implications include:

• Faster deployment cycles for multi-task or multimodal models, as merging can be performed orders of magnitude faster without data or retraining.
• Better compositionality in modular AI systems, since merging can be tuned or adapted post hoc based solely on checkpoint deltas.
• Benchmark support for robust evaluation of merging quality and transfer dynamics across vision, language, and multimodal domains.

Theoretically, the explicit spectral framework provides a unifying language for previously disparate merging methods and informs future work on filter selection, noise modeling, and optimal rank adaptation. As models continue to scale, regularization via principled spectral analysis is expected to become the central engineering variable in merging pipelines, especially in settings with highly heterogeneous or low-rank specializations.

Conclusion

Closed-form spectral filtering, as implemented by SWUDI and its adaptive variant ASWUDI (2606.07289), enables data-free, training-free, and computationally efficient multi-task model merging by replacing iterative proxy minimization with an explicit, analytically justified spectral filter. Across vision, language, and multimodal settings, this paradigm delivers comparable or superior performance to the strongest iterative baselines while dramatically reducing computational cost, paving the way for broader, more efficient, and more principled approaches to large-scale model composition.