Specialist Model Merging Framework

• Specialist Model Merging Frameworks are techniques that combine fine-tuned expert models into one system using dynamic, similarity-based rescaling without further data or retraining.
• The framework leverages representation clustering to align merged outputs with task-specific experts, reducing interference and enhancing multi-task generalization.
• Empirical results in vision and language tasks demonstrate significant performance gains and efficient inference-time adaptation through adaptive rescaling.

Specialist Model Merging Frameworks

Specialist model merging frameworks provide algorithmic and mathematical principles to unify multiple fine-tuned expert models into a single parameterization that exhibits strong performance across all original specialist tasks, typically without further data or retraining. These frameworks address central challenges such as representation interference, task adaptation, parameter scaling, memory efficiency, and compatibility across heterogeneous architectures. Recent advances articulate precise merging criteria rooted in both parameter- and representation-space analyses and offer efficient inference-time or task-aware adaptations, substantially improving multi-task generalization and deployment feasibility.

1. Foundational Mechanisms of Specialist Model Merging

Many specialist model merging frameworks, including SE-Merging (Chen et al., 22 Jun 2025), are grounded in the concept of merging multiple expert model parameters, often fine-tuned from a shared pretrained base $\theta_{\mathrm{PT}}$, by summing scaled task vectors:

$$\theta_{\mathrm{Merged}} = \theta_{\mathrm{PT}} + \sum_{i=1}^T \lambda_i \tau_i,\quad \tau_i = \theta_i - \theta_{\mathrm{PT}}$$

A key observation is that the internal activations $f_\ell(x;\theta_{\mathrm{Merged}})$ typically cluster according to the task source even when no explicit task identifier is available. This phenomenon enables sample-wise auto-adaptation: for each input $x$ originating from task $T_i$, its merged representation $f_\ell(x;\theta_{\mathrm{Merged}})$ closely aligns with that computed under the corresponding fine-tuned expert. Formally,

$$\forall \ell, \forall x\in D_i:\quad \mathrm{Dist}\big( f_\ell(x;\theta_{\mathrm{Merged}}),\, f_\ell(x;\theta_{\mathrm{PT}}+\lambda \tau_i) \big) \approx 0$$

indicating that the merged model's representations are locally near-expert in the relevant region of input space. This intrinsic representation clustering is the basis for dynamic specialist capacity in merged models.

2. Adaptive Rescaling and Dynamic Specialist Enhancement

SE-Merging introduces a self-enhanced model merging framework in which the merging coefficients $\lambda_t(x)$ for each task vector $\tau_t$ are adaptively rescaled for each sample $x$ via inference-time computation. After a static merge $$\theta_{\mathrm{Merged}} = \theta_{\mathrm{PT}} + \lambda \sum_t \tau_t$$ using a base coefficient $\lambda$, the following sample-wise logic is applied:

• Extract merged ($r_{\mathrm{Merged}}$) and per-expert ($r_t$) representations at a chosen layer $\ell$,
• Compute per expert distances $d_t = \|r_{\mathrm{Merged}} - r_t\|_2$,
• Convert $d_t$ to similarities $s_t = (d_{\max}-d_t)+d_{\min}$, normalize via min-max,
• Compute softmax-rescaled coefficients:

$$\lambda_t(x) = T\lambda \,\frac{\exp(\hat{s}_t)}{\sum_{j=1}^T \exp(\hat{s}_j)}$$

• Form the sample-specific merged weight:

$$\theta_{\mathrm{SE}}(x) = \theta_{\mathrm{PT}} + \sum_t \lambda_t(x)\tau_t$$

Inference uses $\theta_{\mathrm{SE}}(x)$ rather than a statically averaged parameter. This mechanism can be plugged on top of any base merging approach (e.g., TIES-Merging or PCB merging), as the dynamic per-sample rescaling is orthogonal to conflict-resolution preprocessing.

Pseudocode (inference-time, training-free):

theta_merged = theta_PT + lambda * sum(tau_t for t in range(T)) for x in test_set: # Representation extraction r_merged = f_l(x, theta_merged) distances = [] for t in range(T): r_t = f_l(x, theta_PT + lambda * tau_t) d_t = l2_norm(r_merged - r_t) distances.append(d_t) # Min-max normalization dmin, dmax = min(distances), max(distances) s = [(dmax - dt) + dmin for dt in distances] s_hat = [(st - min(s)) / (max(s) - min(s)) for st in s] # Softmax rescaling lambdas = [T * lambda * exp(sht) / sum(exp(shj) for shj in s_hat) for sht in s_hat] # Dynamic merge theta_se_x = theta_PT + sum(lam * tau for lam, tau in zip(lambdas, tau_t)) y_hat = f(x, theta_se_x)

Layer choice $\ell$ is critical for optimal adaptation; empirical evidence indicates deeper layers are better for task separation. Cosine distance yields similar results to $\ell_2$.

3. Integration with Static Merging and Compatibility

SE-Merging is fundamentally compatible with all static, training-free merging methods. For instance, if conflict-resolution merging yields post-processed task vectors $\tau_t^0$ and coefficients $\lambda_t^0$, SE-Merging recomputes the dynamic similarity and rescaling on these, producing

$$\theta_{\mathrm{SE}}(x) = \theta_{\mathrm{PT}} + \sum_{t=1}^T \lambda_t(x)\,\tau_t^0$$

This design ensures that any prior improvements or sparsity/control from base methods are inherited, while still permitting per-sample adaptation and specialization.

4. Empirical Outcomes and Performance Gains

Observed performance gains from SE-Merging and its components are substantial and robust across modalities:

Vision (CLIP/ViT, eight tasks):

• Task Arithmetic ($\lambda=0.3$): 70.1% (ViT-B/32), 84.5% (ViT-L/14)
• AdaMerging: 81.1% (B/32), 91.0% (L/14)
• SE-Merging: 84.96% (B/32, +3.86pts), 91.57% (L/14, +0.57)
• Representation bias (layer $\ell_1$) reduced by 20–30% after SE-Merging.

Language (GPT-2, seven GLUE tasks):

• Static merging: 70.0% avg
• TIES-Merging: 70.0% avg
• SE-Merging: 76.86% avg (+6.86 pts)

Additional ablation experiments show that replacing $\ell_2$ with cosine, or skipping normalization, leads to a 1–2% performance decrease. Ensuring scale stability (e.g., using $T\lambda$) is necessary for preservation of the underlying parameter regime. Deep layers in transformers and vision architectures systematically yield better task clustering.

5. Computational Requirements, Scaling, and Application Scope

The SE-Merging strategy is inference-time and training-free; it can be integrated into existing pipelines for vision transformers, LLMs, and multimodal models. The computational overhead consists primarily of a forward pass at chosen layer(s) through $T+1$ models per sample (merged + $T$ experts) to compute layer-level representations, distance matrices, and softmaxes. Memory usage is comparable to static merging, and effective deployment requires no auxiliary data or retraining. Scaling to hundreds of experts is theoretically possible, but with proportional inference-time cost and practical constraints on representation storage and similarity computation. Compatibility is maintained with prior merging improvements (e.g., those from conflict-resolution or sparsity-aware methods).

6. Implications, Trade-Offs, and Practical Considerations

SE-Merging demonstrates that multi-task abilities in merged models emerge chiefly from the dual ability to discriminate input samples by their underlying specialist task and adapt representations toward the corresponding expert model. By leveraging dynamic, similarity-based rescaling of task-vector coefficients, practitioners can realize significant gains in aggregate multi-task accuracy without retraining.

Essential trade-offs include layer selection for representation extraction (trade-off between separation and computation), the method of similarity calculation, normalization procedure, and the size/scale of base coefficients. Empirical results indicate that for both vision and language domains, the framework is robust to these choices within sensible ranges.

In summary, SE-Merging provides a theoretically grounded, empirically validated, and computationally efficient dynamic model merging approach. Its main contributions are: (1) identification of representation-based task separation and auto-adaptation as primary mechanisms, (2) sample-wise reweighting via forward-only similarity, and (3) strict plug-in compatibility with all established merging pipelines. This framework is directly applicable to any LLM, transformer, or multi-expert deployment scenario requiring efficient, high-fidelity multi-task adaptation.