SimMerge: Learning to Select Merge Operators from Similarity Signals

Abstract: Model merging enables multiple LLMs to be combined into a single model while preserving performance. This makes it a valuable tool in LLM development, offering a competitive alternative to multi-task training. However, merging can be difficult at scale, as successful merging requires choosing the right merge operator, selecting the right models, and merging them in the right order. This often leads researchers to run expensive merge-and-evaluate searches to select the best merge. In this work, we provide an alternative by introducing \simmerge{}, \emph{a predictive merge-selection method} that selects the best merge using inexpensive, task-agnostic similarity signals between models. From a small set of unlabeled probes, we compute functional and structural features and use them to predict the performance of a given 2-way merge. Using these predictions, \simmerge{} selects the best merge operator, the subset of models to merge, and the merge order, eliminating the expensive merge-and-evaluate loop. We demonstrate that we surpass standard merge-operator performance on 2-way merges of 7B-parameter LLMs, and that \simmerge{} generalizes to multi-way merges and 111B-parameter LLM merges without retraining. Additionally, we present a bandit variant that supports adding new tasks, models, and operators on the fly. Our results suggest that learning how to merge is a practical route to scalable model composition when checkpoint catalogs are large and evaluation budgets are tight.

• The paper introduces a predictive framework that uses similarity signals to select and order merge operators, eliminating costly merge-and-evaluate loops.
• It leverages lightweight probe passes and similarity metrics to compute functional and structural features for optimal operator prediction across various tasks.
• Empirical results demonstrate significant performance improvements, robust multiway merging, and successful generalization across domains and model scales.

SimMerge: Predictive Merge Operator Selection via Model Similarity Signals

Introduction and Motivation

The proliferation of domain-specialized LLM checkpoints has made post-hoc model merging an essential tool for scalable model composition, supporting flexible, on-demand aggregation of capabilities with minimal retraining. Traditionally, merging relies on fixed weight-space operators (e.g., Linear, SLERP, TIES), often demanding costly merge-and-evaluate searches to select the optimal operator and merge order. However, the interaction between merge operator performance and model/task regimes is highly context-dependent, and naïve uniform application of merge rules often results in suboptimal, brittle outcomes. The paper "SimMerge: Learning to Select Merge Operators from Similarity Signals" (2601.09473) introduces a learned, predictive merge-selection strategy that leverages inexpensive functional and structural similarity measures to select the best merge operator and merge order without downstream evaluation, addressing reliability and scalability challenges for large checkpoint catalogs.

Figure 1: Overview of SimMerge. Given domain-specialized checkpoints and small unlabeled probe sets, SimMerge computes similarity signals, predicts merge operator and order, and produces a merged model—all in a single pass.

Methodology

Similarity-Based Feature Engineering

SimMerge computes a suite of pre-merge similarity features for each ordered checkpoint pair and task, extracted via lightweight probe passes and direct parameter comparisons. Functional features include layer-aggregated KL divergence between model predictions and cosine similarity of hidden activations/attention scores on probes. Structural features comprise parameter cosine similarity, $\ell_2$ distance, and modulewise norms. Statistical summaries (mean, median, quantiles) are concatenated with task encodings when available, yielding fixed-dimensional context vectors for operator selection.

Predictive Offline Operator Selection

For pairwise merges, SimMerge trains an MLP classifier to predict the optimal merge operator from similarity features, using logged downstream utility (performance on held-out benchmarks) as supervision per operator/model pair. At inference, only the model weights and a small probe set are required; no labels or evaluation are performed, enabling instantaneous selection and single-pass merging. For multi-way merges, an analogous plan scorer evaluates candidate merge permutations via concatenated pairwise feature blocks, extending the approach to $k$-way merges without additional supervision.

Efficient Multi-Way Plan Scoring

To avoid recomputation for intermediate merged states, SimMerge approximates similarity measures for hypothetical merges using convex combinations and norm-based inequalities (e.g., upper bounds for KL divergence, triangle inequality for $\ell_2$ distance), capitalizing on the closed-form propagation for mixtures.

Contextual Bandit Extension

Recognizing the dynamism of real-world model and task pools, SimMerge provides an online adaptation mechanism via neural-linear contextual bandits, supporting Thompson sampling (LinTS) and UCB-style policies atop the frozen MLP encoder. The bandit is warm-started with offline data and then updated with partial feedback as new checkpoints and tasks arrive, ensuring continual selection quality under distribution shift.

Empirical Results

Pairwise Merge Performance

SimMerge achieves strong improvements over fixed operators in pairwise expert-auxiliary merges across domains. When evaluated on the percentage of the expert-auxiliary gap closed (GapClosed), SimMerge closes 65.0% of the gap macro-averaged, outperforming the best fixed operator (Linear, 41.8%) and exhibiting robustness across code, math, multilingual, and RAG tasks. Notably, fixed operators demonstrate high sensitivity to domain: Linear excels in code, TIES in math, yet both can regress below the auxiliary on other tasks. SimMerge avoids such failures by instance-specific selection.

Figure 2: Fraction of expert–auxiliary gap closed by merge methods; SimMerge consistently closes more of the gap than fixed baselines.

Per-task changes show that SimMerge simultaneously limits degradation vs. experts and maximizes gains over auxiliaries, never underperforming the auxiliary baseline and maintaining near-expert performance across domains.

Figure 3: Per-task performance changes for each merge method: SimMerge yields the largest auxiliary improvements and minimal expert degradation.

Multi-Way Merging and Order Selection

Extending to 3- and 4-way merges, SimMerge maintains the best expert/auxiliary trade-off as merge complexity increases, demonstrating transferability of its 2-way training to higher-order planning. The number of configurations grows factorially with number of merit models, yet SimMerge's plan scoring circumvents exhaustive evaluation.

Figure 4: Relative performance (vs. expert, vs. auxiliary) for 2-, 3-, and 4-way merges: SimMerge consistently outperforms alternatives at all merge sizes.

Merge order, critical for non-associative operators, is selected via similarity-driven scoring, yielding substantial improvements over random ordering baselines, especially in code and RAG tasks.

Figure 5: Effect of merge order on gap closed for 3-way merges; SimMerge’s plan scoring substantially boosts performance relative to random ordering.

Scalability: Transfer to 111B-Parameter Models

A key claim is that SimMerge generalizes across model scales. The selector trained at 7B applies directly to 111B for 3-way merges, delivering the lowest expert-relative degradation ($-7.8\%$) and largest auxiliary gain ($+42.7\%$), highlighting the generalizability of similarity-based selection mechanisms.

Figure 6: SimMerge yields the smallest expert degradation and largest auxiliary improvements in 111B 3-way merges, despite being trained at 7B.

Online Adaptation

Bandit-based SimMerge quickly matches oracle performance in operator selection, minimizing cumulative regret even under distribution shift with new tasks and checkpoints. LinTS outperforms random and LinUCB policies and achieves near-oracle performance on downstream utility.

Figure 7: Bandit-based operator selection rapidly learns optimal policies, minimizing cumulative regret against the oracle.

Figure 8: Online SimMerge (LinTS) achieves strong expert retention and auxiliary improvement in 3-way merges across domains.

Technical Insights into Similarity-Regime Dependence

Operator efficacy is tightly linked to the structural and functional similarity regime between checkpoints. Correlation and win-probability trends reveal that Linear, SLERP, and TIES have orthogonal regions of success, and fixed rules result in heavy-tailed failures outside their alignment regime. SimMerge’s per-instance selection systematically avoids unfavorable tails, producing robust merged models even as composition complexity grows.

Figure 9: Spearman correlations between similarity metrics and operator selection—distinct metrics favor different operators, validating regime-dependent selection.

Figure 10: Tail effects across similarity regimes—SimMerge’s adaptive selection mitigates severe failures found in fixed operator approaches.

Implications and Future Directions

SimMerge reorients the model merging paradigm from inventing new operators to learning when to optimally deploy existing ones. This establishes a practical route for scalable checkpoint composition, reducing evaluation costs and enabling robust aggregation of large catalogs under tight computational budgets. The empirical transferability across tasks, domains, and model scales, coupled with fast online adaptation, suggests that similarity-conditional selection frameworks can underpin next-generation model deployment pipelines. Extending the operator set, integrating more sophisticated intermediate propagation, and advancing task-agnostic selection are promising avenues, especially for heterogeneously fine-tuned or non-homogeneous architectures.

Conclusion

SimMerge introduces a predictive, similarity-signal-driven framework for merge operator selection and merge ordering in LLM checkpoint composition. It eliminates expensive merge-and-evaluate loops, generalizes robustly to large-scale and multiway merges, and adapts online to streaming tasks and models. The results validate that similarity-dependent selection is essential for reliable model merging—outperforming fixed operators in both numerical performance and regime robustness. SimMerge thus offers scalable, on-demand model composition infrastructure, with immediate relevance for practical deployment and future AI system design.