Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance (2511.13254v1)

Abstract: LLMs have demonstrated remarkable capabilities across diverse domains, but their training remains resource- and time-intensive, requiring massive compute power and careful orchestration of training procedures. Model souping-the practice of averaging weights from multiple models of the same architecture-has emerged as a promising pre- and post-training technique that can enhance performance without expensive retraining. In this paper, we introduce Soup Of Category Experts (SoCE), a principled approach for model souping that utilizes benchmark composition to identify optimal model candidates and applies non-uniform weighted averaging to maximize performance. Contrary to previous uniform-averaging approaches, our method leverages the observation that benchmark categories often exhibit low inter-correlations in model performance. SoCE identifies "expert" models for each weakly-correlated category cluster and combines them using optimized weighted averaging rather than uniform weights. We demonstrate that the proposed method improves performance and robustness across multiple domains, including multilingual capabilities, tool calling, and math and achieves state-of-the-art results on the Berkeley Function Calling Leaderboard.

• The paper formalizes a category-aware, non-uniform model weight averaging technique that achieves state-of-the-art performance on multiple LLM benchmarks.
• It employs benchmark category correlation analysis and Shapley value allocation to select and optimally weight expert models.
• Empirical results demonstrate improved accuracy, consistency, and robustness with minimal computational overhead compared to traditional uniform souping.

Model Souping for LLMs: A Formal Analysis of SoCE

Introduction

This essay analyzes "Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance" (2511.13254), which formalizes enhanced model souping for LLMs. The work introduces Soup Of Category Experts (SoCE), a procedure for principled selection and non-uniform averaging of model weights, underpinned by benchmark category correlation structure and cooperative game theoretic utility. Unlike prior uniform-soup baselines, SoCE demonstrates consistent advancement of accuracy across tool calling, math, multilingual, and long-context reasoning—all with negligible computational overhead relative to conventional post-training. The method establishes novel best-in-class results on several competitive leaderboards.

Motivation and Related Work

Model souping, i.e., weight averaging of multiple checkpoint variants, has shown efficacy for continual learning, catastrophic forgetting, and generalization in classification and vision tasks [wortsman2022model, jang2025modelstock, kleiman2025soup]. Yet for LLMs, prior averaging is predominantly uniform or greedy, lacking principled selection of contributors or dynamic allocation of influence. SoCE departs from this by exploiting the low inter-correlation frequently observed among heterogeneous benchmark category scores—an empirical property visualized for BFCL in the heatmap below.

Figure 1: Pearson correlation structure between category scores in BFCL, highlighting high correlation clusters and anti-correlated regions.

This reveals broad category-specific specialization among model candidates. The paper further integrates game theory (Shapley value allocation) to formalize selection and weighting, providing cohesive theoretical motivation for non-uniform aggregation and expert-based model composition.

SoCE Methodology

SoCE proceeds as follows:

1. Category Correlation Analysis: Determine the Pearson correlation of model scores between all category pairs on a given benchmark, using a threshold τ to identify weakly-correlated clusters.
2. Expert Model Selection: For each uncorrelated cluster, select the model with maximal performance as category expert.
3. Weight Optimization: Search the weight simplex by grid (range [0.1, 0.9]) for each selected expert, seeking the allocation that maximizes aggregate benchmark score.
4. Weighted Model Averaging: Compose the final model by non-uniform averaging of expert weights.

The process is automatable and generalizes to any benchmark with well-defined categorical subdivision. The paper validates the approach extensively—on BFCL (tool calling), MGSM (multilingual math), $\infty$-Bench (long context), and FLORES-36/101 (multilingual translation).

Empirical Results

Benchmark Accuracy & Winrate

SoCE achieves marked improvements:

• BFCL (Tool Calling, 70B Model): SoCE obtains 80.68% overall accuracy, outperforming the prior best xLAM-2-70b (78.56%) and uniform soup (68.33%). Weight tuning delivers an additional 2.28% over selection-only uniform soup. For 8B models, similar absolute and relative gains are shown.
• MGSM (Multilingual Math, 7B Model): SoCE delivers 51.7% accuracy, outperforming best baseline model (50.9%), with uniform soup regressing performance.
• $\infty$-Bench: Model selection and weighting yield nontrivial lifts (+2.05% absolute improvement over best candidate), even for models with similar capabilities due to homogeneous data mixtures.

Across benchmarks, SoCE consistently matches or surpasses every ingredient model in mean, and often in every individual category.

Consistency and Robustness

Post-souping, metric profiles exhibit substantive increases in linear correlation across all categories, as reflected in cross-model Pearson analysis. This signals not only superior mean accuracy, but also greater robustness, stability, and general coherence of capabilities.

Figure 2: Category-level performance for SoCE and ingredient models, evidencing both higher mean and improved consistency.

Shapley Analysis

Cooperative game theoretic analysis of individual, pairwise, and triplet model contributions demonstrates that SoCE-selected candidates often possess maximal Shapley values, i.e., dominant marginal utility. Overweighting weak parents reliably reduces overall score, confirming the criticality of non-uniform weighting for ensemble efficacy.

Ablations and Practical Considerations

The paper conducts extensive ablations, emphasizing the necessity of:

• Category-aware candidate selection (anti-correlation identification)
• Grid-based weight search for optimal allocation
• Direct benchmarking to avoid regression on unrelated tasks

Notably, SoCE avoids overfitting; soup outputs generalize well across tasks not optimized for, with performance at or above ingredient models.

Analysis on FLORES-36

For tasks/benchmarks where candidate models are less diverse (e.g., FLORES-36 translation), SoCE provides only marginal gains, underscoring that maximal benefit occurs when category-wise expertise is pronounced and inter-model heterogeneity is significant.

Figure 3: Analysis of souped and unsouped model performance across FLORES-36 triplets, visualizing BLEU improvement per category and overall average.

Long Context Reasoning

Application to $\infty$-Bench, utilizing multiple fine-tuned Llama3 checkpoints, illustrates the method's utility for sequence continuation, retrieval, and code tasks. SoCE delivers modest but measurable improvements in mean accuracy across categories, confirming that even slight data or recipe variance among models can be synergistically exploited (Figure 4).

Figure 4: Performance profiles of SoCE and constituent models over diverse $\infty$-Bench categories based on long-context Llama3 variants.

Theoretical Implications

The findings reinforce the following:

• Model souping, if formalized around category specialization and anti-correlation, is an efficient route to ensemble-level generalization in LLMs.
• Weight averaging need not be uniform; cooperative expert allocation is superior, especially under multi-benchmark training.
• Post-training aggregation can substitute or complement costly joint training, benefiting open-source and resource-constrained communities.

Future Directions

Potential avenues include:

• Extension to benchmarks without explicit categorical structure (via benchmark clustering)
• Generalization to mixture of experts, adapters, and multimodal architectures
• Systematic paper of scaling laws: number/diversity of models vs. accuracy delta
• Interpretability and mechanistic analysis for soup-derived checkpoints

Conclusion

"Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance" provides a formal foundation for category-aware, weighted model souping and exhibits strong empirical support for the approach's utility. Across a battery of tasks, SoCE reliably enhances performance, robustness, and efficiency, suggesting its broad applicability for advancing open LLM capabilities. The method offers a paradigm for collaborative aggregation and reuse in open-source model development, fundamentally shifting best practices for post-training model composition.