Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancing

Abstract: Model merging has become a practical post-training strategy for building a single multi-task LLM by combining multiple task-specialized models. However, most existing approaches rely on post-hoc merging, in which task-specific models are merged only once after training. This one-shot aggregation often suffers from task interference, leading to information erasure across individual tasks. In this work, we show that replacing post-hoc merging with an iterative many-shot merging protocol is effective in improving multi-task performance. Building on this insight, we propose METIS, Mitigating Erasure from Task Interference for Stable many-shot merging. METIS is a loss-aware many-shot merging method that addresses information erasure in post-hoc merging through task-wise loss-gap weighting and consensus-based masking. Notably, METIS exhibits significant performance improvement on the worst-performing task, effectively mitigating information erasure. (Project page: https://imkyungjin.github.io/METIS/)

• The paper demonstrates that iterative many-shot merging consistently lowers multi-task loss across Task Arithmetic, DARE, TIES, and ConsensusTA, with Task Arithmetic improving normalized performance from 0.706 to 0.857.
• METIS combines loss-gap-aware task weighting with consensus masking to reduce information erasure, achieving the best normalized performance across four backbones and improving worst-task performance to 0.872 on Llama-3.2-3B.
• The results show that METIS scales to eleven tasks, preserves pre-trained knowledge nearly as well as the original models, and adds only moderate runtime overhead, although its worst-case theoretical guarantee depends on unverified assumptions.

Motivation and problem statement

Model merging has become a practical post-training strategy for constructing a single multi-task LLM from independently fine-tuned task-specialized models. The dominant paradigm, however, is post-hoc merging: task models are trained to convergence and merged exactly once. This one-shot aggregation induces abrupt cross-task interference, which the authors attribute to model drift—task-specific updates push parameters toward disparate task-optimal regions, so naive aggregation overwrites task-specific knowledge ("information erasure"). The paper's central claim is that the merging process itself, not merely the merging operator, is a first-order factor: replacing post-hoc aggregation with an iterative many-shot protocol—repeated local updates interleaved with incremental merges—consistently improves multi-task performance across representative merging operators.

Many-shot merging framework and theoretical support

The framework considers $T$ tasks sharing a pre-trained initialization $\Theta^0$. At each round $r$, each task model performs a local gradient update from the previous merged model, computes its task vector $\boldsymbol{v}^r_\tau = \theta^r_\tau - \Theta^0$, and a merging operator $\mathcal{M}$ aggregates them into $\Theta^r$. After $R$ rounds all models share $\Theta^R$. Post-hoc merging is recovered as the special case $R=1$ with matched total local updates.

The authors provide a theoretical guarantee (Theorem 1): under $L$-smoothness of each task loss and learning rate $\Theta^0$0, many-shot merging achieves lower multi-task loss than post-hoc merging whenever

$\Theta^0$1

where $\Theta^0$2 captures the difference in average task losses and $\Theta^0$3 the difference in within-round dispersion of task models around their merge point. The proof follows from applying the smoothness inequality to both merge points and averaging; the linear terms cancel because the merged model is the mean of the task models. Notably, the condition is not vacuous but also not restrictive—the appendix reports that it holds empirically for Task Arithmetic, DARE, TIES, and ConsensusTA on Llama-3.2-3B, with the loss term dominating ($\Theta^0$4 between $\Theta^0$5 and $\Theta^0$6) while the dispersion term is negligible.

Empirically, simply switching from post-hoc to many-shot merging reduces multi-task loss for every baseline; e.g., ConsensusTA drops from 1.83 to 1.49 and Task Arithmetic from 2.97 to 2.00, with normalized performance rising correspondingly (Task Arithmetic 0.706 → 0.857). This establishes the paper's first contribution: iterative integration alone accounts for a substantial share of achievable gains, independent of any new merging operator.

METIS: loss-gap weighting and consensus masking

Many-shot merging constrains drift but does not control how heterogeneous updates are weighted during aggregation. METIS addresses this with two mechanisms.

Loss-gap-aware weighting. The task-wise loss gap $\Theta^0$7 measures how much worse the current merged model fits task $\Theta^0$8 than its locally adapted counterpart—a direct proxy for information erasure incurred in the last round. Merging weights are computed via softmax over scaled loss gaps,

$\Theta^0$9

so underrepresented tasks contribute more strongly in the next round. This signal is only available in the many-shot setting, since it requires access to the previously merged model. Theorem 2 shows that, assuming $r$0-smoothness of the worst-task loss plus bounded-interference and magnitude-control conditions, loss-gap-aware aggregation yields no higher expected worst-task loss than uniform averaging. The proof applies the descent lemma to the two aggregated directions; the key step relies on reweighting aligning the update direction better with the worst task's gradient. These auxiliary assumptions are stated but only informally justified, so the guarantee should be read as conditional rather than unconditional.

Consensus-based masking. Building on ConsensusTA, METIS retains coordinate $r$1 in a task-specific mask when that task's weighted contribution is not dominated by conflicting contributions ($r$2), then activates a coordinate in the consensus mask only if at least $r$3 tasks agree. The final update is $r$4.

Experimental results

Experiments follow the MergeBench protocol on four backbones (Gemma-2-2B, Llama-3.2-3B, Llama-3.1-8B, Qwen-3-4B), four task categories (instruction-following via IFEval, math via GSM8K, multilingual understanding via M-MMLU/M-ARC/M-HellaSwag, safety via XSTest), with 1,000 training instances per task, LoRA rank 16, and $r$5 rounds under matched total update budgets.

Backbone	Best post-hoc baseline	Best many-shot baseline	METIS
Gemma-2-2B	0.752 (ConsensusTA)	0.791 (ConsensusTA)	0.800
Llama-3.2-3B	0.942 (ConsensusTA)	0.945 (ConsensusTA)	1.015
Llama-3.1-8B	0.852 (TIES)	0.902 (TIES)	0.935
Qwen-3-4B	1.108 (TIES)	1.154 (DARE)	1.180

METIS attains the highest average normalized performance on every backbone, and an extended comparison shows it also exceeds recent subspace-based methods (Iso-C, Iso-CTS, TSV-M, Subspace Boosting), federated/iterative baselines (FedMerge, q-FedAvg, ColD Fusion), and data mixing (TULU 3) on Llama-3.2-3B.

The most distinctive result concerns worst-case robustness. On Llama-3.2-3B, METIS's worst-task score is 0.872 versus 0.825 for the next best method, with an average-to-worst gap of only −0.14 compared to gaps of −0.27 to −0.51 for baselines. On Gemma-2-2B the gap is −0.28 versus −0.29 to −0.45. Since information erasure manifests precisely as collapse on individual tasks, this directly supports the claim that loss-gap rebalancing mitigates erasure rather than merely raising averages.

On pre-trained knowledge retention, evaluated out-of-domain on CoQA and PubMedQA, METIS stays closest to the un-fine-tuned backbone—for example, 0.698 vs. 0.697 CoQA on Llama-3.1-8B, and 0.684 vs. 0.683 on Qwen-3-4B—whereas several baselines lose 10–20 points. Scaling to 7, 8, and 11 tasks, METIS remains highest at every task count (e.g., 1.036 at 11 tasks vs. 0.812–1.021 for baselines). Ablations confirm all three components contribute, with removing many-shot merging causing the largest degradation. Sensitivity analyses show graceful behavior across masking thresholds $r$6, scaling factors $r$7, and model sizes from 0.6B to 14B. Runtime overhead is moderate: 918 s versus 875–905 s for standard many-shot baselines and far below TSV-M (3219 s).

Limitations and open questions

The paper concedes that many-shot merging incurs additional training-phase computational overhead from repeated merging steps, though it argues this is modest at $r$8 and could be reduced within LoRA parameter spaces. More substantively, Theorem 2 depends on bounded-interference and magnitude-control assumptions that are asserted rather than verified empirically, unlike the condition of Theorem 1; the worst-case guarantee therefore rests on unvalidated premises. The evaluation uses small per-task datasets (1,000 instances) and a fixed round count, leaving open whether loss-gap weighting remains beneficial at larger data scales or different round schedules, and whether the approach extends beyond LoRA-based adaptation to full fine-tuning.

Conclusion

This paper reframes model merging as an iterative optimization process rather than a one-shot aggregation step. It demonstrates theoretically and empirically that many-shot merging alone lowers multi-task loss across standard merging operators, and introduces METIS, which further exploits the iterative structure through loss-gap-based task reweighting and consensus masking. The consistent gains in average performance, worst-task robustness, and pre-trained knowledge retention across four backbones and up to eleven tasks indicate that controlling the merging schedule—and explicitly compensating for per-round information erasure—is a productive axis for building reliable multi-task LLMs.