Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging

Abstract: Instruction tuning aligns LLMs, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization. We ask whether these two bottlenecks can be addressed jointly by training parts of the mixture independently and reconciling them once in parameter space. We develop a local quadratic theory inside a shared flat basin that yields three results: weight merging produces a curvature-weighted variance reduction; PCA-aligned conflict splitting maximizes this gain along high-curvature directions; and merging additionally acts as spectral filtering with implicit norm regularization. These results directly motivate MERIT, a decentralized merge-ready instruction-tuning pipeline that estimates dataset-level gradient conflicts, partitions the mixture along the top PCA conflict axes, fine-tunes each partition independently with no inter-partition communication, and merges once via token-weighted averaging. On Qwen2.5-VL-3B with 136 Vision-FLAN tasks, MERIT improves the 8-benchmark average from 54.3 (joint training) to 57.0. The same recipe scales to a 7B model on a 1.6M-example, 176-source mixture -- matching or exceeding centralized joint training with minimal cost overhead -- and transfers to text-only FLAN. Our code is available at https://github.com/naver-ai/merit.

• The paper introduces the MERIT pipeline, a decentralized instruction tuning method that splits tasks based on conflict-aware gradient analyses.
• It demonstrates that token-weighted merging of independently fine-tuned branches reduces loss and enhances model generalization in heterogeneous task mixtures.
• Empirical results on both vision-language and text tasks confirm MERIT’s efficiency, matching or exceeding traditional joint training under limited bandwidth.

Decentralized Instruction Tuning via Conflict-Aware Splitting and Weight Merging: The MERIT Pipeline

This paper presents MERIT, a decentralized instruction tuning (IT) pipeline addressing the inherent challenges of gradient interference and communication constraints when post-training large language and vision-LLMs on heterogeneous instruction mixtures. The approach is theoretically motivated and empirically validated, providing new insights into optimally splitting and merging branches for instruction tuning at scale.

Motivation and Problem Analysis

Instruction tuning for LLMs and MLLMs is critical for aligning models to a variety of user intents and deploying practical product-level capabilities. However, as task mixtures grow more heterogeneous, centralized joint training encounters two core bottlenecks: (1) negative gradient interference that slows or disrupts optimization and (2) high-bandwidth requirement for synchronization that is infeasible in fragmented hardware environments. Existing multi-task correction algorithms are computationally infeasible with hundreds of tasks and billions of parameters, and recent mixture design has resorted to extensive manual curation and mixture-ratio tuning.

This work hypothesizes that these two bottlenecks can be addressed jointly if the instruction mixture can be partitioned into branches with maximally disagreeing updates that are then independently fine-tuned and merged once in parameter space. The central question becomes how to design splits that are most beneficial for post-hoc merging, ensuring that the resulting “model soup” does not simply aggregate redundant models but produces a merged model with improved generalization and curvature properties (Figure 1).

Figure 1: Schematic comparison of centralized joint training versus MERIT’s decentralized approach: conflict-aware splitting, independent training, and one-shot merging in parameter space.

Theoretical Framework

The core theoretical contribution is a quadratic analysis of the loss surface in the regime where all fine-tuned branches remain in a connected, flat basin around a merge-ready initialization. This is typical post-pretraining, with independent branches initialized from the same model. The analysis yields three key results:

1. Merging Gain: Merging $K$ checkpoints by weighted averaging provides a deterministic reduction in loss, equal to a curvature-weighted variance across checkpoints. This gain is strictly positive unless all models move identically.
2. Conflict-Aware Splitting: If inter-branch displacements align with dominant curvature axes (high Hessian eigenvalues), the reduction is maximized. Empirically, principal component analysis (PCA) on per-dataset gradients at the initialization identifies these axes.
3. Implicit Regularization: The merge operation also acts as spectral filtering and norm regularization—contracting towards initialization—which improves generalization even if the merged model's training loss exceeds that of joint training (empirically observed).

   Figure 2: Illustration of flat loss basins post-pretraining. Fine-tuning branches diverge along conflicting directions, but all remain connected by low-loss paths suitable for averaging.

   

   Figure 3: The merged model (blue) is consistently less sensitive to weight perturbations than a jointly trained model (red), indicating a flatter loss landscape—a direct empirical signature of implicit regularization via averaging.

MERIT Pipeline

MERIT consists of the following stages:

• Conflict Estimation: Compute dataset-level gradients at the merge-ready initialization on a small calibration subset. Construct the cosine-similarity matrix.
• PCA Decomposition: Run PCA on the similarity matrix to extract the main conflict axes.
• Balanced Partitioning: Recursively split the dataset list along top PCA axes, balancing for sample size.
• Independent Training: Fine-tune one branch per partition, each from the same initialization, with no cross-branch communication.
• Token-Weighted Parameter Merging: Return the token-weighted average of branch checkpoints.

This split-then-merge protocol is communication-free post-split and amenable to decentralized or fragmented clusters.

Figure 4: PCA-based splitting: Dataset groups extend along distinct conflict axes. The merged model sits near the initialization—a geometric interpretation of implicit regularization.

Empirical Evaluation

Empirical studies span both a controlled 3B Qwen2.5-VL experiment and more challenging 7B-scale vision–language settings (with 1.6M examples from 176 datasets), along with text-only FLAN tuning. Key results include:

• On Qwen2.5-VL-3B with 136 Vision-FLAN tasks, MERIT-3D (eight groups) achieves $57.0$ on the benchmark average under identical budgets, versus $54.3$ for one-epoch joint training—exceeding alternatives based on random splitting (random 8-group avg $54.5$), conflict-induced, and classic uniform model soups.
• On a 7B large-scale, multi-seed setting, MERIT-2D (four groups) matches or exceeds joint fine-tuning on all runs and across all benchmark classes (Table results in paper).
• In text-only tuning (Qwen2.5-3B, 66 FLAN tasks), MERIT matches or slightly exceeds 2-epoch joint training, achieving its results at half the computational budget.

  Figure 5: Post-hoc merging baselines applied to MERIT’s 7B branches. Token-weighted averaging outperforms advanced model merging heuristics, confirming the necessity of conflict-aware splits.

Key Claims and Contrasts

• Merging is Always Beneficial in-Basin: Weight-space merging in a flat basin is provably never worse—and typically better—than random selection or joint training, by quadratic variance reduction.
• Conflict-Aware Splitting Outperforms Alternatives: Data splits aligned with dominant PCA axes—i.e., maximizing inter-group gradient conflict along high-curvature directions—outperform random or clustering-based splits with the same resources.
• Superior Profile via Decentralization: MERIT not only lifts benchmark averages but preserves open-ended generation quality that joint training degrades (e.g., user-preference and long-form benchmarks).
• One-Shot Decentralized Training: Unlike federated or multi-round decentralized learning, MERIT assumes the split can be selected to maximize post-hoc merging efficacy, leveraging centrally available data.

Practical and Theoretical Implications

MERIT offers a new perspective on instruction tuning at scale: task conflicts can and should be handled before training, by partitioning along directions where they are most pronounced. This enables scalable decentralized training, relaxing both the optimization and systems constraints typical of current pipelines. The weighted merging operation includes implicit generalization benefit, complements ratio curation and mixture design, and is generic enough to transfer between multimodal and purely textual instruction mixtures.

The theoretical connection to flat-basin quadratic geometry, PCA, and spectral regularization bridges optimization, federated learning, and models soups into one cohesive framework—substantially extending prior work that either did not address task splits or operated in the same-task setting.

Future Directions

Directions for future work include:

• Extension to more aggressive or asymmetric partitioning schemes, including adaptive $K$ per model size or mixture entropy.
• Application to RL-aligned objectives requiring non-differentiable interaction feedback.
• Integration with continual learning, where periodic branch merging could address distribution shift.
• Formalization of the merge-ready property and diagnostics for routine practical checking.

Conclusion

MERIT embodies a theoretically justified, scalable, and empirically validated approach to decentralized instruction tuning. By estimating, splitting, independently training, and merging along conflict axes, it achieves or exceeds joint training performance under realistic compute and bandwidth constraints, with minimal wall-clock overhead. The approach provides actionable new tools for large-scale model post-training and opens avenues for principled, communication-efficient learning in heterogeneous settings.