Merge Module: Integrating Multiple Inputs

• Merge modules are mechanisms that integrate multiple sub-objects—such as models, data streams, or graphs—by fusing parameters, performing unions or intersections, and resolving conflicts.
• They employ diverse strategies including training-free arithmetic operations and gradient-based learning to dynamically adapt and optimize multi-task architectures.
• Empirical results show these modules improve accuracy, latency, and scalability in tasks spanning neural network merging, graph editing, hardware design, and program synthesis.

A merge module is a general paradigm and associated set of mechanisms for systematically combining multiple sub-objects—models, modules, data streams, or graph-theoretic structures—into a coherent single artifact that preserves or synthesizes desired properties from its inputs. The exact semantics of the merge operation are context-specific but usually entail parameter fusion (in models), union or intersection (in graph/module theory), or conflict resolution (in structured data). Merge modules are central to modern model-merging in machine learning, hardware data-path design, graph editing, multi-modal fusion, formal structure theory, and program synthesis. Recent advances have focused on learnable, component-wise, or resource-aware merge modules that can flexibly mediate among diverse sources, optimize trade-offs, and generalize compositional structure.

1. Merge Modules in Model Merging: Formulations and Algorithms

In the context of neural networks and large pre-trained models, merge modules are mechanisms for combining multiple fine-tuned or expert models into a single multi-task model without retraining from scratch. This is motivated by the need to eliminate the high cost of joint fine-tuning, reduce deployment resources, or dynamically adapt to new tasks. Merge modules can be “training-free” (static function, e.g. arithmetic averaging, Fisher-weighted sum, or sign correction) or “gradient-based” (learn an explicit set of fusion weights via a small validation set).

SuperMerge implements a merge module that learns per-task, per-layer fusion weights $w_{i,j}$, merging $k$ task-specific checkpoints $\{f_i\}$ via

$$m(j) = p(j) + \sum_{i=1}^k \tanh(w_{i,j}) \, [f(i,j) - p(j)]$$

where $p(j)$ are the pre-trained layer weights and the $w_{i,j}$ are optimized via gradient descent on a small held-out validation set (Yang et al., 2024). The merge parameters are orders of magnitude fewer than model parameters, yielding a lightweight “merge module” that can be trained in minutes.

Variants include Chain of Merges (CoM) (Buzzega et al., 29 Aug 2025), where model merging is performed layer-wise and sequentially, propagating the new input statistics produced by the merged upstream layers. This addresses the “merging covariate shift” effect and ensures conditionally optimal merging at each layer.

FlexMerge (Dhasade et al., 29 May 2025) organizes the merge module at the block or submodule level, allowing flexible, resource-aware fusion. Blocks (e.g. transformer layers) are grouped and merged by similarity, and the process can be halted at any size to optimize the accuracy-size trade-off.

MERGE (Qiu et al., 6 Feb 2026) and other modular expert recombination methods extend this to fine-grained, component-wise merge modules, using evolutionary search to optimize the selection and grouping of components to minimize loss and storage cost jointly.

2. Merge Modules in Graph Theory and Discrete Structures

In graph theory, a merge module can refer to a formal operation used in modular decomposition and graph editing. For instance, in cograph editing (Hellmuth et al., 2015), the module-merge operation coalesces disjoint modules $\{M_1,\dots,M_k\}$ into a single module $M$ in a new graph $H$ if they are individually modules in both $G$ and $k$0, their union is a module in $k$1, but not in $k$2. This step is equivalent to editing away all induced $k$3 paths that cross the merged submodules. Iterative pairwise module merges transform a general graph into a cograph in a well-defined sequence, providing an algorithmic substrate for graph modification tasks.

Merge-models (Buffière et al., 27 Mar 2026) generalize twin-models for twin-width and allow representation of binary structures via tree-ordered, weakly sparse structures, where the merging process and the associated structural invariants (biclique-number, merge-width) provide sharp structural characterizations for broad classes of graphs.

3. Merge Modules in Multimodal and Personalized Systems

Merge modules play a critical role in multimodal fusion and adaptive personalization. LegoMerge (Hemker et al., 2024) provides a merge module for harmonizing latent representations from arbitrary unimodal encoders by aligning them in the frequency domain and using a harmonic mean of their magnitudes and average phase in a DFT representation. This reduces signal interference and enables merging/fusion with little or no end-to-end retraining.

In personalized LLMs, MTA (“Merge-then-Adapt”) (Li et al., 25 Nov 2025) uses a merge module at the adapter level: meta-LoRA modules learned from anchor users are linearly merged (weighted by user similarity), producing a user-specific LoRA, further refined by ultra-low-rank adaptation for few-shot generalization.

In multi-actor event reasoning (e.g., the MERGE system (Deigmoeller et al., 19 Mar 2026)), the merge module comprises a high-level event abstraction pipeline that collects, disambiguates, and temporally reconciles instance-level grounded actions from disparate low-level perception streams, fusing them into persistent actor/action/object/event relations and higher-level episodic summaries.

4. Design Patterns, Trade-Offs, and Formal Properties

Approaches and Parameterizations

Merge Module Category	Key Example	Typical Parameter or Representation
Gradient-based	SuperMerge (Yang et al., 2024)	Layer-task weights $k$4
Data-free arithmetic	Task Arithmetic, TIES	Scalar or vector merge weights, sign/flip mask
Block/component-wise	FlexMerge (Dhasade et al., 29 May 2025), MERGE (Qiu et al., 6 Feb 2026)	Module grouping matrix, Pareto front
Graph/module-theoretic	Module-merge (Hellmuth et al., 2015)	Modular decomposition trees, twin classes
DFT-based multimodal	LegoMerge (Hemker et al., 2024)	Latent frequency-domain tensors, SLERP heads

Trade-Offs

• Accuracy vs. Size: Fusing more tasks/modules into a single merge module can degrade individual task performance; block-wise and component-wise approaches (e.g., FlexMerge, MERGE) allow tuning of this trade-off.
• Storage vs. Flexibility: Highly modular (or personalized) merge modules support on-demand or input-aware assembly but require a nontrivial expert library and routing overhead.
• Training-Free vs. Learned: Data-free merge modules offer universal applicability but less customization; gradient-based approaches (SuperMerge, CoM) require small validation sets but can substantially close performance gaps.
• Scalability: Hierarchical or greedy merge modules (e.g., hierarchical SuperMerge) support dozens of tasks without prohibitive resource use.

5. Merge Modules in Hardware and Program Synthesis

In hardware, merge modules are instantiated as merge-sort engines—such as LOMS (List Offset Merge Sorters) (Kent et al., 11 Jul 2025)—that physically merge sorted input lists using offset 2D layouts with alternating column and row sort stages. This yields efficient, scalable sort units with superior latency-resource tradeoffs compared to classic Batcher or bitonic networks.

In program synthesis, learning-based merge modules can resolve developer-provided edit conflicts. For example, DeepMerge (Dinella et al., 2021) employs an edit-aware encoding of multiple program variants, decodes merges via pointer networks over aligned diffs, and learns to apply the minimal transformations required to resolve conflicts in collaborative code bases.

6. Comparative Evaluation and Empirical Results

Merge modules yield significant empirical gains across domains:

• SuperMerge (Yang et al., 2024): Ranks best on 8/11 NLP tasks, average accuracy ≈69.6% (next best ≈65.8%), 4–10 points higher on out-of-domain QA datasets.
• FlexMerge (Dhasade et al., 29 May 2025): Allows rapidly approaching fine-tuned oracle accuracy with models only 2–3× larger than a single merge, with performance sharply degrading only near “all-in-one” size.
• CoM (Buzzega et al., 29 Aug 2025): On CLIP ViT, boosts mean accuracy from ≈70.7% (Iso-C) to ≈91.7%.
• LegoMerge (Hemker et al., 2024): Achieves fusion performance matching or exceeding end-to-end training with minimal or no retraining in biomedical multimodal datasets.
• MERGE expert recombination (Qiu et al., 6 Feb 2026): Yields Pareto-optimal performance/storage tradeoffs, regularly dominating static baselines.
• Hardware LOMS (Kent et al., 11 Jul 2025): Achieves up to $k$5 latency speedup over Batcher merge networks for FPGA-synthesized sorters.
• DeepMerge (Dinella et al., 2021): Achieves 36.5% exact match accuracy on non-trivial code merges, vastly exceeding semistructured baselines, and 78% on small (≤3-line) merges.

7. Limitations, Open Problems, and Future Directions

• Homogeneity: Most merge modules (except certain graph-theoretic or symbolic cases) require all participants to share the same architecture or initialization. Merging across heterogeneous models remains an open and challenging problem (Yang et al., 2024, Dhasade et al., 29 May 2025).
• Non-head, non-shared output layers: For instance, merging classification heads with different class spaces is not yet adequately supported (Yang et al., 2024).
• Validation/data requirements: Learned merge modules (SuperMerge, CoM) require small but representative validation sets; unsupervised approaches are less accurate.
• Interpretability and controllability: Optimal merge schedules, especially in large block-wise merge modules (FlexMerge, MERGE), remain only heuristically understood.
• Extensibility to larger and more diverse task sets: Memory and computational limits can arise in fine-grained merge modules; hierarchical and resource-aware designs partially address this but may require new theoretical insights.

Merge modules are thus a rapidly evolving core abstraction underlying much of modern multi-task, multi-source, and collaborative computation, exhibiting deep connections to fundamental theory, empirical optimization, systems design, and learning theory.