GFFMERGE: Efficient Merging for GNNs & LLMs

• GFFMERGE is a framework that employs closed-form convex embedding alignment for graph neural force fields and curvature/Fisher-based averaging for large language model checkpoints.
• In atomistic machine learning, it merges independently fine-tuned neural force fields by solving least-squares problems over linear blocks, achieving performance close to joint fine-tuning with dramatic speedups.
• For large language models, its curvature-driven variant utilizes saliency-based pruning and weighted averaging of task vectors, demonstrating versatility across different architectures and optimization scenarios.

Searching arXiv for the cited papers and closely related context. GFFMERGE denotes two distinct model-merging formulations in recent machine learning literature. In atomistic machine learning, "GFFMERGE: Efficient Merging of Graph Neural Force Fields and Beyond" introduces a closed-form framework for merging Graph Neural Network (GNN) force fields and, more generally, message-passing GNNs by casting merging as a convex embedding-alignment problem over linear blocks in interaction layers (Verma et al., 2 Jun 2026). In a separate line of work on capability composition for LLMs, the term "GFFMERGE" is used as a general Fisher/curvature-driven recipe that coincides with the combination of Fast Fisher Grafting (FFG) and Optimization Trajectory Aware (OTA) Merging, relying on optimizer second moments as diagonal curvature proxies (Mahdavinia et al., 14 Sep 2025). The shared label can invite confusion; in current usage, the graph-learning formulation and the curvature/Fisher formulation are technically different methods that operate on different architectures, objectives, and assumptions.

1. Terminological scope and naming overlap

The primary arXiv usage of GFFMERGE is the graph-centric framework introduced in "GFFMERGE: Efficient Merging of Graph Neural Force Fields and Beyond" (Verma et al., 2 Jun 2026). That work presents GFFMERGE as the first principled framework for closed-form model merging in GNNs, with a specific emphasis on neural force fields (NFFs), where the target models predict energies and forces for atomistic systems. Its generic counterpart is termed GNNMERGE and extends the same embedding-alignment principle to non-atomistic GNN tasks such as node classification and link prediction (Verma et al., 2 Jun 2026).

A second usage appears in "Harnessing Optimization Dynamics for Curvature-Informed Model Merging," where the details explicitly state that a practical variant is termed "GFFMERGE," described there as a curvature/Fisher-driven merging technique that coincides with OTA+FFG (Mahdavinia et al., 14 Sep 2025). In that setting, the method merges supervised fine-tuning checkpoints of a base LLM by saliency-based grafting and curvature-weighted averaging of task vectors.

This suggests that "GFFMERGE" is not yet a uniquely stabilized term across subfields. A plausible implication is that any technical discussion should identify which paper is intended: the GNN force-field method of closed-form embedding alignment (Verma et al., 2 Jun 2026), or the Fisher/curvature-based LLM checkpoint-merging recipe associated with OTA+FFG (Mahdavinia et al., 14 Sep 2025).

2. GFFMERGE for graph neural force fields

In the atomistic setting, GFFMERGE addresses the problem of composing multiple independently fine-tuned GNN force fields into a single model specialized to the union of their domains without joint retraining (Verma et al., 2 Jun 2026). The motivating scenario is practical: foundation models such as M3GNet and Orb contain millions of parameters, perform expensive message passing over atomistic graphs, and are fine-tuned on targeted chemical regimes such as molecular families or materials domains. As coverage requirements broaden, repeated joint fine-tuning becomes prohibitively expensive.

The framework assumes atomistic graphs $G = (V, E, X, r)$, where $X$ denotes node features and $r$ coordinates. Neural force fields predict an energy $\hat{E}$ and forces $F$, with a coupled regression loss

$$C(\theta; G) = \lambda_E |E - \hat{E}|^2 + \lambda_F \|F - \hat{F}\|^2.$$

For $n$ independently trained expert models $O_1,\dots,O_n$, each specialized on domain $D_i$, the merged model $O_M$ is defined by the objective

$X$0

where $X$1 is an indicator and $X$2 measures energy/force discrepancy relative to expert $X$3 (Verma et al., 2 Jun 2026).

The target architectures are message-passing GNNs. A canonical layer is written as

$X$4

while geometry-aware force-field variants use message, aggregate, and update operators driven by relative coordinates $X$5 and geometric encodings such as radial basis features. Readout is permutation-invariant at the energy level, with total energy $X$6, and forces are either predicted directly, as in Orb, or derived by gradients $X$7, as in M3GNet (Verma et al., 2 Jun 2026).

The central technical claim is that large portions of these networks can be merged analytically because the learnable transforms inside interaction layers are linear, even though the overall architecture contains non-linearities, normalizations, and geometric operations (Verma et al., 2 Jun 2026).

3. Convex embedding alignment and the closed-form solution

GFFMERGE reformulates model merging as layerwise embedding alignment. For each source model $X$8, domain $X$9, interaction layer $r$0, and linear block $r$1, the method records a stacked input matrix $r$2 and stacked output matrix $r$3, where rows correspond to atoms and the linear block maps inputs to outputs under the source model’s own computational context (Verma et al., 2 Jun 2026).

The relaxed alignment problem minimizes

$r$4

independently for each linear block. This is a convex quadratic in $r$5, and the blockwise least-squares solution is

$r$6

with nonnegative weights $r$7 reflecting domain weighting or sample counts (Verma et al., 2 Jun 2026). The equivalent transposed form given in the paper is

$r$8

The objective has a unique global minimizer whenever the Gram matrix $r$9 is positive definite, i.e., full column rank. The paper presents this as the formal guarantee behind a stable closed-form merge for the linear message-passing components (Verma et al., 2 Jun 2026).

An important contrast is drawn with Procrustes- and permutation-based alignment strategies common in vision and NLP merging. Those methods constrain the alignment operator to orthogonal or permutation structure, whereas GFFMERGE solves an unconstrained least-squares problem in the actual weight space induced by message-passing computations. The paper argues that orthogonal/permutation constraints are inappropriate for force-field GNN blocks and empirically brittle in this regime (Verma et al., 2 Jun 2026).

Biases can be handled by augmenting $\hat{E}$0 with a column of ones and solving the same joint least-squares system. For multi-model merging with more than two experts, the same normal-equation form applies directly, yielding a barycentric solution without ad hoc averaging (Verma et al., 2 Jun 2026).

4. Algorithmic procedure, complexity, and practical workflow

The operational procedure in GFFMERGE consists of three main stages: extraction of per-block activations, closed-form blockwise merging, and targeted fine-tuning (Verma et al., 2 Jun 2026).

For each interaction layer $\hat{E}$1 and linear block $\hat{E}$2, the source models are run on unlabeled graphs from their respective domains. The inputs $\hat{E}$3 and outputs $\hat{E}$4 are stacked across atoms and sampled graphs to form the matrices $\hat{E}$5 and $\hat{E}$6. The normal equations are then solved to obtain $\hat{E}$7, and the corresponding parameters in the merged model are replaced by these closed-form solutions. After this analytic step, early merged layers are frozen, while only late interaction layers and the energy/force heads are unfrozen; the example procedure specifies unfreezing the last 3 blocks and readout, followed by 10–20 epochs of fine-tuning on the union of domains (Verma et al., 2 Jun 2026).

The same framework is generalized as GNNMERGE for GCN, GraphSAGE, GIN, GAT, and NodeFormer. The paper states that the $\hat{E}$8 derivations can be constructed per architecture: GraphSAGE uses $\hat{E}$9 blocks per layer, GIN uses the layerwise linear transforms in the MLP, GAT derives blocks for the projection $F$0 and attention vector $F$1, and NodeFormer derives blocks for $F$2, $F$3, $F$4, and $F$5 (Verma et al., 2 Jun 2026).

The computational profile is explicitly favorable. Per block, the least-squares solve requires forming the Gram matrix $F$6 and right-hand side $F$7 at cost $F$8, followed by solving $F$9, for instance by Cholesky, at cost $$C(\theta; G) = \lambda_E |E - \hat{E}|^2 + \lambda_F \|F - \hat{F}\|^2.$$0, where $$C(\theta; G) = \lambda_E |E - \hat{E}|^2 + \lambda_F \|F - \hat{F}\|^2.$$1 is the number of samples and $$C(\theta; G) = \lambda_E |E - \hat{E}|^2 + \lambda_F \|F - \hat{F}\|^2.$$2 the channel dimension. Because channel sizes in NFFs are reported to range from tens to hundreds and all blocks and layers are independent, the solves are modest and embarrassingly parallel (Verma et al., 2 Jun 2026).

The paper’s practical guidance specifies Adam for fine-tuning, force weight $$C(\theta; G) = \lambda_E |E - \hat{E}|^2 + \lambda_F \|F - \hat{F}\|^2.$$3 for M3GNet on MD17/MD22, higher domain-specific $$C(\theta; G) = \lambda_E |E - \hat{E}|^2 + \lambda_F \|F - \hat{F}\|^2.$$4 for LiPS20, train/validation/test splits of $$C(\theta; G) = \lambda_E |E - \hat{E}|^2 + \lambda_F \|F - \hat{F}\|^2.$$5, and seeds $$C(\theta; G) = \lambda_E |E - \hat{E}|^2 + \lambda_F \|F - \hat{F}\|^2.$$6 in the illustrative setup. It also recommends reporting MAE, RMSE, $$C(\theta; G) = \lambda_E |E - \hat{E}|^2 + \lambda_F \|F - \hat{F}\|^2.$$7, and rollout diagnostics such as EV/FV and structure metrics including RDF JSD and Wright’s factor (Verma et al., 2 Jun 2026).

5. Empirical performance on force fields and generic GNN tasks

The empirical evaluation in the graph-centric GFFMERGE paper is organized around force-field regression benchmarks and broader GNN tasks. A central claim is that existing merging methods designed for vision and language catastrophically fail on force-field regression, while GFFMERGE recovers performance approaching joint fine-tuning (Verma et al., 2 Jun 2026).

Representative force-field results

Setting	Baselines	GFFMERGE
M3GNet MD17 5-task mix Force MAE	$$C(\theta; G) = \lambda_E |E - \hat{E}|^2 + \lambda_F \|F - \hat{F}\|^2.$$8–$$C(\theta; G) = \lambda_E |E - \hat{E}|^2 + \lambda_F \|F - \hat{F}\|^2.$$9 kcal/mol/Å	$n$0
M3GNet MD17 5-task mix Energy MAE	$n$1–$n$2 kcal/mol	$n$3
M3GNet MD22 3-task mix Force MAE	$n$4–$n$5 kcal/mol/Å	$n$6
M3GNet MD22 3-task mix Energy MAE	not emphasized beyond baseline range context	$n$7
LiPS20 B-Li3PS4 + $n$8-Li3PS4 Force MAE	Fisher $n$9 eV/Å, EMR $O_1,\dots,O_n$0 eV/Å	$O_1,\dots,O_n$1 eV/Å
Orb MD17 5-task mix Force MAE	$O_1,\dots,O_n$2–$O_1,\dots,O_n$3 kcal/mol/Å	$O_1,\dots,O_n$4
Orb MD22 3-task mix Force MAE	$O_1,\dots,O_n$5–$O_1,\dots,O_n$6	$O_1,\dots,O_n$7
Orb LiPS20 merges Force MAE	$O_1,\dots,O_n$8–$O_1,\dots,O_n$9 eV/Å	$D_i$0–$D_i$1 eV/Å

These results are paired with joint fine-tuning references that remain stronger but close: for example, on the M3GNet MD17 5-task mix, joint fine-tuning gives Force MAE $D_i$2 and Energy MAE $D_i$3, while on the MD22 3-task mix it gives Force MAE $D_i$4 and Energy MAE $D_i$5 (Verma et al., 2 Jun 2026).

The speedup claims are similarly explicit. Representative examples include M3GNet MD22 on Ac-Ala3 + DHA + Stachyose, where the reported runtime changes from $D_i$6 s to $D_i$7 s for a $D_i$8 speedup; M3GNet MD17 on Aspirin + Uracil, from $D_i$9 s to $O_M$0 s for $O_M$1; and Orb MD22 3-task merging, from $O_M$2 s to $O_M$3 s for $O_M$4. Across force-field scenarios, the overall speedup range is reported as $O_M$5–$O_M$6 (Verma et al., 2 Jun 2026).

The broader GNNMERGE results extend the same closed-form alignment idea beyond atomistic simulation. In-domain node classification with disjoint label splits shows average gains over SURGERY by $O_M$7, ZIPIT! by $O_M$8, PERMUTE by $O_M$9, GIT-REBASIN by $X$00, and WAVG by $X$01. For merges across different datasets, the average accuracy improvement is reported as approximately $X$02 over the closest baseline, and for different-task merges such as node classification plus link prediction, baselines often collapse link-prediction AUC to near random while GNNMERGE substantially preserves both tasks (Verma et al., 2 Jun 2026).

A particularly strong efficiency example is the large-scale merge of MAG240M and Papers100M, reported to require about $X$03 minutes and about $X$04 GB RAM for GNNMERGE versus more than $X$05 days and about $X$06 GB RAM for joint training, corresponding to up to $X$07 speedups on large-scale graph learning (Verma et al., 2 Jun 2026).

6. Theoretical properties, assumptions, and limitations

The formal argument for GFFMERGE rests on convexity. For each block,

$X$08

is convex in $X$09 because its Hessian is $X$10, which is positive semidefinite. Summing these terms with nonnegative weights preserves convexity, and the normal equations yield the closed-form solution. Uniqueness follows from positive definiteness of the aggregate Gram matrix (Verma et al., 2 Jun 2026).

The paper also provides a transitive-consistency intuition: because input features and geometry are shared, if the merged model approximates each expert’s transform on that expert’s own inputs at layer $X$11, then downstream embeddings remain close, so neighborhood representations stay consistent across layers (Verma et al., 2 Jun 2026). This is an intuition rather than a full end-to-end proof, but it clarifies why independent layerwise alignment can work despite inter-layer dependence.

The method has clear requirements. The merged models must have architectural consistency, including the same topology of interaction layers, identifiable linear blocks, and the same channel sizes $X$12; Appendix C.2 is cited for padding strategies under heterogeneous dimensions. Unlabeled graphs from each domain must be available to extract $X$13, though no labels are required for the closed-form step. The normalization schemes and geometric encodings should also be consistent so that the extracted activations arise from comparable computational contexts (Verma et al., 2 Jun 2026).

Its limitations are equally explicit. Nonlinear components cannot be merged analytically, so targeted fine-tuning is necessary to close residual gaps. Highly incompatible domains can cause the stacked normal equations to overfit dominant regimes, in which case domain weighting or per-domain heads may be needed. Merging heterogeneous architectures or dimension mismatches requires careful padding, and performance depends on representational compatibility. The paper further notes that Orb exhibits high FV in rollouts across all methods, indicating stability issues unrelated to merging (Verma et al., 2 Jun 2026).

On symmetry, permutation invariance in readout is preserved because the method merges per-node embeddings and uses summation, while rotational and translational symmetries remain architectural properties inherited from the base model. The authors explicitly state that extending merging to explicitly equivariant GNNs is future work (Verma et al., 2 Jun 2026).

7. Curvature/Fisher GFFMERGE in checkpoint merging

In the LLM-merging literature, GFFMERGE denotes a different construction. The 2025 paper on optimization dynamics introduces OTA Merging and Fast Fisher Grafting, then states that a practical variant may be termed "GFFMERGE," defined as a general Fisher/curvature-driven recipe for selecting, grafting, and aggregating task vectors (Mahdavinia et al., 14 Sep 2025).

The setup begins with a pretrained base model $X$14 and specialist checkpoints $X$15, with task vectors $X$16. Each specialist also provides Adam second-moment statistics $X$17, treated as diagonal curvature proxies. Fast Fisher Grafting computes saliency

$X$18

retains only the top-$X$19 entries at density $X$20, and defines the pruned vector $X$21. OTA then solves a curvature-weighted aggregation

$X$22

where $X$23 is diagonal, typically $X$24 or $X$25. The final model is $X$26 (Mahdavinia et al., 14 Sep 2025).

That paper further reports an AdaFactor-style rank-1 compression of second moments, reducing fp32 storage for Llama 3.1 8B from $X$27 GB to $X$28 MB while retaining most OTA gains, with stable rank reported below $X$29 across transformer layers and specialists (Mahdavinia et al., 14 Sep 2025). Its evaluation is on capability-based supervised fine-tuning checkpoints for Meta-Llama-3.1-8B, where OTA+FFG attains an average score of $X$30, compared with $X$31 for TIES, $X$32 for Linear, $X$33 for Fisher, $X$34 for Breadcrumbs, and $X$35 for DARE (Mahdavinia et al., 14 Sep 2025).

The relationship between the two GFFMERGE usages is nominal rather than methodological. The graph-force-field GFFMERGE is a closed-form embedding-alignment method over message-passing layers (Verma et al., 2 Jun 2026), whereas the checkpoint-merging GFFMERGE is a curvature-driven edit-selection and weighted-averaging recipe over parameter deltas (Mahdavinia et al., 14 Sep 2025). A plausible implication is that the shared acronym reflects a broader concern with post-training composition, but the underlying mathematics, data requirements, and application domains are distinct.