---
title: 'CanonicalMerge: Invariance in Model Merging'
url: https://www.emergentmind.com/topics/canonicalmerge
type: topic
---

# CanonicalMerge: Invariance in Model Merging

CanonicalMerge is not a single universally standardized algorithm. In recent arXiv literature, the term and closely related constructions denote several distinct ways of making merging *canonical*: a monolithic whole-model operator applied uniformly to all parameters; a merge performed in a canonical shared representation space derived from Canonical Correlation Analysis (CCA); a deterministic render of a CRDT-managed contribution set; and a content-determined cache layout for multi-agent latent reasoning [2602.06552; 2407.05385; 2605.19373; 2607.01308]. Across these uses, the common objective is to remove arbitrary choices—such as neuron ordering, layer-uniform treatment, contribution order, or replica delivery order—while preserving useful merged behavior.

## 1. Terminological scope

The label has several technically distinct meanings in current work.

| Usage | Canonical object | Representative source |
|---|---|---|
| Standard model merging baseline | A single global operator on full parameter tensors | [2602.06552] |
| CCA-based model fusion | A canonical shared basis maximizing cross-model correlation | [2407.05385] |
| CRDT-compliant model merging | A canonically ordered contribution set with deterministic render | [2605.19373] |
| Multi-agent KV-cache merging | A content-determined cache order and deterministic render | [2607.01308] |

This multiplicity is substantive rather than merely terminological. In one line of work, CanonicalMerge is the *baseline* that newer modular methods generalize. In another, it is the *representation space* in which fusion becomes less brittle. In distributed-systems formulations, canonicality is a property of the *state semantics* rather than of the numeric merge itself. In latent reasoning, it is a *layout rule* that makes cache exchange permutation-invariant.

A recurring source of confusion is that these senses are not interchangeable. “Canonical” may refer to parameter-space uniformity, representation-space alignment, order-independent rendering, or state-based convergence. The term therefore indexes a family of invariance principles rather than a single merge operator.

## 2. CanonicalMerge as the monolithic baseline in model merging

In the model-merging literature summarized by "Fine-Grained Model Merging via Modular Expert Recombination" [2602.06552], the canonical or standard view of merging begins with a pre-trained model with parameters $\theta_{\text{pre}}$, followed by task-specific fine-tuning to obtain $\{\theta_t\}_{t=1}^T$. The objective is a single merged model $\theta_m$ obtained without further training by a global merging operator
$$
\theta_m = \mathcal{M}(\theta_1, \dots, \theta_T; \text{hyperparameters}).
$$
This operator acts on full parameter tensors and is applied uniformly across all layers and components.

Within this formulation, simple weight averaging gives
$$
\theta_m = \frac{1}{T}\sum_{t=1}^{T}\theta_t,
$$
while task arithmetic defines task vectors $\tau_t = \theta_t - \theta_{\text{pre}}$ and merges them through
$$
\theta_m = \theta_{\text{pre}} + \lambda\sum_{t=1}^T\tau_t,
$$
with a single global coefficient $\lambda$ [2602.06552]. Fisher-based and RegMean-style weighted averaging remain canonical in this sense because they still output one global $\theta_m$. Pruning-based task-vector methods such as TIES, Breadcrumbs, PCB, and DARE likewise operate on whole-model vectors even when they prune magnitudes or resolve sign conflicts.

Dynamic variants extend the same monolithic premise by making coefficients input-dependent:
$$
\theta_m(x) = \theta_s + \sum_{t=1}^T \lambda_t(x) M_t,
$$
where $\theta_s$ is shared, $M_t$ encodes task-specific deltas, and $\lambda_t(x)$ is predicted by a routing network [2602.06552]. Yet these methods still treat each expert as a full model or full set of task-specific parameters. The merged model for an instance is therefore ephemeral and not reused.

The key limitations identified in this line of work are structural. CanonicalMerge is monolithic; it does not differentiate attention, MLP, normalization, or other homologous components. It typically yields either one merged model for a task set or one dense per-instance combination, and it does not explicitly pose quality versus storage as a bi-objective optimization problem. This framing is important because later methods define themselves by relaxing precisely these assumptions.

## 3. CanonicalMerge as a special case of more modular or sequential frameworks

MERGE decomposes each model into components $\theta_t = \{\theta_t^l\}_{l=1}^L$ and introduces a grouping matrix
$$
\mathbf{G} = [g_{t,l}] \in \mathbb{N}^{T \times L},
$$
where $g_{t,l}$ assigns component $l$ of task $t$ to a group [2602.06552]. For each group, a modular expert is constructed by intra-group merging and optional quantization,
$$
\hat{\theta}_l^n = \mathcal{Q}\left(\mathcal{F}\left(\theta_{\text{pre}}^l, \Theta_l^n\right)\right).
$$
The crucial point is that CanonicalMerge reappears here as a degenerate configuration: if, for every component $l$, one sets $N_l = 1$ and $g_{t,l}=1$ for all tasks, then each layer contributes a single shared expert and the final model is
$$
\widetilde{\theta} = \bigoplus_{l=1}^L \hat{\theta}_l^1,
$$
which is exactly the single merged-model paradigm [2602.06552].

MERGE therefore treats canonical static merging as a boundary point in a larger search space. It also reframes merging as a bi-objective optimization over performance and storage, with a Pareto front of grouping configurations. This changes the meaning of canonicality: instead of a single global merge rule, one obtains a structured family of cost-performance tradeoffs and a reusable modular expert library selected under a storage budget [2602.06552].

A related but distinct critique appears in "Rethinking Layer-wise Model Merging through Chain of Merges" [2508.21421]. There, the target is activation-based layer-wise merging methods such as RegMean and MaTS that solve each layer using activations collected under the original, unmerged network. CoM identifies the resulting distributional mismatch as Merging Covariate Shift and updates activations autoregressively:
$$
\hat X_i^l = \sigma^{l-1}(W_M^{l-1}\hat X_i^{l-1}),
$$
followed by a layerwise regression update for $W_M^l$ using the updated $\hat X_i^l$ [2508.21421]. The paper explicitly presents CoM as a strong candidate for a more canonical activation-based merging procedure because it converts simultaneous layerwise fitting into a chain of conditionally optimal updates. This suggests that, within model merging, canonicality increasingly means *respecting internal structure* rather than enforcing a single whole-model operator.

## 4. CanonicalMerge as canonical shared representation: CCA Merge

"Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysis" replaces one-to-one neuron matching with alignment in a canonical shared basis [2407.05385]. The underlying observation is that permutation-based methods are restrictive: they assume that features in one model correspond to single neurons in another, whereas empirically many neurons have multiple moderately high correlations across independently trained networks.

For a layer $i$, activations from two models are collected as $X_i^\mathcal{A}, X_i^\mathcal{B} \in \mathbb{R}^{m \times n_i}$. CCA computes projection matrices $P_i^\mathcal{A}, P_i^\mathcal{B}$ so that the projected coordinates are maximally correlated. The resulting alignment transform is
$$
T_i = \big(P_i^\mathcal{B}(P_i^\mathcal{A})^{-1}\big)^\top,
$$
and the merged layer weights are formed by
$$
W_i = \frac12\Big(W_i^\mathcal{A} + T_i W_i^\mathcal{B} T_{i-1}^{-1}\Big),
$$
with an analogous update for biases [2407.05385].

Here the word canonical refers to the canonical variables of CCA. The merge is canonical not because it is uniform across all parameters, but because both models are expressed in a maximally correlated shared coordinate system. This is a fundamentally different use of the term from the monolithic baseline in model-merging work.

The empirical consequences are substantial. On VGG11/CIFAR-10, CCA Merge reaches from approximately $82.6\%$ to $85.5\%$ across widths, whereas direct averaging is near random and permutation-style methods remain substantially lower; on ResNet20/CIFAR-100 it reaches up to approximately $75.1\%$; and on a 200-class ImageNet subset with ResNet18x4 it reaches approximately $63.6\%$ top-1 and $85.4\%$ top-5, outperforming permutation-based and OT-based baselines [2407.05385]. The gains become more pronounced when merging many models or models trained on different data splits. The paper also reports that indirect and direct alignments are more consistent under CCA Merge than under permutation-based alignment, which supports the interpretation that CCA is approximating a shared canonical subspace rather than constructing ad hoc pairwise matchings [2407.05385].

## 5. Canonicality as a distributed-systems property

A different notion of CanonicalMerge arises when the core problem is not feature alignment but distributed convergence. "Conflict-Free Replicated Data Types for Neural Network Model Merging" evaluates 26 merge strategies—including weight averaging, SLERP, TIES, DARE, Fisher merging, and evolutionary approaches—and reports that none satisfy the commutativity, associativity, and idempotency required for conflict-free distributed operation [2605.19373]. The paper further states that this failure is structural: normalization-based merges cannot simultaneously satisfy all three properties.

Its solution is a two-layer architecture. Layer 1 stores contributions in an OR-Set-like CRDT state with merge as set union, yielding a state-based CvRDT. In the reconstructed formalization, the state is
$$
S = (A, R, V, H),
$$
and the state merge is
$$
\mathrm{merge}(S_1, S_2) = (A_1\cup A_2,\; R_1\cup R_2,\; \max(V_1,V_2),\; H').
$$
Layer 2 then applies a merge strategy as a deterministic pure function over a canonically ordered contribution set, with randomness seeded from the Merkle root:
$$
\mathcal{R}(C,\sigma,h) = \sigma\!\bigl(\mathrm{sort}_{\mathrm{hash}}(C), \mathrm{seed}(h)\bigr).
$$
Because the wrapper is transparent, downstream performance is identical by construction, and the paper reports byte-identical output verification together with Strong Eventual Consistency under gossip and partition healing [2605.19373].

In this setting, canonicality is a property of the *rendered output as a function of state*. The raw numerical strategy may remain non-associative, but the replicated state has a canonical join, and the resolved model is a deterministic function of the canonically ordered contribution set. This makes canonical merge a systems-level construct rather than an algebraic property of the inner merge rule.

A related distributed-systems formulation appears in "Semantic Conflict Model for Collaborative Data Structures" [2602.19231]. There, conflicts are detected through semantic dependencies between operations, and resolution occurs by rebasing conflicting operations onto a reconciling operation via a three-way merge over a replicated journal. The framework prescribes a canonical *pattern* of merge—identify conflicting premises, close under entailment, synthesize a merge operation, and rebase—while allowing the content of the merge to remain deterministic-policy-driven or user-guided [2602.19231]. This use of canonicality is again distinct: it lies in the structure of reconciliation, not in a fixed tensor operation.

## 6. CanonicalMerge as deterministic cache layout in multi-agent latent reasoning

"Cache Merging as a Convergent Replicated State for Multi-Agent Latent Reasoning" introduces CanonicalMerge for KV-cache composition [2607.01308]. Prior work concatenates caches along the sequence axis with RoPE re-encoding; the paper names this BagMerge and shows that it is non-commutative because the chosen concatenation order determines which fragment occupies the privileged position-0 prefix and how far every other fragment is shifted [2607.01308].

CanonicalMerge replaces caller-specified order with a content-determined permutation. For thinker $i$, at a routing layer $\ell^\star$, it computes a mean K-norm score
$$
\rho_i = \frac{1}{T^\star} \sum_{t = T_i - T^\star + 1}^{T_i}
\left\| K_i^{(\ell^\star)}[\,\cdot, \cdot, t, \cdot\,] \right\|_2,
$$
where $T^\star$ is the minimum sequence length across thinkers, and uses a content hash $h_i$ of the full cache as a deterministic tie-breaker. The canonical order is then
$$
\pi^\star = \operatorname{argsort}_i(-\rho_i, -h_i),
$$
after which BagMerge is applied in that order [2607.01308].

The result is byte-identical under any permutation of inputs. The paper verifies this algorithmically for arity $N \le 5$ and bit-for-bit on real Qwen3-1.7B and 4B KV state [2607.01308]. It then separates the replicated state from the decode-time layout by storing a set of content-addressed latent fragments whose merge is set union, yielding a state-based CvRDT; CanonicalMerge becomes the deterministic render of that set. Re-delivered duplicates are therefore absorbed rather than re-concatenated.

Empirically, on a partitioned-reasoning benchmark, the best BagMerge ordering changes with regime, latent-step budget, and model scale. CanonicalMerge nonetheless matches the best BagMerge ordering in every regime-by-budget-by-ordering cell without knowing which order is best, trading a small, statistically insignificant accuracy margin for an unconditional structural guarantee [2607.01308]. On HotpotQA bridge questions, the behavior transfers to real multi-document QA. Yet the paper also identifies an important limitation: at $k>2$, the approach transports and colocates latent traces but does not by itself compose them. CanonicalMerge therefore solves order-independence and replicated-state convergence, but not the broader problem of higher-arity semantic composition [2607.01308].

## 7. Conceptual synthesis and recurrent misconceptions

Taken together, these works suggest that CanonicalMerge is best understood as a family of invariance-seeking constructions rather than a single algorithmic object. In the monolithic model-merging sense, canonicality means a single whole-model operator applied uniformly across layers [2602.06552]. In CCA-based fusion, it means alignment in canonical variables that maximize cross-model correlation [2407.05385]. In CRDT-based systems, it means that the replicated state has a canonical join and that the render is a deterministic function of canonically ordered contributions [2605.19373]. In latent reasoning, it means that cache layout is fixed by content rather than by caller order [2607.01308].

Several misconceptions follow from conflating these senses. First, canonical does **not** necessarily mean globally optimal. MERGE explicitly treats the canonical monolithic merge as a degenerate point in a larger Pareto space of modular configurations [2602.06552]. Second, canonical does **not** necessarily mean associative at the level of the raw numeric merge. The CRDT work shows that many practical strategies remain non-associative even when the overall system is convergent [2605.19373]. Third, canonical does **not** imply semantic composition. The cache-merging work shows that order-independent transport of latent traces is insufficient for $k>2$ reasoning composition [2607.01308]. Fourth, canonical conflict resolution need not eliminate policy choice; the replicated-journal formulation makes the merge *shape* canonical while leaving reconcile semantics data-type-specific or interactive [2602.19231].

A plausible implication is that future uses of CanonicalMerge will continue to divide along two axes. One axis concerns *representation*: which basis, component granularity, or latent fragment identity should be treated as canonical. The other concerns *systems semantics*: which merge state, ordering, or reconciliation protocol makes distributed behavior invariant to arbitrary history. The recent literature indicates that strong performance increasingly comes from making those invariances explicit, rather than from treating merge as a single undifferentiated arithmetic operation.

Source: https://www.emergentmind.com/topics/canonicalmerge