CanonicalMerge: Invariance in Model Merging

• CanonicalMerge is a framework of invariance principles that removes arbitrary merging choices across diverse neural network fusion techniques.
• It spans methods from monolithic weight averaging, CCA-based shared representations, to CRDT-based state convergence and content-determined cache ordering.
• These strategies yield reproducible, deterministic outputs and define baseline configurations within broader modular, cost-performance trade-off frameworks.

CanonicalMerge is not a single universally standardized algorithm. In recent arXiv literature, the term and closely related constructions denote several distinct ways of making merging canonical: a monolithic whole-model operator applied uniformly to all parameters; a merge performed in a canonical shared representation space derived from Canonical Correlation Analysis (CCA); a deterministic render of a CRDT-managed contribution set; and a content-determined cache layout for multi-agent latent reasoning (Qiu et al., 6 Feb 2026, Horoi et al., 2024, Gillespie, 16 May 2026, Baquero et al., 1 Jul 2026). Across these uses, the common objective is to remove arbitrary choices—such as neuron ordering, layer-uniform treatment, contribution order, or replica delivery order—while preserving useful merged behavior.

1. Terminological scope

The label has several technically distinct meanings in current work.

Usage	Canonical object	Representative source
Standard model merging baseline	A single global operator on full parameter tensors	(Qiu et al., 6 Feb 2026)
CCA-based model fusion	A canonical shared basis maximizing cross-model correlation	(Horoi et al., 2024)
CRDT-compliant model merging	A canonically ordered contribution set with deterministic render	(Gillespie, 16 May 2026)
Multi-agent KV-cache merging	A content-determined cache order and deterministic render	(Baquero et al., 1 Jul 2026)

This multiplicity is substantive rather than merely terminological. In one line of work, CanonicalMerge is the baseline that newer modular methods generalize. In another, it is the representation space in which fusion becomes less brittle. In distributed-systems formulations, canonicality is a property of the state semantics rather than of the numeric merge itself. In latent reasoning, it is a layout rule that makes cache exchange permutation-invariant.

A recurring source of confusion is that these senses are not interchangeable. “Canonical” may refer to parameter-space uniformity, representation-space alignment, order-independent rendering, or state-based convergence. The term therefore indexes a family of invariance principles rather than a single merge operator.

2. CanonicalMerge as the monolithic baseline in model merging

In the model-merging literature summarized by "Fine-Grained Model Merging via Modular Expert Recombination" (Qiu et al., 6 Feb 2026), the canonical or standard view of merging begins with a pre-trained model with parameters $\theta_{\text{pre}}$, followed by task-specific fine-tuning to obtain $\{\theta_t\}_{t=1}^T$. The objective is a single merged model $\theta_m$ obtained without further training by a global merging operator

$$\theta_m = \mathcal{M}(\theta_1, \dots, \theta_T; \text{hyperparameters}).$$

This operator acts on full parameter tensors and is applied uniformly across all layers and components.

Within this formulation, simple weight averaging gives

$\theta_m = \frac{1}{T}\sum_{t=1}^{T}\theta_t,$

while task arithmetic defines task vectors $\tau_t = \theta_t - \theta_{\text{pre}}$ and merges them through

$$\theta_m = \theta_{\text{pre}} + \lambda\sum_{t=1}^T\tau_t,$$

with a single global coefficient $\lambda$ (Qiu et al., 6 Feb 2026). Fisher-based and RegMean-style weighted averaging remain canonical in this sense because they still output one global $\theta_m$. Pruning-based task-vector methods such as TIES, Breadcrumbs, PCB, and DARE likewise operate on whole-model vectors even when they prune magnitudes or resolve sign conflicts.

Dynamic variants extend the same monolithic premise by making coefficients input-dependent:

$$\theta_m(x) = \theta_s + \sum_{t=1}^T \lambda_t(x) M_t,$$

where $\{\theta_t\}_{t=1}^T$0 is shared, $\{\theta_t\}_{t=1}^T$1 encodes task-specific deltas, and $\{\theta_t\}_{t=1}^T$2 is predicted by a routing network (Qiu et al., 6 Feb 2026). Yet these methods still treat each expert as a full model or full set of task-specific parameters. The merged model for an instance is therefore ephemeral and not reused.

The key limitations identified in this line of work are structural. CanonicalMerge is monolithic; it does not differentiate attention, MLP, normalization, or other homologous components. It typically yields either one merged model for a task set or one dense per-instance combination, and it does not explicitly pose quality versus storage as a bi-objective optimization problem. This framing is important because later methods define themselves by relaxing precisely these assumptions.

3. CanonicalMerge as a special case of more modular or sequential frameworks

MERGE decomposes each model into components $\{\theta_t\}_{t=1}^T$3 and introduces a grouping matrix

$\{\theta_t\}_{t=1}^T$4

where $\{\theta_t\}_{t=1}^T$5 assigns component $\{\theta_t\}_{t=1}^T$6 of task $\{\theta_t\}_{t=1}^T$7 to a group (Qiu et al., 6 Feb 2026). For each group, a modular expert is constructed by intra-group merging and optional quantization,

$\{\theta_t\}_{t=1}^T$8

The crucial point is that CanonicalMerge reappears here as a degenerate configuration: if, for every component $\{\theta_t\}_{t=1}^T$9, one sets $\theta_m$0 and $\theta_m$1 for all tasks, then each layer contributes a single shared expert and the final model is

$\theta_m$2

which is exactly the single merged-model paradigm (Qiu et al., 6 Feb 2026).

MERGE therefore treats canonical static merging as a boundary point in a larger search space. It also reframes merging as a bi-objective optimization over performance and storage, with a Pareto front of grouping configurations. This changes the meaning of canonicality: instead of a single global merge rule, one obtains a structured family of cost-performance tradeoffs and a reusable modular expert library selected under a storage budget (Qiu et al., 6 Feb 2026).

A related but distinct critique appears in "Rethinking Layer-wise Model Merging through Chain of Merges" (Buzzega et al., 29 Aug 2025). There, the target is activation-based layer-wise merging methods such as RegMean and MaTS that solve each layer using activations collected under the original, unmerged network. CoM identifies the resulting distributional mismatch as Merging Covariate Shift and updates activations autoregressively:

$\theta_m$3

followed by a layerwise regression update for $\theta_m$4 using the updated $\theta_m$5 (Buzzega et al., 29 Aug 2025). The paper explicitly presents CoM as a strong candidate for a more canonical activation-based merging procedure because it converts simultaneous layerwise fitting into a chain of conditionally optimal updates. This suggests that, within model merging, canonicality increasingly means respecting internal structure rather than enforcing a single whole-model operator.

4. CanonicalMerge as canonical shared representation: CCA Merge

"Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysis" replaces one-to-one neuron matching with alignment in a canonical shared basis (Horoi et al., 2024). The underlying observation is that permutation-based methods are restrictive: they assume that features in one model correspond to single neurons in another, whereas empirically many neurons have multiple moderately high correlations across independently trained networks.

For a layer $\theta_m$6, activations from two models are collected as $\theta_m$7. CCA computes projection matrices $\theta_m$8 so that the projected coordinates are maximally correlated. The resulting alignment transform is

$\theta_m$9

and the merged layer weights are formed by

$$\theta_m = \mathcal{M}(\theta_1, \dots, \theta_T; \text{hyperparameters}).$$0

with an analogous update for biases (Horoi et al., 2024).

Here the word canonical refers to the canonical variables of CCA. The merge is canonical not because it is uniform across all parameters, but because both models are expressed in a maximally correlated shared coordinate system. This is a fundamentally different use of the term from the monolithic baseline in model-merging work.

The empirical consequences are substantial. On VGG11/CIFAR-10, CCA Merge reaches from approximately $$\theta_m = \mathcal{M}(\theta_1, \dots, \theta_T; \text{hyperparameters}).$$1 to $$\theta_m = \mathcal{M}(\theta_1, \dots, \theta_T; \text{hyperparameters}).$$2 across widths, whereas direct averaging is near random and permutation-style methods remain substantially lower; on ResNet20/CIFAR-100 it reaches up to approximately $$\theta_m = \mathcal{M}(\theta_1, \dots, \theta_T; \text{hyperparameters}).$$3; and on a 200-class ImageNet subset with ResNet18x4 it reaches approximately $$\theta_m = \mathcal{M}(\theta_1, \dots, \theta_T; \text{hyperparameters}).$$4 top-1 and $$\theta_m = \mathcal{M}(\theta_1, \dots, \theta_T; \text{hyperparameters}).$$5 top-5, outperforming permutation-based and OT-based baselines (Horoi et al., 2024). The gains become more pronounced when merging many models or models trained on different data splits. The paper also reports that indirect and direct alignments are more consistent under CCA Merge than under permutation-based alignment, which supports the interpretation that CCA is approximating a shared canonical subspace rather than constructing ad hoc pairwise matchings (Horoi et al., 2024).

5. Canonicality as a distributed-systems property

A different notion of CanonicalMerge arises when the core problem is not feature alignment but distributed convergence. "Conflict-Free Replicated Data Types for Neural Network Model Merging" evaluates 26 merge strategies—including weight averaging, SLERP, TIES, DARE, Fisher merging, and evolutionary approaches—and reports that none satisfy the commutativity, associativity, and idempotency required for conflict-free distributed operation (Gillespie, 16 May 2026). The paper further states that this failure is structural: normalization-based merges cannot simultaneously satisfy all three properties.

Its solution is a two-layer architecture. Layer 1 stores contributions in an OR-Set-like CRDT state with merge as set union, yielding a state-based CvRDT. In the reconstructed formalization, the state is

$$\theta_m = \mathcal{M}(\theta_1, \dots, \theta_T; \text{hyperparameters}).$$6

and the state merge is

$$\theta_m = \mathcal{M}(\theta_1, \dots, \theta_T; \text{hyperparameters}).$$7

Layer 2 then applies a merge strategy as a deterministic pure function over a canonically ordered contribution set, with randomness seeded from the Merkle root:

$$\theta_m = \mathcal{M}(\theta_1, \dots, \theta_T; \text{hyperparameters}).$$8

Because the wrapper is transparent, downstream performance is identical by construction, and the paper reports byte-identical output verification together with Strong Eventual Consistency under gossip and partition healing (Gillespie, 16 May 2026).

In this setting, canonicality is a property of the rendered output as a function of state. The raw numerical strategy may remain non-associative, but the replicated state has a canonical join, and the resolved model is a deterministic function of the canonically ordered contribution set. This makes canonical merge a systems-level construct rather than an algebraic property of the inner merge rule.

A related distributed-systems formulation appears in "Semantic Conflict Model for Collaborative Data Structures" (Semenov et al., 22 Feb 2026). There, conflicts are detected through semantic dependencies between operations, and resolution occurs by rebasing conflicting operations onto a reconciling operation via a three-way merge over a replicated journal. The framework prescribes a canonical pattern of merge—identify conflicting premises, close under entailment, synthesize a merge operation, and rebase—while allowing the content of the merge to remain deterministic-policy-driven or user-guided (Semenov et al., 22 Feb 2026). This use of canonicality is again distinct: it lies in the structure of reconciliation, not in a fixed tensor operation.

6. CanonicalMerge as deterministic cache layout in multi-agent latent reasoning

"Cache Merging as a Convergent Replicated State for Multi-Agent Latent Reasoning" introduces CanonicalMerge for KV-cache composition (Baquero et al., 1 Jul 2026). Prior work concatenates caches along the sequence axis with RoPE re-encoding; the paper names this BagMerge and shows that it is non-commutative because the chosen concatenation order determines which fragment occupies the privileged position-0 prefix and how far every other fragment is shifted (Baquero et al., 1 Jul 2026).

CanonicalMerge replaces caller-specified order with a content-determined permutation. For thinker $$\theta_m = \mathcal{M}(\theta_1, \dots, \theta_T; \text{hyperparameters}).$$9, at a routing layer $\theta_m = \frac{1}{T}\sum_{t=1}^{T}\theta_t,$0, it computes a mean K-norm score

$\theta_m = \frac{1}{T}\sum_{t=1}^{T}\theta_t,$1

where $\theta_m = \frac{1}{T}\sum_{t=1}^{T}\theta_t,$2 is the minimum sequence length across thinkers, and uses a content hash $\theta_m = \frac{1}{T}\sum_{t=1}^{T}\theta_t,$3 of the full cache as a deterministic tie-breaker. The canonical order is then

$\theta_m = \frac{1}{T}\sum_{t=1}^{T}\theta_t,$4

after which BagMerge is applied in that order (Baquero et al., 1 Jul 2026).

The result is byte-identical under any permutation of inputs. The paper verifies this algorithmically for arity $\theta_m = \frac{1}{T}\sum_{t=1}^{T}\theta_t,$5 and bit-for-bit on real Qwen3-1.7B and 4B KV state (Baquero et al., 1 Jul 2026). It then separates the replicated state from the decode-time layout by storing a set of content-addressed latent fragments whose merge is set union, yielding a state-based CvRDT; CanonicalMerge becomes the deterministic render of that set. Re-delivered duplicates are therefore absorbed rather than re-concatenated.

Empirically, on a partitioned-reasoning benchmark, the best BagMerge ordering changes with regime, latent-step budget, and model scale. CanonicalMerge nonetheless matches the best BagMerge ordering in every regime-by-budget-by-ordering cell without knowing which order is best, trading a small, statistically insignificant accuracy margin for an unconditional structural guarantee (Baquero et al., 1 Jul 2026). On HotpotQA bridge questions, the behavior transfers to real multi-document QA. Yet the paper also identifies an important limitation: at $\theta_m = \frac{1}{T}\sum_{t=1}^{T}\theta_t,$6, the approach transports and colocates latent traces but does not by itself compose them. CanonicalMerge therefore solves order-independence and replicated-state convergence, but not the broader problem of higher-arity semantic composition (Baquero et al., 1 Jul 2026).

7. Conceptual synthesis and recurrent misconceptions

Taken together, these works suggest that CanonicalMerge is best understood as a family of invariance-seeking constructions rather than a single algorithmic object. In the monolithic model-merging sense, canonicality means a single whole-model operator applied uniformly across layers (Qiu et al., 6 Feb 2026). In CCA-based fusion, it means alignment in canonical variables that maximize cross-model correlation (Horoi et al., 2024). In CRDT-based systems, it means that the replicated state has a canonical join and that the render is a deterministic function of canonically ordered contributions (Gillespie, 16 May 2026). In latent reasoning, it means that cache layout is fixed by content rather than by caller order (Baquero et al., 1 Jul 2026).

Several misconceptions follow from conflating these senses. First, canonical does not necessarily mean globally optimal. MERGE explicitly treats the canonical monolithic merge as a degenerate point in a larger Pareto space of modular configurations (Qiu et al., 6 Feb 2026). Second, canonical does not necessarily mean associative at the level of the raw numeric merge. The CRDT work shows that many practical strategies remain non-associative even when the overall system is convergent (Gillespie, 16 May 2026). Third, canonical does not imply semantic composition. The cache-merging work shows that order-independent transport of latent traces is insufficient for $\theta_m = \frac{1}{T}\sum_{t=1}^{T}\theta_t,$7 reasoning composition (Baquero et al., 1 Jul 2026). Fourth, canonical conflict resolution need not eliminate policy choice; the replicated-journal formulation makes the merge shape canonical while leaving reconcile semantics data-type-specific or interactive (Semenov et al., 22 Feb 2026).

A plausible implication is that future uses of CanonicalMerge will continue to divide along two axes. One axis concerns representation: which basis, component granularity, or latent fragment identity should be treated as canonical. The other concerns systems semantics: which merge state, ordering, or reconciliation protocol makes distributed behavior invariant to arbitrary history. The recent literature indicates that strong performance increasingly comes from making those invariances explicit, rather than from treating merge as a single undifferentiated arithmetic operation.