BagMerge: Transformer Cache Merging

• BagMerge is a cache-composition framework that merges independent KV-caches using RoPE re-encoding to adjust latent positions in decoder-only transformers.
• It employs a concat-with-RoPE method to integrate latent traces, preserving positional details for coherent multi-agent reasoning.
• The approach is order-sensitive and non-commutative, driving innovations like CanonicalMerge for deterministic and permutation-invariant cache rendering.

Searching arXiv for papers on BagMerge and closely related terminology. BagMerge is a cache-composition procedure for multi-agent latent reasoning in decoder-only transformers. In the formulation analyzed in "Cache Merging as a Convergent Replicated State for Multi-Agent Latent Reasoning" (Baquero et al., 1 Jul 2026), multiple parallel agents produce KV-caches on disjoint fragments, and their latent traces are combined into one attended context for a final agent by concatenating caches along the sequence axis with RoPE re-encoding. The same source characterizes BagMerge as simple and effective but also non-commutative and order-sensitive under RoPE, with the best ordering shifting with the regime, the latent-step budget, and the model scale. A secondary, terminological usage appears in heavy-flavor event generation, where the exact name in the underlying paper is “fusing,” and “BagMerge” can be adopted as a label for that merging algorithm (Höche et al., 2019).

1. Transformer-cache formulation

In the latent-reasoning setting, BagMerge operates on the KV-cache of a decoder-only transformer with $L$ layers, $H$ heads per layer, head dimension $d$, and rotary position embedding (RoPE). For an input sequence of length $T$, the cached keys and values at layer $\ell$ are $K_\ell \in \mathbb{R}^{H\times T\times d_k}$ and $V_\ell \in \mathbb{R}^{H\times T\times d_v}$, with keys stored post-rotation. The method assumes that $N$ parallel agents independently run the same model on disjoint fragments and produce caches $\{(K_\ell^{(j)},V_\ell^{(j)})\}_{\ell=1..L}$ with sequence lengths $T^{(j)}$ (Baquero et al., 1 Jul 2026).

RoPE acts pairwise on dimensions $H$0 at position $H$1 and base angular frequency $H$2:

$H$3

Because RoPE geometry makes dot products depend on relative position, reusing stored keys at a new absolute location requires a position shift. BagMerge is therefore not merely concatenation; it is concat-with-RoPE, where successor segments are re-encoded so their positions extend the prefix without overlap.

This construction is the basis for treating intermediate latent traces as composable state. Its importance lies in moving aggregation below the token-output layer: the final agent reads over intermediate K/V geometry rather than only over emitted text or fused logits.

2. Concat-with-RoPE mechanics

BagMerge takes a permutation $H$4 over the $H$5 agent caches, computes cumulative offsets $H$6 and $H$7 for $H$8, and then merges each layer by concatenating values unchanged while re-rotating keys to the new absolute positions. For each layer $H$9:

$d$0

In words, the values pass through unchanged, while the keys are re-rotated to their new absolute positions (Baquero et al., 1 Jul 2026).

If keys in a segment $d$1 were originally stored at absolute positions $d$2, concatenation changes their effective positions to $d$3. RoPE re-encoding applies $d$4 to keys, or equivalently $d$5 to queries at read time. Since practical caching stores keys post-rotation, the operational step is to shift each segment’s keys by the cumulative offset $d$6.

The significance of this mechanism is architectural rather than merely bookkeeping. BagMerge makes several independently produced latent contexts jointly attendable by one downstream decoder pass, but it does so by assigning them an ordered positional layout. That positional assignment is the source of both its utility and its instability.

3. Non-commutativity and order sensitivity

BagMerge is non-commutative because concatenation order determines which fragment becomes the position-0 prefix. Under RoPE, the prefix’s keys retain their original rotation at low absolute positions, while later segments are re-rotated to higher positions. Changing $d$7 changes each segment’s effective position $d$8, hence the rotary phase applied to its keys, and therefore alters attention allocation and outputs (Baquero et al., 1 Jul 2026).

For a query at position $d$9 and a key originally at $T$0, the inner product depends on the reordered absolute position through $T$1 and $T$2:

$T$3

Empirically, the paper reports substantial order sensitivity on a partitioned-reasoning benchmark for Qwen3-1.7B. In the query-known regime, BagMerge-swap beats BagMerge-default, with $T$4 versus $T$5 at $T$6 steps and $T$7 versus $T$8 at $T$9 steps. In the query-blind regime, BagMerge-default beats BagMerge-swap, with $\ell$0 versus $\ell$1 at $\ell$2 steps and $\ell$3 versus $\ell$4 at $\ell$5 steps. At Qwen3-4B, “swap” is best in both regimes, $\ell$6 versus $\ell$7 in query-known and $\ell$8 versus $\ell$9 in query-blind. The paper summarizes these swings as up to $K_\ell \in \mathbb{R}^{H\times T\times d_k}$0–$K_\ell \in \mathbb{R}^{H\times T\times d_k}$1 percentage points across orderings, with the optimal order flipping between regimes at $K_\ell \in \mathbb{R}^{H\times T\times d_k}$2B and changing with scale at $K_\ell \in \mathbb{R}^{H\times T\times d_k}$3B.

A common misconception is that cache concatenation is effectively symmetric once RoPE shifts are applied. The reported behavior contradicts that view: BagMerge preserves a privileged prefix and therefore retains a positional asymmetry inherited from the model’s training and long-context behavior.

4. Canonical ordering and CvRDT reinterpretation

The central response to BagMerge’s order sensitivity is CanonicalMerge, which fixes the layout by content rather than by arrival order. CanonicalMerge scores each cache using mean K-norm at a single middle layer, sorts caches by descending score with deterministic tie-breaking by a full-content hash of all layers’ K and V bytes, and then renders the result via the same concat-with-RoPE mechanism (Baquero et al., 1 Jul 2026).

The formal guarantee is permutation invariance of the rendered cache: CanonicalMerge of any permutation of the same inputs is bit-for-bit identical. This was verified algorithmically on synthetic tensor tests across all $K_\ell \in \mathbb{R}^{H\times T\times d_k}$4 input permutations for $K_\ell \in \mathbb{R}^{H\times T\times d_k}$5, summing to $K_\ell \in \mathbb{R}^{H\times T\times d_k}$6 permutations, using exact equality on raw tensor bytes. Real-model verification established bit-for-bit byte identity across re-orderings on Qwen3-1.7B with $K_\ell \in \mathbb{R}^{H\times T\times d_k}$7 layers and Qwen3-4B with $K_\ell \in \mathbb{R}^{H\times T\times d_k}$8 layers.

The paper then separates replicated state from decode-time layout. The durable state is a set of content-addressed latent fragments,

$K_\ell \in \mathbb{R}^{H\times T\times d_k}$9

where $V_\ell \in \mathbb{R}^{H\times T\times d_v}$0 is a thinker’s unshifted KV cache and $V_\ell \in \mathbb{R}^{H\times T\times d_v}$1 is its content hash. The merge operator is set union,

$V_\ell \in \mathbb{R}^{H\times T\times d_v}$2

which is commutative, associative, idempotent, and absorbing. Render is defined as a deterministic function of the set, $V_\ell \in \mathbb{R}^{H\times T\times d_v}$3. For $V_\ell \in \mathbb{R}^{H\times T\times d_v}$4, the render is byte-identical to canonical-ordered concatenation on the same two caches, so every $V_\ell \in \mathbb{R}^{H\times T\times d_v}$5 accuracy number transfers unchanged to the CvRDT render. Duplicate re-deliveries are absorbed at the state, so they do not re-concatenate.

This formulation changes the status of cache merging from a heuristic exchange protocol into a convergent replicated state. A plausible implication is that the operational problem is no longer “which order should arrive first?” but “which deterministic render should be applied to an order-free fragment set?”

5. Empirical behavior, comparisons, and limits

On the partitioned-reasoning benchmark with $V_\ell \in \mathbb{R}^{H\times T\times d_v}$6 and Qwen3-1.7B, CanonicalMerge is reported to match the best BagMerge ordering in every regime-by-budget-by-ordering cell without knowing which order is best. In the query-known regime, CanonicalMerge achieves $V_\ell \in \mathbb{R}^{H\times T\times d_v}$7 at $V_\ell \in \mathbb{R}^{H\times T\times d_v}$8 steps and $V_\ell \in \mathbb{R}^{H\times T\times d_v}$9 at $N$0 steps, compared with the best fixed BagMerge ordering at $N$1 and $N$2; the differences are $N$3 and $N$4, within bootstrap confidence intervals. In the query-blind regime, CanonicalMerge achieves $N$5 and $N$6, compared with the best fixed BagMerge ordering at $N$7 and $N$8; the differences are $N$9 and $\{(K_\ell^{(j)},V_\ell^{(j)})\}_{\ell=1..L}$0, also within confidence intervals (Baquero et al., 1 Jul 2026).

Multi-seed greedy replication with $\{(K_\ell^{(j)},V_\ell^{(j)})\}_{\ell=1..L}$1 pooled confirms that CanonicalMerge ties the best fixed ordering in each regime, that confidence intervals straddle $\{(K_\ell^{(j)},V_\ell^{(j)})\}_{\ell=1..L}$2, that CanonicalMerge significantly beats the worse ordering by $\{(K_\ell^{(j)},V_\ell^{(j)})\}_{\ell=1..L}$3 with Holm correction, and that BagMerge’s order sensitivity is significant and flips sign across regimes. On HotpotQA bridge with $\{(K_\ell^{(j)},V_\ell^{(j)})\}_{\ell=1..L}$4 and $\{(K_\ell^{(j)},V_\ell^{(j)})\}_{\ell=1..L}$5 problems, the full-context single-agent baseline obtains $\{(K_\ell^{(j)},V_\ell^{(j)})\}_{\ell=1..L}$6 and $\{(K_\ell^{(j)},V_\ell^{(j)})\}_{\ell=1..L}$7; BagMerge default obtains $\{(K_\ell^{(j)},V_\ell^{(j)})\}_{\ell=1..L}$8 and $\{(K_\ell^{(j)},V_\ell^{(j)})\}_{\ell=1..L}$9; CanonicalMerge obtains $T^{(j)}$0 and $T^{(j)}$1, with no detectable degradation versus baseline at that sample.

The paper also contrasts cache-level merging with output-level fusion. At matched $T^{(j)}$2-token budget, PackLLM trails CanonicalMerge by approximately $T^{(j)}$3 percentage points in query-known evaluation over three draws, with CanonicalMerge at $T^{(j)}$4 versus PackLLM at $T^{(j)}$5, and paired difference $T^{(j)}$6. The stated structural reason is that logit-level fusion averages per-step next-token distributions after each thinker compresses its reasoning, whereas cache-level merging preserves intermediate K/V geometry that a judger can still route over.

The limitations are explicit. At $T^{(j)}$7, cache merge transports and colocates latent traces but does not by itself compose them: a flat $T^{(j)}$8-way merge and a pairwise merge tree collapse on chain-structured problems, and composition appears only when the model reprocesses seams. Duplicate deliveries are another edge case: naive re-concatenation degrades accuracy, with an example of $T^{(j)}$9 at $H$00 cache length, and grows memory linearly; the CvRDT set absorbs duplicates and keeps accuracy and size invariant. Heterogeneous hardware can defeat hash-based identity through low-order-bit differences, and semantic near-duplicates remain out of scope because set union is syntactic rather than semantic.

6. Practical settings and alternative terminology

The reported complexity of BagMerge is dominated by RoPE re-encoding of keys, $H$01 across layers, with values concatenated directly. CanonicalMerge adds scoring at the chosen routing layer, hashing of all K/V bytes per fragment, and sorting at $H$02, while preserving the same RoPE re-encode cost once the order is fixed (Baquero et al., 1 Jul 2026). Recommended settings in the paper are a mid-network routing layer, specifically $H$03 for Qwen3-1.7B and $H$04 for Qwen3-4B, tail-aligned averaging over the last $H$05 positions to normalize variable lengths, full-content hashing for deterministic tie-breaking, and the model’s native RoPE base frequencies and rotation kernel without altering the RoPE configuration.

A separate usage of the term appears in collider phenomenology. "Multi-jet merging in a variable flavor number scheme" (Höche et al., 2019) introduces a new event-generation technique whose exact name is “fusing.” In the terminology supplied here, “BagMerge” can denote that fusing algorithm for b-quark–associated generation merging: it combines a multi-jet–merged five-flavor-scheme sample with a four-flavor-scheme calculation of b-associated production, removes overlap through an event-level filter driven by parton-shower histories, and preserves fixed-order and logarithmic accuracy at the level of the ME+PS merging. The paper’s own notation centers on the FONLL master formula,

$H$06

with Sudakov and $H$07 reweighting used to construct the direct component and veto rules applied to the fragmentation component.

The two usages are technically unrelated. In latent reasoning, BagMerge is concat-with-RoPE over transformer KV-caches; in heavy-flavor event generation, the corresponding exact term is “fusing,” and “BagMerge” is only an adopted label. The shared theme is merging structured intermediate states while controlling overlap, but the mathematical objects, guarantees, and failure modes are different.