MB-ToMe: Multi-Bit Token Merging for Quantized Transformers

• MB-ToMe is a cross-bit token fusion strategy that preserves robust high-bit tokens to mitigate inter-bit competition during one-shot calibration.
• It selectively merges token features by computing cosine similarities to retain stable representations while averaging less robust ones across low, mid, and high precisions.
• Integrated with MB-CLoRA in QuEPT, MB-ToMe enables real-time switching between bit-widths while maintaining accuracy and computational efficiency during model calibration.

Multi-Bit Token Merging (MB-ToMe) is the feature-fusion mechanism used in QuEPT to stabilize and improve accuracy under elastic precision quantization, where a single Transformer must operate across several predefined bit-widths and support real-time switching between them. In QuEPT, MB-ToMe is not a token-compression scheme for reducing sequence length; instead, it fuses token features across low-, mid-, and high-bit streams at the same token index, preserving robust high-bit tokens as anchors and softly averaging the remaining tokens across bit-width groups. Its stated function is to mitigate inter-bit “competition” during one-shot calibration, reduce the variance induced by quantization noise, and keep the fused representation close to high-precision geometry (Xu et al., 13 Feb 2026).

1. Role within elastic precision quantization

QuEPT is presented as an efficient post-training scheme that reconstructs block-wise multi-bit errors with one-shot calibration on a small data slice and can dynamically adapt to various predefined bit-widths by cascading different low-rank adapters. It supports real-time switching between uniform quantization and mixed precision quantization without repeated optimization. Within that framework, MB-ToMe is introduced specifically to enhance accuracy and robustness by dynamically fusing token features across different bit-widths, while Multi-Bit Cascaded Low-Rank Adapters (MB-CLoRA) strengthen correlations between bit-width groups (Xu et al., 13 Feb 2026).

The mechanism is motivated by the requirement that one calibrated model serve multiple quantization regimes simultaneously. The details specify that, without additional coordination, optimizing a block jointly for low- and high-precision regimes produces inter-bit “competition”: low-bit quantization injects larger, distribution-shifting noise, while gradients from high-bit reconstructions can be unhelpful for the low-bit slice. MB-ToMe addresses this by selectively aggregating token features across bit-width groups rather than passing a single-precision stream to each block.

A central property of the method is that it acts at calibration time. After one-shot calibration, deployment proceeds by selecting a single precision $b$ or a per-layer $b$ and running one standard forward pass. MB-ToMe is therefore part of the calibration procedure that makes multi-bit switching feasible, rather than part of the final inference path.

2. Source of instability under bit-width switching

The instability that MB-ToMe targets is described in terms of Transformer activations and quantization error. Activations are high-dimensional, exhibit large dynamic ranges, and have dense inter-token dependencies. Changing the quantizer bit-width $b$ alters both the clipping range and the integer lattice on which activations and weights are represented, so the quantization error $e(b)$ is neither small nor stationary across layers (Xu et al., 13 Feb 2026).

Low-bit settings such as 4-bit introduce larger, more structured errors, which can perturb both the attention pattern and the MLP nonlinearity. Mid- and high-bit settings such as 6–8 bits produce smaller error, but may optimize parameters in ways that do not help the low-bit slice. Calibrating one set of adapters and clipping thresholds to satisfy all target $b$ is therefore unstable if each block is optimized from a single-precision input stream.

MB-ToMe is described as a token-level aggregation across bit-width groups that preserves the most robust high-bit tokens and fuses the rest across low, mid, and high precision. The paper characterizes this as a precision-ensemble: aggregation attenuates quantization noise, reduces the variance arising from on-the-fly bit-width switching, and converts conflicting error signals into a more stable target for the low-rank adapters. Figure 1 is summarized as showing that “merging dissimilar tokens leads to a more uniform numerical distribution that is more robust to varying bit-widths,” which empirically illustrates the variance-smoothing effect.

3. Formal definition and selective merge rule

QuEPT partitions the target bit-width set into three groups,

$$\mathcal{B} = \mathcal{B}_L \cup \mathcal{B}_M \cup \mathcal{B}_H,$$

with the main weight-activation experiments using

$$\mathcal{B}_L = \{4\}, \quad \mathcal{B}_M = \{5,6\}, \quad \mathcal{B}_H = \{7,8\}.$$

For a token index $k$ at a given layer or block, $X_k^H$, $X_k^M$, and $b$0 denote the token representations computed under one sampled bit-width from $b$1, $b$2, and $b$3, respectively. MB-ToMe produces a fused token $b$4 at the same position $b$5 (Xu et al., 13 Feb 2026).

The paper defines three strategy cases. Case 1 is Random Selection, which picks one representation at random for each token. Case 2 is Uniform Fusion, which samples one $b$6 from each group and fuses the corresponding representations with equal weights $b$7. Case 3 is Selective Merge, which preserves a subset of robust tokens in their high-bit form and fuses the rest with a weighted average across the three groups. Case 3 is the default and best-performing choice.

The selective merge rule is

$b$8

The robustness set $b$9 is computed by taking the token-wise cosine similarity between high-bit and low-bit representations and selecting the top $b$0 tokens by that score:

$b$1

These top-$b$2 tokens are treated as robust or “foundational” and are kept in high precision, while the remaining tokens are fused. The fusion weights $b$3, $b$4, and $b$5 are hyperparameters, not learned parameters, and are fixed during one-shot calibration; equal weights work well in ablations (Xu et al., 13 Feb 2026).

MB-ToMe is applied between Transformer blocks to the residual stream feeding the next block. It is global over the feature channel and has no per-head specialization. It preserves sequence length, so the output $b$6 has the same shape as $b$7, $b$8, and $b$9. The details explicitly state that there is no per-layer token-merging ratio $e(b)$0; effectively, $e(b)$1 for all layers.

4. Calibration workflow and interaction with MB-CLoRA

At each calibration step, one bit-width is sampled from each group:

$e(b)$2

The previous block or blocks are run under these three settings to produce $e(b)$3, $e(b)$4, and $e(b)$5. These streams are not used independently. MB-ToMe fuses them into a single sequence $e(b)$6, and this fused sequence is then fed to the next block. As a result, every block is optimized on inputs that blend low-, mid-, and high-precision content, rather than on a single bit-width stream (Xu et al., 13 Feb 2026).

QuEPT combines MB-ToMe with MB-CLoRA and per-bit clipping. The MB-CLoRA construction is defined by

$e(b)$7

with a cascaded rank schedule

$e(b)$8

The text explains the division of labor succinctly: MB-ToMe makes the inputs more consistent across precisions, while MB-CLoRA makes parameter updates shared and hierarchically consistent across precisions. There is no direct coupling from MB-CLoRA into the MB-ToMe weights $e(b)$9 or the selection $b$0; the effect is indirect, through improved alignment of features across bit groups.

The adapters are trained with MAE on the block outputs across sampled bit-widths, using the fused input. The calibration recipe is explicitly lightweight in data requirements: examples given are 1024 unlabeled ImageNet images for ViTs, 128 C4 samples for LLaMA, and 128 image-caption pairs for LLaVA-OneVision. After calibration, real-time bit-width switching is implemented by selecting the matching LoRA slice for the desired bit-width via the MB-CLoRA rank cascade; MB-ToMe itself is not used at deployment.

5. Computational characteristics and hyperparameters

MB-ToMe is described as lightweight. If the sequence length is $b$1 and the hidden size is $b$2, computing cosine similarities between $b$3 and $b$4 is $b$5, and the weighted fusion is also $b$6. The calibration-time memory overhead is storing three feature maps, $b$7, $b$8, and $b$9, for a total of $$\mathcal{B} = \mathcal{B}_L \cup \mathcal{B}_M \cup \mathcal{B}_H,$$0 activations, plus the fused map $$\mathcal{B} = \mathcal{B}_L \cup \mathcal{B}_M \cup \mathcal{B}_H,$$1 of size $$\mathcal{B} = \mathcal{B}_L \cup \mathcal{B}_M \cup \mathcal{B}_H,$$2 for the next block (Xu et al., 13 Feb 2026).

The dominant marginal cost does not come from the merge itself, but from producing the three streams $$\mathcal{B} = \mathcal{B}_L \cup \mathcal{B}_M \cup \mathcal{B}_H,$$3, $$\mathcal{B} = \mathcal{B}_L \cup \mathcal{B}_M \cup \mathcal{B}_H,$$4, and $$\mathcal{B} = \mathcal{B}_L \cup \mathcal{B}_M \cup \mathcal{B}_H,$$5 for each block during calibration. QuEPT already samples one bit-width per group and evaluates the block under those precisions to accumulate losses; MB-ToMe reuses those features, so its extra FLOPs are characterized as negligible relative to the three forward evaluations per block.

Because MB-ToMe preserves token count, it does not reduce attention or MLP FLOPs. The details emphasize that its purpose is robustness and accuracy, not acceleration. At inference and deployment, MB-ToMe is absent, so it contributes no runtime overhead and does not alter sequence length or attention FLOPs.

The key hyperparameters are the bit-width partition $$\mathcal{B} = \mathcal{B}_L \cup \mathcal{B}_M \cup \mathcal{B}_H,$$6, the robustness percentile $$\mathcal{B} = \mathcal{B}_L \cup \mathcal{B}_M \cup \mathcal{B}_H,$$7, and the fusion weights $$\mathcal{B} = \mathcal{B}_L \cup \mathcal{B}_M \cup \mathcal{B}_H,$$8. The recommended practical procedure is to sample one bit from each group per calibration step, choose $$\mathcal{B} = \mathcal{B}_L \cup \mathcal{B}_M \cup \mathcal{B}_H,$$9 on a held-out calibration slice, initialize the fusion weights to $$\mathcal{B}_L = \{4\}, \quad \mathcal{B}_M = \{5,6\}, \quad \mathcal{B}_H = \{7,8\}.$$0, and apply MB-ToMe once between consecutive blocks. The method requires no attention-head changes.

6. Empirical evaluation

Two ablations isolate the contribution of MB-ToMe. The first compares the three merge cases on ViT-S with ImageNet top-1 accuracy and on LLaMA2-7B with a 5-task zero-shot average. On ViT-S, Case 1, Case 2, and Case 3 achieve the following for W4A4 through W8A8: 74.5 / 74.9 / 75.1, 79.4 / 79.3 / 79.7, 80.3 / 80.2 / 80.6, 80.5 / 80.6 / 81.0, and 80.9 / 80.8 / 81.2. On LLaMA2-7B, the corresponding values are 55.7 / 60.2 / 61.6, 63.8 / 64.4 / 64.5, 64.7 / 65.2 / 65.5, 64.9 / 65.5 / 65.6, and 65.4 / 65.5 / 65.6. The paper states that Case 3 consistently outperforms the alternatives, with especially strong gains at low bit-widths, including a $$\mathcal{B}_L = \{4\}, \quad \mathcal{B}_M = \{5,6\}, \quad \mathcal{B}_H = \{7,8\}.$$1 improvement over Case 1 at W4A4 on LLaMA2-7B (Xu et al., 13 Feb 2026).

The second ablation is a module-level study on ViT-S. The baseline with no clipping, MB-CLoRA, MB-ToMe, or MAE yields W4A4/W6A6/W8A8 of 69.2 / 80.1 / 81.1. Adding clipping gives 70.1 / 80.2 / 81.1. Adding MB-CLoRA gives 72.2 / 79.2 / 80.7. Adding MB-ToMe gives 73.0 / 80.1 / 80.9. Adding MB-CLoRA and MB-ToMe gives 73.6 / 80.1 / 81.0. Adding MB-CLoRA, MB-ToMe, and MAE gives 74.7 / 80.7 / 81.1. The accompanying summary states that MB-ToMe raises W4A4 by approximately $$\mathcal{B}_L = \{4\}, \quad \mathcal{B}_M = \{5,6\}, \quad \mathcal{B}_H = \{7,8\}.$$2–$$\mathcal{B}_L = \{4\}, \quad \mathcal{B}_M = \{5,6\}, \quad \mathcal{B}_H = \{7,8\}.$$3 depending on the other modules, and that the combination of MB-ToMe, MB-CLoRA, and MAE gives the best overall result.

The interaction with MB-CLoRA is also supported by a LoRA-sharing ablation. The details report that MB-CLoRA improves especially at low bits, including a $$\mathcal{B}_L = \{4\}, \quad \mathcal{B}_M = \{5,6\}, \quad \mathcal{B}_H = \{7,8\}.$$4 gain over Independent sharing at W4A4 on LLaMA2-7B, while MB-ToMe helps recover mid- and high-bit performance when MB-CLoRA places extra emphasis on low-bit ranks. This suggests that the two mechanisms are complementary rather than interchangeable.

7. Relation to other token-merging methods and limitations

As defined in QuEPT, MB-ToMe differs from standard Token Merging (ToMe) in three explicit ways. First, the similarity computation is cross-precision and token-wise—cosine similarity between high-bit and low-bit representations at the same token index—rather than cross-token similarity within a single precision stream. Second, the merge policy preserves an anchor set $$\mathcal{B}_L = \{4\}, \quad \mathcal{B}_M = \{5,6\}, \quad \mathcal{B}_H = \{7,8\}.$$5 of robust tokens at high precision and fuses the rest across precisions, rather than merging tokens pairwise to shrink the sequence. Third, sequence length is unchanged, whereas ToMe typically reduces it through a per-layer ratio $$\mathcal{B}_L = \{4\}, \quad \mathcal{B}_M = \{5,6\}, \quad \mathcal{B}_H = \{7,8\}.$$6 (Xu et al., 13 Feb 2026).

A later diffusion-transformer study contrasts Learnable Global Merging (LGM) with both ToMe and “MB-ToMe” under the broader topic of token merging, but its own objective is different: LGM is a data-independent tokenizer-stage merge designed to align latent representations across variable token lengths, whereas QuEPT’s MB-ToMe is a calibration-time, data-dependent cross-bit feature fusion that preserves token count and is not used at deployment (Lee et al., 18 Jun 2026). This suggests that the shared vocabulary of “merging” covers distinct technical goals: runtime acceleration or variable-length alignment in one setting, and robustness under elastic precision quantization in the other.

The stated limitations of MB-ToMe are correspondingly specific. It does not reduce inference cost, because it does not reduce the number of tokens. If the high-bit and low-bit streams are poorly aligned, such as under extremely low bits or heavily outliered channels, the cosine-selected set $$\mathcal{B}_L = \{4\}, \quad \mathcal{B}_M = \{5,6\}, \quad \mathcal{B}_H = \{7,8\}.$$7 may be small or unstable, which reduces gains. The details also note that outliers in LLMs are not explicitly handled and that combining QuEPT with outlier mitigation, such as SpinQuant, may further improve MB-ToMe’s effect. Sensitivity to the robustness percentile $$\mathcal{B}_L = \{4\}, \quad \mathcal{B}_M = \{5,6\}, \quad \mathcal{B}_H = \{7,8\}.$$8 and fusion weights $$\mathcal{B}_L = \{4\}, \quad \mathcal{B}_M = \{5,6\}, \quad \mathcal{B}_H = \{7,8\}.$$9 is acknowledged: too small a $k$0 may discard useful anchors, too large a $k$1 may under-merge noisy tokens, and severe distribution shifts in calibration data can harm the cosine-based selection.

In summary, MB-ToMe is a selective, training-free, token-level fusion rule for one-shot calibration in elastic precision Transformers. Its defining feature is not token reduction but cross-precision stabilization: robust high-bit tokens are preserved, the remaining tokens are fused across low, mid, and high precision, and the resulting representation is used to optimize a single model that can switch bit-widths in real time without repeated optimization.