Token Norm Imbalance (TNI) in Neural Models

• Token Norm Imbalance (TNI) is defined as inter-token disparity in vector norms that influences quantization fidelity, attention routing, and effective token updates.
• Empirical analyses reveal significant norm gaps between modalities—such as high visual versus low text norms—and identify low-norm sink tokens that slow semantic evolution.
• Mitigation strategies like OScaR and LayerNorm adjustments demonstrate tangible gains in decoding speed, memory reduction, and model robustness.

Token Norm Imbalance (TNI) denotes a family of token-level magnitude disparities that materially alter model behavior. In explicit usage, recent work defines TNI as inter-token disparity in vector norms along the sequence dimension during KV-cache quantization; closely related work studies the same underlying issue as severe norm discrepancy between visual and text tokens in pre-Norm multimodal transformers, low-norm sink tokens in diffusion LLMs, and norm-dependent inhibition in deep residual networks (Su et al., 19 May 2026, Li et al., 9 Dec 2025, Zhang et al., 27 Jan 2026, Figliolia et al., 15 Jun 2026). Although the term is not yet standardized across subfields, these studies converge on a common claim: token norm is not a passive statistic, but a control variable for quantization fidelity, attention routing, cross-modal fusion, and the effective size of token updates.

1. Definition and scope

Current usage does not provide a single canonical definition of TNI. The most explicit formulation appears in KV-cache quantization, where TNI is defined as inter-token disparity in vector norms along the sequence dimension. In multimodal LLMs, an allied formulation is a severe norm disparity between the high-norm visual tokens and the low-norm text tokens. In diffusion LLMs, the relevant object is a cross-token disparity in value-space norms, where sink tokens have unusually low norm. In deep residual transformers, the corresponding pathology is that tokens with larger residual norm are harder to move, so identical sublayer outputs have different effective impact when added back to the residual stream (Su et al., 19 May 2026, Li et al., 9 Dec 2025, Zhang et al., 27 Jan 2026, Figliolia et al., 15 Jun 2026).

Formulation	Object of imbalance	Immediate consequence
Explicit TNI	Token-vector norms across a sequence block	Quantization error under shared group scales
Cross-modal norm discrepancy	Visual-token norms vs text-token norms	Asymmetric update rates and visual information loss
Sink-token norm disparity	Low-norm value vectors within a sequence	Attention offloading to low-information tokens
Residual-stream norm imbalance	Residual magnitude across tokens and depth	Norm-dependent suppression of later updates

This scope is narrower than a generic notion of token salience. The cited work does not treat all token-level imbalance as TNI. Some papers study neighboring phenomena—frequency imbalance, attention imbalance, or norm homogenization—and are relevant because they show how token-level disparities can be operationalized, measured, or corrected even when token norm itself is not the direct observable.

2. Mechanistic formulations

In pre-Norm multimodal LLMs, the central mechanism is an asymmetric update dynamic. The residual update is written as

$$\Delta h^{(l)} = \mathrm{Sublayer}(\mathrm{Norm}(h^{(l)})), \qquad h^{(l+1)} = h^{(l)} + \Delta h^{(l)},$$

and the effective angular change satisfies

$$\tan(\theta_{\text{eff}})= \frac{C \sin(\phi)}{\|h\|_2+ C\cos(\phi)}.$$

For fixed update magnitude $C$ and geometry $\phi$, larger $\|h\|_2$ implies smaller angular motion. The paper’s conclusion is that high-norm visual tokens develop representational inertia, so they transform semantically much slower than lower-norm text tokens, weakening cross-modal alignment and attention to relevant visual evidence (Li et al., 9 Dec 2025).

In diffusion LLMs, TNI is localized to the Transformer value space. Per-token value vectors are

$$\mathbf{v}^{(\ell,m)}_j = \mathbf{h}^{(\ell)}_j \mathbf{W}^{V}_m,$$

and the attention output obeys

$$\mathbf{o}^{(\ell,m)}_i = \sum_{j=1}^{n} \alpha^{(\ell,m)}_{ij}\,\mathbf{v}^{(\ell,m)}_j,\qquad \left\|\mathbf{o}^{(\ell,m)}_i\right\|_2 \le \sum_{j=1}^{n}\alpha^{(\ell,m)}_{ij}\,\left\|\mathbf{v}^{(\ell,m)}_j\right\|_2.$$

Low-norm value vectors are therefore attractive no-op destinations for excess attention mass. The paper interprets the moving sink phenomenon as a protective mechanism against excessive information mixing, but also as a robustness problem because the sink role moves across diffusion steps and layers (Zhang et al., 27 Jan 2026).

In KV-cache quantization, TNI enters through shared quantization parameters over token groups. For a Key block $g$, the paper derives

$$\overline{\mathrm{MSE}_g} \gtrsim \frac{\bigl( \|\mathbf{k}_m\|_2 - \|\mathbf{k}_n\|_2 \bigr)^2}{12(2^b-1)^2},$$

where $m$ and $$\tan(\theta_{\text{eff}})= \frac{C \sin(\phi)}{\|h\|_2+ C\cos(\phi)}.$$0 are the largest- and smallest-norm tokens in the block. This makes within-block token norm spread a lower bound on reconstruction error. The mechanism is not channel-wise outlierness but sequence-dimensional variance: shared per-channel scales become too coarse when token magnitudes differ substantially inside the same group (Su et al., 19 May 2026).

In norm-agnostic residual networks, the relevant mechanism is norm-dependent inhibition. Standard residual accumulation is

$$\tan(\theta_{\text{eff}})= \frac{C \sin(\phi)}{\|h\|_2+ C\cos(\phi)}.$$1

while NAG rewrites the state as $$\tan(\theta_{\text{eff}})= \frac{C \sin(\phi)}{\|h\|_2+ C\cos(\phi)}.$$2 and updates it via

$$\tan(\theta_{\text{eff}})= \frac{C \sin(\phi)}{\|h\|_2+ C\cos(\phi)}.$$3

The induced norm growth ratio is

$$\tan(\theta_{\text{eff}})= \frac{C \sin(\phi)}{\|h\|_2+ C\cos(\phi)}.$$4

The paper’s claim is that pre-norm sublayers cannot see residual magnitude differences across tokens, even though those differences determine how much a given update actually rotates the residual stream after addition (Figliolia et al., 15 Jun 2026).

3. Empirical signatures

The multimodal pre-Norm evidence is unusually direct. The motivating example reports $$\tan(\theta_{\text{eff}})= \frac{C \sin(\phi)}{\|h\|_2+ C\cos(\phi)}.$$5, and broad surveys of vision encoders and LLM embeddings show visual norms far above text norms. Representative values include CLIP-ViT-large-patch14 at 29.30 ± 17.12, SigLIP-SO-400m-patch14-384 at 71.78 ± 13.95, and text embedding norms of 0.80 for Qwen2.5-7B-Instruct and 1.09 for Llama3.2-3B-Instruct. Projectors reduce but do not eliminate the mismatch: for example, in LLaVA-v1.5 visual norms are 39.96 ± 45.58 after the projector against text 1.08, and in Qwen-2.5-VL they are 56.88 ± 25.73 against text 0.86. Inter-layer cosine similarity is then used as a proxy for update speed, and image tokens consistently exhibit higher inter-layer similarity than text tokens, indicating slower semantic evolution (Li et al., 9 Dec 2025).

The diffusion-language-model evidence identifies sink tokens by value-space norm. Figure 1 reports mean sink-token norms of 2.36 versus 7.60 for normal tokens in LLaDA, and 3.15 versus 7.79 in Dream. The same analysis shows bright vertical attention stripes whose positions change across denoising steps, supporting the claim that sink roles move over time rather than remaining tied to a fixed location (Zhang et al., 27 Jan 2026).

The KV-cache quantization evidence isolates TNI through controlled error comparisons. For per-channel Key quantization, MSE scaled by 100 is 0.23 with outlier tokens versus 0.16 without at 4-bit, 1.34 versus 0.78 at 3-bit, and 5.92 versus 3.52 at 2-bit. Mixed-modality grouping is especially costly for the visual component at 2-bit: 5.87 in mixed groups versus 2.45 in single-modality vision groups. By contrast, per-token Value quantization remains much less sensitive, consistent with the paper’s claim that TNI becomes harmful specifically when quantization parameters are shared across tokens (Su et al., 19 May 2026).

Residual-stream measurements show that norm variation persists even among angularly similar tokens. Using von Mises–Fisher clustering, NAG reports average normalized 10th percentiles around 0.723–0.809 and 90th percentiles around 1.209–1.300 within clusters of similar residual direction. The same study also reports an approximately 120× reduction in residual-stream norm relative to the baseline for the model with aspect ratio 80, indicating that uncontrolled norm growth is not a marginal artifact but a large-scale structural effect (Figliolia et al., 15 Jun 2026).

4. Mitigation strategies

The most direct TNI-oriented mitigation is OScaR, which combines Canalized Rotation with Omni-Token Scaling. The method first applies a Hadamard transform to Queries and Keys, then factors each Key into a normalized direction and a stored token-wise norm: $$\tan(\theta_{\text{eff}})= \frac{C \sin(\phi)}{\|h\|_2+ C\cos(\phi)}.$$6 The paper argues that direct token-wise scaling alone creates a Scaling-Induced Outlier Artifact, so rotation must precede scaling. The ablation is decisive: on WorldSense with INT2 KCVT, baseline INT2 gives 22.64 / 39.62, scaling only gives 17.61 / 35.22, rotation only gives 37.74 / 41.51, and full OScaR gives 38.36 / 42.77. At system level, the implementation reports up to 3.0× decoding speedup, 5.3× memory reduction, and 4.1× throughput increase relative to the BF16 FlashDecoding-v2 baseline (Su et al., 19 May 2026).

In pre-Norm MLLMs, the proposed intervention is to insert a single LayerNorm after the visual projector and initialize its gain by

$$\tan(\theta_{\text{eff}})= \frac{C \sin(\phi)}{\|h\|_2+ C\cos(\phi)}.$$7

Because this tiny gain would otherwise suppress gradients, the paper adds Global Weight Compensation in the backward pass. On the LLaVA-1.5 framework with a Llama backbone, the average score rises from 59.01 to 62.04 / 61.27; with a Qwen backbone, the average rises from 68.49 to 69.41 when GWC is used. The gains include substantial jumps on multimodal benchmarks and notable improvements on text-oriented evaluations such as MMLU (Li et al., 9 Dec 2025).

For diffusion LLMs, the proposed remedy is not to remove sink behavior but to stabilize it with one extra token. The augmented sequence is $$\tan(\theta_{\text{eff}})= \frac{C \sin(\phi)}{\|h\|_2+ C\cos(\phi)}.$$8 or $$\tan(\theta_{\text{eff}})= \frac{C \sin(\phi)}{\|h\|_2+ C\cos(\phi)}.$$9, and the special attention mask makes the sink token self-only as a query while remaining globally visible as key/value: $C$0 The effect is robust across placements and does not depend on semantic content. In AR-initialized 0.5B DLMs, ARC-e improves from 44.70 to 55.47, HellaSwag from 35.21 to 48.91, and PIQA from 55.28 to 63.44. A zero-value sink remains competitive, supporting the claim that the token’s utility is structural rather than semantic (Zhang et al., 27 Jan 2026).

At the architectural level, NAG attempts to remove norm-dependent suppression by separating magnitude and direction in the residual stream, orthogonalizing layer outputs, and modulating relative rotation rather than raw additive scale. Its gains increase with depth: in the aspect-ratio study, the deepest configuration improves final training loss from 2.35618 to 2.32769, a gain of 0.02849. The same formulation yields a Mixture-of-Depths mechanism in which rates of approximately 20%–25% match full-depth baseline performance under equal training compute while reducing executed layer parameters and forward-pass FLOPs (Figliolia et al., 15 Jun 2026).

5. Relation to frequency imbalance, attention imbalance, and norm preservation

TNI is distinct from token frequency imbalance. “Token Imbalance Adaptation for Radiology Report Generation” studies skewed token counts in report vocabularies, not norms. Its central result is that current radiology report generators overfit frequent tokens and fail on clinically important infrequent ones, with low-frequency F1 as low as 3.85 versus 47.31 for Bi-LSTM on IU X-RAY under the $C$1 split, and 1.27 versus 37.48 on MIMIC. The proposed TIMER method uses unlikelihood loss and reinforcement-learned dynamic token-set selection, but the paper explicitly does not measure or theorize embedding norms, logit norms, or gradient norms (Wu et al., 2023).

A later pre-training study provides a direct bridge from frequency imbalance to norm imbalance. It shows that once common words are already atomic tokens at 24K vocabulary with over 95% efficiency for the top 2,500 frequent words, further vocabulary growth mainly increases token-frequency skew rather than segmentation quality. The paper then argues that frequent tokens acquire larger input and output embedding norms, and decomposes the target logit as

$C$2

Constraining input and output embeddings to unit norm reverses the gain, which the authors interpret as causal evidence that pre-training exploits rather than suffers from frequency-induced norm asymmetry (Chung et al., 21 Aug 2025).

Attention imbalance is a neighboring but distinct notion. In large vision-LLMs, MAI measures modality-wise attention imbalance and TAI measures token-wise mismatch between received attention and information contribution. AIR shows that correcting these imbalances at decoding time can reduce object hallucination by up to 35.1% on CHAIR $C$3 and improve general capability by up to 15.9%, but the paper never defines imbalance in terms of hidden-state norm or residual magnitude (Sun et al., 25 Mar 2026).

The opposite pathology also appears in the normalization literature. Dynamic Token Normalization argues that ordinary LayerNorm in ViTs makes tokens at different positions similar in magnitude, weakening local positional inductive bias. DTN restores inter-token variation by mixing intra-token and inter-token normalization, improving ImageNet top-1 by 0.5%–1.2%, COCO box AP by 1.2–1.4, ImageNet-C mCE by 2.3%–3.9%, and Long ListOps accuracy by 0.5%–0.8% (Shao et al., 2021). Taken together, these papers show that the relevant question is not simply whether token norms differ, but whether the induced norm structure matches the objective and architecture.

6. Limitations, scope conditions, and open questions

The literature does not yet support a universal theory of TNI. A particularly important scope condition comes from Ovis-2.5-9B: the paper reports a large static norm gap between visual and text vocabularies—64.00 ± 0.71 versus 1.38 ± 0.32—yet visual and text update rates remain synchronized. This indicates that a large norm disparity is not, by itself, sufficient to produce the same asymmetric update dynamics observed in LLaVA-style continuous projector architectures (Li et al., 9 Dec 2025).

Methodological under-specification remains common. In the vocabulary-scaling study, the unit-norm intervention is described conceptually, but the text does not specify whether normalization is implemented after every optimizer step, by reparameterization, by penalty, or by clipping. The same paper also notes that its output-layer gradient derivation assumes untied embeddings and does not provide a tied-versus-untied empirical comparison (Chung et al., 21 Aug 2025).

The diffusion-language-model sink analysis is deliberately narrow. Its strongest evidence is in value space, not in queries, keys, or the residual stream more broadly; sink movement is shown visually rather than quantified with a formal instability metric; and the reported experiments cover only 0.5B and 1.5B DLMs. The paper therefore establishes a specific low-norm sink mechanism rather than a general law of token norms in bidirectional diffusion transformers (Zhang et al., 27 Jan 2026).

Finally, current evidence does not justify the claim that any norm imbalance is inherently pathological. Pre-training results argue that models can benefit from frequency-induced norm and logit asymmetry because the same frequent words dominate both training and downstream coverage, whereas ViT normalization results identify the converse failure mode of token-magnitude homogenization (Chung et al., 21 Aug 2025, Shao et al., 2021). A plausible implication is that future TNI research will need to distinguish at least four cases: imbalance that degrades shared quantization, imbalance that slows semantic rotation, imbalance that creates useful structural sinks, and homogenization that erases informative token-level variation.