---
title: Token Norm Imbalance (TNI) in Neural Models
url: https://www.emergentmind.com/topics/token-norm-imbalance-tni
type: topic
---

# Token Norm Imbalance (TNI) in Neural Models

Token Norm Imbalance (TNI) denotes a family of token-level magnitude disparities that materially alter model behavior. In explicit usage, recent work defines TNI as inter-token disparity in vector norms along the sequence dimension during KV-cache quantization; closely related work studies the same underlying issue as severe norm discrepancy between visual and text tokens in pre-Norm multimodal transformers, low-norm sink tokens in diffusion language models, and norm-dependent inhibition in deep residual networks [2605.19660][2512.08374][2601.19657][2606.16112]. Although the term is not yet standardized across subfields, these studies converge on a common claim: token norm is not a passive statistic, but a control variable for quantization fidelity, attention routing, cross-modal fusion, and the effective size of token updates.

## 1. Definition and scope

Current usage does not provide a single canonical definition of TNI. The most explicit formulation appears in KV-cache quantization, where TNI is defined as **inter-token disparity in vector norms along the sequence dimension**. In multimodal large language models, an allied formulation is a **severe norm disparity between the high-norm visual tokens and the low-norm text tokens**. In diffusion language models, the relevant object is a cross-token disparity in **value-space** norms, where sink tokens have unusually low norm. In deep residual transformers, the corresponding pathology is that tokens with larger residual norm are harder to move, so identical sublayer outputs have different effective impact when added back to the residual stream [2605.19660][2512.08374][2601.19657][2606.16112].

| Formulation | Object of imbalance | Immediate consequence |
|---|---|---|
| Explicit TNI | Token-vector norms across a sequence block | Quantization error under shared group scales |
| Cross-modal norm discrepancy | Visual-token norms vs text-token norms | Asymmetric update rates and visual information loss |
| Sink-token norm disparity | Low-norm value vectors within a sequence | Attention offloading to low-information tokens |
| Residual-stream norm imbalance | Residual magnitude across tokens and depth | Norm-dependent suppression of later updates |

This scope is narrower than a generic notion of token salience. The cited work does not treat all token-level imbalance as TNI. Some papers study neighboring phenomena—frequency imbalance, attention imbalance, or norm homogenization—and are relevant because they show how token-level disparities can be operationalized, measured, or corrected even when token norm itself is not the direct observable.

## 2. Mechanistic formulations

In pre-Norm multimodal LLMs, the central mechanism is an **asymmetric update dynamic**. The residual update is written as
\[
\Delta h^{(l)} = \mathrm{Sublayer}(\mathrm{Norm}(h^{(l)})), \qquad
h^{(l+1)} = h^{(l)} + \Delta h^{(l)},
\]
and the effective angular change satisfies
\[
\tan(\theta_{\text{eff}})= \frac{C \sin(\phi)}{\|h\|_2+ C\cos(\phi)}.
\]
For fixed update magnitude \(C\) and geometry \(\phi\), larger \(\|h\|_2\) implies smaller angular motion. The paper’s conclusion is that high-norm visual tokens develop **representational inertia**, so they transform semantically much slower than lower-norm text tokens, weakening cross-modal alignment and attention to relevant visual evidence [2512.08374].

In diffusion language models, TNI is localized to the Transformer **value space**. Per-token value vectors are
\[
\mathbf{v}^{(\ell,m)}_j = \mathbf{h}^{(\ell)}_j \mathbf{W}^{V}_m,
\]
and the attention output obeys
\[
\mathbf{o}^{(\ell,m)}_i = \sum_{j=1}^{n} \alpha^{(\ell,m)}_{ij}\,\mathbf{v}^{(\ell,m)}_j,\qquad
\left\|\mathbf{o}^{(\ell,m)}_i\right\|_2 \le \sum_{j=1}^{n}\alpha^{(\ell,m)}_{ij}\,\left\|\mathbf{v}^{(\ell,m)}_j\right\|_2.
\]
Low-norm value vectors are therefore attractive **no-op destinations** for excess attention mass. The paper interprets the moving sink phenomenon as a protective mechanism against excessive information mixing, but also as a robustness problem because the sink role moves across diffusion steps and layers [2601.19657].

In KV-cache quantization, TNI enters through shared quantization parameters over token groups. For a Key block \(g\), the paper derives
\[
\overline{\mathrm{MSE}_g} \gtrsim \frac{\bigl( \|\mathbf{k}_m\|_2 - \|\mathbf{k}_n\|_2 \bigr)^2}{12(2^b-1)^2},
\]
where \(m\) and \(n\) are the largest- and smallest-norm tokens in the block. This makes within-block token norm spread a lower bound on reconstruction error. The mechanism is not channel-wise outlierness but **sequence-dimensional variance**: shared per-channel scales become too coarse when token magnitudes differ substantially inside the same group [2605.19660].

In norm-agnostic residual networks, the relevant mechanism is **norm-dependent inhibition**. Standard residual accumulation is
\[
R_{l+1} = R_l + f_l(\bar R_l),
\]
while NAG rewrites the state as \(R_l = \rho_l \bar R_l\) and updates it via
\[
R_{l+1} = \rho_l \bar R_l + \rho_l \alpha_l m_l(\bar R_l) N_{\mathrm{out}\left(f_l^\perp(\bar R_l)\right)}.
\]
The induced norm growth ratio is
\[
\frac{\|R_{l+1}\|_2}{\|R_l\|_2} = \sqrt{1 + \alpha_l^2 m_l^2(\bar R_l)}.
\]
The paper’s claim is that pre-norm sublayers cannot see residual magnitude differences across tokens, even though those differences determine how much a given update actually rotates the residual stream after addition [2606.16112].

## 3. Empirical signatures

The multimodal pre-Norm evidence is unusually direct. The motivating example reports \( E\|text\|_2 \sim 0.86 < 56.88 \sim E\|image\|_2 \), and broad surveys of vision encoders and LLM embeddings show visual norms far above text norms. Representative values include CLIP-ViT-large-patch14 at **29.30 ± 17.12**, SigLIP-SO-400m-patch14-384 at **71.78 ± 13.95**, and text embedding norms of **0.80** for Qwen2.5-7B-Instruct and **1.09** for Llama3.2-3B-Instruct. Projectors reduce but do not eliminate the mismatch: for example, in LLaVA-v1.5 visual norms are **39.96 ± 45.58** after the projector against text **1.08**, and in Qwen-2.5-VL they are **56.88 ± 25.73** against text **0.86**. Inter-layer cosine similarity is then used as a proxy for update speed, and image tokens consistently exhibit higher inter-layer similarity than text tokens, indicating slower semantic evolution [2512.08374].

The diffusion-language-model evidence identifies sink tokens by value-space norm. Figure 1 reports mean sink-token norms of **2.36** versus **7.60** for normal tokens in LLaDA, and **3.15** versus **7.79** in Dream. The same analysis shows bright vertical attention stripes whose positions change across denoising steps, supporting the claim that sink roles move over time rather than remaining tied to a fixed location [2601.19657].

The KV-cache quantization evidence isolates TNI through controlled error comparisons. For per-channel Key quantization, MSE scaled by 100 is **0.23** with outlier tokens versus **0.16** without at 4-bit, **1.34** versus **0.78** at 3-bit, and **5.92** versus **3.52** at 2-bit. Mixed-modality grouping is especially costly for the visual component at 2-bit: **5.87** in mixed groups versus **2.45** in single-modality vision groups. By contrast, per-token Value quantization remains much less sensitive, consistent with the paper’s claim that TNI becomes harmful specifically when quantization parameters are shared across tokens [2605.19660].

Residual-stream measurements show that norm variation persists even among angularly similar tokens. Using von Mises–Fisher clustering, NAG reports average normalized 10th percentiles around **0.723–0.809** and 90th percentiles around **1.209–1.300** within clusters of similar residual direction. The same study also reports an approximately **120×** reduction in residual-stream norm relative to the baseline for the model with aspect ratio 80, indicating that uncontrolled norm growth is not a marginal artifact but a large-scale structural effect [2606.16112].

## 4. Mitigation strategies

The most direct TNI-oriented mitigation is **OScaR**, which combines **Canalized Rotation** with **Omni-Token Scaling**. The method first applies a Hadamard transform to Queries and Keys, then factors each Key into a normalized direction and a stored token-wise norm:
\[
n_k = \|\mathbf{K}_h\|_2, \qquad \mathbf{K}_u = \frac{\mathbf{K}_h}{n_k}.
\]
The paper argues that direct token-wise scaling alone creates a **Scaling-Induced Outlier Artifact**, so rotation must precede scaling. The ablation is decisive: on WorldSense with INT2 KCVT, baseline INT2 gives **22.64 / 39.62**, scaling only gives **17.61 / 35.22**, rotation only gives **37.74 / 41.51**, and full OScaR gives **38.36 / 42.77**. At system level, the implementation reports up to **3.0×** decoding speedup, **5.3×** memory reduction, and **4.1×** throughput increase relative to the BF16 FlashDecoding-v2 baseline [2605.19660].

In pre-Norm MLLMs, the proposed intervention is to insert a single **LayerNorm after the visual projector** and initialize its gain by
\[
T = \frac{1}{|W^*|}\sum_{w \in W^*}\|w\|_2, \qquad
g_{\text{init}} = T/\sqrt{D}.
\]
Because this tiny gain would otherwise suppress gradients, the paper adds **Global Weight Compensation** in the backward pass. On the LLaVA-1.5 framework with a Llama backbone, the average score rises from **59.01** to **62.04 / 61.27**; with a Qwen backbone, the average rises from **68.49** to **69.41** when GWC is used. The gains include substantial jumps on multimodal benchmarks and notable improvements on text-oriented evaluations such as MMLU [2512.08374].

For diffusion language models, the proposed remedy is not to remove sink behavior but to stabilize it with one extra token. The augmented sequence is \(\tilde{X} = [s; X]\) or \([X; s]\), and the special attention mask makes the sink token self-only as a query while remaining globally visible as key/value:
\[
M_{ij} =
\begin{cases}
-\infty, & \text{if } i = k \text{ and } j \neq k \\
0, & \text{otherwise}.
\end{cases}
\]
The effect is robust across placements and does not depend on semantic content. In AR-initialized 0.5B DLMs, ARC-e improves from **44.70** to **55.47**, HellaSwag from **35.21** to **48.91**, and PIQA from **55.28** to **63.44**. A zero-value sink remains competitive, supporting the claim that the token’s utility is structural rather than semantic [2601.19657].

At the architectural level, NAG attempts to remove norm-dependent suppression by separating magnitude and direction in the residual stream, orthogonalizing layer outputs, and modulating relative rotation rather than raw additive scale. Its gains increase with depth: in the aspect-ratio study, the deepest configuration improves final training loss from **2.35618** to **2.32769**, a gain of **0.02849**. The same formulation yields a Mixture-of-Depths mechanism in which rates of approximately **20%–25%** match full-depth baseline performance under equal training compute while reducing executed layer parameters and forward-pass FLOPs [2606.16112].

## 5. Relation to frequency imbalance, attention imbalance, and norm preservation

TNI is distinct from token **frequency imbalance**. “Token Imbalance Adaptation for Radiology Report Generation” studies skewed token counts in report vocabularies, not norms. Its central result is that current radiology report generators overfit frequent tokens and fail on clinically important infrequent ones, with low-frequency F1 as low as **3.85** versus **47.31** for Bi-LSTM on IU X-RAY under the \(1/8\) split, and **1.27** versus **37.48** on MIMIC. The proposed TIMER method uses unlikelihood loss and reinforcement-learned dynamic token-set selection, but the paper explicitly does **not** measure or theorize embedding norms, logit norms, or gradient norms [2304.09185].

A later pre-training study provides a direct bridge from frequency imbalance to norm imbalance. It shows that once common words are already atomic tokens at **24K** vocabulary with over **95%** efficiency for the top **2,500** frequent words, further vocabulary growth mainly increases token-frequency skew rather than segmentation quality. The paper then argues that frequent tokens acquire larger input and output embedding norms, and decomposes the target logit as
\[
\ell_t = \|h\|_2 \,\| E_{\text{out}_t}\|_2\cos\theta_t.
\]
Constraining input and output embeddings to unit norm **reverses the gain**, which the authors interpret as causal evidence that pre-training exploits rather than suffers from frequency-induced norm asymmetry [2508.15390].

Attention imbalance is a neighboring but distinct notion. In large vision-language models, **MAI** measures modality-wise attention imbalance and **TAI** measures token-wise mismatch between received attention and information contribution. AIR shows that correcting these imbalances at decoding time can reduce object hallucination by up to **35.1%** on CHAIR \(C_S\) and improve general capability by up to **15.9%**, but the paper never defines imbalance in terms of hidden-state norm or residual magnitude [2603.24058].

The opposite pathology also appears in the normalization literature. Dynamic Token Normalization argues that ordinary LayerNorm in ViTs **makes tokens at different positions similar in magnitude**, weakening local positional inductive bias. DTN restores inter-token variation by mixing intra-token and inter-token normalization, improving ImageNet top-1 by **0.5%–1.2%**, COCO box AP by **1.2–1.4**, ImageNet-C mCE by **2.3%–3.9%**, and Long ListOps accuracy by **0.5%–0.8%** [2112.02624]. Taken together, these papers show that the relevant question is not simply whether token norms differ, but whether the induced norm structure matches the objective and architecture.

## 6. Limitations, scope conditions, and open questions

The literature does not yet support a universal theory of TNI. A particularly important scope condition comes from Ovis-2.5-9B: the paper reports a large static norm gap between visual and text vocabularies—**64.00 ± 0.71** versus **1.38 ± 0.32**—yet visual and text update rates remain synchronized. This indicates that a large norm disparity is not, by itself, sufficient to produce the same asymmetric update dynamics observed in LLaVA-style continuous projector architectures [2512.08374].

Methodological under-specification remains common. In the vocabulary-scaling study, the unit-norm intervention is described conceptually, but the text does not specify whether normalization is implemented after every optimizer step, by reparameterization, by penalty, or by clipping. The same paper also notes that its output-layer gradient derivation assumes untied embeddings and does not provide a tied-versus-untied empirical comparison [2508.15390].

The diffusion-language-model sink analysis is deliberately narrow. Its strongest evidence is in value space, not in queries, keys, or the residual stream more broadly; sink movement is shown visually rather than quantified with a formal instability metric; and the reported experiments cover only **0.5B** and **1.5B** DLMs. The paper therefore establishes a specific low-norm sink mechanism rather than a general law of token norms in bidirectional diffusion transformers [2601.19657].

Finally, current evidence does not justify the claim that any norm imbalance is inherently pathological. Pre-training results argue that models can benefit from frequency-induced norm and logit asymmetry because the same frequent words dominate both training and downstream coverage, whereas ViT normalization results identify the converse failure mode of token-magnitude homogenization [2508.15390][2112.02624]. A plausible implication is that future TNI research will need to distinguish at least four cases: imbalance that degrades shared quantization, imbalance that slows semantic rotation, imbalance that creates useful structural sinks, and homogenization that erases informative token-level variation.

Source: https://www.emergentmind.com/topics/token-norm-imbalance-tni