---
title: Dynamic Token/Tile Adaptive Quantization
url: https://www.emergentmind.com/topics/dynamic-token-tile-adaptive-quantization
type: topic
---

# Dynamic Token/Tile Adaptive Quantization

Dynamic Token/Tile Adaptive Quantization refers to a class of methods that allocate quantization precision or bit-rate in a spatially or semantically adaptive manner at the granularity of tokens (elements in a sequence, typically in language models) or tiles (spatial blocks, as in images or hidden activations). Rather than assigning a fixed number of quantization bits uniformly, these methods dynamically adjust bit-width or allocate quantization resources according to the local information content, token- or tile-wise sensitivity, or saliency, enabling improved trade-offs between model accuracy, computational/memory efficiency, and communication overhead.

## 1. Motivation and Overview of Dynamic Quantization

Conventional quantization schemes apply a static bit-width uniformly across all data elements (tokens or tiles) within a model component, disregarding the heterogeneity in signal sensitivity or statistical distribution. However, empirical studies show that both in large language models (LLMs) and vision-language models (VLMs), quantization error is highly concentrated in specific locations: either on rare "outlier" tokens or spatial/image regions with increased local complexity. This motivates adaptive allocation of bit-width or quantization parameters, tailored to token-level or tile-level statistics, to minimize overall error for a given resource budget [2602.20191], [2506.01352], [2410.05265], [1802.02629].

Token-adaptive quantization targets the dynamic, context-sensitive distribution of activations in LLMs or VLMs, while tile-adaptive quantization is particularly valuable in imaging and vision, where spatial complexity or saliency can vary at fine granularity.

## 2. Algorithmic Approaches and Mathematical Formulation

### Adaptive Bit Allocation and Quantization Operators

**Token- or Tile-wise Bit-width Selection:** A central mechanism is to determine, for each token or tile \( t \), the required number of quantization bits \( b_t \) or, equivalently, the number of quantization "slices" or blocks to employ. Allocation is typically driven by measurements of local information content, such as activation entropy [2506.01352], token-level sensitivity [2602.20191], or local distortion/residual error [1802.02629].

**Quantization Operator:** Consider a data vector \( x_t \in \mathbb{R}^d \) (e.g., token embedding or image tile), quantized using a per-token or per-tile operator:
\[
Q(x_t; b_t) = \mathrm{clip}\left(\mathrm{round}\left(\frac{x_t}{s_t}\right) + z_t, 0, 2^{b_t}-1 \right),
\]
where \( s_t \) and \( z_t \) are local scale and zero-point parameters, which may be dynamically determined per token/tile. Some methods hierarchically decompose quantization into multiple residual slices, each capturing finer quantization detail, to facilitate elastic allocation [2602.20191].

**Token/Tile Selection Criteria:**
- Entropy-based: Assign higher bit-widths to tokens/tiles with larger activation entropy,
  \(
  H(x_t) = -\sum_k p_{t,k} \log p_{t,k}
  \)
  where \( p_{t,k} \) are normalized component magnitudes [2506.01352].
- Saliency-based: For tokens in attention mechanisms, assign bit-widths based on normalized attention scores, reflecting their downstream impact [2405.14256].
- Residual/distortion-based: Dynamically increase bits for a tile until its local distortion (e.g., MSE) falls below a threshold [1802.02629].

### Routing and MoE-style Quantization
Several frameworks employ token- or tile-aware routers: small neural networks that predict, per data element, the necessary quantization configuration—such as number of bit slices, selection of specialized experts for error compensation, or block-wise allocation [2602.20191], [2602.24059].

A common structure is a gating function
\[
G_{t,e} = \mathbf{1}(S_{t,e} > \delta)
\]
where \( S_{t,e} \) are router logits indicating the need for slice \( e \) for token \( t \). The bit allocation then follows as \( k_t = \sum_e G_{t,e} \).

## 3. Key Techniques and Methodological Innovations

### Recursive Residual Quantization
MoBiQuant introduces a "many-in-one" recursive residual quantization scheme (MoBiSlice) that decomposes weights into multiple quantized residual slices \( W_e \). At inference, the cumulative sum of a selected subset of slices yields the quantized weights at the requested bit-width. This enables smooth, calibration-free precision switching and token-adaptive bit-width allocation, mitigating "precision-dependent outlier migration" [2602.20191].

### Tile-wise Adaptive Quantization in Vision
TAH-Quant partitions activations into tiles, and allocates bit-width per token based on a tile's entropy, assigning higher bits to "information-rich" tokens/tiles and lower bits where activations are sparse or dominated by outliers. A pivot-based Hadamard transform further suppresses the effect of outliers within a tile [2506.01352].

In image compression, spatial tiles are processed by a recurrent autoencoder, with early stopping based on local distortion thresholds. Bit-rate is thus spatially adapted without explicit per-tile scale step size [1802.02629].

### Token/Tile-aware Error Compensation
Quant Experts applies a mixture-of-experts approach: channels are partitioned into globally important (token-independent) vs. locally important (token-dependent) sets using frequency statistics over token activations. A shared expert compensates quantization error on token-independent channels, while routed experts adaptively compensate errors for input-dependent (token/tile) channel patterns, with a lightweight routing matrix selecting the best expert per input [2602.24059].

## 4. Salient Token/Tile Identification and Outlier Handling

Dynamic token/tile quantization often relies on identification of critical elements:
- Saliency metrics (ZipCache): Normalized attention scores provide an unbiased estimate of each token's importance, mitigating position bias from causal attention masking. Tokens are ranked and assigned either high or low bit-width quantization accordingly [2405.14256].
- Outlier isolation (PrefixQuant): PrefixQuant isolates rare token-wise outliers by identifying, via a fixed calibration pass, which tokens exhibit activation maxima exceeding a factor threshold over the median. These tokens are prefixed in the KV cache, separating them from the quantization scale used for the rest, thereby enabling static low-bit quantization for non-outlier tokens without per-token dynamic scaling [2410.05265].

## 5. Memory, Computational, and Implementation Considerations

Adaptive quantization methods must carefully balance overhead:
- Parameter storage: Fine-grained groupwise quantization incurs high storage for per-token/group scalars. Channel-separable schemes (ZipCache) employ per-channel normalization pre-quantization, reducing parameter count to \( H d + 2L \), orders of magnitude less than groupwise per-token scalars [2405.14256].
- Computation: Online routers and reconstruction experts contribute little overhead compared to matrix-matrix multiplications when configured with modest rank (e.g., adapter rank \( r=64 \), number of experts \( N_r=8 \)), keeping total compute sub-1% of baseline [2602.24059], [2602.20191].
- Memory: Methods such as TAH-Quant avoid extra activation or gradient storage by working on current tiles only, in contrast to methods requiring activation error compensation buffers [2506.01352].

Implementation for large-scale or bandwidth-limited settings incorporates hardware-awareness—e.g., tensor packing for coalesced memory in GPU (ZipCache), bit-major kernel optimization (MoBiQuant), and fusion of quantization normalization into kernel launches.

## 6. Empirical Performance and Trade-offs

Dynamic token/tile adaptive quantization methods consistently demonstrate that careful, input-adaptive allocation achieves superior accuracy-compression trade-offs over static quantization:

| Method        | Key Metric                 | Speedup/Compression | Accuracy Degradation |
|---------------|---------------------------|---------------------|---------------------|
| MoBiQuant     | LLaMA3-8B, PPL=7.31       | 2.7× vs FP16        | None at matched bpp |
| ZipCache      | KV cache, 4.98× compress  | 19.8% memory save   | 0.38% drop          |
| PrefixQuant   | LLaMA3-8B, 2.81× speedup  | >2.7× GEMM speedup  | ≤1% (with finetune) |
| QuantExperts  | Qwen2VL-72B, W4A6         | 3.5–4.5× vs full-16 | +5.44% over MBQ     |
| TAH-Quant     | GPT-2XL pipeline, INT3/4  | Up to 4.3×          | Matches FP16, AQ-SGD|

A common finding is that the memory and throughput savings scale nearly linearly with the fraction of elements assigned low bit-width, while empirical accuracy displays an "elbow": beyond a threshold reduction in high-precision assignments, error begins to rise sharply [2405.14256].

## 7. Limitations, Applicability, and Extensions

Applicability is greatest in domains with high variability in token/tile information content, notably LLM inference/generation (where token-wise activation distribution can vary widely with prompt structure, language, etc.) and image/vision tasks exhibiting spatial heterogeneity. 

Limitations include:
- Dependence on precomputed or stable outlier/token saliency statistics (PrefixQuant): if outlier identities are highly dynamic, offline isolation may not capture all problematic elements [2410.05265].
- Overhead: For models with massive input spaces, the per-token routing logic and scale/bias updates may impose nontrivial inference cost.
- Calibration: While calibration-free switching is enabled in some frameworks (MoBiQuant), others require per-configuration grid search or block-wise finetuning to optimally set static quantization parameters.

Generalizations include extension to other axes (time, batch, etc.), and across modalities (tile-adaptive quantization of vision-layer patches; channel-adaptive quantization for multi-head attention). Several methods directly support both token-adaptive (sequence) and tile-adaptive (spatial/image) regimes [2602.24059], [2506.01352].

In summary, dynamic token/tile adaptive quantization represents a confluence of input-aware resource allocation, hierarchical or MoE compensation, and context-sensitive error control, supporting robust low-bit inference and training with minimal accuracy loss across a spectrum of neural architectures and modalities [2602.20191], [2506.01352], [2410.05265], [1802.02629], [2602.24059], [2405.14256].

Source: https://www.emergentmind.com/topics/dynamic-token-tile-adaptive-quantization