High-Granularity Quantization

• High-granularity quantization is a technique that assigns finer precision units—such as per-channel or per-token scales—to reduce quantization error compared to conventional layer-wise methods.
• It encompasses diverse methods including per-channel activation scaling, token-level calibration, patch-wise mixed precision, and graph-structured approaches to tailor quantization to local data dynamics.
• Empirical studies show that finer granularity improves trade-offs between accuracy and efficiency, though the optimal approach depends on model architecture, data characteristics, and hardware constraints.

High-granularity quantization denotes quantization schemes that reduce the scope over which numerical parameters are shared, or increase the resolution at which precision, calibration, or reconstruction is assigned. In recent work, this idea appears in several technically distinct forms: per-channel activation scaling in zero-shot quantization, token-level calibration in large vision-LLMs, block-level mixed-precision assignment in LLMs, per-weight and per-activation bitwidth learning for on-chip deployment, mixed-granularity reconstruction in post-training quantization for vision transformers, and module-level transition granularity in elastic deployment systems (Hong et al., 24 Mar 2025, Xiang et al., 18 Mar 2026, Hooper et al., 19 Apr 2025, Sun et al., 2024, Yang et al., 2024, Chai et al., 13 Jan 2025). The common objective is to localize quantization error, sensitivity estimation, or resource allocation more precisely than coarse layer-wise or per-tensor schemes.

1. Definitions and scope

In recent literature, granularity refers to the unit over which a quantization-related decision is shared. In the most conventional sense, it is the scale-sharing domain: one scale and zero-point for a whole tensor, one per channel, or one per smaller group. In other work, it is the unit of sensitivity attribution, the unit of precision assignment, the unit of reconstruction supervision, or the step size between adjacent deployable models (Hong et al., 24 Mar 2025, Xiang et al., 18 Mar 2026, Yang et al., 2024, Chai et al., 13 Jan 2025).

Context	Granularity unit	What is shared or assigned
Activation quantization	Tensor, channel, group, head, token	Scale and zero-point
Mixed-precision PTQ/QAT	Layer, block, weight, activation	Bitwidth or datatype
LVLM calibration	Modality or token	Sensitivity weight
ViT reconstruction	Global, block, intra-block layer	Supervision loss
Elastic deployment	Module	Memory-footprint transition step

The scale-sharing interpretation is explicit in "GranQ" (Hong et al., 24 Mar 2025). There, per-tensor quantization uses one scale and zero-point for all channels in a layer, whereas per-channel quantization assigns each channel its own parameters. The paper also notes finer variants such as per-group, per-head, or per-token, with higher parameter and compute overhead. In the scaling-law analysis for quantization-aware training, granularity is formalized as group size $G$, the number of elements sharing a single scale or clipping parameter; smaller $G$ means higher granularity (Chen et al., 20 May 2025).

Other papers broaden the term. "Fine-Grained Post-Training Quantization for Large Vision LLMs with Quantization-Aware Integrated Gradients" defines granularity at the calibration level: modality-level sensitivity versus token-level sensitivity (Xiang et al., 18 Mar 2026). "MGRQ" uses granularity to describe reconstruction supervision at global, block, and intra-block levels (Yang et al., 2024). "FlexQuant" defines transition granularity as the maximum memory-footprint difference between adjacent deployable models in an elastic ensemble (Chai et al., 13 Jan 2025). "HGQ" pushes precision assignment to per-weight and per-activation resolution (Sun et al., 2024).

2. Error mechanisms and quantitative laws

The central rationale for high granularity is that coarse sharing enlarges quantization step size in the presence of heterogeneous dynamic ranges. In low-bit regimes, if a single layer scale must cover outlier channels, the scale $s$ becomes large and small changes in activations are lost; per-channel scaling shrinks ranges and reduces activation distortion (Hong et al., 24 Mar 2025). For activations $x_l \in \mathbb{R}^{C\times H\times W}$, GranQ computes

$$\vec{x}_{\min}[c] = \min_{h,w} x_l[c,h,w],\quad \vec{x}_{\max}[c] = \max_{h,w} x_l[c,h,w],$$

then

$$\vec{s}[c] = \frac{\vec{x}_{\max}[c] - \vec{x}_{\min}[c]}{2^b - 1},\quad \vec{z}[c] = \left\lfloor -\frac{\vec{x}_{\min}[c]}{\vec{s}[c]} \right\rceil.$$

The reported activation-fidelity metrics on ResNet-20 show the same trend numerically: layer-wise quantization gives cosine similarity $0.5111$ and relative error $0.3129$, channel-wise gives $0.5602$ and $0.2528$, and GranQ gives $G$0 and $G$1 (Hong et al., 24 Mar 2025).

A complementary formulation appears in FGMP, which models the expected loss increase from quantization perturbations with a diagonal Fisher approximation. For a block $G$2 of size $G$3, the block impact score is

$G$4

where $G$5 is the per-element Fisher weight and $G$6 is the incremental error from using low precision instead of high precision. This makes high granularity valuable when sensitive values and outliers are unstructured and scattered across tensors, because small blocks can be selectively retained in higher precision (Hooper et al., 19 Apr 2025).

Scaling-law work makes the dependence on granularity explicit. For W4A4 QAT, the unified error law is

$G$7

with fitted parameters $G$8, $G$9, $s$0, and $s$1 for W4A4 (Chen et al., 20 May 2025). The same paper decomposes W4A4 error into weight and activation components and reports that activation error is especially sensitive to $s$2, with $s$3 for W16A4. It further identifies FC2 input activation outliers as the primary bottleneck and shows that keeping the FC2 input at 8-bit reduces the activation-side granularity sensitivity (Chen et al., 20 May 2025).

A related PTQ study for mixed quantization writes degradation as a power law in block size,

$s$4

and reports that larger models are less sensitive to small $s$5 than smaller ones at the same quantization ratio $s$6 (Cao et al., 2024). This suggests that granularity is not a universal constant of the method alone; it interacts with model scale, data scale, and the specific source of quantization error.

3. Granularities in scale sharing and calibration

Per-channel activation scaling. GranQ operationalizes high granularity by assigning per-channel activation scales and zero-points for every layer and vectorizing the scaling operation itself, not only the quantization operator. The method forms channel vectors of scales and zero-points, broadcasts them across spatial positions, and uses a single fused tensor kernel for scale application, rounding, clipping, and dequantization. All arithmetic is performed via tensor broadcasting, with no per-channel scalar loops (Hong et al., 24 Mar 2025). This is specifically motivated by zero-shot quantization, where synthetic inputs may contain outliers that distort layer-wise activation ranges.

Token-level calibration. QIG moves calibration in large vision-LLMs from modality level to token level. It defines Quantization-aware Integrated Gradients as

$s$7

then applies IQR clipping and normalization to obtain token importances $s$8. Those sensitivities reweight token reconstruction errors in channel-wise equalization, or the GPTQ Hessian through $s$9 (Xiang et al., 18 Mar 2026). The paper uses 128 image-caption pairs and an $x_l \in \mathbb{R}^{C\times H\times W}$0-step Riemann approximation.

Patch-wise, layer-invariant dynamic quantization. Granular-DQ assigns activation bitwidths patch-wise rather than layer-wise for image super-resolution. A granularity-bit controller analyzes multi-granularity patch representations and assigns preliminary bits from $x_l \in \mathbb{R}^{C\times H\times W}$1, then an entropy-to-bit mechanism refines high-bit patches using $x_l \in \mathbb{R}^{C\times H\times W}$2 (Wang et al., 2024). A defining property is layer invariance: the same patch-specific bit code is used across all quantized layers for that patch, rather than reselecting per layer.

Graph-structured granularity. SGQuant applies quantization at three levels in GNNs: component-level, topology-level, and layer-level. Attention values and embeddings can use different bitwidths, low-degree and high-degree nodes can use different bitwidths, and each layer can use its own bitwidth. Crucially, weights remain full precision, and differently quantized quantities are rematched to float32 immediately before aggregation and combination (Feng et al., 2020). Automatic Bit Selecting uses a regression-tree cost model with $x_l \in \mathbb{R}^{C\times H\times W}$3, $x_l \in \mathbb{R}^{C\times H\times W}$4, and $x_l \in \mathbb{R}^{C\times H\times W}$5 to search bit configurations.

These cases illustrate that high granularity is not restricted to a single operator class. It can target activation ranges, token sensitivities, local image regions, or graph-structural strata, provided the grouping aligns with the dominant source of quantization error.

4. Learnable mixed precision, reconstruction, and elastic deployment

Differentiable per-layer mixed precision. WaveQ introduces a sinusoidal adaptive regularizer,

$x_l \in \mathbb{R}^{C\times H\times W}$6

so that per-layer bitwidth proxies $x_l \in \mathbb{R}^{C\times H\times W}$7, scales, and the quantization lattice are learned jointly with model weights (Elthakeb et al., 2020). The minima occur at $x_l \in \mathbb{R}^{C\times H\times W}$8, which makes the lattice itself part of the optimization.

Per-weight and per-activation precision learning. HGQ pushes granularity further by making bitwidths trainable at arbitrary resolution, including per-weight and per-activation. During training it optimizes fractional bits $x_l \in \mathbb{R}^{C\times H\times W}$9 via a straight-through estimator and a surrogate gradient

$$\vec{x}_{\min}[c] = \min_{h,w} x_l[c,h,w],\quad \vec{x}_{\max}[c] = \max_{h,w} x_l[c,h,w],$$0

while integer bits are determined by post-training calibration (Sun et al., 2024). The method regularizes effective bit operations,

$$\vec{x}_{\min}[c] = \min_{h,w} x_l[c,h,w],\quad \vec{x}_{\max}[c] = \max_{h,w} x_l[c,h,w],$$1

to target FPGA resource use.

Block-level mixed precision with hardware co-design. FGMP assigns mixed numeric precision to contiguous 1D blocks of size $$\vec{x}_{\min}[c] = \min_{h,w} x_l[c,h,w],\quad \vec{x}_{\max}[c] = \max_{h,w} x_l[c,h,w],$$2 along the dot-product dimension of weights and activations. Weight-block assignments are computed offline; activation blocks are assigned online by hardware. Blocks with $$\vec{x}_{\min}[c] = \min_{h,w} x_l[c,h,w],\quad \vec{x}_{\max}[c] = \max_{h,w} x_l[c,h,w],$$3 above a global threshold are kept in FP8, while others use NVFP4 with microscaling and sensitivity-weighted clipping (Hooper et al., 19 Apr 2025).

Mixed-granularity reconstruction. In ViT PTQ, MGRQ does not redefine the quantizer itself; it changes the supervisory granularity used to reconstruct the quantized model. The objective for block $$\vec{x}_{\min}[c] = \min_{h,w} x_l[c,h,w],\quad \vec{x}_{\max}[c] = \max_{h,w} x_l[c,h,w],$$4 is

$$\vec{x}_{\min}[c] = \min_{h,w} x_l[c,h,w],\quad \vec{x}_{\max}[c] = \max_{h,w} x_l[c,h,w],$$5

combining optimized block-wise reconstruction, logits-level global supervision, and intra-block layer-wise feature matching (Yang et al., 2024). Here, “high granularity” refers to where the reconstruction signal is applied.

Elastic transition granularity. FlexQuant relocates the concept again: it defines transition granularity as the maximum memory-footprint difference between adjacent models in an ordered elastic ensemble. Hybrid models are constructed by replacing parameters module-by-module from a higher-bit checkpoint with lower-bit equivalents, so the transition step is bounded by the size of a single module (Chai et al., 13 Jan 2025). The search is constrained to one-way transitions to reduce transient memory pressure during hot swaps.

Taken together, these methods show that high granularity can be learned, calibrated, reconstructed, or scheduled. The object of control varies, but the underlying principle is the same: replace coarse uniformity with localized decisions.

5. Empirical behavior across domains

Reported results consistently show that finer granularity improves the accuracy-efficiency trade-off when the target error is highly localized, especially in low-bit settings. The effect is visible in activation-fidelity metrics, accuracy under W4A4 or 3-bit regimes, memory-footprint reductions, and deployment elasticity.

Setting	Granularity intervention	Reported outcome
Zero-shot QAT for ResNet-20	Vectorized per-channel activation scaling	CIFAR-100 3w3a accuracy $$\vec{x}_{\min}[c] = \min_{h,w} x_l[c,h,w],\quad \vec{x}_{\max}[c] = \max_{h,w} x_l[c,h,w],$$6; cosine similarity $$\vec{x}_{\min}[c] = \min_{h,w} x_l[c,h,w],\quad \vec{x}_{\max}[c] = \max_{h,w} x_l[c,h,w],$$7; relative error $$\vec{x}_{\min}[c] = \min_{h,w} x_l[c,h,w],\quad \vec{x}_{\max}[c] = \max_{h,w} x_l[c,h,w],$$8 (Hong et al., 24 Mar 2025)
LVLM PTQ	Token-level sensitivity via QIG	LLaVA-onevision-7B W3A16 average accuracy $$\vec{x}_{\min}[c] = \min_{h,w} x_l[c,h,w],\quad \vec{x}_{\max}[c] = \max_{h,w} x_l[c,h,w],$$9 vs $$\vec{s}[c] = \frac{\vec{x}_{\max}[c] - \vec{x}_{\min}[c]}{2^b - 1},\quad \vec{z}[c] = \left\lfloor -\frac{\vec{x}_{\min}[c]}{\vec{s}[c]} \right\rceil.$$0 for MBQ, a $$\vec{s}[c] = \frac{\vec{x}_{\max}[c] - \vec{x}_{\min}[c]}{2^b - 1},\quad \vec{z}[c] = \left\lfloor -\frac{\vec{x}_{\min}[c]}{\vec{s}[c]} \right\rceil.$$1 gain (Xiang et al., 18 Mar 2026)
ViT PTQ	Global + block + intra-block reconstruction	ViT-B W4/A4 top-1 $$\vec{s}[c] = \frac{\vec{x}_{\max}[c] - \vec{x}_{\min}[c]}{2^b - 1},\quad \vec{z}[c] = \left\lfloor -\frac{\vec{x}_{\min}[c]}{\vec{s}[c]} \right\rceil.$$2; RepQ-ViT at W4/A4 reports $$\vec{s}[c] = \frac{\vec{x}_{\max}[c] - \vec{x}_{\min}[c]}{2^b - 1},\quad \vec{z}[c] = \left\lfloor -\frac{\vec{x}_{\min}[c]}{\vec{s}[c]} \right\rceil.$$3 (Yang et al., 2024)
Image super-resolution	Patch-wise, layer-invariant dynamic quantization	EDSR ×4 on Urban100: FAB $$\vec{s}[c] = \frac{\vec{x}_{\max}[c] - \vec{x}_{\min}[c]}{2^b - 1},\quad \vec{z}[c] = \left\lfloor -\frac{\vec{x}_{\min}[c]}{\vec{s}[c]} \right\rceil.$$4, PSNR $$\vec{s}[c] = \frac{\vec{x}_{\max}[c] - \vec{x}_{\min}[c]}{2^b - 1},\quad \vec{z}[c] = \left\lfloor -\frac{\vec{x}_{\min}[c]}{\vec{s}[c]} \right\rceil.$$5, SSIM $$\vec{s}[c] = \frac{\vec{x}_{\max}[c] - \vec{x}_{\min}[c]}{2^b - 1},\quad \vec{z}[c] = \left\lfloor -\frac{\vec{x}_{\min}[c]}{\vec{s}[c]} \right\rceil.$$6 (Wang et al., 2024)
GNN quantization	Component + topology + layer granularity	Memory reduction from $$\vec{s}[c] = \frac{\vec{x}_{\max}[c] - \vec{x}_{\min}[c]}{2^b - 1},\quad \vec{z}[c] = \left\lfloor -\frac{\vec{x}_{\min}[c]}{\vec{s}[c]} \right\rceil.$$7 to $$\vec{s}[c] = \frac{\vec{x}_{\max}[c] - \vec{x}_{\min}[c]}{2^b - 1},\quad \vec{z}[c] = \left\lfloor -\frac{\vec{x}_{\min}[c]}{\vec{s}[c]} \right\rceil.$$8 with accuracy drop about $$\vec{s}[c] = \frac{\vec{x}_{\max}[c] - \vec{x}_{\min}[c]}{2^b - 1},\quad \vec{z}[c] = \left\lfloor -\frac{\vec{x}_{\min}[c]}{\vec{s}[c]} \right\rceil.$$9 on average (Feng et al., 2020)
LLM mixed precision	Block-level FP8/NVFP4 assignment	Llama-2-7B: $0.5111$0 perplexity degradation, $0.5111$1 less energy, $0.5111$2 less weight memory (Hooper et al., 19 Apr 2025)
Elastic LLM deployment	Module-level transition granularity	$0.5111$3 granularity improvement and $0.5111$4 storage reduction (Chai et al., 13 Jan 2025)

Additional studies reinforce the same pattern. WaveQ learns heterogeneous per-layer bitwidths in the range 2–8 bits with average learned bitwidths of approximately 3.85 for AlexNet, 3.57 for ResNet-18, and 3.95 for MobileNet, while reporting about 4.8% average accuracy improvements when incorporated into DoReFa and WRPN at fixed 3- to 5-bit settings (Elthakeb et al., 2020). HGQ reports up to a factor of 20 resource reduction and a factor of 5 latency improvement while preserving accuracy on FPGA-oriented deployments (Sun et al., 2024). In W4A4 QAT, the scaling-law study reports that quantization error decreases by 34% on average when model size increases from 74M to 594M, rises by 22% on average when training tokens increase from 10B to 100B, and differs by 0.037 loss between the coarsest and finest tested granularities (Chen et al., 20 May 2025).

These results do not imply that the best granularity is always the finest available. They show, rather, that the optimal granularity depends on where the dominant error source resides: channel outliers, token sensitivity, patch complexity, graph degree, block-local Fisher sensitivity, or deployment-time memory fluctuations.

6. Systems implications, misconceptions, and future directions

A recurrent misconception is that high granularity always means per-element numeric scales. The literature is broader. In some papers it means per-channel or per-token scales; in others it means per-weight mixed precision, reconstruction supervision at multiple levels, or module-level deployment elasticity (Hong et al., 24 Mar 2025, Sun et al., 2024, Yang et al., 2024, Chai et al., 13 Jan 2025). Another misconception is that finer granularity is uniformly superior. The mixed-quantization scaling law shows that very small block sizes are not essential for large models, and may unnecessarily complicate hardware design (Cao et al., 2024). The W4A4 QAT scaling law shows that the benefit of finer grouping is much stronger for activations than for weights, and that the FC2 input in SwiGLU-based models is a specific activation bottleneck because of heavy-tailed outliers (Chen et al., 20 May 2025).

The systems trade-off is explicit in nearly every method. GranQ stores two length-$0.5111$5 vectors per layer for scales and zero-points and keeps latency close to layer-wise quantization by vectorizing the scaling step (Hong et al., 24 Mar 2025). FGMP uses one precision bit per block and an FP8 microscale for NVFP4 blocks, which yields an average per-block storage of 73 bits for NVFP4 versus 129 bits for FP8 blocks, but requires hardware support for block-granularity mixed datapaths and an online post-processing unit (Hooper et al., 19 Apr 2025). FlexQuant reduces transition steps to approximately 100 MB and relies on one-way module swaps to limit peak unified-memory pressure, but its search and storage strategy remain calibration- and checkpoint-dependent (Chai et al., 13 Jan 2025).

Several limitations recur. Zero-shot methods remain dependent on synthetic-data quality; GranQ mitigates activation distortion but still benefits from better synthetic distributions (Hong et al., 24 Mar 2025). Token-level PTQ requires backpropagation through a small calibration set and has not been evaluated at extremely low precisions such as 2-bit (Xiang et al., 18 Mar 2026). QAT scaling results are reported for dense Llama3-style transformers rather than MoE models (Chen et al., 20 May 2025). Fine-grained hardware schemes assume support for mixed datapaths, arbitrary-bit arithmetic, or fused per-channel kernels (Hooper et al., 19 Apr 2025, Sun et al., 2024).

Current research directions are correspondingly hybrid. Proposed extensions include token × channel or head-level sensitivity for LVLMs, mixed-precision allocation driven by token importance, dedicated kernels that fuse per-channel broadcast with rounding and clipping, joint optimization of quantization and token pruning, and scaling-law-guided selection of group size and mixed precision (Xiang et al., 18 Mar 2026, Hong et al., 24 Mar 2025, Chen et al., 20 May 2025). Outside mainstream neural compression, the term also appears with different semantics: multi-granularity vector quantization in speech enhancement uses multiple codebooks at multiple hierarchy points to extract local and global discrete speech features, while reconfigurable reflectarray radar studies use spatial and phase quantization to reduce beam granularity and improve angular resolution (Zhao et al., 2023, Zong et al., 21 May 2025). This suggests that high-granularity quantization has become a general design pattern: identify the unit at which error, information density, or control sensitivity is genuinely nonuniform, and place the quantizer there rather than at a coarser proxy.