---
title: High-Granularity Quantization
url: https://www.emergentmind.com/topics/high-granularity-quantization
type: topic
---

# High-Granularity Quantization

High-granularity quantization denotes quantization schemes that reduce the scope over which numerical parameters are shared, or increase the resolution at which precision, calibration, or reconstruction is assigned. In recent work, this idea appears in several technically distinct forms: per-channel activation scaling in zero-shot quantization, token-level calibration in large vision-language models, block-level mixed-precision assignment in large language models, per-weight and per-activation bitwidth learning for on-chip deployment, mixed-granularity reconstruction in post-training quantization for vision transformers, and module-level transition granularity in elastic deployment systems [2503.18339], [2603.17809], [2504.14152], [2405.00645], [2406.09229], [2501.07139]. The common objective is to localize quantization error, sensitivity estimation, or resource allocation more precisely than coarse layer-wise or per-tensor schemes.

## 1. Definitions and scope

In recent literature, *granularity* refers to the unit over which a quantization-related decision is shared. In the most conventional sense, it is the scale-sharing domain: one scale and zero-point for a whole tensor, one per channel, or one per smaller group. In other work, it is the unit of sensitivity attribution, the unit of precision assignment, the unit of reconstruction supervision, or the step size between adjacent deployable models [2503.18339], [2603.17809], [2406.09229], [2501.07139].

| Context | Granularity unit | What is shared or assigned |
|---|---|---|
| Activation quantization | Tensor, channel, group, head, token | Scale and zero-point |
| Mixed-precision PTQ/QAT | Layer, block, weight, activation | Bitwidth or datatype |
| LVLM calibration | Modality or token | Sensitivity weight |
| ViT reconstruction | Global, block, intra-block layer | Supervision loss |
| Elastic deployment | Module | Memory-footprint transition step |

The scale-sharing interpretation is explicit in "GranQ" [2503.18339]. There, per-tensor quantization uses one scale and zero-point for all channels in a layer, whereas per-channel quantization assigns each channel its own parameters. The paper also notes finer variants such as per-group, per-head, or per-token, with higher parameter and compute overhead. In the scaling-law analysis for quantization-aware training, granularity is formalized as group size $G$, the number of elements sharing a single scale or clipping parameter; smaller $G$ means higher granularity [2505.14302].

Other papers broaden the term. "Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients" defines granularity at the calibration level: modality-level sensitivity versus token-level sensitivity [2603.17809]. "MGRQ" uses granularity to describe reconstruction supervision at global, block, and intra-block levels [2406.09229]. "FlexQuant" defines transition granularity as the maximum memory-footprint difference between adjacent deployable models in an elastic ensemble [2501.07139]. "HGQ" pushes precision assignment to per-weight and per-activation resolution [2405.00645].

## 2. Error mechanisms and quantitative laws

The central rationale for high granularity is that coarse sharing enlarges quantization step size in the presence of heterogeneous dynamic ranges. In low-bit regimes, if a single layer scale must cover outlier channels, the scale $s$ becomes large and small changes in activations are lost; per-channel scaling shrinks ranges and reduces activation distortion [2503.18339]. For activations $x_l \in \mathbb{R}^{C\times H\times W}$, GranQ computes
$$
\vec{x}_{\min}[c] = \min_{h,w} x_l[c,h,w],\quad
\vec{x}_{\max}[c] = \max_{h,w} x_l[c,h,w],
$$
then
$$
\vec{s}[c] = \frac{\vec{x}_{\max}[c] - \vec{x}_{\min}[c]}{2^b - 1},\quad
\vec{z}[c] = \left\lfloor -\frac{\vec{x}_{\min}[c]}{\vec{s}[c]} \right\rceil.
$$
The reported activation-fidelity metrics on ResNet-20 show the same trend numerically: layer-wise quantization gives cosine similarity $0.5111$ and relative error $0.3129$, channel-wise gives $0.5602$ and $0.2528$, and GranQ gives $0.6835$ and $0.1063$ [2503.18339].

A complementary formulation appears in FGMP, which models the expected loss increase from quantization perturbations with a diagonal Fisher approximation. For a block $b$ of size $N$, the block impact score is
$$
\delta_b = \sum_{i=1}^{N} g_i^2 \big(\Delta_{p_h \to p_l} v_i\big)^2,
$$
where $g_i^2$ is the per-element Fisher weight and $\Delta_{p_h \to p_l} v_i$ is the incremental error from using low precision instead of high precision. This makes high granularity valuable when sensitive values and outliers are unstructured and scattered across tensors, because small blocks can be selectively retained in higher precision [2504.14152].

Scaling-law work makes the dependence on granularity explicit. For W4A4 QAT, the unified error law is
$$
\delta_p(N,D,G) = \frac{k \cdot D^{\gamma_D} \cdot (\log_2 G)^{\gamma_G}}{N^{\gamma_N}},
$$
with fitted parameters $k=0.1582$, $\gamma_N=0.2186$, $\gamma_D=0.0745$, and $\gamma_G=0.7779$ for W4A4 [2505.14302]. The same paper decomposes W4A4 error into weight and activation components and reports that activation error is especially sensitive to $G$, with $\gamma_G=0.9812$ for W16A4. It further identifies FC2 input activation outliers as the primary bottleneck and shows that keeping the FC2 input at 8-bit reduces the activation-side granularity sensitivity [2505.14302].

A related PTQ study for mixed quantization writes degradation as a power law in block size,
$$
\Delta \mathrm{PPL}_{\%} \approx A(P,Q_r)\, Q_b^{\kappa(P,Q_r)},
$$
and reports that larger models are less sensitive to small $Q_b$ than smaller ones at the same quantization ratio $Q_r$ [2410.06722]. This suggests that granularity is not a universal constant of the method alone; it interacts with model scale, data scale, and the specific source of quantization error.

## 3. Granularities in scale sharing and calibration

**Per-channel activation scaling.** GranQ operationalizes high granularity by assigning per-channel activation scales and zero-points for every layer and vectorizing the scaling operation itself, not only the quantization operator. The method forms channel vectors of scales and zero-points, broadcasts them across spatial positions, and uses a single fused tensor kernel for scale application, rounding, clipping, and dequantization. All arithmetic is performed via tensor broadcasting, with no per-channel scalar loops [2503.18339]. This is specifically motivated by zero-shot quantization, where synthetic inputs may contain outliers that distort layer-wise activation ranges.

**Token-level calibration.** QIG moves calibration in large vision-language models from modality level to token level. It defines Quantization-aware Integrated Gradients as
$$
\mathrm{QIG}(x) = (x - x^q)\int_0^1 \frac{\partial (f(x_\alpha,w)-f(x_\alpha,w^q))}{\partial x_\alpha} d\alpha,
\quad x_\alpha = x^q + \alpha(x-x^q),
$$
then applies IQR clipping and normalization to obtain token importances $\lambda_i$. Those sensitivities reweight token reconstruction errors in channel-wise equalization, or the GPTQ Hessian through $H' = X^\top \Lambda X$ [2603.17809]. The paper uses 128 image-caption pairs and an $m=32$-step Riemann approximation.

**Patch-wise, layer-invariant dynamic quantization.** Granular-DQ assigns activation bitwidths patch-wise rather than layer-wise for image super-resolution. A granularity-bit controller analyzes multi-granularity patch representations and assigns preliminary bits from $\{4,6,8\}$, then an entropy-to-bit mechanism refines high-bit patches using $\{4,5,8\}$ [2409.14330]. A defining property is layer invariance: the same patch-specific bit code is used across all quantized layers for that patch, rather than reselecting per layer.

**Graph-structured granularity.** SGQuant applies quantization at three levels in GNNs: component-level, topology-level, and layer-level. Attention values and embeddings can use different bitwidths, low-degree and high-degree nodes can use different bitwidths, and each layer can use its own bitwidth. Crucially, weights remain full precision, and differently quantized quantities are rematched to float32 immediately before aggregation and combination [2007.05100]. Automatic Bit Selecting uses a regression-tree cost model with $N_{\text{mea}}=40$, $N_{\text{iter}}=5$, and $N_{\text{sample}}=2000$ to search bit configurations.

These cases illustrate that high granularity is not restricted to a single operator class. It can target activation ranges, token sensitivities, local image regions, or graph-structural strata, provided the grouping aligns with the dominant source of quantization error.

## 4. Learnable mixed precision, reconstruction, and elastic deployment

**Differentiable per-layer mixed precision.** WaveQ introduces a sinusoidal adaptive regularizer,
$$
L_{\text{total}}(W,\beta)=E_0(W)+
\lambda_w \sum_i \sum_j \frac{\sin^2\!\big(\pi w_{ij}(2^{\beta_i}-1)\big)}{2^{\beta_i}}
+\lambda_\beta \sum_i \beta_i,
$$
so that per-layer bitwidth proxies $\beta_i$, scales, and the quantization lattice are learned jointly with model weights [2003.00146]. The minima occur at $w_{ij}=n/(2^{\beta_i}-1)$, which makes the lattice itself part of the optimization.

**Per-weight and per-activation precision learning.** HGQ pushes granularity further by making bitwidths trainable at arbitrary resolution, including per-weight and per-activation. During training it optimizes fractional bits $f$ via a straight-through estimator and a surrogate gradient
$$
\frac{\partial \delta_f}{\partial f} \leftarrow -\log 2 \cdot \delta_f,
$$
while integer bits are determined by post-training calibration [2405.00645]. The method regularizes effective bit operations,
$$
\mathrm{EBOPs} = \sum_{i,j\in\mathcal{M}} b_i b_j,
$$
to target FPGA resource use.

**Block-level mixed precision with hardware co-design.** FGMP assigns mixed numeric precision to contiguous 1D blocks of size $\mathrm{BS}=16$ along the dot-product dimension of weights and activations. Weight-block assignments are computed offline; activation blocks are assigned online by hardware. Blocks with $\delta_b$ above a global threshold are kept in FP8, while others use NVFP4 with microscaling and sensitivity-weighted clipping [2504.14152].

**Mixed-granularity reconstruction.** In ViT PTQ, MGRQ does not redefine the quantizer itself; it changes the supervisory granularity used to reconstruct the quantized model. The objective for block $l$ is
$$
\mathcal{L}=
\mathcal{L}_{\mathrm{OBWR}^{(l)}}+\alpha \mathcal{L}_{\mathrm{EBGS}}+\beta \mathcal{L}_{\mathrm{IBLS}^{(l)}},
$$
combining optimized block-wise reconstruction, logits-level global supervision, and intra-block layer-wise feature matching [2406.09229]. Here, “high granularity” refers to where the reconstruction signal is applied.

**Elastic transition granularity.** FlexQuant relocates the concept again: it defines transition granularity as the maximum memory-footprint difference between adjacent models in an ordered elastic ensemble. Hybrid models are constructed by replacing parameters module-by-module from a higher-bit checkpoint with lower-bit equivalents, so the transition step is bounded by the size of a single module [2501.07139]. The search is constrained to one-way transitions to reduce transient memory pressure during hot swaps.

Taken together, these methods show that high granularity can be learned, calibrated, reconstructed, or scheduled. The object of control varies, but the underlying principle is the same: replace coarse uniformity with localized decisions.

## 5. Empirical behavior across domains

Reported results consistently show that finer granularity improves the accuracy-efficiency trade-off when the target error is highly localized, especially in low-bit settings. The effect is visible in activation-fidelity metrics, accuracy under W4A4 or 3-bit regimes, memory-footprint reductions, and deployment elasticity.

| Setting | Granularity intervention | Reported outcome |
|---|---|---|
| Zero-shot QAT for ResNet-20 | Vectorized per-channel activation scaling | CIFAR-100 3w3a accuracy $62.73\%$; cosine similarity $0.6835$; relative error $0.1063$ [2503.18339] |
| LVLM PTQ | Token-level sensitivity via QIG | LLaVA-onevision-7B W3A16 average accuracy $72.04\%$ vs $70.44\%$ for MBQ, a $+1.60\%$ gain [2603.17809] |
| ViT PTQ | Global + block + intra-block reconstruction | ViT-B W4/A4 top-1 $75.59\%$; RepQ-ViT at W4/A4 reports $68.48\%$ [2406.09229] |
| Image super-resolution | Patch-wise, layer-invariant dynamic quantization | EDSR ×4 on Urban100: FAB $4.97$, PSNR $26.01$, SSIM $0.784$ [2409.14330] |
| GNN quantization | Component + topology + layer granularity | Memory reduction from $4.25\times$ to $31.9\times$ with accuracy drop about $0.4\%$ on average [2007.05100] |
| LLM mixed precision | Block-level FP8/NVFP4 assignment | Llama-2-7B: $<1\%$ perplexity degradation, $14\%$ less energy, $30\%$ less weight memory [2504.14152] |
| Elastic LLM deployment | Module-level transition granularity | $15\times$ granularity improvement and $10\times$ storage reduction [2501.07139] |

Additional studies reinforce the same pattern. WaveQ learns heterogeneous per-layer bitwidths in the range 2–8 bits with average learned bitwidths of approximately 3.85 for AlexNet, 3.57 for ResNet-18, and 3.95 for MobileNet, while reporting about 4.8% average accuracy improvements when incorporated into DoReFa and WRPN at fixed 3- to 5-bit settings [2003.00146]. HGQ reports up to a factor of 20 resource reduction and a factor of 5 latency improvement while preserving accuracy on FPGA-oriented deployments [2405.00645]. In W4A4 QAT, the scaling-law study reports that quantization error decreases by 34% on average when model size increases from 74M to 594M, rises by 22% on average when training tokens increase from 10B to 100B, and differs by 0.037 loss between the coarsest and finest tested granularities [2505.14302].

These results do not imply that the best granularity is always the finest available. They show, rather, that the optimal granularity depends on where the dominant error source resides: channel outliers, token sensitivity, patch complexity, graph degree, block-local Fisher sensitivity, or deployment-time memory fluctuations.

## 6. Systems implications, misconceptions, and future directions

A recurrent misconception is that high granularity always means per-element numeric scales. The literature is broader. In some papers it means per-channel or per-token scales; in others it means per-weight mixed precision, reconstruction supervision at multiple levels, or module-level deployment elasticity [2503.18339], [2405.00645], [2406.09229], [2501.07139]. Another misconception is that finer granularity is uniformly superior. The mixed-quantization scaling law shows that very small block sizes are not essential for large models, and may unnecessarily complicate hardware design [2410.06722]. The W4A4 QAT scaling law shows that the benefit of finer grouping is much stronger for activations than for weights, and that the FC2 input in SwiGLU-based models is a specific activation bottleneck because of heavy-tailed outliers [2505.14302].

The systems trade-off is explicit in nearly every method. GranQ stores two length-$C$ vectors per layer for scales and zero-points and keeps latency close to layer-wise quantization by vectorizing the scaling step [2503.18339]. FGMP uses one precision bit per block and an FP8 microscale for NVFP4 blocks, which yields an average per-block storage of 73 bits for NVFP4 versus 129 bits for FP8 blocks, but requires hardware support for block-granularity mixed datapaths and an online post-processing unit [2504.14152]. FlexQuant reduces transition steps to approximately 100 MB and relies on one-way module swaps to limit peak unified-memory pressure, but its search and storage strategy remain calibration- and checkpoint-dependent [2501.07139].

Several limitations recur. Zero-shot methods remain dependent on synthetic-data quality; GranQ mitigates activation distortion but still benefits from better synthetic distributions [2503.18339]. Token-level PTQ requires backpropagation through a small calibration set and has not been evaluated at extremely low precisions such as 2-bit [2603.17809]. QAT scaling results are reported for dense Llama3-style transformers rather than MoE models [2505.14302]. Fine-grained hardware schemes assume support for mixed datapaths, arbitrary-bit arithmetic, or fused per-channel kernels [2504.14152], [2405.00645].

Current research directions are correspondingly hybrid. Proposed extensions include token × channel or head-level sensitivity for LVLMs, mixed-precision allocation driven by token importance, dedicated kernels that fuse per-channel broadcast with rounding and clipping, joint optimization of quantization and token pruning, and scaling-law-guided selection of group size and mixed precision [2603.17809], [2503.18339], [2505.14302]. Outside mainstream neural compression, the term also appears with different semantics: multi-granularity vector quantization in speech enhancement uses multiple codebooks at multiple hierarchy points to extract local and global discrete speech features, while reconfigurable reflectarray radar studies use spatial and phase quantization to reduce beam granularity and improve angular resolution [2302.08342], [2505.15029]. This suggests that high-granularity quantization has become a general design pattern: identify the unit at which error, information density, or control sensitivity is genuinely nonuniform, and place the quantizer there rather than at a coarser proxy.

Source: https://www.emergentmind.com/topics/high-granularity-quantization