---
title: Group-wise Quantization Techniques
url: https://www.emergentmind.com/topics/group-wise-quantization
type: topic
---

# Group-wise Quantization Techniques

Group-wise quantization is a quantization paradigm in which a weight or activation tensor is partitioned into multiple non-overlapping groups, and each group is quantized independently—typically with its own scale and/or zero-point. This fine-grained approach sharply reduces quantization error compared to layer-wise or channel-wise quantization, especially under low-bit regimes, and has become the prevailing method for quantizing large neural networks such as diffusion models, large language models (LLMs), and vision transformers, supporting both high-fidelity and efficient inference [2412.19867].

## 1. Mathematical Formulation and Core Principles

Given any vectorized tensor $X\in\mathbb{R}^N$ (weight or activation), group-wise quantization partitions it into $M$ groups of size $G$:
\[
X = \bigl[ X^{(1)},\, X^{(2)},\,\dots,\,X^{(M)} \bigr],\quad X^{(m)}\in\mathbb{R}^G,\, M=N/G.
\]
For each group, a scale $s_m$ and (optionally) zero-point $z_m$ are adaptively computed:
\[
s_m = \frac{\max_i x_i - \min_i x_i}{q_{\max} - q_{\min}},\quad z_m = \min_i x_i.
\]
The quantization and dequantization equations per group are
\[
x^{(m)}_{\mathrm{int},i} = \mathrm{clamp}\left(\mathrm{round}\left(\frac{x_i-z_m}{s_m}\right), q_{\min}, q_{\max}\right),
\]
\[
\hat{x}_i = s_m\, x^{(m)}_{\mathrm{int},i} + z_m.
\]
This quantization is performed for each group independently; scale granularity can be per group, per block, or even adaptive across unstructured subsets as in recent binary quantization [2509.03054].

Unlike traditional channel-wise (per-column) or layer-wise (whole tensor) quantization, group-wise schemes capitalize on the reduced intra-group variance, leading to better quantization error/accuracy trade-offs, particularly in low-precision settings [2412.19867, 2604.18556, 2502.05376].

## 2. Implementation Methodologies

### Tensor Partitioning

Group-wise quantization is highly configurable in how groups are defined:
- **Contiguous blocks**: Most methods flatten tensors and partition into consecutive groups aligned to hardware SIMD widths (e.g., 32/64/128 elements). This maximizes vectorized kernel efficiency [2412.19867, 2604.18556].
- **Per-kernel groups**: In convolutional layers, a group may correspond to each output-channel kernel [1902.05690], capturing kernel-specific statistics.
- **Block and cluster**: BCQ [2502.05376] slices tensors into small blocks, then clusters blocks across the tensor, applying optimized codebooks per cluster.
- **Unstructured or adaptive grouping**: Recent works such as [2509.03054] introduce algorithms that adaptively partition (possibly non-contiguous) entries into groups based on statistical similarity.

### Scale/Zero-point Computation

Quantization parameters are derived by solving per-group objectives, typically minimizing MSE between the original and quantized weights/activations. For group $m$:
\[
s_m^*, z_m^* = \arg\min_{s,z} \| X^{(m)} - s Q(X^{(m)}, s, z) - z \|_2^2.
\]
Advanced methods further refine these via two-stage or coordinate descent optimizations that account for inter-group correlations and input statistics [2602.02126], or use post-training stochastic relaxation (e.g., Gumbel-Softmax in GSQ [2604.18556]).

### Hardware Alignment

Groups are chosen to align with inference kernel memory layouts, ensuring low execution overhead when switching between scale factors. For example, group sizes of 32, 64, or 128 are matched to the vector units of CPUs/GPUs [2412.19867].

In specialized settings, such as quantized Winograd convolution, only the scales of transform matrices are learned, preserving domain-specific invariants [2412.19867].

## 3. Domain-Specific Applications

### Large Language Models (LLMs)

For LLMs, group-wise quantization (often with group sizes 32–128) has become the dominant approach in W4A4, INT3, or even binary regimes [2604.18556, 2602.02126, 2310.04836, 2502.05376]. Recent advances include:
- **Per-group adaptive data types**: M-ANT [2502.18755] introduces a parameterized numeral system per group, adaptively interpolating between grid types (e.g., uniform, power-of-two, flint) to match local value distributions.
- **Block clustering**: BCQ [2502.05376] clusters blocks by similarity, learning per-cluster codebooks, which yields state-of-the-art compression/accuracy at W4A4.
- **Dynamic unstructured grouping**: Binary quantization [2509.03054] sorts weights and adaptively assigns groups minimizing within-group variance, enabling nearly lossless 1-bit quantization.

### Vision and Diffusion Models

In large vision or diffusion models, group-wise quantization is essential for handling heavy-tailed distributions and outliers—channel-wise or layer-wise quantizers severely degrade output quality. DGQ [2501.04304] and GDRQ [1908.01477] dynamically assign groups per channel or pixel dimension, with scales matched to the local value spread to faithfully preserve extremes affecting perceptual fidelity.

### Medical Foundation Models

Permutation-COMQ [2604.07674] further enhances per-channel quantization by permuting weights prior to quantization so that each group (column) comprises weights of similar magnitude, reducing the impact of outliers on scale selection and benefiting medical segmentation tasks.

### Transformers & Vision Transformers

Instance-aware group quantization [2404.00928] dynamically clusters activation channels on a per-instance basis and applies group-specific quantization, greatly improving quantized ViT performance under 4/4-bit constraints.

## 4. Optimization Algorithms

The methods for finding optimal groupings and quantization parameters have advanced significantly:
- **Greedy, dynamic, and windowed grouping**: Algorithms span from dynamic programming (exact, cubic complexity) to windowed greedy merge (efficient, scalable) for adaptively partitioning tensors into low-variance groups [2509.03054].
- **Clustered assignment**: Blocks are clustered via K-means or Lloyd-Max to maximize codebook utility (BCQ [2502.05376]).
- **Gumbel-Softmax relaxation**: In GSQ [2604.18556], per-group scales and per-weight assignments are learned by differentiable, noise-annealed softmax relaxation of the discrete grid, then collapsed to hard assignments.
- **Two-stage coordinate descent/refinement**: [2602.02126] proposes a two-phase algorithm: initialization using group-wise input statistics, followed by closed-form coordinate-descent refinement using the full block Hessian to minimize true layer-wise loss.

## 5. Comparative Empirical Performance

Group-wise quantization consistently narrows the performance gap to full-precision baselines at low bit-widths:
- **LLMs (2–4 bits)**: M-ANT [2502.18755] at group size 64 achieves <0.2 PPL loss on LLaMA-1/2, with 2–4× throughput/energy improvement over fixed-type or INT-only schemes, and GSQ [2604.18556] achieves 4–5 point accuracy gains over prior scalar quantizers at 2b.
- **Diffusion models**: Fully quantized Winograd convolution with group-wise scales [2412.19867] achieves FID and CLIP scores within 1–4 points of FP16, outperforming all prior Winograd quantization.
- **Computer vision**: GDRQ [1908.01477] and DGQ [2501.04304] maintain accuracy within 1% of float for classification/detection, and group quantization is indispensable for handling rare but impactful activation outliers.

A representative summary of empirical gains:

| Method / Paper         | Architecture                   | Setting   | Metric                | Gap to FP16/32 / Notes              |
|----------------------- |-------------------------------|-----------|-----------------------|-------------------------------------|
| GSQ [2604.18556]       | LLaMA-8B/70B                  | 2b/3b     | Zero-shot accuracy    | 2b GSQ: +4.8p over GPTQ; Δ≈1–2% FP  |
| BCQ [2502.05376]       | LLaMA2-70B                    | W4A4      | Perplexity            | Δ=0.09, accuracy loss <1%           |
| DGQ [2501.04304]       | Stable Diffusion, UNet        | 8b W/A    | FID, CLIP             | FID gap −1.3, CLIP gap −0.001       |
| M-ANT [2502.18755]     | LLaMA-1/2/OPT                 | INT4      | PPL                   | <0.2 PPL loss, up to 4x speedup     |
| Permutation-COMQ [2604.07674] | MedSAM (ViT-B)          | 2b/4b W   | DSC/NSD (segmentation)| DSC: −0.07 at 4b; near-float at 2b  |

## 6. Extensions and Theoretical Considerations

### Outlier Handling and Distribution Reshaping

Group-wise quantization schemes often integrate outlier detection and distribution reshaping:
- **Activation/weight outlier preservation**: Cluster-based grouping dynamically isolates outlier vectors, preventing over-compression of extreme values that can compromise output fidelity (DGQ [2501.04304], GDRQ [1908.01477]).
- **Distribution reshaping**: Scale-Clip in GDRQ drives each group toward a uniform distribution (by adaptive clipping), making quantization more robust at low bits.

### Mixed-Precision and Adaptive Types

Emerging work explores per-group adaptive numeric types beyond fixed INT grids (e.g., M-ANT’s parametric polynomial grid [2502.18755]). Methods also allow mixed-precision assignment per group, finding layer- and group-specific bit-widths to further optimize the accuracy/compression trade-off [1902.05690].

### Dynamic/Instance-Aware Grouping

For architectures with highly instance-dependent activation distributions (e.g., ViTs), online grouping and group-size adaptation per input have been shown to substantially outperform static grouping [2404.00928].

## 7. Practical Implications and Limitations

Group-wise quantization is now standard in high-performance model deployment pipelines, offering:
- Enhanced accuracy at low bit-width with negligible additional inference cost when group sizes align with hardware data paths [2412.19867, 2604.18556].
- Robustness to outlier-induced quantization artifacts in both weights and activations [2501.04304].
- High compatibility with modern PTQ workflows, including efficient calibration and minimal code change from channel-wise baselines [2512.01546, 2604.18556].
- Empirically, the total storage overhead for per-group metadata is marginal (e.g., ~0.007 bits/weight in WGM [2509.03054]).

Limitations include slight increases in parameter storage and kernel complexity when group sizes are very small, and the need for hardware support to efficiently handle scales and potential type heterogeneity. *This suggests* that further research on mixed-type and group-wise fused compute kernels could unlock even greater benefits.

In summary, group-wise quantization delivers order-of-magnitude improvements in quantized model fidelity in deep learning, especially in low-precision and resource-constrained domains, and has reshaped the technical landscape of both training-free and hardware-optimized compression pipelines across domains [2412.19867, 2604.18556, 2602.02126, 2502.05376, 2509.03054].

Source: https://www.emergentmind.com/topics/group-wise-quantization