---
title: Group-wise Weight Quantization
url: https://www.emergentmind.com/topics/group-wise-weight-quantization
type: topic
---

# Group-wise Weight Quantization

Group-wise weight quantization is a set of methodologies in low-precision neural network inference that partitions the weight tensor of a neural network into small groups, blocks, or clusters. Each group is quantized using distinct parameters—typically its own scale (and possibly zero-point/codebook)—allowing the quantizer to better match local data statistics and thereby reducing overall quantization error relative to global, layer-wise, or channel-wise quantization schemes. Group-wise quantization is now standard in efficient large language model (LLM), vision, and generative network deployment, owing to its favorable trade-off between accuracy, memory, compute efficiency, and hardware practicalities.

## 1. Mathematical Formulation and Partitioning Strategies

Group-wise quantization divides a weight tensor $W$ into $G$ non-overlapping groups—indexed as $\{ W_g \}_{g=1}^G$. The choice of partitioning depends on the layer topology and desired hardware compatibility:
- **Spatially contiguous blocks**: 1D or 2D sub-blocks in flattened tensors.
- **Per-output channel**: Each output channel (i.e., convolutional kernel or linear layer row) forms a group.
- **K-means/group clustering**: Clustering based on statistics (e.g., per-channel max, variance) or range variability [2501.04304, 2212.10200].

For each group, a dedicated quantizer is constructed—typically characterized by a per-group scale $s_g$, offset/zero-point $z_g$, or (for VQ methods) a group codebook.

**General framework**:

For $b$-bit quantization, the typical affine quantizer is:
$$
Q_g(w) = s_g \cdot \text{clip}\big(\text{round}\big(\frac{w}{s_g}\big)+z_g, a_\text{min}, a_\text{max}\big)
$$
with dequantization $w \approx s_g (q - z_g)$, $q \in \mathbb{Z}, a_\text{min}\leq q \leq a_\text{max}$, all parameters local to group $g$.

This design enables isolating outlier-heavy regions, exploiting local redundancy, and increasing quantization granularity where necessary.

## 2. Group-wise Quantization Algorithms and Variants

### 2.1 Uniform and Asymmetric Group-wise Quantization

The baseline is uniform, symmetric (or asymmetric) quantization applied per group, with scales and possible zero-points adapted to the local statistics. For group $g$ of weights $W_g$:

- **Uniform, symmetric**: $s_g = \frac{\max(W_g) - \min(W_g)}{2^b - 1}$; $z_g = \text{round}(-\min(W_g)/s_g)$ [2310.04836, 2212.10200, 2501.04304].
- **Dynamic axis selection**: In DGQ for diffusion models, the maximal range axis is automatically detected to capture outlier structure, and groups are formed by K-means on local range tuples $(\max, \min)$, with quantization scales set accordingly [2501.04304].
- **Permutation-based grouping**: In Permutation-COMQ, rows are permuted to cluster similar-magnitude entries per column before independent quantization, minimizing within-group scale range and maximizing dynamic resolution [2604.07674].

### 2.2 Non-uniform, Adaptive, and Codebook Approaches

- **Mathematically adaptive numeric types**: MANT parameterizes the group-wise quantization grid by a per-group parameter $a$, allowing the quantizer to smoothly interpolate between log-uniform, power-of-two, and normal-float behavior [2502.18755]. This is learned or selected per group based on minimizing the induced output error.
- **Block/codebook clustering**: BCQ (LO-BCQ) partitions blocks of weights, clusters blocks into a small set of codebooks, and quantizes each block with its corresponding codebook, yielding near-optimal MSE [2502.05376]. This is iteratively optimized via block–cluster re-assignment and Lloyd–Max codebook updates.
- **Gumbel-Softmax relaxation**: GSQ directly learns discrete grid assignments for each weight (within a group having a shared scale) via differentiable Gumbel-Softmax relaxation, jointly optimizing per-group scales and per-coordinate grid points [2604.18556].

### 2.3 Group-wise PTQ and QAT Extensions

- **Two-stage grid optimization**: Stage 1 minimizes group-wise reconstruction loss using calibration activations; Stage 2 uses coordinate descent to jointly refine all group scales to minimize the full layer-wise output error, incorporating Hessian structure and propagation of quantization error from prior layers [2602.02126].
- **DL-QAT**: Assigns each group a learnable scale (quantization magnitude) and corrects remaining quantization error with a local low-rank LoRA update (trained with QAT, touching <1% parameters), yielding state-of-the-art low-bit accuracy with extreme compute efficiency [2504.09223].
- **Kernel-wise quantization via DRL**: AutoQ uses hierarchical RL to simultaneously allocate per-kernel QBN (bit number) given target accuracy/latency/energy, and can automatically discover non-uniformly quantized configurations per group [1902.05690].

## 3. Hardware Implications, Efficiency, and Folding Strategies

### 3.1 Inference-Efficient Parameterization

- **Per-group scaling**: Design accommodates efficient integer GEMM/conv kernels by storing per-group scales, which can be folded into downstream components—such as BatchNorm $\gamma$ in vision nets [1908.01477] or fused as runtime per-block coefficients in transformer matmuls [2502.18755].
- **Adaptive decode efficiency**: MANT’s a-param grid yields integer/shift MACs fused within systolic arrays, enabling high-throughput 4-bit inference [2502.18755]. GSQ and group-wise scalar/clustered approaches are drop-ins for standard INT4/INT8 GEMM kernels [2604.18556, 2502.05376].
- **Dual-grained recasting**: In LLMs, dual-grained quantization dequantizes groupwise INT4 weights to INT8 and performs all inference with a single INT8 kernel (CUTLASS/GPU), leveraging both groupwise accuracy and coarse-grained hardware efficiency [2310.04836].

### 3.2 Training-Time and Runtime Complexity

- Group-wise parameter optimizations can be performed via:
  - Closed-form/minimum error projections [2602.02126, 2604.07674]
  - Lightweight K-means (blocks/ranges/statistics) [2501.04304, 2212.10200]
  - Multi-objective or RL-based search (when joint hardware and accuracy optimization required) [1902.05690].

Compared to layer-wise approaches, group-wise variants introduce modest additional parameters (e.g., per-group scale, small codebook/parameter tables), but have negligible runtime or memory impact on contemporary hardware for group sizes $32\leq G\leq 128$ [2502.18755, 2502.05376].

## 4. Empirical Performance and Trade-offs

Extensive experiments across LLMs, vision, and generative models consistently establish that group-wise or block-quantization is a dominant regime for sub-8-bit and especially sub-4-bit quantization, with notable findings:
- **Vision**: On ResNet-18 (CIFAR-100, 2-bit), per-filter (gs=1) groupwise quantization attains 71.3% top-1 (float: 73%) versus 64.9% for layerwise [1908.01477]. AdderNet, with group-quantization and lossless clamp/outlier handling, recovers 66.5% top-1 at 4-bit PTQ, outperforming global scaling by 8.5 points [2212.10200].
- **LLMs**: Groupwise INT4 in LLaMA-7B improves PPL from 6.85 (channelwise) to 5.8 (group size 128), while MANT-W4A8 further narrows the gap to FP16 (PPL 5.79) [2502.18755]. Sophisticated grid or codebook approaches close the delta to “vector quantization” at $<1\%$ as in BCQ (LO-BCQ) and GSQ [2520.05376, 2604.18556]. Two-stage scale optimization provides PPL and accuracy enhancements over GPTQ at 2–3 bits [2602.02126].
- **Diffusion & Generative Models**: DGQ with dynamic axis/group selection and prompt-specific log quantization attains $\mathit{FID}\approx 13.15$ in W8A8 StableDiffusion, matching FP. At W4A6, using 16 groups, FID drops from $>200$ to 43.66 (baseline), with group-wise DJQ at 0.263 CLIP (vs 0.127 baseline) [2501.04304].

**Key trade-offs**:
- **Group size**: Smaller groups (32–128) capture more local variability but increase metadata/storage; larger groups have lower overhead but higher error.
- **Codebook/parameter overhead**: Codebook-based methods (BCQ) and per-group-adaptive types (MANT) trade extra per-group parameters for significant accuracy at ultra-low bitwidths.
- **Hardware interface**: Selection of quantization format and group size must balance between inference kernel compatibility, scale lookup efficiency, and accelerator/ASIC resource usage.

## 5. Algorithmic and Implementation Workflows

### 5.1 Unified Pipelines

The group-wise quantization process involves several typical algorithmic phases:

| Phase                     | Description                                                                                                             | Representative Approaches                         |
|---------------------------|-------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------|
| 1. Grouping/Clustering    | Partition weights by axis, clustering, or contiguous blocks                                                            | [2501.04304, 2212.10200, 2310.04836, 2604.07674]  |
| 2. Scale/Parameter search | Determine per-group quantizer scales or codebooks (possibly via output-layer error minimization)                        | [2502.18755, 2602.02126, 2502.05376]              |
| 3. Quantization           | Apply group-wise quantization, coordinate/codebook lookup, and optional outlier correction or clamp                     | [1908.01477, 2212.10200, 2504.09223]              |
| 4. Post-processing        | Fold scales into downstream normalization/Bias (e.g., BatchNorm $\gamma$ adjustment), group-aware dequantization logic  | [1908.01477, 2502.18755]                          |
| 5. (Optional) QAT         | Fine-tune group magnitudes and LoRA low-rank correction for error minimization in the deployment regime                 | [2504.09223]                                      |

**Pseudocode snippets** for selected approaches can be found in [1908.01477, 2501.04304, 2212.10200, 2602.02126, 2502.05376], formally specifying the groupwise quantization, block clustering, and scale/codebook updates.

### 5.2 Outlier Handling and Lossless Clamp

Handling outliers by groupwise clustering (K-means on range, magnitude permutation [2501.04304, 2604.07674]), lossless clamping plus bias compensation [2212.10200], or activation percentile-based squeezing [2310.04836] is central to preventing representational collapse with global scaling.

### 5.3 Hardware Folding and Efficient Execution

Folding per-group scale factors into BatchNorm parameters during inference ensures parameter-free runtime cost, as shown in GDRQ [1908.01477]. Parallel systolic hardware paths, as in MANT, enable high-throughput 4-bit MACs via simultaneous accumulation/shift logic [2502.18755].

## 6. Applications, Impact, and Best Practices

Group-wise weight quantization is adopted across a diversity of application areas:

- **Computer vision**: ResNet, VGG, and AdderNet architectures—group-wise approaches consistently deliver higher accuracy at 2–4 bit regimes than layer/row-wise approaches [1908.01477, 2212.10200].
- **LLMs and transformers**: Standard for sub-8-bit quantization in LLaMA, OPT, GPT-3, Kimi-K2.5, and Mixture-of-Experts models, with kernels and quantization toolkits directly supporting group/block formats [2502.05376, 2604.18556, 2502.18755, 2602.02126].
- **Diffusion and generative models**: Distribution-aware group selection achieves state-of-the-art low bitwidth deployment without architectural surgery or retraining [2501.04304].
- **Medical foundation models**: Permuted group-wise quantization delivers SOTA DSC and NSD in calibration-only settings [2604.07674].

**Best practices**:
- For sub-8-bit deployment, $G=64$–$128$ is widely adopted, balancing metadata and quantization error [2502.18755, 2502.05376].
- Outlier-aware grouping and per-group parameterization are crucial for regimes below 4 bits [2501.04304, 2212.10200].
- For mixed-precision and hardware-optimized inference, group-wise dequantization and folding should align with kernel organization (row/column blocks, scale folding).

## 7. Limitations, Extensions, and Ongoing Research

- **Overhead**: Small group sizes increase metadata per parameter; however, this is negligible (e.g., 0.1% model size at 16 groups/layer for UNet [2501.04304]; single-parameter $a$ and scale per group for MANT [2502.18755]).
- **Activation quantization**: Most approaches treat activations separately; joint group-wise schemes are area of active research [2310.04836, 2502.05376].
- **Extensions**: K-means and permutation strategies can generalize to blocks of arbitrary shapes and even non-linear group selection (e.g., attention outlier detection in DGQ) [2501.04304]. Unifying runtime scale/shift logic is ongoing in LLM accelerator design [2502.18755].
- **Codebook/parameter LUTs**: Clustering methods with per-group codebooks introduce ROM/codebook lookup paths, which must be balanced against kernel simplicity [2502.05376].
- **Task transfer**: Empirical evidence suggests that group-wise schemes often generalize across tasks and domains, and can be composed with QAT or LoRA/post-training finetuning for further accuracy boosts [2504.09223, 2212.10200].

Group-wise and block quantization continue to evolve as the empirical and algorithmic backbone for ultra-low bitwidth neural network deployment across modalities and scales, underlying both academic innovation and industry platforms [1908.01477, 2502.18755, 2602.02126, 2604.18556].

Source: https://www.emergentmind.com/topics/group-wise-weight-quantization