---
title: Multi-Group Quantization (MGVQ)
url: https://www.emergentmind.com/topics/multi-group-quantization-mgvq
type: topic
---

# Multi-Group Quantization (MGVQ)

Multi-Group Quantization (MGVQ) is a set of quantization paradigms in which tensor values—weights, activations, or latents—are partitioned into small "groups," with each group assigned its own quantization parameters, codebook, or numeric type. This approach generalizes the classic layerwise/channelwise quantization, enabling fine-grained adaptation to local statistics, and is now foundational across efficient large language model (LLM) inference, vision transformer calibration, and vector quantized generative models. MGVQ can employ uniform or non-uniform grids, instance-dependent grouping, and dynamic parameterization to optimize rate–distortion trade-offs and hardware efficiency.

## 1. Fundamental Concepts and Mathematical Framework

MGVQ partitions a vector or tensor $w \in \mathbb{R}^d$ into $n_g$ disjoint groups $\{w_i\}_{i=1}^{n_g}$, each of size $g$ ($d = n_g \cdot g$), with independent quantization parameters per group. The general groupwise quantization mapping, as used in LLM quantization, is
$$
q_i = s_i \cdot \mathrm{clamp}_Q\left( \frac{w_i}{s_i} \right), \quad i=1,\ldots,n_g,
$$
where $s_i>0$ is a trainable group scale and $\mathrm{clamp}_Q(\cdot)$ denotes rounding and clamping to the target bitwidth. In generative modeling contexts, groupwise quantization may instead use sub-codebooks per group, with each input vector split into $G$ sub-vectors, each quantized independently via assignment to the closest codebook vector [2507.07997][2510.13331].

The efficacy of MGVQ relies on the observation that local statistics—such as dynamic range or variance—can vary significantly even across small groups. Adapting parameters to each group reduces quantization error and prevents information loss that would otherwise be incurred by forced global quantization.

## 2. MGVQ in Parameter and Activation Quantization

Fine-grained groupwise quantization is a de facto standard in low-bit LLM inference [2502.18755][2602.02126], as well as in post-training calibration for vision transformers [2404.00928]. The core motivations and methodologies include:

- **LLM Groupwise Schemes**: LLMs commonly use group sizes $g=32$ or $g=64$ per output channel. Each group is assigned a scale and potentially a non-uniform grid to match its range and distribution. Recent LLM accelerators also quantize KV caches groupwise, with real-time group parameter selection [2502.18755].
- **Vision Transformer Activation Quantization**: IGQ-ViT introduces *instance-aware* dynamic group splitting, where both activations and attention maps are divided into groups for each input instance. Group assignments are updated with an EM-style algorithm to minimize distributional discrepancies and optimize uniform quantizer fit [2404.00928].
- **Groupwise Grid Optimization**: Two-stage methods further minimize reconstruction loss by first initializing group scales with an input-aware local objective, then refining all scales via coordinate descent on the *global* layerwise loss, using block-partitioned input Hessians [2602.02126].

**Table 1: Representative MGVQ Methods in Parameter/Activation Quantization**

| Method       | Group Approach         | Assignment         |
|--------------|-----------------------|--------------------|
| MANT [2502.18755]    | Static, 64-weights | Grid search on MSE |
| Two-Stage [2602.02126] | Static, 32/64     | Loss minimization  |
| IGQ-ViT [2404.00928]  | Dynamic, per-input | EM-based, BOP-constrained |

## 3. Multi-Group Vector Quantization in Representation Learning

In vector-quantized generative models such as VQ-VAEs and VQGANs, MGVQ replaces a single monolithic codebook with $G$ sub-codebooks ("multi-group vector quantization"), each operating on a sub-vector of encoder outputs [2507.07997][2510.13331]. The encoder output $z \in \mathbb{R}^{C_l}$ is split into $G$ chunks of size $C_l/G$, and each chunk is quantized independently:
$$
z_{q,i} =  e_i^{k_i}, \quad k_i = \arg\min_{j} \|\ z_i - e_{i}^{j}\|_2\,,~i=1,...,G,
$$
with each sub-codebook $E_i$ of size $K$. The quantized codes are concatenated. This increases representational capacity ($K^G$ combinations) without incurring codebook collapse or excessive per-codebook dimensionality.

Group-VQ [2510.13331] generalizes this to allow groupwise optimization of codebook segments, enabling higher code utilization and post-hoc codebook resampling or self-extension for capacity tuning.

**Table 2: MGVQ in VQ-VAEs**

| System       | No. Groups  | Codebook Size per Group | Total Capacity  |
|--------------|-------------|------------------------|-----------------|
| MGVQ-G4 [2507.07997] | 4       | 8192                  | $8192^4$        |
| MGVQ-G8 [2507.07997] | 8       | 2048                  | $2048^8$        |
| Group-VQ [2510.13331] | $k$     | $\sim n/k$            | $n$             |

## 4. Algorithmic Techniques for Group Assignment and Parameterization

MGVQ covers a spectrum of assignment mechanisms:

- **Static Partitioning**: Fixed group boundaries (by position, channel count, or latent dimension).
- **Dynamic Grouping**: Assignment at runtime based on per-instance statistics, using distance measures (e.g., min/max spread or KL divergence in output space) and iterative EM steps [2404.00928].
- **Groupwise Grid/Codebook Optimization**: Search or descent algorithms per group for optimal quantizer parameters (scales, grid coefficients, codebooks), with group-specific calibration loss, often involving grid search or closed-form updates [2602.02126][2502.18755].
- **Within-Group Joint Adaptation**: Learned affine projectors shared within groupwise codebooks to capture heterogeneous data subdistributions [2510.13331].

Dynamically assigned groups can be optimized subject to computational or hardware constraints, such as limiting extra Bit-Operations (BOP) via integer programming [2404.00928].

## 5. Hardware and Inference Considerations

MGVQ is central in hardware-efficient model deployment:

- **Low-cost Metadata Storage**: Scales or group identifiers are small, often amortized over large tensor tiles.
- **Fused Dequantization**: Hardware pipelines (e.g., systolic arrays) can interleave decode and compute operations where group parameters are locally buffered, as in MANT [2502.18755].
- **No Inference Cost Overhead**: Approaches such as GDRQ [1908.01477] merge per-group scales into BatchNorm during inference, retaining hardware simplicity.
- **Dynamic Real-Time Support**: For LLMs, real-time groupwise quantization is deployed for KV cache updates, requiring lightweight metadata computation and buffering [2502.18755].

## 6. Empirical Performance and Trade-offs

MGVQ methods consistently report accuracy close to full-precision baselines at aggressive bitwidths ($\leq$4 bits), often with minimal overhead:

- **LLMs**: W4A8 groupwise quantization with adaptive types (MANT) yields $<$0.12 PPL loss versus float on LLaMA and OPT, and $2.81 \times$ energy reduction over state-of-the-art accelerators [2502.18755]. Two-stage optimization narrows the performance gap to full precision at INT3 [2602.02126].
- **Vision Transformers**: IGQ-ViT achieves $4\text{–}9$\% top-1 gains over prior PTQ methods at 4 bits, with $<3\%$ BOP overhead [2404.00928].
- **Generative Models**: MGVQ-G8 reaches $r\mathrm{FID}=0.49$ and PSNR=24.70 on ImageNet, outperforming both classic VQ-GANs and continuous-latent VAEs. Ablation shows near-100\% code utilization and stable training even at large capacity [2507.07997][2510.13331].

Smaller group size typically improves fidelity at the cost of additional metadata and potential statistical instability. The number of groups $k$ is commonly tuned to maximize utilization and minimize distortion [2510.13331].

## 7. Limitations, Variants, and Future Directions

MGVQ frameworks assume that efficient folding of groupwise parameters into post-processing (such as BatchNorm) is available, and require access to per-group statistics during training or calibration. In some cases, very small groups may induce noisy estimates or excessive shape overhead [1908.01477]. Support for dynamic assignment on-device introduces marginal hardware complexity [2404.00928].

Variants include instance-aware grouping, hierarchical grouping strategies, adaptive selection of group count per layer, and integration of MGVQ with codebook resampling or self-extension for post-hoc capacity scaling [2510.13331]. Extending MGVQ concepts to other model components (e.g., LayerNorm, MLP outputs), and tightly coupling group assignment with quantization-aware training, represent ongoing areas of research [2404.00928][2507.07997].

---
**References:**
- [2502.18755] "M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type"
- [2602.02126] "Two-Stage Grid Optimization for Group-wise Quantization of LLMs"
- [2404.00928] "Instance-Aware Group Quantization for Vision Transformers"
- [1908.01477] "GDRQ: Group-based Distribution Reshaping for Quantization"
- [2507.07997] "MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantization"
- [2510.13331] "Group-Wise Optimization for Self-Extensible Codebooks in Vector Quantized Models"

Source: https://www.emergentmind.com/topics/multi-group-quantization-mgvq