Group-Wise Quantization & Adaptation

• Group-wise quantization and adaptation is a set of techniques that partition neural network weights or activations into disjoint groups, each with its own quantization parameters for tailored precision.
• These approaches employ optimization methods such as coordinate descent and variance-based grouping to minimize reconstruction loss and adapt to local statistical characteristics.
• They enable efficient hardware mapping and reduced runtime overhead, making them ideal for large-scale architectures like transformers, CNNs, and diffusion models.

Group-wise quantization and adaptation denote a collection of techniques for compressing neural network models by partitioning weights (or activations, codebooks, or deltas) into small, disjoint groups and assigning adaptive quantization parameters or adaptation strategies to each group. This paradigm is now central to state-of-the-art post-training quantization (PTQ) and parameter-efficient adaptation for transformers, CNNs, diffusion models, vector quantized autoencoders, and other large-scale architectures. Group-wise schemes address the inherent statistical heterogeneity, reduce precision loss, allow mixed adaptation flexibility, and enable sophisticated hardware mappings—yielding highly compressed yet accurate networks with minimal runtime penalty.

1. Principles of Group-wise Quantization

The foundational idea in group-wise quantization is to partition model parameters (weights, activations, codebook entries, or task-specific deltas) into non-overlapping groups along a well-chosen dimension (e.g., output channels, spatial blocks, vectors in a VQ codebook). Each group receives its own quantization scale (and, in some methods, codebook, zero-point, or even quantization format).

Let $w \in \mathbb{R}^d$ be a weight vector, partitioned into $n_g$ groups $w_i$ of possibly varying size. The quantized representation per group is typically

$q_i = s_i \cdot w_{\text{int},i}$

where $s_i$ is the group scale, and $w_{\text{int},i}$ is the quantized integer representation of $w_i$. For full matrices, groupings are often column-wise or row-wise for LLMs (Kim et al., 2 Feb 2026), or along flattened spatial channels in vision models (Pan et al., 2024, Yu et al., 2019). For codebooks, groups correspond to disjoint subcodebooks or code clusters (Zheng et al., 15 Oct 2025).

This contrasts with tensor-wise (single scale per parameter set), channel-wise, or per-layer quantization which all fail to adapt to local outliers and fine-scale heterogeneity. Group sizes are chosen to balance statistical adaptation, metadata overhead, and hardware SIMD compatibility.

2. Mathematical Foundations and Algorithmic Frameworks

Group-wise quantization supports a diverse set of mathematical objectives and algorithmic realizations, including:

• Quantization Objective: Most methods minimize reconstruction loss, e.g., expected output error

$L(\{s_i\}) = (q - w)^\top H (q - w)$

where $H = \mathbb{E}[X X^\top]$ is the Hessian/covariance estimator, and $q, w$ stack all groupwise quantized and original weights respectively (Kim et al., 2 Feb 2026). Gaussian/loss-propagating variants further introduce cross-layer error terms to handle upstream quantization artifacts.

• Scale and Codebook Learning: Scales are initialized (via min–max, variance, groupwise optimization) and then often refined by closed-form minimization or coordinate descent. For high flexibility, additional group-adaptive parameters or even full codebooks (cores + projectors) can be learned (Zheng et al., 15 Oct 2025).
• Adaptivity to Distributional Properties: Offline quantization can use calibration data to optimize per-group parameters for a target layer or function. Online or run-time strategies update quantization types for activations/KV cache using streaming statistics, e.g., variance or range (Hu et al., 26 Feb 2025).
• Group-wise Dropout and Sparsification: In delta-compressed models, groupwise dropout sparsifies fine-tuned weights with contiguous masked groups, followed by group-wise quantization and multi-bin low-bit decomposition (Jiang et al., 2024).

Table: Typical Group-wise Quantization Formulas

Operation	Formula (per group $g$)	Reference
Linear quantization	$$\hat X_g = s_g \cdot \operatorname{round}(X_g / s_g)$$	(Pan et al., 2024)
Layerwise loss	$L = (q - w)^\top H (q - w)$	(Kim et al., 2 Feb 2026)
Adaptive format (MANT)	$x^* = s_g \cdot (a_g \cdot |i^*| + 2^{|i^*|})$	(Hu et al., 26 Feb 2025)

This principled formalization enables fine-grained control over quantization error and supports advanced adaptation pipelines.

3. Representative Group-wise Quantization and Adaptation Methods

A range of group-wise approaches have emerged for diverse settings:

• Two-stage Layerwise Optimization: A two-phase scheme initializes group scales for minimum local reconstruction error (weighted by group input covariances) and then globally optimizes all scales to minimize full layerwise output loss via coordinate descent, with analytic updates. Upstream quantization error is explicitly corrected in later layers using additional Hessian cross-terms (Kim et al., 2 Feb 2026).
• QA-LoRA: Combines group-wise INT4 quantization (scales and zero-points per-row-group) with a group-constrained LoRA adaptation, ensuring merged FP16 weights after adaptation remain quantized exactly on the group integer grid (Xu et al., 2023).
• M-ANT: Each group selects its own numeric type (e.g., INT4, PoT, NF4-like) via MSE-optimized coefficient search, and real-time streaming statistics for KV-caches, supporting on-the-fly mixed-format quantization with hardware acceleration (Hu et al., 26 Feb 2025).
• Distribution-aware Grouping: In diffusion models, groupings are selected based on outlier detection, using per-layer channel/pixel dimensions with clustering (e.g., K-means) on ranges; prompt-specific nonlinear scaling may be applied to cross-attention components (Ryu et al., 8 Jan 2025).
• Delta Compression with Group-wise Dropout: Fine-tuning deltas are aggressively group-sparsified (optimal groups selected via attention block error proxy), then quantized and bit-decomposed for compression ratios up to 128×+ with minimal loss (Jiang et al., 2024).
• VQ Codebooks (Group-VQ): Codebooks are partitioned into groups, each with its own projector/bias; groupwise optimization ensures mutual independence, improved utilization, and permits post-hoc codebook size adaptation via parametric resampling, without retraining (Zheng et al., 15 Oct 2025).

4. Empirical Performance and Practical Trade-offs

Empirical analyses across the literature establish clear accuracy, efficiency, and system-level benefits:

• LLMs: Two-stage optimized group quantization achieves up to 6 percentage points improvement in zero-shot accuracy at INT2/INT3 over standard GPTQ, with almost full FP32 recovery at INT3 and minimal runtime overhead (quantization time increases from 5.8 min to 7.5 min for Llama-3 8B) (Kim et al., 2 Feb 2026). Group-wise adaptation with QA-LoRA outperforms QLoRA by 3–4 points on MMLU at 4-bit in LLaMA-7B (Xu et al., 2023). M-ANT matches FP16 task metrics at 4/8 bits and delivers 2.99× speedup and 2.81× energy reduction over specialized accelerators (Hu et al., 26 Feb 2025).
• Diffusion/text-to-image: Group-wise quantization with outlier-grouping and prompt-specific scaling achieves FID improvements (MS-COCO, FID↓ from 26.12 to 13.15 at 8/8 bits; 31.36 at 8/6 bits; Table 1) and negligible CLIP drops compared to full-precision (Ryu et al., 8 Jan 2025). Data-free scale adaptation for Winograd F(6,3) reduces catastrophic FID collapse (>300 to ≈27) (Pan et al., 2024).
• ASR/Edge: Block-wise NormalFloat4 quantization with LoRA adaptation (P4Q) reduces WER by 24.2%/25.3% relative over quantized-only baselines at only 1% parameter overhead (Zhao et al., 2024).
• VQ Models: Group-wise codebooks in VQ-VAEs (Group-VQ) approach 100% utilization and minimize FID, outperforming both vanilla and fully coupled schemes. Codebook size can be halved or doubled post-training with predictable rFID impact and no retraining (Zheng et al., 15 Oct 2025).
• Delta compression: Group-wise mask/quantization yields up to 512× compression (WizardMath-70B) with <2pt absolute performance degradation versus full-precision fine-tuned models (Jiang et al., 2024).

Choice of group size is critical: too small increases scale overhead and potential underfitting (especially in VQ); too large exposes to local outliers and higher per-group quantization noise. Typical recommended sizes are 32 or 64 for LLMs (hardware-aligned), 128 for VQ/VQGAN, and in diffusion, 16–64 groups per layer.

5. Hardware Mapping, Systems, and Efficiency Considerations

Group-wise quantization is inherently suitable for efficient hardware mapping:

• SIMD/Vectorization Alignment: Groups are sized to match hardware vector widths (e.g., 32/64 for ARM, 128 for AVX512, 32×32 tiles in systolic arrays), enabling block matrix-multiply with minimal padding (Pan et al., 2024, Hu et al., 26 Feb 2025).
• Quantization/Dequantization Fusion: Scales and formats can be fused into post-accumulation steps, so compute overhead is <1% even in highly pipelined architectures (Hu et al., 26 Feb 2025).
• Bit-width and Format Flexibility: Per-group adaptation (e.g., M-ANT) supports runtime assignment of numeric formats, safely mixing INT4/PoT/NF4 in the same operator, with coefficients selected by fast LUTs trained on group variance (Hu et al., 26 Feb 2025).
• Delta/adapter fusion: Group-constrained adaptation (QA-LoRA), zero-point shifting, and LoRA rank grouping ensure merging into quantized kernels without FP cycles or extra memory lookups (Xu et al., 2023).

This enables quantized models to maintain full throughput, minimize latency, and permit aggressive run-time adaptation, even on edge devices.

6. Limitations, Open Problems, and Theoretical Insights

While empirically successful, several aspects of group-wise quantization and adaptation remain areas of active investigation:

• Group Formation and Selection: Most methods use heuristics (channel, block, or K-means clustering among outlier metrics), but the optimal partitioning remains open. In binary quantization, dynamic grouping by minimizing groupwise variance is shown to be optimal under the adopted loss, and realized via DP/greedy/windowed grouping (Zheng et al., 3 Sep 2025).
• Error Propagation: Accumulation of quantization error from earlier layers is an active concern. Recent work shows incorporating upstream error cross-terms yields further improvements (Kim et al., 2 Feb 2026).
• Adaptation Capacity: The balance of quantization and adaptation degrees-of-freedom is crucial; groupwise LoRA-type adapters can improve this, but overly small groups may underfit or collapse (Xu et al., 2023).
• Codebook Collapse and Utilization: In VQ models, too many (or too small) groups diminish codebook utilization; conversely, too few groups degrade expressivity. Group-VQ interpolates between these extremes to find the best utilization-quality tradeoff (Zheng et al., 15 Oct 2025).
• Metadata and Overhead: Storage and communication of per-group scales, coefficients, and indices is a negligible fraction of weights for $G \lesssim 128$ in contemporary LLMs or CNNs.
• Generalization to Unseen Domains: Data-free fine-tuning of groupwise scales (not the weight/activation scales, but Winograd transform scales) can maintain generalization, as shown for diffusion models and Winograd convolution (Pan et al., 2024).

7. Applications, Generalization, and Outlook

The group-wise quantization and adaptation paradigm is broadly applicable:

• Language modeling, pretraining, and fine-tuning: LLMs, transformer-based encoders/decoders, and ASR models leverage groupwise schemes for robust quantization and lossless, parameter-efficient adaptation (Kim et al., 2 Feb 2026, Xu et al., 2023, Zhao et al., 2024).
• Diffusion and vision models: Groupwise activation/weight quantization supports high-quality text-to-image synthesis under severe resource constraints and in quantized convolution (standard and Winograd) (Ryu et al., 8 Jan 2025, Pan et al., 2024).
• VQ-VAEs and generative models: Self-extensible, resamplable group codebooks enable flexible bitrate adaptation post-training, combining the advantages of both vanilla and joint codebook learning (Zheng et al., 15 Oct 2025).
• Delta compression: Enables ultra-high model multiplexing for personalized or task-adapted LLMs, with only a small (~2pt) headroom in end-task accuracy even at 128×+ delta compression (Jiang et al., 2024).
• Cross-modal and multimodal adaptation: Groupwise scale learning combined with structured warm-up enables full or better task accuracy on VL-instruction tuning under 4-bit quantization (Xie et al., 2024).

A plausible implication is that as groupwise techniques continue to mature, they will become the default mechanism for low-cost deployment across modalities, tasks, and adaptation settings, with further theoretical work needed on optimal group partitioning, error propagation, and the joint design of groupwise quantization with hardware and task-specific adapters.