Group-wise Lookup-free Quantization (GQ)

• Group-wise Lookup-free Quantization (GQ) is a technique that partitions tensors into groups to enable lookup-free execution while preserving fine-grained accuracy in quantized representations.
• GQ methods, such as DGQ, WeTok, Winograd GQ, and GSQ, shift complexity to offline preprocessing using scale absorption, implicit codebooks, or fixed grids to maintain runtime efficiency.
• Applications span LLM inference, visual tokenization, and CNN acceleration, offering improved speed, reduced memory usage, and maintained accuracy in low-bit quantization settings.

Group-wise Lookup-free Quantization (GQ) denotes a family of quantization schemes in which a tensor or latent representation is partitioned into groups, but the resulting discrete representation is executed without runtime nearest-neighbor search, prototype-table access, or per-group scale switching inside the arithmetic hot path. The term is used across several distinct settings—mixed-precision LLM inference, discrete visual tokenization, fully quantized Winograd convolution, and ultra-low-bit scalar LLM quantization—but the shared objective is consistent: preserve the accuracy advantages of finer granularity while retaining hardware-efficient execution paths (Zhang et al., 2023, Zhuang et al., 7 Aug 2025, Pan et al., 2024, Dadgarnia et al., 20 Apr 2026).

1. Core definition and semantic scope

In GQ, the phrase group-wise refers to quantization granularity that is finer than tensor-wise or channel-wise quantization, but coarser than per-element quantization. The precise grouping depends on the application. In LLM weight quantization, groups are typically formed along the reduction axis or row-wise over consecutive entries; in WeTok, grouping is channel-wise over latent dimensions; in Winograd convolution, groups are aligned with row, column, or vector-width structure in transformed tensors; in GSQ, groups are formed row-wise with group size 128 by default (Zhang et al., 2023, Zhuang et al., 7 Aug 2025, Pan et al., 2024, Dadgarnia et al., 20 Apr 2026).

The phrase lookup-free is not identical across these works. In DGQ for A8W4 LLM inference, lookup-free means that the INT8 GEMM inner loop consumes only integer tensors and does not read or switch scales per group; all scales needed at inference reduce to a single per-output-channel FP16 post-scale applied outside the inner GEMM loop (Zhang et al., 2023). In WeTok, lookup-free means that quantization does not perform nearest-neighbor search against stored code vectors, and decoding into the decoder input is also lookup-free because the quantized latent is the binary tensor itself (Zhuang et al., 7 Aug 2025). In fully quantized Winograd convolution, lookup-free means the method uses standard uniform affine quantizers with per-group scales and optional zero-points, with no codebooks or learned LUT indices (Pan et al., 2024). In GSQ, lookup-free means runtime dequantization is the product of one per-group scale and one small integer code on a fixed symmetric grid, so kernels remain standard scalar weight-only kernels (Dadgarnia et al., 20 Apr 2026).

A common misconception is that lookup-free quantization is necessarily scale-free. These papers show the opposite. DGQ retains a per-output-channel FP16 post-scale, Winograd GQ tracks fused scales through transform stages, and GSQ uses one real scale per group. Lookup-free therefore refers to the elimination of expensive runtime indirections, not to the absence of scaling metadata.

2. Common algorithmic patterns

Across the cited works, GQ appears in three main technical forms. This suggests a useful taxonomy rather than a single canonical algorithm.

The first form is scale absorption into arithmetic-friendly tensors. DGQ starts from fine-grained INT4 weights, then converts them into a coarse-grained INT8 representation so that inference uses continuous INT8 GEMM kernels without per-group scale lookups. The central mapping is

$$S^{(2)} = \mathrm{round}\Big(\frac{S'}{s^{(1)}}\Big), \quad q^{(8)} = \mathrm{clip}\Big(S^{(2)}\cdot\big(q^{(4)} + zp^{(4)}\big), -127, 127\Big), \quad \hat{w} \approx s^{(1)}\cdot q^{(8)}.$$

The group-wise scales are absorbed offline into $q^{(8)}$, leaving only a single channel-wise post-scale at runtime (Zhang et al., 2023). Winograd GQ follows a related idea, but at the level of transformed convolutional operands: scale factors from quantized transform matrices, transformed inputs, and transformed weights are fused so that the integer path never needs table lookups, and final dequantization multiplies by the accumulated scale (Pan et al., 2024).

The second form is implicit codebooks without prototype storage. WeTok partitions the latent tensor into groups and quantizes each group with a sign operator over an implicit binary hypercube. For each group,

$$t^{(g)}[i, j, m] = \mathrm{sign}(z^{(g)}[i, j, m]),$$

so the per-group code lies in $\{-1,+1\}^{d'}$, and the decoder input is simply the concatenated binary tensor (Zhuang et al., 7 Aug 2025). No stored prototypes and no nearest-neighbor search are involved.

The third form is fixed scalar grids with learned assignments. GSQ uses a symmetric integer grid $G=\{g_k\}$ and group-wise scales $s_g$, with dequantization

$\hat w_i = s_g \cdot q_i,$

where $q_i \in G$. Runtime remains lookup-free because the kernel performs integer code unpacking plus per-group scale multiplication, but the assignments are optimized with a Gumbel-Softmax relaxation rather than simple local rounding (Dadgarnia et al., 20 Apr 2026).

These forms share a systems-level principle: they push complexity into offline preprocessing, training-time optimization, or entropy regularization so that inference-time execution can remain vectorizable and kernel-compatible.

3. DGQ and lookup-free A8W4 inference for LLMs

"Dual Grained Quantization: Efficient Fine-Grained Quantization for LLM" introduces a mixed-precision A8W4 scheme designed to preserve the accuracy of fine-grained group-wise INT4 weight quantization while avoiding the inference inefficiency that ordinarily arises when different segments of the reduction axis use different scales (Zhang et al., 2023).

The starting point is the standard contrast between coarse-grained and fine-grained quantization. Channel-wise quantization assigns one scale per output channel and aligns naturally with integer GEMM because the inner loop does not need to change scales mid-accumulation. Group-wise quantization splits the input-channel dimension into groups with separate scales, which reduces quantization error but disrupts continuous integer matmul. DGQ resolves this by introducing dual scales for weights: a channel-wise FP16 scale $s^{(1)}$ and a group-wise integer scale $S^{(2)}$ stored as INT8. Fine-grained INT4 codes are lifted into an INT8 tensor offline, after which runtime uses standard CUTLASS INT8 kernels.

At inference, integer GEMM computes

$q^{(8)}$0

and the final output is approximated by

$q^{(8)}$1

In the paper’s notation, the post-scale is $q^{(8)}$2, and the kernel path requires only a single per-output-channel FP16 scale in the epilogue. This is the sense in which DGQ realizes group-wise lookup-free quantization: group-wise scales are not read inside the GEMM hot loop (Zhang et al., 2023).

Scale determination is handled by a two-phase grid search. Phase 1 computes original group-wise INT4 scales and zero-points by minimizing the reconstruction error of the quantized matmul for each group. Phase 2 decouples the original group-wise scale tensor into $q^{(8)}$3 and $q^{(8)}$4, enforces integer-range constraints so that $q^{(8)}$5, and optimizes the channel-wise scale over a small grid. The method also introduces percentile clipping for activations, using a simple percentile smoothing scheme to suppress activation outliers without complex optimization.

The reported empirical results are framed around both quality and systems performance. DGQ consistently outperforms prior A8W4 methods across LLaMA and OPT families and tasks including WikiText-2, C4, PTB, CSQA, and MMLU. With efficient CUTLASS kernels, it achieves up to $q^{(8)}$6 speedup versus A16W4 for long sequences, and the implementation reports $q^{(8)}$7 memory reduction and $q^{(8)}$8 speed gains compared to A16W4. The paper also states that DGQ reaches A8W8-level inference time while using $q^{(8)}$9 the memory of A8W8 (Zhang et al., 2023).

The main trade-off is that complexity is shifted from runtime to preprocessing. Projection error can still arise when group-wise INT4 codes are mapped to INT8 through $$t^{(g)}[i, j, m] = \mathrm{sign}(z^{(g)}[i, j, m]),$$0 and $$t^{(g)}[i, j, m] = \mathrm{sign}(z^{(g)}[i, j, m]),$$1, grid-search ranges and resolutions remain design sensitivities, and the benefits depend on robust INT8 GEMM kernels.

4. GQ as compositional discrete tokenization in WeTok

"WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstruction" uses GQ in a substantially different regime: discrete visual tokenization rather than arithmetic acceleration. Here GQ converts continuous encoder latents into extremely large compositional codebooks without storing prototypes or performing nearest-neighbor lookup (Zhuang et al., 7 Aug 2025).

Let the encoder produce

$$t^{(g)}[i, j, m] = \mathrm{sign}(z^{(g)}[i, j, m]),$$2

WeTok partitions $$t^{(g)}[i, j, m] = \mathrm{sign}(z^{(g)}[i, j, m]),$$3 along the channel dimension into $$t^{(g)}[i, j, m] = \mathrm{sign}(z^{(g)}[i, j, m]),$$4 groups, with each group quantized directly by sign. Each group has a fixed, non-learnable, implicit binary codebook

$$t^{(g)}[i, j, m] = \mathrm{sign}(z^{(g)}[i, j, m]),$$5

Concatenating the per-group binary vectors yields a token $$t^{(g)}[i, j, m] = \mathrm{sign}(z^{(g)}[i, j, m]),$$6 for each spatial location, and the quantized latent fed to the decoder is simply $$t^{(g)}[i, j, m] = \mathrm{sign}(z^{(g)}[i, j, m]),$$7. The effective codebook cardinality per location is

$$t^{(g)}[i, j, m] = \mathrm{sign}(z^{(g)}[i, j, m]),$$8

which scales exponentially with the latent channel count while avoiding stored code vectors (Zhuang et al., 7 Aug 2025).

The principal technical contribution is not only sign-based quantization, but grouped entropy regularization. WeTok follows lookup-free quantization practice by adding an entropy loss on the code distribution, but factorizes the distribution across groups:

$$t^{(g)}[i, j, m] = \mathrm{sign}(z^{(g)}[i, j, m]),$$9

This allows the per-token entropy across the full code space to decompose into a sum of group entropies, and the global codebook entropy to be approximated by a sum of grouped terms. The resulting regularizer,

$\{-1,+1\}^{d'}$0

is computed over grouped subspaces of size $\{-1,+1\}^{d'}$1 instead of the full $\{-1,+1\}^{d'}$2 space. The paper states that this reduces memory from $\{-1,+1\}^{d'}$3 to $\{-1,+1\}^{d'}$4, eliminating the entropy loss as the bottleneck while avoiding the stronger independence assumption of BSQ (Zhuang et al., 7 Aug 2025).

WeTok couples GQ with a Generative Decoder (GD), which reconstructs from grouped binary latents together with an extra noise input $\{-1,+1\}^{d'}$5 and models $\{-1,+1\}^{d'}$6. The decoder is trained in a second stage by zero-initializing the new input channels so that it initially behaves identically to the pre-trained deterministic version, then gradually learns to use $\{-1,+1\}^{d'}$7.

The reported reconstruction numbers are unusually strong for a discrete tokenizer. On the ImageNet 50k validation set, WeTok achieves zero-shot rFID $\{-1,+1\}^{d'}$8, compared with FLUX-VAE $\{-1,+1\}^{d'}$9 and SD-VAE 3.5 $G=\{g_k\}$0. Its highest compression model, at compression ratio $G=\{g_k\}$1, achieves zero-shot rFID $G=\{g_k\}$2, surpassing Cosmos at ratio $G=\{g_k\}$3 with rFID $G=\{g_k\}$4. In-distribution reconstruction results include rFID $G=\{g_k\}$5 and PSNR $G=\{g_k\}$6 at ratio $G=\{g_k\}$7, and rFID $G=\{g_k\}$8 and PSNR $G=\{g_k\}$9 at ratio $s_g$0. The ablations isolate GQ itself: with $s_g$1 fixed and increasing $s_g$2, LFQ hits OOM beyond $s_g$3, while GQ remains flat at approximately $s_g$4 GB like BSQ; increasing the number of groups consistently improves reconstruction metrics (Zhuang et al., 7 Aug 2025).

The main limitation is intrinsic to sign quantization. Because amplitude is discarded, fidelity depends on the decoder’s ability to recover magnitudes and details from binary latents, and the factorized approximation used for codebook entropy remains an assumption.

5. Fully quantized Winograd convolution via group-wise affine quantization

"Data-Free Group-Wise Fully Quantized Winograd Convolution via Learnable Scales" extends GQ into convolutional acceleration for diffusion models and CNNs. In this setting, the objective is not prototype-free tokenization or LLM GEMM restructuring, but numerically stable fully quantized Winograd convolution using standard integer arithmetic and no training data (Pan et al., 2024).

For Winograd $s_g$5 convolution, the transformed quantities are

$s_g$6

The paper identifies a key failure mode of low-precision Winograd: in the output-domain tensor $s_g$7, different Winograd taps can have very different variances and ranges, empirically exhibiting a cross-like pattern with orders-of-magnitude differences. Pixel-wise scales would alleviate the numerical problem but would destroy vectorization and kernel efficiency. Group-wise quantization over row, column, or vector-width aligned groups instead offers a hardware-aware compromise (Pan et al., 2024).

The quantizer itself is lookup-free affine quantization:

$s_g$8

Weights, activations, transform matrices, and Winograd-domain intermediates are all quantized group-wise, with int8 operands and int32 accumulators. Activations use dynamic per-group min–max at inference time. To reduce range differences in the Winograd domain, the paper parameterizes the transforms with diagonal scales:

$s_g$9

subject to

$\hat w_i = s_g \cdot q_i,$0

Only $\hat w_i = s_g \cdot q_i,$1 and $\hat w_i = s_g \cdot q_i,$2 are learned, while $\hat w_i = s_g \cdot q_i,$3. The learning is data-free: for random Gaussian or uniform inputs, the method minimizes the discrepancy between full-precision convolution output and fully quantized Winograd output, using one shared set of scales across all Winograd layers (Pan et al., 2024).

The integerized pipeline quantizes the input transform, the Hadamard product, and the output transform, while tracking fused scales such as $\hat w_i = s_g \cdot q_i,$4, $\hat w_i = s_g \cdot q_i,$5, and $\hat w_i = s_g \cdot q_i,$6. Because these scales are fused, the arithmetic path never requires codebook access or table lookup. For $\hat w_i = s_g \cdot q_i,$7, the paper notes that the inner-product length is $\hat w_i = s_g \cdot q_i,$8, and the worst-case int8×int8 sum $\hat w_i = s_g \cdot q_i,$9 is well within int32 limits (Pan et al., 2024).

The reported results are strikingly dependent on whether learnable scales are used. For Stable Diffusion v1.5 with the AKL autoencoder, FP16 obtains FID $q_i \in G$0 and CLIP $q_i \in G$1; W8A8 group-wise standard convolution obtains FID $q_i \in G$2 and CLIP $q_i \in G$3; W8A8 Winograd $q_i \in G$4 with standard scales collapses to FID $q_i \in G$5 and CLIP $q_i \in G$6; W8A8 Winograd $q_i \in G$7 with learned scales recovers to FID $q_i \in G$8 and CLIP $q_i \in G$9; and W8A8 Winograd $s^{(1)}$0 with learned scales gives FID $s^{(1)}$1 and CLIP $s^{(1)}$2. On ImageNet classification with ResNet-18 and ResNet-34 under W8A8, GQ with learned scales outperforms PAW+FSQ by $s^{(1)}$3 and $s^{(1)}$4 in top-1 accuracy for Winograd $s^{(1)}$5, respectively. Highly optimized integer kernels reduce convolution-layer wall-clock by $s^{(1)}$6 versus standard convolution and produce a $s^{(1)}$7 end-to-end improvement on an InstaFlow-0.9B pipeline on Arm Graviton3, single thread (Pan et al., 2024).

The method’s assumptions are explicit. Larger tiles accentuate range imbalance, 8-bit is the demonstrated sweet spot, group size must align with vector width, and applicability is natural for CNN-like blocks rather than transformer MLPs.

6. GSQ and ultra-low-bit scalar GQ for LLMs

"GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling" places GQ in the regime of 2–3 bit weight-only quantization for LLMs. Its central claim is that a carefully optimized scalar, group-wise, lookup-free format can close most of the accuracy gap between conventional scalar quantization and more complex vector- or trellis-based methods while remaining fully compatible with existing scalar inference kernels (Dadgarnia et al., 20 Apr 2026).

GSQ partitions weights into groups and assigns each group a single real scale $s^{(1)}$8, with no per-level codebook and no zero-point. Each weight $s^{(1)}$9 is mapped to a small symmetric integer grid $S^{(2)}$0, and runtime dequantization remains the single fused multiply

$S^{(2)}$1

Because the grid is fixed and scalar, the format is compatible with llama.cpp- and vLLM-style low-bit GEMM backends.

The novelty lies in how the assignments are optimized. For each weight, GSQ introduces logits $S^{(2)}$2 over grid levels and uses a Gumbel-Softmax relaxation:

$S^{(2)}$3

with relaxed dequantization

$S^{(2)}$4

As the temperature $S^{(2)}$5 is annealed and the scale factor $S^{(2)}$6 increased, the relaxed assignments harden toward one-hot vectors. For bit-widths above 2, GSQ replaces the global $S^{(2)}$7-way relaxation with a five-way local-shift relaxation around an initialized grid index, reducing parameter count from $S^{(2)}$8 to approximately $S^{(2)}$9 while still allowing accurate local moves (Dadgarnia et al., 20 Apr 2026).

The calibration objective is distillation-style reconstruction:

$q^{(8)}$00

The paper uses group-wise symmetric quantization with group size $q^{(8)}$01 by default, warm-starts the logits from a GPTQ solution, anneals $q^{(8)}$02 linearly from $q^{(8)}$03 to $q^{(8)}$04, increases $q^{(8)}$05 from $q^{(8)}$06 to $q^{(8)}$07, uses Lion rather than AdamW, and performs a final scale-only fine-tuning pass with discrete assignments fixed (Dadgarnia et al., 20 Apr 2026).

On Llama-3.1-8B/70B-Instruct, the reported zero-shot averages show that GSQ narrows or nearly closes the gap to QTIP at very low precision. In uniform 3-bit quantization, GSQ reaches average $q^{(8)}$08 on 8B and $q^{(8)}$09 on 70B, compared with QTIP at $q^{(8)}$10 and $q^{(8)}$11. In uniform 2-bit quantization, GSQ reaches $q^{(8)}$12 on 8B and $q^{(8)}$13 on 70B, compared with EfficientQAT at $q^{(8)}$14 and $q^{(8)}$15, and QTIP at $q^{(8)}$16 and $q^{(8)}$17. The paper also reports ternary results and non-uniform 2/3-bit configurations, including GSQ averages of $q^{(8)}$18 at $q^{(8)}$19 bpp and $q^{(8)}$20 at $q^{(8)}$21 bpp on 70B. On L40s GPUs with vLLM plus Humming, throughput rises from $q^{(8)}$22 TPS/GPU for BF16 to $q^{(8)}$23 for uniform 3-bit, $q^{(8)}$24 for $q^{(8)}$25 bpp, $q^{(8)}$26 for $q^{(8)}$27 bpp, and $q^{(8)}$28 for uniform 2-bit. The same format scales to trillion-parameter MoE quantization in Kimi-K2.5, where the paper reports mixed gains and losses across reasoning, coding, and long-context benchmarks, underscoring the importance of calibration data composition (Dadgarnia et al., 20 Apr 2026).

GSQ therefore broadens the meaning of GQ: it shows that lookup-free execution need not imply crude or greedy scalar assignment, provided the training-time discrete optimization is strong enough.

7. Comparative interpretation, misconceptions, and limitations

Taken together, these works indicate that GQ is better understood as a design principle than as a single method. The unifying principle is not a common formula, but a common constraint: group-level expressivity must be retained without introducing runtime lookup overheads that negate the systems benefits of quantization. This suggests a cross-domain pattern in which the quantizer becomes more structured offline so that the deployed representation becomes simpler online (Zhang et al., 2023, Zhuang et al., 7 Aug 2025, Pan et al., 2024, Dadgarnia et al., 20 Apr 2026).

A second clarification concerns the meaning of “lookup-free.” In WeTok, the eliminated operation is nearest-neighbor search over learned prototypes. In DGQ, the eliminated operation is per-group scale lookup or dequantization inside the GEMM inner loop. In Winograd GQ, the eliminated objects are codebooks and LUT indices, while affine quantization metadata remains. In GSQ, the eliminated structure is any per-level table or vector-code decoding step, leaving only integer codes and group scales. The phrase therefore refers to the removal of runtime indirection, but the source of that indirection differs by domain.

The limitations are correspondingly domain-specific. DGQ remains sensitive to projection error, grid-search resolution, activation outliers, and hardware availability of efficient INT8 kernels (Zhang et al., 2023). WeTok’s sign quantizer discards amplitude information and relies on grouped entropy approximations and decoder capacity to recover fine detail (Zhuang et al., 7 Aug 2025). Winograd GQ depends on learned transform scales to tame severe range imbalance, with larger tiles making the problem harder and group size constrained by vector-width considerations (Pan et al., 2024). GSQ remains dependent on calibration quality, can underperform under domain shift or at extreme context lengths, and still trails the strongest vector or trellis methods in some low-bit regimes (Dadgarnia et al., 20 Apr 2026).

The broader significance of GQ lies in this repeated compromise. Purely coarse-grained quantization is kernel-friendly but often accuracy-limited at low precision; purely fine-grained or prototype-heavy quantization is accurate but can disrupt continuous integer execution or create codebook overheads. GQ methods occupy the middle ground: they preserve much of the statistical benefit of grouping while restoring a runtime path that is compatible with standard kernels, vectorized operators, or implicit-code decoding.