Fused Two-Level Table-Based Dequantization

• The paper presents a novel dequantization scheme that fuses dual-level lookup tables to directly transform low-bit codes without materializing full floating-point tensors.
• It achieves significant speedups—up to 48× in attention performance and 10.2× faster dequantization—while drastically reducing memory bandwidth usage.
• The method balances trade-offs between arithmetic reuse and lookup fusion, optimizing hardware-specific operations across GPUs, NPUs, and custom accelerators.

Searching arXiv for the cited papers and closely related work on fused dequantization and LUT-based low-bit inference. arXiv search: "fused dequantization lookup table low-bit LLM inference" Fused two-level table-based dequantization denotes a low-bit inference design in which dequantization is not executed as an independent tensor-wide pass that materializes floating-point weights or KV tensors. Instead, low-bit codes are transformed through two levels of lookup or parameter application, and the resulting values are consumed immediately by attention or matrix multiplication. In the literature, the term is explicit in T-MAN, where dequantization on NPUs is split into a bit-repacking LUT and a block-specific integer-to-FP affine LUT (Wei et al., 14 Nov 2025). Closely related systems implement the same structure algebraically rather than as explicit tables: Open-TQ-Metal dequantizes packed int4 KV “in registers” inside a fused compressed-domain attention kernel, and FLUTE realizes a global codebook plus group-wise scaling inside a fused matmul kernel (Vegasena, 18 Apr 2026, Guo et al., 2024).

1. Definition and formal structure

The basic dequantization map in these systems is the affine form

$\hat{x} = (q - z)\cdot s,$

where $q$ is a low-bit integer code, $z$ is a zero-point, and $s$ is a scale. Open-TQ-Metal uses per-group asymmetric int4 quantization with group size $g$, computing

$$s = \frac{\max - \min}{15}, \qquad z = \frac{-\min}{s},$$

followed by forward quantization

$$q = \text{clamp}\!\left( \left\lfloor \frac{x}{s} + z \right\rceil, 0, 15 \right),$$

and inverse mapping

$\hat{x} = (q - z)\cdot s.$

The same paper notes that this affine map can be rewritten as a group-specific 16-entry lookup table,

$\text{LUT}_g[q] = (q - z_g)\cdot s_g,$

which is mathematically equivalent to one-level table-based dequantization even though the implementation performs arithmetic rather than literal LUT fetches (Vegasena, 18 Apr 2026).

A two-level interpretation arises when the mapping is factored into a codebook stage and a scale/bias stage. Open-TQ-Metal makes this explicit as

$\hat{x} = s_g\cdot c[q] + b_g,\qquad b_g=-s_g z_g,$

with a global codebook $q$0 and group-specific parameters $q$1 (Vegasena, 18 Apr 2026). FLUTE presents an analogous decomposition for lookup-table-quantized weights: $q$2 where $q$3 is a tensor-level LUT and $q$4 is a per-group scale. In that formulation, the first level supplies the quantile-coded “shape,” and the second level supplies the group-wise “magnitude” (Guo et al., 2024).

Two-level table-based dequantization therefore refers less to one fixed data structure than to a class of fused mappings in which low-bit codes are first decoded into a codebook domain and then modulated by block-, group-, or channel-specific parameters. This structure may be realized by explicit LUTs, by sign logic plus a base LUT, or by inline arithmetic that is functionally equivalent.

2. Fusion semantics and execution model

The defining property of the fused variant is that dequantization does not produce a persistent floating-point tensor. Open-TQ-Metal contrasts a textbook pipeline—packed int4 KV, then dequantized FP16/FP32 KV, then scaled dot-product attention—with a single sdpa_int4 kernel that reads packed int4 keys and values directly from GPU memory, unpacks and dequantizes in registers, immediately uses the reconstructed values in the dot products and value accumulation, and never writes an intermediate dequantized matrix to memory (Vegasena, 18 Apr 2026). The decode-time algorithm combines register-level dequantization with online softmax: $q$5

T-MAN uses the phrase “fused two-level LUT-based dequantization” for a different but structurally related purpose. There, fusion occurs both within dequantization and across layout transformation. Level-1 LUT replaces cascades of bitwise shifts, masks, and ORs by bit-repacking a bit-serial representation into bit-parallel quantized integers. Level-2 LUT maps each small integer code directly to an FP16 value that already incorporates block-wise scale and zero-point: $q$6 The result is that bit extraction, integer-to-float conversion, and affine scaling are subsumed into two lookup stages rather than fine-grained scalar operations (Wei et al., 14 Nov 2025).

LUT Tensor Core generalizes the same fusion principle to mixed-precision GEMM. Instead of dequantizing low-bit weights and then multiplying, it precomputes small activation-dependent tables of partial dot products and replaces multiply-accumulate with table lookups plus additions. The paper states that table precomputation is “an element-wise operation” and can be fused with the preceding normalization operator, reducing precompute overhead “to almost zero.” In this design, the explicit $q$7 step disappears from the execution graph; its effect is encoded in LUT construction, indexing, and bit-serial accumulation (Mo et al., 2024).

Across these systems, fusion means that dequantization is consumed at the point of arithmetic. The floating-point value exists only as a transient register quantity or as the semantic output of a LUT access; it is not materialized as a separate buffer.

3. Two-level organizations in representative systems

The phrase “two-level” is instantiated differently across current implementations. The common pattern is a first stage that resolves low-bit symbolic structure and a second stage that applies group- or block-specific numeric semantics.

System	First level	Second level
Open-TQ-Metal (Vegasena, 18 Apr 2026)	int4 codebook or implicit $q$8	per-group $q$9 or $z$0
T-MAN (Wei et al., 14 Nov 2025)	bit-repacking LUT	block-specific FP16 affine LUT
LUT Tensor Core (Mo et al., 2024)	base LUT for 1-bit patterns	sign logic and bit-serial combination
FLUTE (Guo et al., 2024)	tensor-level LUT $z$1	group-wise scale $z$2

In Open-TQ-Metal, the paper does not describe a literal two-level LUT, but it states that the affine int4 quantizer “naturally admits such an interpretation.” A per-group table $z$3 can be precomputed, or the mapping can be decomposed into a fixed codebook plus group-specific scaling and bias (Vegasena, 18 Apr 2026). This makes the system a canonical example of an algebraically fused two-level design.

In T-MAN, the first level is not a numerical codebook at all but a structural one. A 4-bit LUT index representing one bitplane across four successive 4-bit weights can return an already shifted bit-parallel pattern such as 0b0000 0000 1000 1000, replacing twelve separate bit manipulation operations with one lookup for that step. Only after this reconstruction does the second-level LUT apply the quantization parameters and produce FP16 (Wei et al., 14 Nov 2025).

LUT Tensor Core realizes two-level behavior through table symmetrization. After symmetric reinterpretation, only half of the pattern table must be stored, and the highest bit determines whether the fetched value is negated: $z$4 The paper further extends this to bit-serial multi-bit accumulation, so a weight code is interpreted as a sum over bit planes, each reusing the same base LUT (Mo et al., 2024).

FLUTE provides a simpler but important variant. Its “two-level structure” is the pairing of a global NormalFloat-style LUT and per-group scales. Operationally, it also introduces a vectorized LUT over pairs of codes—256 entries for 4-bit codes—so that two dequantized values can be obtained with one shared-memory lookup before group scaling is applied (Guo et al., 2024).

4. Hardware realization and memory behavior

The operational value of fused two-level table-based dequantization is strongly hardware-dependent. On Apple Silicon, Open-TQ-Metal tailors sdpa_int4 to 32-wide SIMD groups, packed int4 storage, and a unified-memory system with approximately 400 GB/s bandwidth. The kernel uses vectorized nibble extraction, register-only temporaries, and split-K parallelism, and it avoids the bandwidth cost of writing and rereading a full $z$5 dequantized matrix. This is why the fused kernel shows super-linear speedup as context length grows: the baseline dequantize-then-attend path pays for separate materialization, whereas the fused path streams compressed KV once and performs dequantization in arithmetic and registers (Vegasena, 18 Apr 2026).

On Snapdragon-class NPUs, T-MAN maps the two levels onto distinct execution resources. HVX vector cores perform bit-repacking and FP16 LUT-based dequantization; HMX performs the resulting FP16 GEMM in prefill. The system is organized as a three-stage DMA–HVX–HMX pipeline: compressed weights are moved from DDR to TCM, dequantized by HVX using VLUT16, and then consumed by HMX. The same underlying bit-serial layout is also reused for decoding, where HVX interprets the weights directly as LUT-GEMV indices. The architectural point is that one stored weight layout serves both prefill and decoding, avoiding a second copy (Wei et al., 14 Nov 2025).

LUT Tensor Core proposes a software-hardware co-design rather than a software kernel alone. Its hardware adopts an elongated tile shape, with small $z$6, modest $z$7, and large $z$8, because table size grows exponentially with $z$9 while each LUT entry can be reused across many columns. The reported optimal tile shape is $s$0. Bit-serial decomposition then turns multi-bit weights into repeated accesses to a 1-bit-pattern LUT, and custom LMMA instructions expose the resulting lookup-based tensor-core primitive to the compilation stack (Mo et al., 2024).

FLUTE, by contrast, is tuned for GPUs where Tensor Core compute is abundant but memory movement dominates. It restructures the quantized weight matrix offline so that packed codes arrive in the register layout already required by MMA fragments, and it duplicates vectorized LUTs in shared memory to mitigate bank conflicts. In this setting, the benefit of fusion derives from reducing global-memory traffic and minimizing on-chip unpacking overhead rather than from replacing the entire matmul primitive (Guo et al., 2024).

5. Accuracy, throughput, and design trade-offs

Open-TQ-Metal reports that the fused sdpa_int4 kernel achieves a 48x attention speedup at 128K context over the dequantize-then-attend baseline, reduces KV cache memory from 40 GB to 12.5 GB, and maintains identical top-1 token predictions to FP16 inference. The same work also reports that attention scale factor, rather than model size, determines whether angular quantization schemes such as PolarQuant succeed or fail; under Gemma 4’s $s$1, directional error is amplified much more strongly than under Llama’s standard $s$2 scaling. The paper consequently reports a context ceiling of approximately 950 tokens for int4 KV on Gemma 4 in its experiments, whereas Llama works to 128K+ (Vegasena, 18 Apr 2026).

T-MAN isolates the dequantization contribution directly. On Snapdragon 8 Gen 3, LUT-based dequant is reported as 10.2x faster than a “ConvertDQ” baseline and 4.9x faster than loading pre-converted FP weights. End to end, the system reports 1.4x and 3.1x speedup for prefill and decoding, respectively, and 84% energy savings compared to baseline NPU methods. The method is described as lossless with respect to the underlying quantization scheme, because it implements $s$3 in FP16 up to standard FP16 rounding, and the paper reports substantially better perplexity than hardware-constrained per-channel INT4/INT16 formats for Qwen3-8B and Llama-3.1-8B (Wei et al., 14 Nov 2025).

LUT Tensor Core evaluates a more aggressive LUT-substituted mpGEMM path. It reports that fusion reduces table precompute overhead from 16–24% to approximately 2.5%, and that an A100-like GPU equipped with LUT Tensor Cores can achieve up to 6.93x inference speedup, with approximately 20.9x compute density and 11.2x energy efficiency gains. The same paper also reports that INT8 table quantization changes WikiText2 perplexity only from 7.68 to 7.69 for a 2-bit LLAMA2-7B model, with zero-shot and MMLU accuracy essentially unchanged or slightly improved (Mo et al., 2024).

FLUTE targets the memory-bound decoding regime on GPUs and reports that its kernel can be 2–4x faster than existing GEMM kernels at batch sizes below 32 and quantization group size 128, with end-to-end throughput increases of 1.5 to 2 times for LLaMA3 deployments. The trade-off is that LUT kernels introduce shared-memory pressure, bank-conflict sensitivity, and register pressure; the paper notes that uniform-quantization kernels can still be faster on A100 in some cases (Guo et al., 2024).

These results show that the main trade-off is not whether lookup is cheaper than arithmetic in isolation, but whether the chosen formulation reduces the dominant bottleneck for the target hardware. On Apple Silicon, the bottleneck is memory bandwidth for KV materialization; on Snapdragon NPUs, it is unsupported dequantization math on vector units; on LUT Tensor Core, it is lack of native mpGEMM support; and on GPUs targeted by FLUTE, it is global-memory movement into on-chip fragments.

6. Relation to adjacent approaches and conceptual boundaries

Fused two-level table-based dequantization should not be conflated with every LUT-based low-bit inference method. T-MAN explicitly distinguishes its design from LUT-GEMM-style approaches: in prefill, LUTs are used only for dequantization and layout transformation, while the main GEMM still runs on the native matrix core. In contrast, LUT Tensor Core uses LUTs to replace multiply-accumulate behavior inside mpGEMM itself, and FLUTE fuses LUT dequantization directly into Tensor Core matmul without an intermediate floating-point weight tensor (Wei et al., 14 Nov 2025, Mo et al., 2024, Guo et al., 2024).

Nor is an explicit LUT necessary for a design to belong to this family. Open-TQ-Metal states that its affine int4 quantizer is “fully compatible with a two-level table-based dequantization architecture,” but the authors favor in-register arithmetic over table lookups on Apple Silicon because bandwidth is scarce and arithmetic is relatively cheap. In this sense, fused two-level table-based dequantization names a structural equivalence class rather than a single implementation tactic (Vegasena, 18 Apr 2026).

A further boundary is exposed by Multi-Scale Dequant, which removes weight and KV dequantization from the GEMM critical path without using lookup tables. MSD decomposes BF16 activations into multiple low-precision components and performs multiple native low-precision GEMMs, reporting near 16 effective bits for a two-pass INT8 scheme, near 6.6 effective bits for a two-pass MXFP4 scheme, and up to 2.5 times lower KV-cache HBM traffic in attention relative to dequant baselines (Zheng et al., 13 May 2026). This suggests that the broader objective is dequantization elimination rather than LUT use as such. A plausible implication is that future fused two-level LUT designs will increasingly be evaluated against activation-side decomposition methods, not only against conventional dequantize-then-GEMM baselines.

In current practice, the concept therefore spans three closely related meanings: an explicit two-LUT dequant pipeline as in T-MAN, an implicit codebook-plus-scale decomposition as in FLUTE and Open-TQ-Metal, and a hierarchical sign/bit-plane organization as in LUT Tensor Core. The unifying criterion is that low-bit representations are mapped to compute-ready values through a compact multi-stage mechanism that is fused into the consuming kernel and avoids materializing a full dequantized tensor.