BlockDialect: Efficient 4-Bit Quantization

• BlockDialect is a block-wise quantization method that uses specialized 4-bit floating-point dialects to enhance numerical representation.
• It partitions tensors into blocks (e.g., B=16,32,64) and selects an optimal numeric format per block, reducing quantization error and boosting energy efficiency.
• The approach employs a two-stage selection process for activations and demonstrates significant improvements in accuracy and power efficiency over traditional methods.

BlockDialect is a block-wise quantization method for LLM inference that introduces fine-grained, mixed-format representation using a library of specialized 4-bit floating-point variants. This approach enables each block of weights or activations to select not just a dedicated scale, but also an optimal numeric format (“dialect”) from a pre-defined set, with the goal of minimizing representation error and improving energy efficiency in full-path, low-precision matrix multiplication. BlockDialect expands upon traditional quantization strategies by focusing on adaptive format selection within blocks, resulting in significant gains in accuracy and energy savings relative to prior 4-bit and mixed-precision schemes (Jang et al., 2 Jan 2025).

1. Quantization Challenges in LLMs

The size of contemporary LLMs, such as LLaMA-3, which can reach tens of billions of parameters, introduces formidable memory and compute demands. Traditional 16-bit or 32-bit formats for weights and activations impose tens of gigabytes of storage and high energy costs due to frequent data movement and multiplication–accumulation (MAC) operations. Reducing operand bit-width to 4 bits can, in principle, provide 2–4× improvements in memory and computation efficiency.

However, coarse quantization strategies are limited by outliers. Global scaling factors become overly permissive, compressing most values into a narrow subinterval and causing substantial rounding error. Block-wise scaling addresses some of this, but uniform numeric formats (e.g., INT4 or FP4) for every block still lead to poor matching to each block’s distribution, wasting representational capacity—particularly when block data diverges markedly in range or clustering.

2. BlockDialect Framework and DialectFP4 Formatbook

BlockDialect partitions tensors into 1D blocks of size $B\in\{16,32,64\}$ (along channel or token axes), enabling each block $b$ to be assigned an optimal dialect $f_b$ and a shared power-of-two scale $s_b=2^{e_b}$. The innovation centers on the use of DialectFP4, a “formatbook” comprising 16 FP4-like 4-bit formats, where each dialect maintains the same granularity (0.5) as standard FP4 but reallocates the largest representable values.

In standard FP4 E2M1, the range is up to 6.0 with steps of 0.5; however, profiling reveals that some blocks rarely approach this maximum, while others frequently surpass it. DialectFP4 dialects, each defined by a mapping $v_f(m)$ for $m \in [0..7]$, flexibly cover $[0,8)$ by shifting which of the highest magnitudes (e.g., 5.0, 6.0, 7.5, etc.) are precisely representable.

<table> <thead> <tr><th>Dialect Index</th><th>Unique Largest Values</th><th>Primary Target Range</th></tr> </thead> <tbody> <tr><td\>0</td><td\>6.0</td><td>[0,6)</td></tr> <tr><td\>3</td><td\>7.5</td><td>[0,7.5)</td></tr> <tr><td\>8</td><td\>4.0,6.5</td><td>[0,6.5)</td></tr> </tbody> </table>

This structure allows allocation of extra resolution or headroom for blocks whose magnitudes cluster in different subintervals of $[0,8)$, minimizing either wasted range or clipping. The encoding cost is $4$ bits per element for the sign and index, plus $4$ bits per block for the dialect ID.

3. Two-Stage Block Dialect Selection for Activations

For weight quantization, dialect selection is performed offline via an exact mean squared error (MSE) search across the 16 candidates. For activations, where calibration is impractical, BlockDialect uses a two-stage on-the-fly method:

Stage 1: Range Matching

Identify the pair of dialects whose maximum representable value matches the block’s maximum preprocessed value $M_b$ (rounded to the nearest $0.5$ in quantized fixed-point form).

Stage 2: Distribution Matching

The candidate dialects differ in only one large-value slot. For each, determine its beneficial range (the half-interval around the unique large representable), and count how many block entries fall within. Select the dialect covering the greater share, thereby minimizing block-wise MSE without per-sample calibration.

This process removes the need for computationally expensive per-dialect MSE evaluation and sample-based calibration while preserving near-optimal quantization for activations.

4. Mathematical Formulation and Quantization Properties

The quantized block $\hat{w}_b$ is stored as

$$\hat{w}_b = s_b \cdot Q_{f_b}\left(w_b / s_b\right)$$

where $Q_f(x) = v_f(\operatorname{round}(x/0.5))$ with clamping to valid indices. Offline, blockwise dialect selection for weights proceeds as

$$f_b = \arg\min_{f \in \mathcal{F}} \|w_b - s_b \cdot Q_f(w_b / s_b)\|^2$$

For activations, the same formula holds, but dialect selection follows the two-stage logical method.

Scaling and rounding are implemented efficiently, since division by $s_b \in 2^\mathbb{Z}$ is executed as exponent subtraction, and rounding exploits just two fraction bits. The per-element quantization error is bounded by $0.25$ (half the granularity), plus possible out-of-range clipping.

Effective storage overhead consists of $4$ bits per element plus $$(5\ \text{bits for exponent} + 4\ \text{bits for dialect}) / B$$.

5. Hardware Efficiency and MAC Design

BlockDialect is hardware efficient because all arithmetic is mapped to integer-friendly operations:

• Dequantization reconstructs $(–1)^{\text{sign}}\cdot(\mu\cdot 0.5)\cdot s_b$, with sign recovery by a 1-bit XOR, and the magnitude component $\mu$ is a $4$-bit unsigned integer.
• Matrix multiplies are performed as $4$-bit unsigned $\times$ $4$-bit unsigned ($8$-bit product), right-shifted to account for the $0.5$ scaling, and accumulated in FP16 or INT16 sum.

A custom “DialectFP4 MAC” block matches the area and power of a basic 4-bit integer MAC (and can achieve slightly lower cost than a 4-bit FP MAC), leading to $4\times$ throughput compared to INT16/FP16 MACs. In synthesized 45 nm logic at 0.5 GHz, this MAC is approximately $1.58\times$ more power-efficient than the FP6 MAC required by prior-mixed format quantization (MXFP6) to match the same accuracy, and $2.45\times$ more power-efficient than an INT8 MAC.

6. Empirical Results and Ablation Analysis

BlockDialect was evaluated on full-path quantization for both weights ($W$) and activations ($A$) on LLaMA3-8B and LLaMA2-7B, quantizing every matrix multiplication. For $B=32$, results include:

• LLaMA3-8B: Perplexity $7.87$ (vs. MXFP4(32) $16.69$ and FP16 $6.14$); zero-shot accuracy $68.57\%$ (vs. MXFP4(32) $58.89\%$, FP16 $74.46\%$).
• LLaMA2-7B: BlockDialect yields $+6.90\%$ accuracy over MXFP4, and a gap of $-3.31\%$ w.r.t. FP16.

The effective bitwidth at $B=32$ is $4.28$ for BlockDialect versus $4.16$ for MXFP4, accounting for dialect and exponent overhead.

Ablation studies demonstrate that smaller block size ($B=16$) gives slightly higher accuracy at the cost of higher overhead, while $B=64$ favors better compression and throughput. The two-stage quantization for activations incurs only minimal accuracy loss ($-0.61\%$) and perplexity increase ($+0.04$) relative to full MSE search. Sixteen dialects strike a balance; fewer under-cover block distributions, while more (e.g., $24$) introduce noise. Partial combinations with SmoothQuant provide only marginal additional gains ($\sim0.4\%$).

7. Extensions and Future Directions

BlockDialect alters the central quantization paradigm from focusing on scaling to emphasizing high-fidelity numeric representability at the block level, utilizing a compact library of 4-bit floating-point variants. This achieves state-of-the-art results for joint 4-bit weight and activation quantization in LLM inference, with only single-digit percentage loss relative to FP16 and an order-of-magnitude gain over previous FP4 approaches.

Promising avenues for further development include automatic selection of block size per layer or sublayer, extending the dialect concept to other bitwidths (2-bit, 6-bit), integration with quantization-aware training regimes, and the design of dialectbooks that adapt dynamically over the course of training or fine-tuning (Jang et al., 2 Jan 2025).