---
title: 'Q-Palette: Fractional-Bit PTQ Framework'
url: https://www.emergentmind.com/topics/q-palette
type: topic
---

# Q-Palette: Fractional-Bit PTQ Framework

Q-Palette is a framework for **weight-only post-training quantization (PTQ)** of large language models that targets **memory-bound inference**, especially **small-batch decoding** such as edge or personalized inference on laptops and phones. It studies the regime in which model weights are quantized **without retraining**, and with **little or no calibration data**, then connects that practical problem to an information-theoretic analysis of **Gaussianized** weight distributions produced by rotation-based incoherence processing. Its central claim is that, under a fixed memory budget, the optimal allocation of quantization rates across layers is generally **fractional**, not integer-valued, and that practical deployment therefore requires both **fine-grained fractional-bit quantizers** and an optimization framework that can exploit them. To realize this, the work introduces a “palette” of quantizers—**trellis-coded quantization (TCQ)**, **vector quantization (VQ)**, and **non-uniform scalar quantization (NUQ)**—together with optimized CUDA kernels and a mixed-scheme quantization framework that jointly selects quantizer choices, and in one variant also layer fusion, under resource constraints [2509.20214].

## 1. Problem setting and motivation

Q-Palette studies **weight-only PTQ** for LLM deployment. In this setting, only the model weights are quantized, while activations remain unquantized, and the procedure avoids retraining. The motivating deployment regime is **memory-bound inference**, particularly **small-batch decoding**, where moving weights from memory dominates runtime. In that setting, weight-only PTQ is presented as important not only for compression but also for reducing bandwidth demand and improving decoding latency and throughput [2509.20214].

The paper identifies a specific obstacle: LLM weights often exhibit **irregular, heavy-tailed distributions with outliers**. These outliers widen the effective quantization range and increase distortion for simple quantizers. This motivates **rotation-based incoherence processing**, which transforms weights into a representation with fewer outliers and an approximately Gaussian distribution. The paper refers to these transformed weights as **Gaussianized**. Once weights are Gaussianized, quantization can be analyzed through the lens of **Gaussian source coding**, which yields a clean distortion-rate benchmark and leads directly to the paper’s bit-allocation theory [2509.20214].

This suggests that Q-Palette is not merely a new quantizer implementation. It is a framework built around a specific thesis: if Gaussianization makes the weight distribution resemble a Gaussian source, then practical PTQ should approximate the rate–distortion-optimal solution for that source as closely as hardware and inference constraints allow.

## 2. Information-theoretic formulation and optimal bit allocation

The theoretical starting point is a linearized approximation to the loss increase from quantization. For layers \(W_l\), quantizers \(Q(W_l)\), and layer sensitivities \(a_l\), the paper writes:
\[
\mathcal{L}(\{Q(W_l)\}_{l=1}^L) - \mathcal{L}(\{W_l\}_{l=1}^L) \approx \sum_{l=1}^L a_l \underbrace{\|Q(W_l)-W_l\|^2/\|W_l\|^2}_{\eqqcolon~\mathrm{err}(Q;W_l)}.
\]
Under a finite quantizer set \(\mathcal{Q}\), this yields a multiple-choice knapsack problem over layerwise quantizer assignments. Q-Palette then relaxes the discrete problem to a continuous one by assuming ideal Gaussian quantizers available at arbitrary real-valued bitwidths \(b_l\) and using the Gaussian distortion-rate law
\[
\mathbb{E}[\mathrm{err}(Q)] \ge 2^{-2\,\mathrm{bit}(Q)}.
\]
This leads to the constrained optimization problem
\[
\begin{aligned}
\min_{b_l \ge \eta}\quad & \sum_{l=1}^L a_l 2^{-2b_l} \\
\text{s.t.}\quad & \sum_{l=1}^L b_l\, d_l^{\mathrm{in}}d_l^{\mathrm{out}} \le M,
\end{aligned}
\]
where \(M\) is the total bit budget and \(\eta\) is a minimum bitwidth floor [2509.20214].

The paper derives a closed-form solution:
\[
b_l^* = \max\left\{ \eta,\; \frac{1}{2\ln 2}\left( \ln \frac{a_l}{d_l^{\mathrm{in}}d_l^{\mathrm{out}}} + C \right) \right\},
\]
with \(C\) chosen so that
\[
\sum_l b_l^*\, d_l^{\mathrm{in}}d_l^{\mathrm{out}} = M.
\]
The crucial consequence is that the optimal \(b_l^*\) is generally **real-valued**, so the theoretically optimal solution assigns rates such as \(2.87\) or \(3.23\) bits rather than restricting layers to \(2\), \(3\), or \(4\) bits. This is the formal basis for the paper’s claim that **fractional-bit quantizers are essential** [2509.20214].

The paper further decomposes the practical gap to the continuous optimum into two terms:
\[
\underbrace{ \sum_{l=1}^L a_l \left(\mathrm{err}(Q_l^*) - 2^{-2b_l^*}\right) }_{\text{Total gap}}
=
\underbrace{ \sum_{l=1}^L a_l \left(\mathrm{err}(Q_l^*) - 2^{-2\mathrm{bit}(Q_l^*)}\right) }_{\text{Distortion gap}}
+
\underbrace{ \sum_{l=1}^L a_l \left(2^{-2\mathrm{bit}(Q_l^*)} - 2^{-2b_l^*}\right) }_{\text{Bit allocation gap}}.
\]
Here, the **distortion gap** measures how far a practical quantizer is from the Gaussian bound at its own bitwidth, while the **bit allocation gap** measures how poorly the available bitwidth grid approximates the ideal \(b_l^*\). This decomposition is central to Q-Palette’s design: strong quantizers reduce the first gap, and dense fractional-bit support reduces the second [2509.20214].

## 3. Quantizer families and fractional-bit construction

Q-Palette is a **collection** of quantizers rather than a single quantization scheme. The three main families are **NUQ**, **VQ**, and **TCQ**, with **half-TCQ** added to achieve finer granularity.

| Quantizer | Main role | Supported bitwidths |
|---|---|---|
| NUQ | low dequantization overhead, fast inference | \(2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0\) |
| VQ | middle ground between distortion and speed | \(1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5, 6.0\) |
| TCQ / half-TCQ | near-optimal distortion for Gaussianized weights | TCQ: \(1.5\) to \(5.0\); half-TCQ: \(1.75, 2.25, 2.75, 3.25, 3.75, 4.25, 4.75\) |

**Non-uniform scalar quantization (NUQ)** maps each scalar to a learned lookup table \(\mathrm{LUT}\in\mathbb{R}^{2^b}\), built by \(k\)-means on Gaussian samples. Quantization uses the round-to-nearest rule
\[
\mathrm{RTN}(v;\mathrm{LUT}) \coloneqq \arg\min_{\mathbf{r}\in\{0,1\}^b} \left|v-\mathrm{LUT}[\mathrm{int}(\mathbf{r})]\right|,
\]
and dequantization is a direct table lookup:
\[
\mathrm{dq}(\mathbf{r};\mathrm{LUT}) \coloneqq \mathrm{LUT}[\mathrm{int}(\mathbf{r})].
\]
NUQ is positioned as the simplest family, with low dequantization overhead and high inference efficiency, but weaker distortion than TCQ [2509.20214].

**Vector quantization (VQ)** groups weights into 2D vectors. For scalar bitwidth \(b\), the codebook has \(2^{2b}\) entries:
\[
\mathrm{LUT}\in\mathbb{R}^{2^{2b}\times 2}.
\]
Quantization is again nearest-neighbor:
\[
\mathrm{RTN}(\mathbf{v};\mathrm{LUT}) \coloneqq \arg\min_{\mathbf{r}\in\{0,1\}^{2b}} \|\mathbf{v}-\mathrm{LUT}[\mathrm{int}(\mathbf{r})]\|_2.
\]
VQ naturally supports **half-bit increments**. A \(2^3\)-entry codebook over 2D vectors uses 3 bits for 2 scalars, i.e. \(1.5\) bits per scalar. The paper places VQ between NUQ and TCQ in the distortion-latency trade-off [2509.20214].

**Trellis-coded quantization (TCQ)** is the highest-fidelity family in Q-Palette and is described as near-optimal for Gaussian sources. Using the bitshift variant with tail-biting, dequantization is defined over a sliding window:
\[
\hat{\mathbf{v}}[i\cdot V : (i+1)\cdot V] = \mathrm{LUT}\!\left(\mathbf{r}[i\cdot s : i\cdot s + L]\right),
\]
with \(V=2\), \(L=16\), \(T=256\), and effective bitwidth
\[
\frac{s}{V}.
\]
Because \(V=2\), odd \(s\) produces half-bit rates such as \(1.5\), \(2.5\), and \(3.5\). Encoding uses a Viterbi-based round-to-nearest operator. TCQ is the quantizer family that most closely tracks the Gaussian-optimal RMSE curve \(2^{-x}\), corresponding to the Gaussian MSE law \(2^{-2b}\) [2509.20214].

**Half-TCQ** is introduced to realize quarter-step average bitwidths. A weight matrix is split row-wise; one part is quantized at \(b\) bits and the other at \(b+0.5\) bits, yielding average rate \(b+0.25\). This produces bitwidths such as \(2.25\), \(2.75\), and \(3.25\), which allow Q-Palette to approximate the continuous optimum more closely than a half-bit grid alone [2509.20214].

The paper also describes both **data-free** and **data-aware** quantization modes. In data-free mode, weights are partitioned and quantized directly with the quantizer’s round-to-nearest rule. In data-aware mode, the same quantizer families are used inside a block LDLQ framework with Hessian guidance, with block sizes 1 for NUQ, 2 for VQ, and 16 for TCQ [2509.20214].

## 4. Systems implementation and kernel design

A central practical claim of Q-Palette is that sophisticated Gaussian quantizers need not be confined to narrow rate sets or batch size 1. The paper therefore devotes substantial effort to systems design, combining quantizer families with optimized CUDA kernels [2509.20214].

To reduce the overhead of incoherence processing, the paper rotates only along the **input dimension**,
\[
W \to RW,
\]
then applies **per-output-channel scaling** so each rotated column is approximately standard Gaussian. It also shares rotation matrices among layers with the same input, such as \(q/k/v\) and \(u/g\) projections. This reduces the number of online rotations per transformer block from **14 to 4** [2509.20214].

Two kernel families are implemented: **Tensor Core-based** kernels and **CUDA Core-based** kernels. Tensor Core kernels support **TCQ, half-TCQ, VQ, and NUQ**. They extend the QTIP-style warp-level MMA design by performing **register-level dequantization** directly on load, traversing each quantized weight only once for larger batch sizes, and caching input activations in shared memory. CUDA Core kernels support **NUQ and VQ**, extend Any-Precision LLM kernels, replace bit-plane encoding with bit-packing, and use shared-memory LUTs together with `half2` and `hfma2` operations [2509.20214].

The reported throughput results on **LLaMA 3.1-8B** and **RTX 4090** show that these implementation choices materially affect deployment. At **batch size 1**, relative to FP16, **QTIP 2-bit** achieves \(2.91\times\), **Ours-TCQ-TC 2-bit** achieves \(3.57\times\), and **Ours-NUQ-CC 2-bit** achieves \(3.70\times\). At **batch size 8**, **QTIP 2-bit** drops to \(0.74\times\), while **Ours-TCQ-TC 2-bit** reaches \(3.16\times\) and **Ours-NUQ-TC 2-bit** reaches \(3.28\times\) [2509.20214].

The paper states that **CUDA Core kernels** tend to be better for **batch size 1**, whereas **Tensor Core kernels** tend to be better for **larger batch sizes**. This suggests that Q-Palette is designed not just as an abstract quantization method but as a deployment-oriented toolkit whose best operating point depends on the inference regime.

## 5. Mixed-scheme quantization and fusion-aware optimization

Q-Palette uses its quantizer collection inside a **mixed-scheme quantization (MSQ)** framework. The generic resource-constrained problem is written as
\[
\begin{aligned}
\min_{P_{lq}\in\{0,1\}} \quad & \sum_{l=1}^{L}\sum_{q=1}^{|\mathcal{Q}|} P_{lq}\,\ell_{lq} \\
\text{s.t.}\quad & \sum_{q=1}^{|\mathcal{Q}|} P_{lq}=1,\quad \forall l, \\
& \sum_{l=1}^{L}\sum_{q=1}^{|\mathcal{Q}|} P_{lq}\,c_{lq}\le C,
\end{aligned}
\]
where \(P_{lq}\) selects quantizer \(Q_q\) for layer \(l\), \(\ell_{lq}\) is estimated loss, and \(c_{lq}\) is either memory or latency cost. In data-free settings, the paper uses
\[
\ell_{lq} = a_l \cdot \mathrm{err}(Q_q;W_l),
\]
and estimates \(\mathrm{err}(Q_q;W_l)\) from quantizer distortion on random Gaussian matrices, exploiting the Gaussianization assumption [2509.20214].

The more novel extension is **fusion-aware MSQ**, which jointly chooses quantizer assignments and layer fusion patterns. Transformer layers such as
\[
XW_q,\; XW_k,\; XW_v
\quad\to\quad
X(W_q \oplus W_k \oplus W_v)
\]
and
\[
XW_u,\; XW_g
\quad\to\quad
X(W_u \oplus W_g)
\]
can be fused to reduce kernel launches, memory accesses, and latency. The paper defines a set of fusible groups \(\mathcal{G}\), including singleton groups and fused \(q/k/v\) or \(u/g\) groups, and optimizes
\[
\begin{aligned}
\min_{P_{gq}\in\{0,1\}} \quad & \sum_{g\in\mathcal{G}}\sum_{q=1}^{|\mathcal{Q}|} P_{gq}\cdot \sum_{l\in g}\ell_{lq} \\
\text{s.t.}\quad & \sum_{g\in\mathcal{G}: l\in g}\sum_{q=1}^{|\mathcal{Q}|} P_{gq}=1, \quad \forall l, \\
& \sum_{g\in\mathcal{G}}\sum_{q=1}^{|\mathcal{Q}|} P_{gq}\,c_{gq}\le C.
\end{aligned}
\]
Constraint (C1) enforces exclusive assignment of each layer to one active group, and (C2) enforces the resource budget [2509.20214].

This framework is important because quantization and fusion are not separable. A fused group may require a shared quantizer assignment, which can reduce fidelity, but the same fusion may reduce latency enough to permit better bit allocation elsewhere. The paper solves both MSQ and fusion-aware MSQ with **SCIP in OR-Tools**, using a 60-second time limit for the fusion-aware ILP [2509.20214].

## 6. Empirical performance, trade-offs, and limitations

The experiments cover **LLaMA 3.1-8B**, **LLaMA 3.1-70B**, **LLaMA 3.2-1B**, **LLaMA 3.2-3B**, **LLaMA 2-7B**, **LLaMA 2-13B**, and **Qwen 2.5-7B**. Evaluation uses **WikiText2 perplexity** and average zero-shot accuracy over **ARC-easy, ARC-challenge, HellaSwag, PiQA, and WinoGrande**, with throughput measured mainly on **RTX 4090** and additionally on **RTX 3090** [2509.20214].

In the **data-free** regime, the main pattern is that single-scheme **TCQ** already improves over strong baselines, while **MSQ** improves further. On **LLaMA 3.1-8B** at **3.25 bits**, **NF** gives Wiki2 \(7.70\), **HIGGS** gives \(6.64\), **Ours-TCQ-3.25** gives \(6.48\), **HIGGS-MSQ** gives \(6.39\), and **Ours-MSQ-Mem** gives \(6.10\). At **4.0 bits**, **Data-free QTIP** gives \(5.94\), **Ours-TCQ-4** gives \(5.92\), and **Ours-MSQ-Mem** gives \(5.81\) [2509.20214].

The paper emphasizes Pareto improvements under memory constraints. One highlighted example is that its **2.875-bit** model attains perplexity comparable to **3.25-bit HIGGS-MSQ**, implying about **\(1.13\times\)** higher compression at similar or better quality. Similar trends are reported for **Qwen 2.5-7B** and **LLaMA 3.1-70B**, where Q-Palette’s MSQ results match or exceed higher-bit HIGGS-MSQ models, implying up to **12.5%** and **15%** memory savings, respectively [2509.20214].

Under latency constraints, the gains are also pronounced. On **LLaMA 3.1-8B**, **RTX 4090**, and **batch size 1**, the paper reports: **NF3G64 + FLUTE** at Wiki2 \(7.70\) and speedup \(2.33\times\); **TCQ-3.25** at \(6.48\) and \(2.72\times\); **MSQ with Q-Palette** at \(6.46\) and \(2.99\times\); and **Fusion-aware MSQ with Q-Palette** at \(6.39\) and \(3.17\times\). In the data-aware setting on **LLaMA 2-7B**, **QTIP 2-bit** gives Wiki2 \(6.84\), accuracy \(58.9\), and throughput \(209/386\) tokens/s at batch sizes \(1/8\), while **Ours-MSQ-Mem** gives \(6.47\), \(60.3\), and \(272/1684\) [2509.20214].

Ablations support the paper’s two-gap theory. Comparing **VQ-2,3,4**, **TCQ-2,3,4**, and **TCQ-ALL**, the broader **TCQ-ALL** set consistently performs best, indicating that richer fractional-bit support matters in addition to the underlying quantizer quality. The Gaussian quantization-error curves show **uniform** quantization as worst, **NUQ** better, **VQ** better still, and **TCQ** closest to the Gaussian-optimal line \(2^{-x}\). This aligns with the interpretation that TCQ minimizes the **distortion gap**, while dense rate support minimizes the **bit allocation gap** [2509.20214].

The paper also notes several limitations. First, the framework is designed for **one-shot PTQ/MSQ**, not retraining-based quantization. Second, estimating sensitivity coefficients \(a_l\) incurs a one-time cost that scales with the number of layers. Third, the work focuses on **weight-only PTQ**, not weight-activation quantization. Fourth, some hardware environments require integer GEMMs, where this style of dequantize-on-the-fly weight-only quantization may not directly fit [2509.20214].

A common misconception is that Q-Palette is simply “another low-bit quantizer.” The paper’s own structure argues against that interpretation. Q-Palette is simultaneously a **theory of fractional-bit optimality for Gaussianized weights**, a **family of Gaussian-aware quantizers**, a **CUDA implementation**, and an **optimization framework** that chooses quantizers and, in one variant, fusion patterns under realistic memory or latency budgets. This suggests that its contribution lies as much in connecting rate–distortion theory to deployment decisions as in any individual quantizer design.

Source: https://www.emergentmind.com/topics/q-palette