Q-Palette: Fractional-Bit PTQ Framework

• Q-Palette is a framework for weight-only PTQ that leverages Gaussianization to transform irregular weight distributions into nearly Gaussian ones, enabling fractional-bit quantization.
• It integrates multiple quantizer families (NUQ, VQ, TCQ, and half-TCQ) within an optimization framework to assign real-valued bit allocations across layers for optimal rate–distortion performance.
• Optimized CUDA and Tensor Core kernels support both singleton and fused quantization schemes, balancing speed and accuracy in memory-bound, small-batch inference scenarios.

Q-Palette is a framework for weight-only post-training quantization (PTQ) of LLMs that targets memory-bound inference, especially small-batch decoding such as edge or personalized inference on laptops and phones. It studies the regime in which model weights are quantized without retraining, and with little or no calibration data, then connects that practical problem to an information-theoretic analysis of Gaussianized weight distributions produced by rotation-based incoherence processing. Its central claim is that, under a fixed memory budget, the optimal allocation of quantization rates across layers is generally fractional, not integer-valued, and that practical deployment therefore requires both fine-grained fractional-bit quantizers and an optimization framework that can exploit them. To realize this, the work introduces a “palette” of quantizers—trellis-coded quantization (TCQ), vector quantization (VQ), and non-uniform scalar quantization (NUQ)—together with optimized CUDA kernels and a mixed-scheme quantization framework that jointly selects quantizer choices, and in one variant also layer fusion, under resource constraints (Lee et al., 24 Sep 2025).

1. Problem setting and motivation

Q-Palette studies weight-only PTQ for LLM deployment. In this setting, only the model weights are quantized, while activations remain unquantized, and the procedure avoids retraining. The motivating deployment regime is memory-bound inference, particularly small-batch decoding, where moving weights from memory dominates runtime. In that setting, weight-only PTQ is presented as important not only for compression but also for reducing bandwidth demand and improving decoding latency and throughput (Lee et al., 24 Sep 2025).

The paper identifies a specific obstacle: LLM weights often exhibit irregular, heavy-tailed distributions with outliers. These outliers widen the effective quantization range and increase distortion for simple quantizers. This motivates rotation-based incoherence processing, which transforms weights into a representation with fewer outliers and an approximately Gaussian distribution. The paper refers to these transformed weights as Gaussianized. Once weights are Gaussianized, quantization can be analyzed through the lens of Gaussian source coding, which yields a clean distortion-rate benchmark and leads directly to the paper’s bit-allocation theory (Lee et al., 24 Sep 2025).

This suggests that Q-Palette is not merely a new quantizer implementation. It is a framework built around a specific thesis: if Gaussianization makes the weight distribution resemble a Gaussian source, then practical PTQ should approximate the rate–distortion-optimal solution for that source as closely as hardware and inference constraints allow.

2. Information-theoretic formulation and optimal bit allocation

The theoretical starting point is a linearized approximation to the loss increase from quantization. For layers $W_l$, quantizers $Q(W_l)$, and layer sensitivities $a_l$, the paper writes: $$\mathcal{L}(\{Q(W_l)\}_{l=1}^L) - \mathcal{L}(\{W_l\}_{l=1}^L) \approx \sum_{l=1}^L a_l \underbrace{\|Q(W_l)-W_l\|^2/\|W_l\|^2}_{\eqqcolon~\mathrm{err}(Q;W_l)}.$$ Under a finite quantizer set $\mathcal{Q}$, this yields a multiple-choice knapsack problem over layerwise quantizer assignments. Q-Palette then relaxes the discrete problem to a continuous one by assuming ideal Gaussian quantizers available at arbitrary real-valued bitwidths $b_l$ and using the Gaussian distortion-rate law

$$\mathbb{E}[\mathrm{err}(Q)] \ge 2^{-2\,\mathrm{bit}(Q)}.$$

This leads to the constrained optimization problem

$$\begin{aligned} \min_{b_l \ge \eta}\quad & \sum_{l=1}^L a_l 2^{-2b_l} \ \text{s.t.}\quad & \sum_{l=1}^L b_l\, d_l^{\mathrm{in}}d_l^{\mathrm{out}} \le M, \end{aligned}$$

where $M$ is the total bit budget and $\eta$ is a minimum bitwidth floor (Lee et al., 24 Sep 2025).

The paper derives a closed-form solution: $Q(W_l)$0 with $Q(W_l)$1 chosen so that

$Q(W_l)$2

The crucial consequence is that the optimal $Q(W_l)$3 is generally real-valued, so the theoretically optimal solution assigns rates such as $Q(W_l)$4 or $Q(W_l)$5 bits rather than restricting layers to $Q(W_l)$6, $Q(W_l)$7, or $Q(W_l)$8 bits. This is the formal basis for the paper’s claim that fractional-bit quantizers are essential (Lee et al., 24 Sep 2025).

The paper further decomposes the practical gap to the continuous optimum into two terms: $Q(W_l)$9 Here, the distortion gap measures how far a practical quantizer is from the Gaussian bound at its own bitwidth, while the bit allocation gap measures how poorly the available bitwidth grid approximates the ideal $a_l$0. This decomposition is central to Q-Palette’s design: strong quantizers reduce the first gap, and dense fractional-bit support reduces the second (Lee et al., 24 Sep 2025).

3. Quantizer families and fractional-bit construction

Q-Palette is a collection of quantizers rather than a single quantization scheme. The three main families are NUQ, VQ, and TCQ, with half-TCQ added to achieve finer granularity.

Quantizer	Main role	Supported bitwidths
NUQ	low dequantization overhead, fast inference	$a_l$1
VQ	middle ground between distortion and speed	$a_l$2
TCQ / half-TCQ	near-optimal distortion for Gaussianized weights	TCQ: $a_l$3 to $a_l$4; half-TCQ: $a_l$5

Non-uniform scalar quantization (NUQ) maps each scalar to a learned lookup table $a_l$6, built by $a_l$7-means on Gaussian samples. Quantization uses the round-to-nearest rule

$a_l$8

and dequantization is a direct table lookup: $a_l$9 NUQ is positioned as the simplest family, with low dequantization overhead and high inference efficiency, but weaker distortion than TCQ (Lee et al., 24 Sep 2025).

Vector quantization (VQ) groups weights into 2D vectors. For scalar bitwidth $$\mathcal{L}(\{Q(W_l)\}_{l=1}^L) - \mathcal{L}(\{W_l\}_{l=1}^L) \approx \sum_{l=1}^L a_l \underbrace{\|Q(W_l)-W_l\|^2/\|W_l\|^2}_{\eqqcolon~\mathrm{err}(Q;W_l)}.$$0, the codebook has $$\mathcal{L}(\{Q(W_l)\}_{l=1}^L) - \mathcal{L}(\{W_l\}_{l=1}^L) \approx \sum_{l=1}^L a_l \underbrace{\|Q(W_l)-W_l\|^2/\|W_l\|^2}_{\eqqcolon~\mathrm{err}(Q;W_l)}.$$1 entries: $$\mathcal{L}(\{Q(W_l)\}_{l=1}^L) - \mathcal{L}(\{W_l\}_{l=1}^L) \approx \sum_{l=1}^L a_l \underbrace{\|Q(W_l)-W_l\|^2/\|W_l\|^2}_{\eqqcolon~\mathrm{err}(Q;W_l)}.$$2 Quantization is again nearest-neighbor: $$\mathcal{L}(\{Q(W_l)\}_{l=1}^L) - \mathcal{L}(\{W_l\}_{l=1}^L) \approx \sum_{l=1}^L a_l \underbrace{\|Q(W_l)-W_l\|^2/\|W_l\|^2}_{\eqqcolon~\mathrm{err}(Q;W_l)}.$$3 VQ naturally supports half-bit increments. A $$\mathcal{L}(\{Q(W_l)\}_{l=1}^L) - \mathcal{L}(\{W_l\}_{l=1}^L) \approx \sum_{l=1}^L a_l \underbrace{\|Q(W_l)-W_l\|^2/\|W_l\|^2}_{\eqqcolon~\mathrm{err}(Q;W_l)}.$$4-entry codebook over 2D vectors uses 3 bits for 2 scalars, i.e. $$\mathcal{L}(\{Q(W_l)\}_{l=1}^L) - \mathcal{L}(\{W_l\}_{l=1}^L) \approx \sum_{l=1}^L a_l \underbrace{\|Q(W_l)-W_l\|^2/\|W_l\|^2}_{\eqqcolon~\mathrm{err}(Q;W_l)}.$$5 bits per scalar. The paper places VQ between NUQ and TCQ in the distortion-latency trade-off (Lee et al., 24 Sep 2025).

Trellis-coded quantization (TCQ) is the highest-fidelity family in Q-Palette and is described as near-optimal for Gaussian sources. Using the bitshift variant with tail-biting, dequantization is defined over a sliding window: $$\mathcal{L}(\{Q(W_l)\}_{l=1}^L) - \mathcal{L}(\{W_l\}_{l=1}^L) \approx \sum_{l=1}^L a_l \underbrace{\|Q(W_l)-W_l\|^2/\|W_l\|^2}_{\eqqcolon~\mathrm{err}(Q;W_l)}.$$6 with $$\mathcal{L}(\{Q(W_l)\}_{l=1}^L) - \mathcal{L}(\{W_l\}_{l=1}^L) \approx \sum_{l=1}^L a_l \underbrace{\|Q(W_l)-W_l\|^2/\|W_l\|^2}_{\eqqcolon~\mathrm{err}(Q;W_l)}.$$7, $$\mathcal{L}(\{Q(W_l)\}_{l=1}^L) - \mathcal{L}(\{W_l\}_{l=1}^L) \approx \sum_{l=1}^L a_l \underbrace{\|Q(W_l)-W_l\|^2/\|W_l\|^2}_{\eqqcolon~\mathrm{err}(Q;W_l)}.$$8, $$\mathcal{L}(\{Q(W_l)\}_{l=1}^L) - \mathcal{L}(\{W_l\}_{l=1}^L) \approx \sum_{l=1}^L a_l \underbrace{\|Q(W_l)-W_l\|^2/\|W_l\|^2}_{\eqqcolon~\mathrm{err}(Q;W_l)}.$$9, and effective bitwidth

$\mathcal{Q}$0

Because $\mathcal{Q}$1, odd $\mathcal{Q}$2 produces half-bit rates such as $\mathcal{Q}$3, $\mathcal{Q}$4, and $\mathcal{Q}$5. Encoding uses a Viterbi-based round-to-nearest operator. TCQ is the quantizer family that most closely tracks the Gaussian-optimal RMSE curve $\mathcal{Q}$6, corresponding to the Gaussian MSE law $\mathcal{Q}$7 (Lee et al., 24 Sep 2025).

Half-TCQ is introduced to realize quarter-step average bitwidths. A weight matrix is split row-wise; one part is quantized at $\mathcal{Q}$8 bits and the other at $\mathcal{Q}$9 bits, yielding average rate $b_l$0. This produces bitwidths such as $b_l$1, $b_l$2, and $b_l$3, which allow Q-Palette to approximate the continuous optimum more closely than a half-bit grid alone (Lee et al., 24 Sep 2025).

The paper also describes both data-free and data-aware quantization modes. In data-free mode, weights are partitioned and quantized directly with the quantizer’s round-to-nearest rule. In data-aware mode, the same quantizer families are used inside a block LDLQ framework with Hessian guidance, with block sizes 1 for NUQ, 2 for VQ, and 16 for TCQ (Lee et al., 24 Sep 2025).

4. Systems implementation and kernel design

A central practical claim of Q-Palette is that sophisticated Gaussian quantizers need not be confined to narrow rate sets or batch size 1. The paper therefore devotes substantial effort to systems design, combining quantizer families with optimized CUDA kernels (Lee et al., 24 Sep 2025).

To reduce the overhead of incoherence processing, the paper rotates only along the input dimension,

$b_l$4

then applies per-output-channel scaling so each rotated column is approximately standard Gaussian. It also shares rotation matrices among layers with the same input, such as $b_l$5 and $b_l$6 projections. This reduces the number of online rotations per transformer block from 14 to 4 (Lee et al., 24 Sep 2025).

Two kernel families are implemented: Tensor Core-based kernels and CUDA Core-based kernels. Tensor Core kernels support TCQ, half-TCQ, VQ, and NUQ. They extend the QTIP-style warp-level MMA design by performing register-level dequantization directly on load, traversing each quantized weight only once for larger batch sizes, and caching input activations in shared memory. CUDA Core kernels support NUQ and VQ, extend Any-Precision LLM kernels, replace bit-plane encoding with bit-packing, and use shared-memory LUTs together with half2 and hfma2 operations (Lee et al., 24 Sep 2025).

The reported throughput results on LLaMA 3.1-8B and RTX 4090 show that these implementation choices materially affect deployment. At batch size 1, relative to FP16, QTIP 2-bit achieves $b_l$7, Ours-TCQ-TC 2-bit achieves $b_l$8, and Ours-NUQ-CC 2-bit achieves $b_l$9. At batch size 8, QTIP 2-bit drops to $$\mathbb{E}[\mathrm{err}(Q)] \ge 2^{-2\,\mathrm{bit}(Q)}.$$0, while Ours-TCQ-TC 2-bit reaches $$\mathbb{E}[\mathrm{err}(Q)] \ge 2^{-2\,\mathrm{bit}(Q)}.$$1 and Ours-NUQ-TC 2-bit reaches $$\mathbb{E}[\mathrm{err}(Q)] \ge 2^{-2\,\mathrm{bit}(Q)}.$$2 (Lee et al., 24 Sep 2025).

The paper states that CUDA Core kernels tend to be better for batch size 1, whereas Tensor Core kernels tend to be better for larger batch sizes. This suggests that Q-Palette is designed not just as an abstract quantization method but as a deployment-oriented toolkit whose best operating point depends on the inference regime.

5. Mixed-scheme quantization and fusion-aware optimization

Q-Palette uses its quantizer collection inside a mixed-scheme quantization (MSQ) framework. The generic resource-constrained problem is written as

$$\mathbb{E}[\mathrm{err}(Q)] \ge 2^{-2\,\mathrm{bit}(Q)}.$$3

where $$\mathbb{E}[\mathrm{err}(Q)] \ge 2^{-2\,\mathrm{bit}(Q)}.$$4 selects quantizer $$\mathbb{E}[\mathrm{err}(Q)] \ge 2^{-2\,\mathrm{bit}(Q)}.$$5 for layer $$\mathbb{E}[\mathrm{err}(Q)] \ge 2^{-2\,\mathrm{bit}(Q)}.$$6, $$\mathbb{E}[\mathrm{err}(Q)] \ge 2^{-2\,\mathrm{bit}(Q)}.$$7 is estimated loss, and $$\mathbb{E}[\mathrm{err}(Q)] \ge 2^{-2\,\mathrm{bit}(Q)}.$$8 is either memory or latency cost. In data-free settings, the paper uses

$$\mathbb{E}[\mathrm{err}(Q)] \ge 2^{-2\,\mathrm{bit}(Q)}.$$9

and estimates $$\begin{aligned} \min_{b_l \ge \eta}\quad & \sum_{l=1}^L a_l 2^{-2b_l} \ \text{s.t.}\quad & \sum_{l=1}^L b_l\, d_l^{\mathrm{in}}d_l^{\mathrm{out}} \le M, \end{aligned}$$0 from quantizer distortion on random Gaussian matrices, exploiting the Gaussianization assumption (Lee et al., 24 Sep 2025).

The more novel extension is fusion-aware MSQ, which jointly chooses quantizer assignments and layer fusion patterns. Transformer layers such as

$$\begin{aligned} \min_{b_l \ge \eta}\quad & \sum_{l=1}^L a_l 2^{-2b_l} \ \text{s.t.}\quad & \sum_{l=1}^L b_l\, d_l^{\mathrm{in}}d_l^{\mathrm{out}} \le M, \end{aligned}$$1

and

$$\begin{aligned} \min_{b_l \ge \eta}\quad & \sum_{l=1}^L a_l 2^{-2b_l} \ \text{s.t.}\quad & \sum_{l=1}^L b_l\, d_l^{\mathrm{in}}d_l^{\mathrm{out}} \le M, \end{aligned}$$2

can be fused to reduce kernel launches, memory accesses, and latency. The paper defines a set of fusible groups $$\begin{aligned} \min_{b_l \ge \eta}\quad & \sum_{l=1}^L a_l 2^{-2b_l} \ \text{s.t.}\quad & \sum_{l=1}^L b_l\, d_l^{\mathrm{in}}d_l^{\mathrm{out}} \le M, \end{aligned}$$3, including singleton groups and fused $$\begin{aligned} \min_{b_l \ge \eta}\quad & \sum_{l=1}^L a_l 2^{-2b_l} \ \text{s.t.}\quad & \sum_{l=1}^L b_l\, d_l^{\mathrm{in}}d_l^{\mathrm{out}} \le M, \end{aligned}$$4 or $$\begin{aligned} \min_{b_l \ge \eta}\quad & \sum_{l=1}^L a_l 2^{-2b_l} \ \text{s.t.}\quad & \sum_{l=1}^L b_l\, d_l^{\mathrm{in}}d_l^{\mathrm{out}} \le M, \end{aligned}$$5 groups, and optimizes

$$\begin{aligned} \min_{b_l \ge \eta}\quad & \sum_{l=1}^L a_l 2^{-2b_l} \ \text{s.t.}\quad & \sum_{l=1}^L b_l\, d_l^{\mathrm{in}}d_l^{\mathrm{out}} \le M, \end{aligned}$$6

Constraint (C1) enforces exclusive assignment of each layer to one active group, and (C2) enforces the resource budget (Lee et al., 24 Sep 2025).

This framework is important because quantization and fusion are not separable. A fused group may require a shared quantizer assignment, which can reduce fidelity, but the same fusion may reduce latency enough to permit better bit allocation elsewhere. The paper solves both MSQ and fusion-aware MSQ with SCIP in OR-Tools, using a 60-second time limit for the fusion-aware ILP (Lee et al., 24 Sep 2025).

6. Empirical performance, trade-offs, and limitations

The experiments cover LLaMA 3.1-8B, LLaMA 3.1-70B, LLaMA 3.2-1B, LLaMA 3.2-3B, LLaMA 2-7B, LLaMA 2-13B, and Qwen 2.5-7B. Evaluation uses WikiText2 perplexity and average zero-shot accuracy over ARC-easy, ARC-challenge, HellaSwag, PiQA, and WinoGrande, with throughput measured mainly on RTX 4090 and additionally on RTX 3090 (Lee et al., 24 Sep 2025).

In the data-free regime, the main pattern is that single-scheme TCQ already improves over strong baselines, while MSQ improves further. On LLaMA 3.1-8B at 3.25 bits, NF gives Wiki2 $$\begin{aligned} \min_{b_l \ge \eta}\quad & \sum_{l=1}^L a_l 2^{-2b_l} \ \text{s.t.}\quad & \sum_{l=1}^L b_l\, d_l^{\mathrm{in}}d_l^{\mathrm{out}} \le M, \end{aligned}$$7, HIGGS gives $$\begin{aligned} \min_{b_l \ge \eta}\quad & \sum_{l=1}^L a_l 2^{-2b_l} \ \text{s.t.}\quad & \sum_{l=1}^L b_l\, d_l^{\mathrm{in}}d_l^{\mathrm{out}} \le M, \end{aligned}$$8, Ours-TCQ-3.25 gives $$\begin{aligned} \min_{b_l \ge \eta}\quad & \sum_{l=1}^L a_l 2^{-2b_l} \ \text{s.t.}\quad & \sum_{l=1}^L b_l\, d_l^{\mathrm{in}}d_l^{\mathrm{out}} \le M, \end{aligned}$$9, HIGGS-MSQ gives $M$0, and Ours-MSQ-Mem gives $M$1. At 4.0 bits, Data-free QTIP gives $M$2, Ours-TCQ-4 gives $M$3, and Ours-MSQ-Mem gives $M$4 (Lee et al., 24 Sep 2025).

The paper emphasizes Pareto improvements under memory constraints. One highlighted example is that its 2.875-bit model attains perplexity comparable to 3.25-bit HIGGS-MSQ, implying about $M$5 higher compression at similar or better quality. Similar trends are reported for Qwen 2.5-7B and LLaMA 3.1-70B, where Q-Palette’s MSQ results match or exceed higher-bit HIGGS-MSQ models, implying up to 12.5% and 15% memory savings, respectively (Lee et al., 24 Sep 2025).

Under latency constraints, the gains are also pronounced. On LLaMA 3.1-8B, RTX 4090, and batch size 1, the paper reports: NF3G64 + FLUTE at Wiki2 $M$6 and speedup $M$7; TCQ-3.25 at $M$8 and $M$9; MSQ with Q-Palette at $\eta$0 and $\eta$1; and Fusion-aware MSQ with Q-Palette at $\eta$2 and $\eta$3. In the data-aware setting on LLaMA 2-7B, QTIP 2-bit gives Wiki2 $\eta$4, accuracy $\eta$5, and throughput $\eta$6 tokens/s at batch sizes $\eta$7, while Ours-MSQ-Mem gives $\eta$8, $\eta$9, and $Q(W_l)$00 (Lee et al., 24 Sep 2025).

Ablations support the paper’s two-gap theory. Comparing VQ-2,3,4, TCQ-2,3,4, and TCQ-ALL, the broader TCQ-ALL set consistently performs best, indicating that richer fractional-bit support matters in addition to the underlying quantizer quality. The Gaussian quantization-error curves show uniform quantization as worst, NUQ better, VQ better still, and TCQ closest to the Gaussian-optimal line $Q(W_l)$01. This aligns with the interpretation that TCQ minimizes the distortion gap, while dense rate support minimizes the bit allocation gap (Lee et al., 24 Sep 2025).

The paper also notes several limitations. First, the framework is designed for one-shot PTQ/MSQ, not retraining-based quantization. Second, estimating sensitivity coefficients $Q(W_l)$02 incurs a one-time cost that scales with the number of layers. Third, the work focuses on weight-only PTQ, not weight-activation quantization. Fourth, some hardware environments require integer GEMMs, where this style of dequantize-on-the-fly weight-only quantization may not directly fit (Lee et al., 24 Sep 2025).

A common misconception is that Q-Palette is simply “another low-bit quantizer.” The paper’s own structure argues against that interpretation. Q-Palette is simultaneously a theory of fractional-bit optimality for Gaussianized weights, a family of Gaussian-aware quantizers, a CUDA implementation, and an optimization framework that chooses quantizers and, in one variant, fusion patterns under realistic memory or latency budgets. This suggests that its contribution lies as much in connecting rate–distortion theory to deployment decisions as in any individual quantizer design.