CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs

Abstract: In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs. Unlike existing state-of-the-art ternary quantization methods that rely on data-intensive and costly quantization-aware training to mitigate severe performance degradation, CAT-Q is a simple yet effective post-training quantization scheme that is readily applicable to LLMs with diverse architectures and model sizes. It has two key components, learnable modulation (LM) and softened ternarization (ST), which are coupled from an optimization perspective. LM leverages a composition of learnable factors to modulate the distribution of pre-trained high-precision weights and the ternary threshold, making them less sensitive to ternarization. ST further introduces a differentiable transition function to guide the ternarization process toward stable convergence. We show that, for pre-trained LLMs with 1.7B to 8B parameters, CAT-Q can efficiently quantize them into ternary models using only 512 calibration samples, while achieving superior performance than the seminal BitNet 1.58-bit v1 and v2 families (with 1.3B to 7B parameters) trained with 100B tokens, yielding about a 100,000X reduction in training tokens. Moreover, we show for the first time that CAT-Q can quantize much larger pre-trained LLMs having 14B to 235B parameters into leading ternary models within just 8 to 60 hours on 8 A100-80GB GPUs. Code is available at https://github.com/IntelChina-AI/BitTern.

• The paper presents CAT-Q, a novel PTQ framework that achieves cost-efficient, accurate ternary quantization for LLMs by overcoming distributional mismatches and convergence issues.
• The methodology leverages learnable modulation with adaptive scaling and a soft-to-hard annealing process to improve gradient flow and stability during calibration.
• Experimental results show CAT-Q outperforms state-of-the-art QAT and PTQ methods across various LLM sizes, with minimal accuracy drops even on large-scale models.

CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs

Motivation and Problem Formulation

Ternary (1.58-bit) quantization offers an appealing trade-off for scaling LLM inference: it yields a $>10\times$ reduction in model memory footprint relative to FP16 and enables efficient hardware acceleration by mapping weights to $\{-1, 0, +1\}$, introducing inherent sparsity. Conventional state-of-the-art ternary quantization approaches for LLMs, such as "BitNet 1.58-bit" and "TriLM," depend on data- and compute-intensive quantization-aware training (QAT), frequently leveraging up to hundreds of billions of tokens and being generally restricted to sub-10B parameter models. Prior post-training quantization (PTQ) methods have demonstrated competitive performance in higher-precision (4/8-bit), but fail under extreme bit-widths due to outlier-driven distributional mismatch and hard quantization convergence.

"CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs" (2606.26650) interrogates these limitations by re-examining PTQ design for ultra-low bit settings. The key technical bottlenecks tackled are the distributional misalignment between ternary and high-precision weight distributions and non-differentiability-induced convergence failures of discrete ternarization.

CAT-Q: Methodology

The CAT-Q framework couples two complementary modules—Learnable Modulation (LM) and Softened Ternarization (ST)—into a sliding-window, group-wise PTQ pipeline for LLMs of arbitrary architecture and scale using modest calibration sets (e.g., 512 sequences of 2048 tokens).

Learnable Modulation

LM introduces three learnable scaling factors $(\delta_\mu, \delta_\alpha, \delta_\Delta)$ that transform the weight centroid, normalization, and ternarization threshold, respectively. Specifically, weights $W$ are linearly modulated to $\hat{W} = (W-\mu)/\alpha$ (with learnable mean and scale) before ternarization; the threshold for ternary assignment is also learned (not fixed). This technique adaptively corrects for layer/group-level distributional outliers, aligning the quantizer input to be maximally robust to information loss under ternarization. This is critical for high-fidelity reconstruction at extreme bit widths.

Figure 1: Overview of the CAT-Q learning flow, highlighting the sequence for learning the modulation factors used to ternarize pre-trained weights and achieve hardware-friendly group-wise output.

Figure 2 demonstrates empirically that CAT-Q’s learnable modulation sharply reduces weight reconstruction errors compared to both static analytical and direct parameter learning approaches.

Figure 2: Quantitative comparison of weight reconstruction error across ternarization strategies, establishing lower error with CAT-Q’s LM over static/direct baselines.

Softened Ternarization

ST introduces a two-stage, differentiable relaxation-to-discrete ternarization process. A smooth transition function $f(\cdot)$, parameterized by sharpness $s$, gradually morphs from the identity (full-precision) mapping ($s \to 0$) to a piecewise assignment mirroring hard ternary quantization ($s \to \infty$). This "soft-to-hard" annealing, scheduled across calibration epochs, enables stable gradient propagation in the calibration phase, overcoming the convergence barriers endemic to prior non-differentiable ternarizers.

Figure 3: The ST process visualized, showing the progressive transition from continuous/differentiable quantization output toward discrete ternary assignment.

Figure 4 provides insight into how increasing the sharpness $s$ on $\{-1, 0, +1\}$0 tightens the mapping, empirically validating its effect in approaching hard quantization while maintaining smooth optimization.

Figure 4: Output curves of the smooth transition function $\{-1, 0, +1\}$1 as $\{-1, 0, +1\}$2 increases, transitioning from near-identity to sharp ternarization.

Combining LM and ST, CAT-Q optimizes ternarization in a group-wise sliding window across all LLM layers, with calibration loss directly computed on the output between full-precision groups and their ternary reconstructions.

Experimental Analysis and Performance

CAT-Q is evaluated on 10 LLMs (including Qwen3, Llama2, Ring-flash-2.0) ranging from 1.7B to 235B parameters, mixing dense and MoE architectures. On standard reasoning benchmarks (PIQA, ARC-e/c, HellaSwag, Winogrande), CAT-Q’s ternary models (W1.58A16) exhibit average accuracy drops diminishing with model scale—on Llama2-70B, the drop is only 3.81%, and even 235B-parameter models are efficiently quantized within 60 GPU-hours on 8×A100-80GB, a scale not approached by prior ternary PTQs.

CAT-Q’s bold quantitative result: For 1.7B–8B LLMs, CAT-Q with only 1M calibration tokens outperforms BitNet 1.58-bit and TriLM, which use >100B tokens, corresponding to a 100,000× reduction in data demand while yielding superior accuracy under identical precision settings. This is in strong tension with the standard view that QAT is required for lossless ternarization.

On direct PTQ comparisons, CAT-Q substantially outperforms all prior 2-bit and dual-binarization LLM PTQs, both in accuracy (e.g., vs. DB-LLM, SliderQuant, PB-LLM) and weight memory footprint, with hardware benchmarks showing superior inference throughput across both CPU and GPU in llama.cpp/BitNet kernels.

Ablations validate that the synergy of all three learnable LM factors, the soft-to-hard annealing schedule, and group size/epoch/calsize parameters is necessary for optimality. CAT-Q’s deployment remains zero-point free, preserving add-subtract-only inference and enabling direct integration into hardware-optimized ternary LLM backends.

Theoretical and Practical Implications

CAT-Q establishes, for the first time, that PTQ can match or surpass QAT-based ternary quantization in LLMs at scale, provided the quantization process: (1) actively corrects for outlier-driven distributional misalignment using learnable transformations, and (2) employs differentiable annealed ternarization to enable smooth calibration. This contradicts the prevailing assumption that high-fidelity ternarization necessarily mandates full QAT.

For practice, this renders efficient, privacy-preserving LLM compression feasible in restricted-data or time-constrained settings. Hardware accelerators can leverage high-sparsity ternary weights with minimal software engineering overhead. Limitations persist for complex domains (math, coding), where additional domain-aware calibration is required; this is being actively addressed (see appendix).

Future Directions

Key open areas include further reducing the accuracy gap to FP16 for mathematically and programmatically intensive tasks, the development of universal ternary hardware kernels (with BitNet’s being model-specific), and extension of the CAT-Q calibration protocol to automated sample-efficient strategies, possibly leveraging synthetic or task-adaptive calibration.

Conclusion

CAT-Q defines a new benchmark for PTQ in ultra-low-bit LLMs, providing a repeatable, cost-efficient methodology for ternary quantization using modest calibration data, while matching or exceeding the accuracy of prior QAT-based and PTQ baselines across both dense and MoE transformer architectures. The approach reconfigures the efficiency frontier for practical LLM deployment in constrained environments.

(2606.26650)