TernaryLM: Efficient Ternary LLM Quantization

• TernaryLM is a language model that quantizes weights (and sometimes activations) to {-1, 0, +1}, reducing memory footprint, energy consumption, and computational cost.
• It employs advanced techniques like quantization-aware training, dynamic thresholds, and rotation-domain shaping to closely approximate full-precision performance.
• Specialized hardware designs including FPGA/ASIC kernels and efficient packed encoding schemes facilitate real-time, low-latency inference and significant speedups.

A Ternary LLM (TernaryLM) is a LLM whose parameters— and, in some systems, activations—are quantized to the ternary alphabet $\{-1, 0, +1\}$. The motivation for TernaryLMs is to drastically reduce memory footprint, energy consumption, and computational complexity compared to conventional 8–16-bit or floating-point LLMs, while retaining a substantial fraction of their language modeling, reasoning, and transfer capabilities. TernaryLM research encompasses quantization-aware training, post-training quantization, advanced packing/compression schemes, hardware/FPGA/ASIC/CPU/GPU kernel design, and scaling law analysis. Despite the apparent extremeness of 1-bit (ternary) quantization, careful algorithmic and systems co-design enables these models to approach full-precision performance in key metrics such as perplexity, downstream task accuracy, and inference throughput (Nargund et al., 7 Feb 2026, Vaidhya et al., 28 Jun 2025, Chen et al., 2024, Crulis et al., 7 Apr 2025, Huang et al., 17 Sep 2025, Qiao et al., 22 Apr 2025, Oh et al., 17 Nov 2025, Zhao et al., 11 Jun 2026, Zuo et al., 22 Apr 2026).

1. Ternary Quantization Methodologies and Theory

Ternary quantization constrains weights (and sometimes activations) to the set $\{-1, 0, +1\}$. The basic mapping for a real-valued parameter $w$ is: $\hat{w} = \alpha \cdot \mathrm{sign}_\tau(w)$ where $\alpha$ is a learned or per-group/channel scale and the ternary sign function applies a threshold $\tau$: $$\mathrm{sign}_\tau(x) = \begin{cases} +1 & x > \tau \ 0 & |x| \le \tau \ -1 & x < -\tau \end{cases}$$ Sophisticated methods extend this mapping by introducing learnable channel-wise shifts (DLT) (Chen et al., 2024), asymmetric thresholds, or signed-zero states (SZT, a 2-bit extension providing two bit-exact "zero" codes for enhanced backward signal) (Uhlmann, 8 Aug 2025).

Ternarization may be applied during post-training quantization (PTQ) (Chen et al., 2024, Malhotra et al., 1 Jun 2025, Zhao et al., 11 Jun 2026), quantization-aware training (QAT) (Nargund et al., 7 Feb 2026, Vaidhya et al., 28 Jun 2025, Malhotra et al., 1 Jun 2025), or via hybrid approaches. Training often leverages the straight-through estimator (STE) so that gradients are not zeroed out by the piecewise-constant ternary function.

TWLA (Zhao et al., 11 Jun 2026) incorporates Kronecker-structured orthogonal transformations ("rotation-domain shaping") to induce ternary-friendly tri-modal distributions in both weight and activation statistics, enabling stable joint quantization of weights to 1.58 bits and activations down to 4-bit.

Signed-Zero Ternary (SZT) (Uhlmann, 8 Aug 2025) generalizes balanced ternary by splitting the dead-zone zero state into “$+$” and “$-$,” providing one extra bit of sign information for improved gradient flow during QAT.

2. Training, Post-training Quantization, and Optimization

TernaryLMs can be either:

• Quantized from full-precision checkpoints: PTQ methods such as E2M-ATQ (Zhao et al., 11 Jun 2026), DLT + OFF (Chen et al., 2024), and asymmetric ternarization (Malhotra et al., 1 Jun 2025) optimize scale, shift, and thresholds to minimize pre- and post-activation errors, often using calibration data to relocate centroids and assess bit allocation (e.g., via ILA-AMP (Zhao et al., 11 Jun 2026)).
• Trained natively under ternary constraints: TernaryLM (Nargund et al., 7 Feb 2026) and Spectra 1.1 (Vaidhya et al., 28 Jun 2025) incorporate quantization in their forward computation, adapt scaling factors per-layer (or per-group), and propagate gradients with STE. This approach eliminates the post-training conversion step and enables the model to adapt to the quantization regime "from scratch."

Key optimization features include: per-neuron or per-channel scaling factors, per-channel shift, dynamic thresholds (often a fraction of activation or weight statistics), and dual-stage clustering or K-means for initialization (Crulis et al., 7 Apr 2025, Chen et al., 2024).

E2M-ATQ (Zhao et al., 11 Jun 2026) uses a two-stage process, first minimizing reconstruction loss in weight space, then relocating weight centroids on the manifold to directly minimize the layer's output error under the calibrating activation statistics. KOTMS rotation (Zhao et al., 11 Jun 2026) and similar rotation-domain approaches (Yoon, 30 Mar 2026) quantitatively shape the weight (and activation) distribution to maximize the efficiency of ternary codes.

3. Memory, Compute, and Information Efficiency

Ternary quantization offers a theoretical storage requirement of $\log_2(3) \approx 1.585$ bits/weight. Practical implementations approach this bound via packed encodings (e.g., five ternaries per byte (Vaidhya et al., 28 Jun 2025, Huang et al., 17 Sep 2025)). Hybrid formats use 2 bits/weight (for fast decoding (Qiao et al., 22 Apr 2025, Oh et al., 17 Nov 2025), sometimes augmented with per-row or per-channel scales), or 1.6 bits/weight for optimal entropy packing (Vaidhya et al., 28 Jun 2025, Qiao et al., 22 Apr 2025, Huang et al., 17 Sep 2025).

For a 132 M parameter TernaryLM (Nargund et al., 7 Feb 2026), memory usage drops from 1.2 GB (FP32 BERT-Base) to 498 MB, a 2.4× reduction. Tenet and TeLLMe demo 64 B → 80 B decompression for storage-optimal inference on FPGAs and ASICs (Huang et al., 17 Sep 2025, Qiao et al., 22 Apr 2025).

Compute savings arise because ternary dot products need no multiplications (only adds and subtracts), reducing energy per operation by ∼5× over FP16 (Chen et al., 2024, Zuo et al., 22 Apr 2026). On CPUs and FPGAs, multiplication-free matmul kernels yield 30× speedup in GEMV/GEMM kernel latency and drive the system from memory-bound to compute-bound regimes (Zuo et al., 22 Apr 2026, Oh et al., 17 Nov 2025, Qiao et al., 22 Apr 2025). Specialized kernels exploit AVX-512 (FairyFuse (Zuo et al., 22 Apr 2026)), CPU SIMD with in-register LUTs (T-SAR (Oh et al., 17 Nov 2025)), and custom table-LUT matmul (TeLLMe (Qiao et al., 22 Apr 2025), TENET (Huang et al., 17 Sep 2025)).

Packing Efficiency Table:

Scheme	Bits/Weight	Comment
Naive ternary	2.0	Direct 2-bit encoding (Zuo et al., 22 Apr 2026)
Base-3 packing	1.6	Entropy-optimal, 5/byte (Vaidhya et al., 28 Jun 2025, Huang et al., 17 Sep 2025)
ITQ3_S	3.125	Ternary in rotation domain (Yoon, 30 Mar 2026)

4. Model Quality, Scaling, and Task Performance

Empirical findings indicate that TernaryLMs achieve high accuracy on both pretraining and downstream tasks with only moderate perplexity overhead. For example:

• TernaryLM (1-bit native QAT) obtains PPL 58.42 (TinyStories), MRPC F1 82.47, SST-2 88.92, CoLA 47.23, with 2.4× reduction in memory (Nargund et al., 7 Feb 2026).
• TernaryLLM (W1.58A16) on LLaMA-3-8B achieves PPL(C4)=13.4 versus DB-LLM’s 19.2 (W2A16) (Chen et al., 2024).
• TWLA (W1.58A4) on LLaMA2-13B achieves 86.4 tok/s at 3.3 GB and 58–73% accuracy on zero-shot tasks, recovering ∼80–92% FP16 performance (Zhao et al., 11 Jun 2026).
• Spectra-1.1 TriLMs (QAT ternary) retain full-precision scaling law behavior, with only a +0.02 nats/token irreducible loss gap, and downstream MMLU accuracy equal or superior to FP16 baselines in the 1–3B parameter regime (Vaidhya et al., 28 Jun 2025).

Layer-wise analysis shows that middle transformer blocks achieve the highest sparsity (∼60–62%), enabling more aggressive quantization in these layers without severe performance loss (Nargund et al., 7 Feb 2026). Scaling law fits in Spectra-1.1 further indicate that, under aggressive ternary quantization, data scaling is notably more profitable than parameter scaling—suggesting large-token-count pretraining is essential (Vaidhya et al., 28 Jun 2025).

5. Accelerator, Kernel, and Inference System Design

TernaryLMs require highly specialized hardware or kernel support to unlock their efficiency. Key developments include:

• FPGAs/ASICs: Table-LUT matmul (TeLLMe (Qiao et al., 22 Apr 2025)), sparse ternary LUT cores (TENET (Huang et al., 17 Sep 2025)), full on-chip and HBM-assisted dataflows (TerEffic (Yin et al., 23 Feb 2025)), and 64 B→80 B base-3 decompression for storage-optimal decoding (Huang et al., 17 Sep 2025).
• CPUs: FairyFuse fuses all matrix-vector multiplications in a Transformer block into a single AVX-512 loop with zero multiplies, achieving 32.4 tok/s on a Xeon 8558P (WikiText-2 PPL gap to FP16 is only +0.05) (Zuo et al., 22 Apr 2026). T-SAR introduces in-register LUTs and novel SIMD instructions (TLUT, TGEMV) for dynamic kernel construction, avoiding memory LUT bottlenecks and yielding up to 24.5× GEMM latency reduction (Oh et al., 17 Nov 2025).
• GPUs: TriRun (Spectra 1.1 (Vaidhya et al., 28 Jun 2025)) fuses INT2 weight unpacking and matmul with double-buffered, tile-based shared-memory loading, exploiting tensor core throughput for more than 5× layer throughput over FP16 baselines.

Key architectural innovations include dynamic activation sparsity via N:M block schemes (Huang et al., 17 Sep 2025), rotation-domain shaping (Yoon, 30 Mar 2026), and fused normalization–quantization pipelines minimizing off-chip traffic (Qiao et al., 22 Apr 2025).

6. Fault Tolerance, Reliability, and Edge Deployment

TernaryLMs are well-positioned for deployment on resource-limited and unreliable hardware. ReTern exploits redundancy in ternary codes and column-wise sign flipping to achieve up to 35% perplexity reduction under 10% stuck-at-faults in compute-in-memory (CiM) accelerators with <3% overhead (Malhotra et al., 1 Jun 2025). Practical systems such as TeLLMe and TENET demonstrate real-time inference (10–16k tok/s) and energy efficiency up to 21× over NVIDIA A100 on edge FPGAs and ASICs, with full prefill and decoding support (Qiao et al., 22 Apr 2025, Huang et al., 17 Sep 2025, Yin et al., 23 Feb 2025).

CPU-based frameworks (T-SAR) provide viable alternatives to NPU and FPGA for platforms with only SIMD units, unlocking up to 8×–13× speedups over previous software kernels on mainstream x86 and ARM edge devices (Oh et al., 17 Nov 2025).

7. Limitations, Open Challenges, and Future Research

Remaining challenges include:

• Activation quantization: Most practical TernaryLMs retain FP16/INT8 activations owing to catastrophic errors from outlier spikes; only TWLA demonstrates stable 4-bit activation quantization with ternary weights (Zhao et al., 11 Jun 2026).
• Extreme outlier handling: Dual learnable scale/shift (DLT), orthogonal/rotation domain shaping, and group-wise threshold adaptation mitigate but do not eliminate performance losses from highly asymmetric outliers (Zhao et al., 11 Jun 2026, Chen et al., 2024, Yoon, 30 Mar 2026).
• Hardware support: Full realization of the theoretical efficiency requires hardware support for ternary arithmetic, bit-fused LUTs, or custom packing/decoding, which is only partially available in present CPUs/GPUs/NPUs.
• Model architecture generality: Most results are demonstrated on decoder-only transformer LLMs; extending TernaryLMs to encoder-decoder or retrieval-augmented architectures is an open avenue (Chen et al., 2024, Nargund et al., 7 Feb 2026).
• Scalability: TernaryLMs exhibit unique scaling law behaviors—efficient scaling with data, not just model size—and methods for trillion-parameter TernaryLMs are under active investigation (Vaidhya et al., 28 Jun 2025).
• Joint weight-activation quantization: Co-design of ternary weights and low-bit (≤4b) activations remains at research frontiers, with only TWLA showing practical high-accuracy joint quantization (Zhao et al., 11 Jun 2026).

Empirical and analytical research continues on mixed-precision policies, hybrid codebooks (e.g., SZT (Uhlmann, 8 Aug 2025)), edge-device kernels, and hardware-fault adaptive quantization (Malhotra et al., 1 Jun 2025, Vaidhya et al., 28 Jun 2025, Li et al., 16 Jun 2025).