---
title: 'TernaryLLM: Low-Bit Language Models'
url: https://www.emergentmind.com/topics/ternaryllm
type: topic
---

# TernaryLLM: Low-Bit Language Models

A TernaryLLM is a large language model (LLM) in which the majority of weights are quantized to a ternary alphabet, typically $\{-1, 0, +1\}$, and encoded using dense sub-2-bit representations (e.g., 1.6 or 2 bits/weight). These models achieve a drastic reduction in memory footprint and remove most floating-point multiplications from inference, while preserving a high degree of model expressiveness and accuracy. TernaryLLMs exploit advances in post-training quantization, quantization-aware training, hardware design (CPU, GPU, FPGA, ASIC), and information-theoretically motivated schemes to realize language model inference at orders-of-magnitude lower computational cost than full-precision or even 4-bit models.

## 1. Mathematical Foundations of Ternary Quantization

The core operation in TernaryLLMs is the quantization of neural weights to the ternary set $\{-1, 0, +1\}$. The forward path of a linear or projection layer with floating-point weights $W\in\mathbb{R}^{n\times d}$ is approximated as: 
\[
W \approx \widetilde{W} = \alpha \cdot T
\]
where $T \in \{-1,0,1\}^{n\times d}$ and $\alpha$ is a learnable or derived scaling factor (often applied per-row, per-column, or per-group) [2406.07177][2509.16989][2506.23025].

Several quantization procedures are in use:

- **Hard thresholding:** $T_{ij} = \text{sign}(W_{ij})$ if $|W_{ij}| > \Delta$, $T_{ij}=0$ otherwise. The scale $\alpha$ is set to minimize $\|W - \alpha T\|_F^2$ [2504.16266][2502.16473].
- **Dual Learnable Ternarization (DLT):** Both scale $\alpha$ and shift $\gamma$ parameters are learned for each group, allowing the quantized-and-reconstructed weight to be $D_i = \alpha T_i + \gamma$ [2406.07177].
- **Structured Trit-Plane Decomposition:** Advanced schemes such as PTQTP represent every row of $W$ as a sum of two ternary planes weighted by learned scales:
  \[
  W_i \approx \alpha_i^{(1)} T^{(1)}_i + \alpha_i^{(2)} T^{(2)}_i
  \]
  yielding an effective storage of $2\cdot 1.585 \approx 3.17$ bits/weight, or 1.585 bits per plane [2509.16989].

- **Signed-Zero Ternary (SZT):** Encodes four states (using two bits), allowing additional sign information for sub-threshold weights, improving gradient flow and information density [2508.05905].

Activations are usually left in higher precision (e.g., FP16 or INT8), as quantizing activations to ternary remains an outstanding challenge due to heavy-tailed distributions and significant dynamic range [2406.07177][2509.16989].

## 2. Quantization Methodologies: Post-Training and Quantization-Aware Training

Two principal quantization strategies are prominent:

- **Post-Training Quantization (PTQ):** Applies quantization to a pretrained LLM (e.g., LLaMA, Qwen) without further gradient updates. Algorithms such as PTQTP use a monotonic, globally consistent, group-wise progressive approximation loop: alternating ridge regression updates for scale and exhaustive search for ternary assignments per group, with convergence guarantees [2509.16989].

- **Quantization-Aware Training (QAT):** Modifies the forward pass to simulate ternary weights and employs straight-through estimators for the backward pass, learning to compensate for quantization error during training [2506.23025][2406.07177]. DLT augments this process with learnable shifts to better fit asymmetric weight distributions, while Outlier-Friendly Feature Distillation (OFF) guides the quantized student toward teacher representations using cosine similarity, addressing information loss due to extreme quantization [2406.07177].

Knowledge distillation and fine-tuning techniques (e.g., LoTA-QAF) employ low-rank trainable adapters in the ternary domain, supporting lossless merging and integer-only inference [2505.18724].

## 3. Packing Schemes and Hardware Implementation

Efficiently storing and operating over ternary weights is critical for realizing the theoretical savings. Key approaches:

- **Bit-packing:** Blocks of 5 ternary values ($3^5=243$) are packed into a single 8-bit byte, yielding 1.6 bits/weight ("TQ1" scheme); using two bits per value ("TQ2") reaches 2 bits/weight. These methods are implemented both on CPUs and GPUs for fast unpacking and high memory bandwidth utilization [2506.23025][2509.13765].

- **Matrix-vector multiplication (GEMM):** Inference kernels are redesigned to exploit the ternary structure:
  - On CPUs (e.g., Apple Silicon), custom sparse GEMM kernels using blocked, interleaved storage, loop unrolling, and NEON vectorization deliver 5–6× speedup over default libraries [2510.06957].
  - On FPGAs/ASICs, accelerators such as TENET and TeLLMe use table-lookup engines and LUT-centric ternary matmuls, slashing the need for multipliers and reducing DRAM access via specialized weight packing [2509.13765][2504.16266][2510.15926]. Dynamic N:M activation sparsity further reduces compute [2509.13765].
  - On GPUs, TriRun offers a mixed-precision CUDA kernel (FP16 activations × INT2 weights) leveraging shared memory and pipeline parallelism, achieving up to 4.9× end-to-end throughput gains [2506.23025].

- **Indexing algorithms:** For fixed ternary weight matrices, block-indexed GEMV algorithms achieve $O(n^2/\log n)$ time and memory by precomputing permutation and segmentation indices, with up to $29\times$ speedup and $6\times$ memory reduction in software-only settings [2411.06360].

| Packing Method        | Bits/Weight  | Packing Unit | Main Platform         |
|----------------------|--------------|--------------|----------------------|
| 2-bit ("TQ2")        | 2            | k=256        | CPU, GPU             |
| 1.6-bit ("TQ1")      | 1.6          | k=5          | CPU, FPGA, ASIC      |
| PTQTP Trit-Plane     | 3.17         | group=128    | GPU, FPGA, ASIC      |

## 4. Empirical Scaling Laws and Model Behavior

Recent empirical analysis reveals that ternary models exhibit distinctly different scaling behavior compared to their full-precision counterparts. For ternary LLMs (TriLMs) [2506.23025]:
\[
\text{Loss}(N, D) \approx 2.19 + 4.73 N^{-0.32} + 5.18 D^{-0.81}
\]
where $N$ is parameters (M), $D$ pretraining tokens (B). The data exponent ($\beta=0.81$) dominates the parameter exponent ($\alpha=0.32$), implying that expanding the dataset, rather than the model size, yields greater returns for ternary LLMs at fixed FLOPs.

For FloatLMs, the exponents are nearly matched ($\alpha=0.56$, $\beta=0.53$).

A practical implication is that TernaryLLMs should allocate training computation towards increasing data rather than model width/depth, diverging from established scaling rules for float-precision models.

## 5. Accuracy-Complexity Tradeoffs and Experimental Results

Comprehensive benchmarks show that TernaryLLMs typically retain $>90\%$ of baseline FP16 accuracy at 1.58 bits/weight, and dramatically outperform earlier binary or poorly compensated ternary/PTQ methods.

- On Qwen3-14B, PTQTP achieves $82.4\%$ retention of mathematical reasoning test accuracy compared to FP16, versus $0\%$ for baseline 3-bit GPTQ under the same conditions [2509.16989].
- LLaMA-3-8B with QAT (DLT+OFF) matches or outperforms 2-bit quantization, reaching $+8.2\%$ higher zero-shot accuracy than the best 2-bit method [2406.07177].
- Language modeling perplexity increases <0.5 PPL for BitNet-1.58 quantization [2504.16266][2510.15926].
- For quantization-aware fine-tuning, LoTA-QAF enables lossless merging of ternary adapters, recovering or surpassing LoRA (16-bit) accuracy by up to $+5.14\%$ on downstream tasks [2505.18724].
- FPGA and ASIC accelerators using optimized ternary GEMM consistently deliver $2-6\times$ end-to-end speedup and $4-21\times$ energy efficiency over A100-class GPUs [2509.13765][2504.16266][2510.15926][2502.16473].

## 6. Hardware Integration and Edge Deployment

TernaryLLMs are highly amenable to deployment on resource-constrained hardware due to their uniform, low-bit arithmetic and multiplication-free operations:

- **Edge FPGAs:** Engines such as TeLLMe and TerEffic store weights on-chip or in HBM, implement pipelined table-lookup matmul, and achieve $>16\times$ the throughput and $>8\times$ the efficiency of Jetson-class SoCs at equivalent or lower power [2504.16266][2510.15926][2502.16473].
- **ASICs:** TENET-ASIC deploys a heterogeneous architecture (Sparse Ternary LUT arrays plus FP16 attention blocks), reaching $2.7\times$ end-to-end inference speedup and $21\times$ higher energy efficiency than the NVIDIA A100 GPU, supported by custom 1.6-bit packing and decompression [2509.13765].
- **CPUs/GPUs:** Dedicated CPU kernels and the TriRun CUDA kernel unlock prompt and decode speedups of 1.5–7.9×, with dense or sparse storage for ternary weights [2510.06957][2506.23025].

## 7. Information-Theoretic and Theoretical Advances

TernaryLLM quantization is increasingly positioned as an information-theoretically optimal representation under resource constraints.

- **Entropy:** Log-base-three entropy yields $\log_2 3 \approx 1.585$ bits/trit, realized asymptotically by 1.6-bit packing [2506.23025][2508.05905].
- **SZT encoding adds “signed-zero” states**, recovering additional redundancy available in the unused 2-bit codeword, greatly enhancing gradient feedback for sub-threshold weights, reducing mean-squared-error in the STE, and tightening PAC–Bayes bounds [2508.05905].
- **Convergence dynamics:** Progressive trit-plane and DLT-decompositions are theoretically guaranteed to converge monotonically, with bounded scaling parameters [2509.16989].

TernaryLLMs thus represent not only an engineering compromise for edge or memory-bounded deployments, but also a theoretically motivated, rigorously analyzed quantization regime.

## References

- PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models [2509.16989]
- TernaryLLM: Ternarized Large Language Model [2406.07177]
- Accelerating Sparse Ternary GEMM for Quantized LLM inference on Apple Silicon [2510.06957]
- The Fourth State: Signed-Zero Ternary for Stable LLM Quantization (and More) [2508.05905]
- LoTA-QAF: Lossless Ternary Adaptation for Quantization-Aware Fine-Tuning [2505.18724]
- TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge [2509.13765]
- TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAs [2504.16266]
- TeLLMe v2: An Efficient End-to-End Ternary LLM Prefill and Decode Accelerator with Table-Lookup Matmul on Edge FPGAs [2510.15926]
- TerEffic: Highly Efficient Ternary LLM Inference on FPGA [2502.16473]
- An Efficient Matrix Multiplication Algorithm for Accelerating Inference in Binary and Ternary Neural Networks [2411.06360]
- Spectra 1.1: Scaling Laws and Efficient Inference for Ternary Language Models [2506.23025]

Source: https://www.emergentmind.com/topics/ternaryllm