--- title: Ternary Weight Quantization url: https://www.emergentmind.com/topics/ternary-weight-quantization-scheme type: topic --- # Ternary Weight Quantization A ternary weight quantization scheme constrains neural network weights to the discrete set $\{-1, 0, +1\}$, typically combined with one or more scaling factors for representational fidelity. This systematic quantization dramatically reduces model size (generally to 2 bits per weight), eliminates most multiplications in inference, and yields hardware-friendly sparsity and energy savings. Modern ternary schemes optimize the quantizer and quantization threshold using minimum mean-square error criteria, distributional matching, layer-wise statistics, or direct training via straight-through estimators. Variant schemes integrate pruning, hyperspherical norm constraints, hybrid filter banks, and quantization-aware fine-tuning. Ternary quantization is empirically validated for classification, detection, segmentation, generative diffusion transformers, spiking neural networks, and transformer LLMs. ## 1. Mathematical Formulation of Ternary Quantization Ternary quantization is typically expressed as mapping each scalar weight $w_i$ in a full-precision tensor $w$ to a quantized value $w_{q,i}\in\{-s, 0, +s\}$. The mapping uses a symmetric threshold $\Delta$ and positive scaling $s$: \[ w_{q,i} = \begin{cases} +s & \text{if } w_i > +\Delta, \\ -s & \text{if } w_i < -\Delta, \\ 0 & \text{otherwise}. \end{cases} \] Equivalently, $w_q = s \cdot \mathrm{sign}(w) \cdot 1\{|w| > \Delta\}$, with the indicator function $1\{\cdot\}$ [2107.10998], [1605.04711]. The threshold $\Delta$ and scaling $s$ are derived under a mean-square-error criterion or optimized by closed-form or numerically. Common choices: - $\Delta = \alpha \cdot E[|w|]$ with hyperparameter $\alpha\in(0,1)$. - $s = E[ |w| \cdot 1\{|w| > \Delta\} ] / E[ 1\{|w| > \Delta\} ]$. Some approaches admit per-group, per-channel, or per-layer adaptation, and allow for asymmetric positive/negative scale parameters [1802.08635], [1612.01064]. Alternative formulations—such as fine-grained (group-wise) quantization, truncated Gaussian optimization, cosine-similarity-based assignment, and two-branch binary decomposition—yield additional flexibility, regularization, or training stability [1705.01462], [1810.01018], [1912.09236], [2204.01234]. ## 2. Quantization-Aware Training and Optimization Methods In contrast to static post-training quantization, quantization-aware training (QAT) integrates the quantizer into the forward pass and propagates gradients using a straight-through estimator (STE), which approximates derivatives of the piecewise-constant quantizer as $\frac{\partial w_q}{\partial w} \approx 1$ in the clipped range [2107.10998], [2405.14854]. The loss typically combines the primary task objective with regularization terms: \[ \min_{w}\; \mathcal{L}(f(x; w_q)) + \eta\|w - w_q\|_2^2 + \lambda\|w\|_2^2, \] where $\mathcal{L}$ is the training loss, $\eta$ controls pruning pressure, and $\lambda$ is weight decay [2107.10998]. Variations include: - Simultaneous optimization of quantizer thresholds with truncated Gaussian approximations, allowing back-propagation into threshold parameters [1810.01018]. - Pruning and re-initialization cycles to drive weights toward angular (cosine) alignment with ternary codebooks, minimizing the bias induced by the STE [2212.12653]. - Integration with complex models (e.g., diffusion transformers, spiking neural networks) by replacing all linear and projection layers with ternary quantized counterparts, sometimes with architectural adjustments such as RMS-norm layers for robust training [2405.14854], [2409.15849]. Sparsity induced by ternary quantization can be further controlled by scheduling the threshold parameter $\Delta$, either statically, learned, or gradually increased during training [2107.10998]. ## 3. Algorithmic Schemes and Implementation Pipelines Below is a general skeleton for ternary QAT, applicable to convolutional, transformer, or feed-forward architectures: ```python # Inputs: full-precision weights w, hyperparameters α, η, λ for epoch in range(num_epochs): for minibatch x, y: μ = mean(abs(w)) Δ = α * μ # layerwise threshold s = sum(abs(w_i) * (abs(w_i) > Δ)) / sum(abs(w_i) > Δ) w_q = s * sign(w) * (abs(w) > Δ) out = model(x, w_q) loss = loss_fn(out, y) + η * norm(w - w_q) ** 2 + λ * norm(w) ** 2 grad_wq = backprop(loss, w_q) # Straight-through estimator grad_w = grad_wq * (abs(w) <= 1) * s + 2 * η * (w - w_q) + 2 * λ * w w -= learning_rate * grad_w # Optionally adjust α, η, λ per layer or per epoch ``` This can be fused with architectural modules such as batch-norm folding, hybrid filter banks (mixing full-precision and ternary filters), or activation quantization as needed [2107.10998], [1911.01028], [2204.01234], [1705.01462]. Classic post-training approaches (e.g., TNT, FGQ) replace each weight (or group of weights) by its optimal ternary proxy according to closed-form statistics, sometimes with per-group scaling factors or cosine similarity maximization [1912.09236], [1705.01462]. ## 4. Pruning, Hyperspherical, and Hybrid Techniques Recent advances combine ternary quantization with additional constraints, regularizers, and computational constructs: - **Pruning Ternary Quantization (PTQ):** Embeds L2-norm regularization and pruning penalties, reducing weight discrepancy in the gradient estimator, offering compression rates up to $49\times$ with modest accuracy drops (e.g., $<2\%$ top-1 for ResNet-18/ImageNet) [2107.10998]. - **Hyperspherical Quantization (HQ, HLA):** Constrains weights to live on the unit sphere ($\|w_j\|_2=1$), incorporates iterative column-wise pruning and angular discrepancy penalties, enabling up to $48\times$ compression with accuracy retention superior to prior work [2212.12653], [2212.12649]. - **Hybrid Filter Banks:** Layerwise assignment of full-precision and ternary filters—optimized to retain sensitive filters in float, quantizing the rest—delivers adjustable energy and model-size savings (e.g., $51\%$ reduction, $28\%$ energy savings in MobileNets) [1911.01028]. - **Twin Network Augmentation (TNA):** For spiking neural networks, co-training a full-precision "twin" model alongside a ternary-quantized base with logit-matching losses, enhances performance of the compressed SNN, often matching or exceeding FP accuracy [2409.15849]. ## 5. Scaling, Thresholding, and Adaptation Mechanisms Robust quantization depends critically on threshold selection, scaling, and adaptation strategy: - Fixed heuristics (e.g., $\Delta \sim 0.7$–$0.8$ times mean $|w|$) [2107.10998], [1605.04711]. - Adaptive learning of thresholds via truncated Gaussian matching, cosine similarity, or minimum mean/maximum error [1810.01018], [2306.17442]. - Group-wise scale and threshold optimization in post-training conversion (FGQ), trading compute savings for fine control over accuracy [1705.01462]. - Loss-aware (LAT) and trained ternary quantization (TTQ) methods use per-layer or per-sign scaling, explicitly optimizing for network-level loss during assignment [1802.08635], [1612.01064]. - Ternary adaptation for fine-tuning quantized LLMs (LoTA-QAF) aligns ternary weights with the quantization grid for lossless merging and efficient inference [2505.18724]. Empirical analysis favors worst-case error minimization (TQuant) for robustness in data-free/QAT settings and mean-error minimization (MQuant) for PTQ with limited data [2306.17442]. ## 6. Hardware Efficiency, Sparsity, and Energy Savings Ternary quantization offers compelling hardware advantages: - **Storage:** $2$ bits/weight achieves $16\times$–$49\times$ model compression over $32$-bit float baselines [2107.10998], [1605.04711]. - **Compute:** Inference eliminates nearly all multiplies, relying largely on additions and sign operations—custom tensor accelerators, FPGAs, and ASICs exploit this for $3$–$10\times$ energy savings, $4$–$15\times$ throughput [1911.01028], [1605.04711], [2502.11880]. - **Packing:** Efficient runtime packing and unpacking (2-bit/word schemes) further shrink memory bandwidth requirements, as in ternary LLMs and diffusion transformers [2405.14854], [2502.11880]. - **Sparsity:** Induced by thresholding, can reach $63\%$ zero weights in typical architectures (AlexNet), or be scheduled per layer or group for optimal energy/accuracy trade-off [1612.01064], [1705.01462]. - **Bitwise Operations:** Dedicated computation patterns (bitwise XNOR, popcount, lookup-tables) replace floating-point MACs, scaling with activation bit-width and packing strategy [1912.02057], [2502.11880]. ## 7. Empirical Performance, Benchmark Results, and Limitations Representative task performance for ternary models is as follows: - **ResNet-18/ImageNet:** PTQ achieves $68.7\%$ top-1 ($-1.4\%$), $2.75$ MB ($16\times$ smaller) [2107.10998]; Hyperspherical Quantization yields $67.0$–$65.5\%$ at $37\times$–$48\times$ size reduction [2212.12653]. - **Mask R-CNN/COCO:** PTQ compresses $170$ MB to $5$ MB ($34\times$) with only $2.8\%$ drop in AP [2107.10998]. - **MobileNets:** Hybrid filter banks retain accuracy with $51\%$ size and $28\%$ energy savings [1911.01028]. - **Language Models/LLMs:** LoTA-QAF recovers and sometimes exceeds the accuracy of full-precision LoRA in quantized Llama-3.1/QWen-2.5. Inference speedup $1.7\times$–$2\times$ over low-bit adapters [2505.18724]. Bitnet.cpp achieves up to $6.25\times$ speedup, sub-2-bit lossless inference over baseline [2502.11880]. - **Fine-grained quantization (FGQ):** Post-hoc conversion with group size $N=4$ preserves accuracy within $4\%$ of baseline on ImageNet, with $9\times$ speedup [1705.01462]. - **Spiking Neural Networks (TNA):** Ternary SNN outperforms binary and matches or exceeds full precision in several benchmarks (CIFAR-10, -100, Fashion-MNIST, CIFAR10-DVS), with energy-sparse inference [2409.15849]. Performance gaps remain most pronounced in ultra-large models and challenging quantization of early/final layers, where mixed-precision or layerwise sensitivity scheduling is recommended. Post-training quantization may require light retraining for largest group sizes or non-Gaussian weight distributions [1705.01462], [2306.17442]. Block-fitting, lookup-table bandwidth, and inference stage optimization remain active development targets in emerging accelerators [2502.11880]. --- This entry consolidates mathematical formalism, algorithmics, engineering practices, and empirical findings in ternary weight quantization. The referenced schemes can be implemented per layer or architecture, adapted to domain-specific accuracy constraints, and deployed across diverse hardware with predictable benefits in compression, latency, and power consumption. Source: https://www.emergentmind.com/topics/ternary-weight-quantization-scheme