Ternary Neural Networks

• Ternary neural networks are models with weights and activations limited to three states (-1, 0, +1), balancing sparse computation with expressive capacity.
• They reduce computational complexity by replacing multiplications with sign changes, zero-skips, and simple arithmetic, which supports energy-efficient hardware designs.
• Advanced training methods, including supervised conversion, sparsity-induced regularization, and post-training quantization, help mitigate accuracy loss while optimizing efficiency.

Ternary neural networks (TNNs) are neural models in which weights, activations, or emitted spikes are constrained to three discrete states, most commonly $\{-1,0,+1\}$. In the feed-forward setting, this constraint is used to replace floating-point multiplication with sign changes, zero-skips, and additions or subtractions; in spiking settings, it enlarges the signaling alphabet beyond binary spikes while retaining event-driven computation. The ternary regime has therefore been studied as a middle ground between binary networks and higher-precision quantization, with goals that include model compression, multiplier-free inference, sparsity induction, and efficient realization on FPGA, ASIC, resistive-memory, computing-in-memory, and printed-electronics platforms (Alemdar et al., 2016, Faraone et al., 2017, Thakuria et al., 2024, Mrazek et al., 27 Aug 2025).

1. Core definitions and quantization models

A standard definition of a TNN constrains each deployed weight to the alphabet $\{-1,0,+1\}$. In sparse ternary training, a real-valued weight $w_r$ is mapped to a ternary weight $w_q$ by thresholding:

$$w_q= \begin{cases} +1, & w_r>\eta\ 0, & -\eta\le w_r\le \eta\ -1, & w_r<-\eta, \end{cases}$$

so the zero state explicitly encodes pruned or inactive parameters rather than merely low precision (Faraone et al., 2017). In the original resource-efficient TNN formulation, both weights and activations are ternary, and inference is performed with integer accumulation followed by a ternary activation defined by lower and upper thresholds on the post-synaptic sum (Alemdar et al., 2016).

Several later formulations preserve the ternary deployed model while modifying how ternary values are produced. RTN applies the same hard-threshold quantizer to both weights and activations, then reparameterizes the fixed ternary vector by learnable scale and offset terms so that range and magnitude are decoupled from direction (Li et al., 2019). STTN removes the explicit hard threshold $\Delta$ by decomposing a ternary kernel as the sum of two binary kernels, $T=B_1+B_2$ with $B_1,B_2\in\{+1,-1\}$, and at inference recombines them into a single ternary kernel with one shared scale (Xu et al., 2022). FGQ instead partitions a pretrained tensor into groups of size $N$ and solves an independent ternarization problem in each group, storing a ternary pattern and a group-specific scale $\alpha_j$ (Mellempudi et al., 2017).

These formulations show that “ternary” does not denote a single training rule. It denotes a representational constraint that can be imposed by global thresholding, neuron-wise or group-wise scaling, reparameterization, or binary decompositions. A plausible implication is that many empirical differences between TNN papers arise less from the alphabet itself than from how the threshold, scale, and sparsity pattern are parameterized.

2. Training paradigms and post-training conversion

One major line of work treats ternarization as a supervised conversion problem. The teacher-student method for resource-efficient TNNs first trains a full-precision teacher with standard techniques such as dropout and batch normalization, then converts the model layer by layer into a fully ternary student by greedily searching weight thresholds and output thresholds so that each student neuron mimics its teacher counterpart (Alemdar et al., 2016). A different training-oriented approach introduces sparsity-induced regularization: initial training with $\{-1,0,+1\}$0 and a quantization-threshold regularizer is followed by quantization pruning and then retraining with the pruning mask held fixed. In that setting, networks reached up to $\{-1,0,+1\}$1 sparsity and became approximately $\{-1,0,+1\}$2 smaller than an equivalent binary-weight network and approximately $\{-1,0,+1\}$3 smaller than a uniform ternary network $\{-1,0,+1\}$4 after compression (Faraone et al., 2017).

A second line of work begins from pretrained floating-point models and ternarizes them post hoc. FGQ ternarizes groups of weights in pretrained convolutional networks without additional training when group size is small. With $\{-1,0,+1\}$5, the reported ResNet-101 and ResNet-50 ImageNet-1K results are within $\{-1,0,+1\}$6 and $\{-1,0,+1\}$7 Top-1 accuracy of the full-precision baselines for 2-bit weights and 8-bit activations, while eliminating $\{-1,0,+1\}$8 of multiplications; with $\{-1,0,+1\}$9, the no-retraining drop is much larger, but fine-tuning recovers ResNet-50 to within $w_r$0 of baseline (Mellempudi et al., 2017). TNT replaces Euclidean reconstruction by cosine-similarity maximization,

$w_r$1

which reduces the search space from $w_r$2 to $w_r$3 candidates and yields an $w_r$4 algorithm via sorting and prefix sums. In the reported no-finetuning experiments, LeNet-5 on MNIST dropped from $w_r$5 to $w_r$6, VGG-7 on CIFAR-10 from $w_r$7 to $w_r$8, and VGG-16 on ImageNet Top-1 from $w_r$9 to $w_q$0 (Zhang et al., 2019).

Other methods move the adaptation into the quantizer itself. RTN introduces learnable scale and offset parameters for activations and weights, so that gradients continue to update dynamic range even when the straight-through estimator saturates; on ResNet-18 for ImageNet, RTN-R reports $w_q$1 Top-1 at 2/2 bits versus $w_q$2 for the non-reparameterized RTN-F (Li et al., 2019). FATNN uses two learnable step sizes per layer, constrains each quantized layer to the ternary set together with exactly one high-precision scale coefficient, and derives an $w_q$3 ternary inner product rather than the naive $w_q$4 bit-operation cost. On ImageNet, FATNN reports $w_q$5 Top-1/Top-5 for ResNet-18 and $w_q$6 for ResNet-50 in the ternary-weight/ternary-activation regime (Chen et al., 2020). STTN, trained from scratch without a manual hard threshold, reports ResNet-18 Top-1 accuracy up to $w_q$7 on ImageNet in the 2+2 regime (Xu et al., 2022).

3. Expressivity and approximation-theoretic properties

For several years, the empirical success of TNNs outpaced their formal analysis. A recent expressivity result studies ternary ReLU-regression networks in which each affine layer uses weight matrices in $w_q$8, odd-indexed layers have identity activation, even-indexed layers use ReLU, and the last layer is affine. If $w_q$9 is the input dimension, $$w_q= \begin{cases} +1, & w_r>\eta\ 0, & -\eta\le w_r\le \eta\ -1, & w_r<-\eta, \end{cases}$$0 the width, $$w_q= \begin{cases} +1, & w_r>\eta\ 0, & -\eta\le w_r\le \eta\ -1, & w_r<-\eta, \end{cases}$$1 the depth, $$w_q= \begin{cases} +1, & w_r>\eta\ 0, & -\eta\le w_r\le \eta\ -1, & w_r<-\eta, \end{cases}$$2 the largest odd integer not exceeding $$w_q= \begin{cases} +1, & w_r>\eta\ 0, & -\eta\le w_r\le \eta\ -1, & w_r<-\eta, \end{cases}$$3, and

$$w_q= \begin{cases} +1, & w_r>\eta\ 0, & -\eta\le w_r\le \eta\ -1, & w_r<-\eta, \end{cases}$$4

then the maximum number of linear regions satisfies

$$w_q= \begin{cases} +1, & w_r>\eta\ 0, & -\eta\le w_r\le \eta\ -1, & w_r<-\eta, \end{cases}$$5

This yields polynomial growth in width and exponential growth in depth, mirroring the classical behavior of unrestricted ReLU networks (Nakahara et al., 21 Jul 2025).

The construction adapts the Montúfar et al. integer-coefficient argument to the ternary setting by replacing large integer edge weights with constant-size ternary gadgets. In the resulting emulation, each real-valued ReLU layer is implemented by two ternary layers—an identity layer plus a ReLU layer—so the depth exponent is halved, while each gadget requires $$w_q= \begin{cases} +1, & w_r>\eta\ 0, & -\eta\le w_r\le \eta\ -1, & w_r<-\eta, \end{cases}$$6 units, replacing the real-valued width parameter by $$w_q= \begin{cases} +1, & w_r>\eta\ 0, & -\eta\le w_r\le \eta\ -1, & w_r<-\eta, \end{cases}$$7 (Nakahara et al., 21 Jul 2025). The same analysis shows that a ternary network can recover a comparable lower bound on the maximum number of linear regions of a general ReLU regression network either by squaring the width or by doubling the depth.

This result does not claim equality of all approximation properties between ternary and unrestricted networks, but it directly addresses a common misconception that severe weight quantization necessarily destroys the depth advantage of piecewise-linear architectures. Within the linear-region framework, the classical “deep-is-better” phenomenon survives the restriction to $$w_q= \begin{cases} +1, & w_r>\eta\ 0, & -\eta\le w_r\le \eta\ -1, & w_r<-\eta, \end{cases}$$8 (Nakahara et al., 21 Jul 2025).

4. Ternary signaling in spiking neural networks

In spiking neural networks, ternarization can apply not only to synaptic weights but also to the spike alphabet itself. The ternary-spike LIF neuron emits $$w_q= \begin{cases} +1, & w_r>\eta\ 0, & -\eta\le w_r\le \eta\ -1, & w_r<-\eta, \end{cases}$$9 by comparing the membrane potential to symmetric thresholds $\Delta$0, resetting after a nonzero spike. The associated information-capacity argument counts $\Delta$1 possible ternary maps versus $\Delta$2 binary maps, giving

$\Delta$3

whereas binary maps carry $\Delta$4 bits. On ImageNet with 4 timesteps, ResNet18 improves from $\Delta$5 Top-1 for binary spikes to $\Delta$6 for ternary spikes and $\Delta$7 for trainable ternary spikes; the ResNet20 CIFAR-10 energy estimate rises only from approximately $\Delta$8 to approximately $\Delta$9, an approximately $T=B_1+B_2$0 increase (Guo et al., 2023).

Subsequent work argues that the basic ternary spike still suffers from iterative information loss, temporal gradient vanishing, and irregular membrane-potential distributions. CTSN augments the ternary neuron with a learnable complemental term $T=B_1+B_2$1 that stores historical input contributions and never resets, while TMPR adds a time-varying membrane-potential regularizer that creates extra backpropagation paths. In the reported results, CIFAR-100 with ResNet19 at $T=B_1+B_2$2 improves from $T=B_1+B_2$3 to $T=B_1+B_2$4, CIFAR10-DVS with ResNet20 at $T=B_1+B_2$5 improves from $T=B_1+B_2$6 to $T=B_1+B_2$7, and ImageNet-100 with SEW-ResNet34 at $T=B_1+B_2$8 reaches $T=B_1+B_2$9 (Zhang et al., 22 Jan 2026).

A separate reinforcement-learning study shows that symmetric ternary spikes can behave poorly in deep Q-learning because the expected surrogate gradient becomes approximately zero when positive and negative thresholds are symmetric. The proposed asymmetric ternary neuron trains only the negative threshold, breaking the $B_1,B_2\in\{+1,-1\}$0 symmetry. In seven Atari games, the symmetric ternary DTSQN averages $B_1,B_2\in\{+1,-1\}$1 of binary DSQN performance, whereas the asymmetric DATSQN averages $B_1,B_2\in\{+1,-1\}$2 and reaches $B_1,B_2\in\{+1,-1\}$3 on Beam Rider (Ghoreishee et al., 3 Jun 2025). These results indicate that ternary signaling in SNNs is not a uniform modification: the choice between symmetric and asymmetric thresholds, fixed versus trainable spike amplitude, and explicit membrane regularization materially changes trainability.

5. Hardware realization and accelerator co-design

The hardware appeal of TNNs follows directly from the semantics of the alphabet. When $B_1,B_2\in\{+1,-1\}$4, multiplication reduces to a sign flip, a zero skip, or a simple addition/subtraction, and the zero state induces sparsity in both storage and execution (Alemdar et al., 2016, Faraone et al., 2017). This basic observation has supported several distinct accelerator styles.

One approach specializes the datapath to a frozen ternary model. In FPGA unrolling, a convolution with known ternary weights is compiled into a pruned adder tree in which every zero tap is removed offline and every $B_1,B_2\in\{+1,-1\}$5 tap becomes an add or subtract. On an AWS F1 instance, this eliminated $B_1,B_2\in\{+1,-1\}$6 of the operations in convolutional layers, achieved $B_1,B_2\in\{+1,-1\}$7 accuracy on CIFAR-10, and delivered $B_1,B_2\in\{+1,-1\}$8 frames per second with $B_1,B_2\in\{+1,-1\}$9 latency (Tridgell et al., 2019). RTN complements this systems view with a bitwise ternary encoding that uses a nonzero indicator and a sign bit; in FPGA validation, the corresponding ternary dot-product unit provides $N$0 and $N$1 savings on power and area, respectively, compared with full-precision convolution (Li et al., 2019).

Another approach embeds ternary computation into memory arrays. In differential 2T2R RRAM synapses, the state $N$2 encodes $N$3, $N$4 encodes $N$5, and $N$6 encodes $N$7, allowing a precharge sense amplifier to extract the ternary weight in a single sense operation (Laborieux et al., 2020). On CIFAR-10, TNNs implemented with this substrate outperform corresponding BNNs, and the measured zero-to-nonzero error modes cause only a $N$8 accuracy loss at realistic error rates, so no ECC is needed (Laborieux et al., 2020). A related near-threshold study reports that moving from binary to ternary networks increases CIFAR-10 accuracy by about $N$9 absolute while reducing average per-MAC energy from about $\alpha_j$0 to about $\alpha_j$1 because zero weights can skip sensing and accumulation stages (Laborieux et al., 2020).

Computing-in-memory generalizes the same principle to signed ternary inputs and weights. SiTe-CiM stores a ternary weight using differential binary cells and cross-coupling circuitry, with two design points: SiTe-CiM I adds two transistors per cell and SiTe-CiM II adds four transistors per 16 cells. Across 8T-SRAM, 3T-eDRAM, and 3T-FEMFET technologies, the arrays achieve up to $\alpha_j$2 lower CiM latency and $\alpha_j$3 CiM energy savings relative to near-memory baselines; at the accelerator level they provide up to $\alpha_j$4 throughput boost and up to $\alpha_j$5 energy reduction (Thakuria et al., 2024).

Printed-electronics work extends ternary hardware beyond silicon-centric assumptions. Approximate printed TNNs with custom analog-to-binary conversion have been reported as at least $\alpha_j$6 smaller in area and $\alpha_j$7 lower in power than prior printed MLPs (Mrazek et al., 2024). A later holistic flow that co-optimizes ADC precision, approximate LTG and popcount blocks, and whole-network architecture reports average improvements of $\alpha_j$8 in area and $\alpha_j$9 in power over existing approximate printed neural networks, while enabling printed-battery-powered operation with under $\{-1,0,+1\}$00 accuracy loss (Mrazek et al., 27 Aug 2025).

6. Empirical trade-offs, misconceptions, and open directions

The empirical record shows that TNNs are neither uniformly accuracy-preserving nor uniformly accuracy-limited. On one end of the spectrum, no-retraining TNT produces only a $\{-1,0,+1\}$01 absolute drop for LeNet-5 on MNIST and a $\{-1,0,+1\}$02 drop for VGG-7 on CIFAR-10, but an $\{-1,0,+1\}$03 Top-1 drop for VGG-16 on ImageNet (Zhang et al., 2019). On the other end, more elaborate training and quantizer design narrow the gap substantially: FATNN reaches $\{-1,0,+1\}$04 Top-1/Top-5 on ImageNet with ternary weights and activations for ResNet-50, and STTN reaches $\{-1,0,+1\}$05 Top-1 on ResNet-18 in the same low-bit regime (Chen et al., 2020, Xu et al., 2022). These results indicate that the main question is not whether ternary quantization works in the abstract, but under which quantization rule, scaling policy, activation precision, and retraining budget it works.

A recurrent misconception is that ternary automatically dominates binary because it adds a zero state. Several papers support the practical value of the zero code for pruning and information capacity, but they also show nontrivial caveats. FATNN explicitly notes that conventional TNNs still require 2 bits to encode three levels, so they can have memory consumption and speed similar to standard 2-bit models while offering worse representational capability unless the computation pattern is redesigned (Chen et al., 2020). In spiking reinforcement learning, symmetric ternary neurons can perform markedly worse than binary ones unless the threshold asymmetry is introduced to correct surrogate-gradient bias (Ghoreishee et al., 3 Jun 2025). Conversely, sparse ternary regularization and CTSN-style state augmentation show that the zero state can be exploited as a structural degree of freedom rather than a mere midpoint code (Faraone et al., 2017, Zhang et al., 22 Jan 2026).

Open problems identified across the literature are correspondingly diverse. Feed-forward TNN work raises questions about end-to-end learnable thresholds, mixed-precision allocation, multi-scale ternary parameterizations, and grouping strategies for post-training ternarization (Alemdar et al., 2016, Mellempudi et al., 2017, Chen et al., 2020). Spiking work points to per-channel or per-neuron amplitudes, asymmetric thresholds or levels such as $\{-1,0,+1\}$06, and tighter analyses of channel capacity and dynamical stability (Guo et al., 2023, Zhang et al., 22 Jan 2026). Hardware work leaves unresolved how to preserve ternary efficiency under larger models, routing constraints, analog variation, and broader sensor-interface co-design (Tridgell et al., 2019, Thakuria et al., 2024, Mrazek et al., 27 Aug 2025). Taken together, these directions show that TNNs are best understood not as a single compression trick, but as a broad research program in discrete neural representation, training under extreme quantization, and algorithm-hardware co-optimization.