Soft Threshold Ternary Networks

• STTN is a quantization method for CNNs that replaces hard thresholds with a soft, learnable binary decomposition to create efficient ternary representations.
• It decomposes full-precision weights into dual binary proxies that are fused into a ternary model, reducing computational cost while maintaining high accuracy on benchmarks like ImageNet.
• STTN integrates seamlessly into CNNs, such as ResNet-18, offering simplified inference with a single 2-bit convolution and competitive Top-1/Top-5 performance.

Soft Threshold Ternary Networks (STTN) are a quantization method for convolutional neural networks (CNNs) that eliminates the need for a manually selected hard threshold during ternarization. Instead, STTN decomposes network weights into two parallel binary domains during training, which are then fused into a ternary representation for inference. This relaxation enables efficient model deployment while maintaining competitive accuracy, particularly for deployment scenarios constrained by memory and computational resources. STTN achieves state-of-the-art performance with ternary weights and activations, outperforming prior quantization methods on benchmarks such as ImageNet (Xu et al., 2022).

1. Mathematical Formulation of Soft Threshold Ternarization

STTN replaces the rigid threshold-based quantization $\Delta \to \{-1,0,+1\}$ found in prior ternary neural network (TNN) work with a soft, learnable alternative. The core observation is that any ternary tensor $T\in\{-1,0,+1\}^n$ can be expressed as the sum of two binary tensors $B_1, B_2\in\{-1,+1\}^n$:

$T = B_1 + B_2.$

At training time, a full-precision weight tensor $W \in \mathbb{R}^n$ is decomposed as follows:

• Create two copies $W_1 = W_2 = W$.
• Compute $B_1 = \mathrm{sign}(W_1)$ and $B_2 = \mathrm{sign}(W_2)$.
• Determine the shared scaling factor:

$\alpha = \frac{1}{N}\sum_{i=1}^N |W_i|,$

with $N$ being the number of weights in $W$.

The sum $T = B_1 + B_2$ takes values in $\{-2, 0, +2\}$, which via re-scaling yields the ternary quantized weight:

$W_t = \alpha T \in \{-2\alpha, 0, +2\alpha\}.$

Alternatively, absorbing the factor into $\alpha$, one may denote $W_t \in \{-\alpha, 0, +\alpha\}$. The crucial distinction is that the zero output arises "softly" wherever $B_1$ and $B_2$ disagree, thus obviating the need for explicit threshold tuning.

2. Training Algorithm and Backpropagation

STTN employs a training algorithm with explicit logic for forward and backward passes, summarized as follows for each layer $\ell$:

• Forward pass:
  • Compute scaling factor $$\alpha^{(\ell)} = \frac{1}{N}\sum_{i=1}^{N}|W^{(\ell)}_i|$$.
  • Compute binary proxies $B_1 = \mathrm{sign}(W^{(\ell)})$, $B_2 = \mathrm{sign}(W^{(\ell)})$.
  • Convolve with input $X$: $Y = \alpha(B_1 * X + B_2 * X)$.
  • Apply batch normalization and ternary activation quantization.
• Activation ternarization (training and inference):

$$X^t_i = \begin{cases} \mathrm{sign}(X_i), & |X_i| > 0.5;\ 0, & \text{otherwise} \end{cases}$$

with the straight-through estimator (STE) for quantized backpropagation.

• Backward pass:
  • For each scaled binary weight $\widehat{W}_k = \alpha B_k$,

  $$\frac{\partial \ell}{\partial W_i} = \frac{1}{2N} \,\mathrm{sign}(W_i)\sum_{k,j} \left(\frac{\partial \ell}{\partial \widehat{W}_{k,j}}\mathrm{sign}(W_j)\right) + \alpha \,\mathbb{I}_{|W_i|\leq 1}\sum_{k,j}\left(\frac{\partial \ell}{\partial \widehat{W}_{k,j}}\right).$$ - The indicator for $|W| \leq 1$ replaces the derivative of the sign function in practice. - Ternary activations are back-propagated using STE.

• Optimization: Network weights, batch normalization, and scaling factors are updated (Adam optimizer is used).

A high-level pseudocode block describes this process, aligning with the structure observed in the data (Xu et al., 2022).

3. Integration into Convolutional Neural Networks

STTN is integrated into standard CNNs, such as ResNet-18, by adapting the convolutional and normalization sequence:

• Block modification: For each 3×3 convolution (omitting the initial and final layers):

  • Apply batch normalization
  • Quantize the activation via ternary thresholding
  • Apply the STTN-based TernaryConv (training: dual binary branches; inference: fused ternary kernel)
  • (Optional) Follow with ReLU
• Inference efficiency: Post-training, the two binary kernels $B_1$ and $B_2$ are combined offline to form a single ternary kernel $T = B_1 + B_2$. This reduces inference to a single 2-bit convolution, requiring no run-time dual-branch computation.
• Downsampling layers: 1×1 convolutions are replaced by max pooling and identity connections, enabling more straightforward quantization.

This design removes the need for double-convolutions at inference, allowing deployment with only one ternary-valued convolution per layer.

4. Experimental Results and Ablation Analyses

Multiple ImageNet experiments demonstrate STTN’s empirical effectiveness:

Network	Precision	Top-1 (%)	Top-5 (%)
ResNet-18	STTN 2+2	66.2	86.4
ResNet-18	STTN 2+2 (fine-tuned)	68.2	87.9
ResNet-18	TWN 2+32	61.8	84.2
ResNet-18	TTQ 2+32	66.6	87.2
ResNet-18	FATNN 2+2	66.0	86.4
ResNet-18	XNOR 1+1	51.2	73.2

On AlexNet, STTN (2+2) achieves 55.6% Top-1 and 78.9% Top-5, exceeding previous binary/ternary models.

L₂ distance ablation measuring the quantization approximation $\|W-\alpha T\|_2^2$ indicates STTN (379.3) has lower error than TWN (537.8) and TTQ (439.5), suggesting a tighter approximation to full-precision weights. A layerwise sparsity analysis confirms that the rate of zero-valued weights is automatically adapted per layer.

A plausible implication from these findings is that STTN provides improved accuracy for a given bit-width without necessitating hand-tuned thresholds or full-precision pretraining.

5. Advantages, Limitations, and Deployment

STTN demonstrates several key advantages in low-bit neural network quantization:

• No explicit threshold hyper-parameter: STTN circumvent the need for the threshold $\Delta$ entirely, learning quantization intervals via the binary decomposition.
• Self-contained training: Models are trainable from scratch, not dependent on pre-trained full-precision weights.
• Inference simplicity: Only a single ternary convolution is required at inference, which can be implemented using two bit-planes and masking to realize computational savings.
• Accuracy: For ResNet-18 on ImageNet, the accuracy drop relative to full-precision is reduced to approximately 1–2%.

Potential drawbacks include a slight increase in hardware complexity for ternary versus strictly binary convolutions, as zeros necessitate dynamic masking or skipping. Additionally, STE-based gradient approximations may percolate bias, requiring learning rate adjustments.

Deployment considerations: On CPU or embedded hardware, the ternary kernel $T$ can be mapped to sign and magnitude bit-planes, enabling acceleration via bitwise operations, vectorized popcount, and 2-bit fused MAC instructions. Memory and speedup figures can approach the theoretical maxima of prior binary and TNN schemes.

6. Significance, Context, and Future Directions

STTN marks a progression in quantized neural network research by removing the dependency on hard thresholds without sacrificing accuracy or incurring inference-time penalties. This suggests broader applicability for deployment in resource-limited environments and simplifies hyper-parameter optimization. The competitive empirical results on large-scale benchmarks indicate state-of-the-art performance among ternary networks. A plausible implication is that future research may further generalize the soft decomposition approach beyond ternary quantization, or combine it with advanced gradient estimation for improved stability and convergence.

For further technical detail and code, see the work of Xu et al. (Xu et al., 2022).