Error Decay Estimator (EDE)

• EDE is a surrogate gradient method for binary neural networks that dynamically interpolates between identity mapping and a tight sign approximation using a tanh-based schedule.
• It preserves broad gradient support during early training and minimizes gradient mismatch later, ensuring stable and precise weight updates.
• Integrated in IR-Net, EDE reduces the accuracy gap between 1-bit and full-precision models, achieving up to +2.7% top-1 accuracy gain on CIFAR-10 benchmarks.

The Error Decay Estimator (EDE) is a surrogate gradient methodology introduced for binary neural networks (BNNs) to address information loss in backward propagation caused by non-differentiability of the sign function. EDE is a core component of the Information Retention Network (IR-Net), designed to reduce the accuracy gap between 1-bit and full-precision deep neural networks by employing a dynamic, tanh-based schedule for surrogate gradients. Its principal innovation is bridging the fundamental trade-off in straight-through estimator (STE) design: maintaining strong weight update capacity early in training and minimizing gradient mismatch late in training by continuously transitioning from an identity mapping to a tight sign approximation (Qin et al., 2019).

1. Background and Motivation

Binarization of weights and activations enables deep neural network compression and fast inference by leveraging efficient bitwise operations. However, mapping real-valued weights and activations to $\{\pm1\}$ introduces information loss. While forward-propagation quantization error has been extensively studied, the fundamental challenge in the backward pass stems from the fact that the true derivative of the sign function is zero almost everywhere, precluding standard gradient-based optimization. Existing approaches employ surrogate straight-through estimators, but the two most common variants have mutually exclusive drawbacks:

• Identity STE: $g(x)=x$. This surrogate disregards the hard binary clipping, causing large gradient-mismatch error between the true and surrogate gradients.
• Clip (Hardtanh) STE: $g(x) = \mathrm{HardTanh}(x)$. The derivative is 1 for $|x| \leq 1$ and 0 otherwise, suppressing updates for weights outside $[-1,1]$ and hindering out-of-range optimization.

Neither surrogate maintains update flexibility throughout training while ensuring alignment with the true gradient structure of the sign function, motivating the development of EDE (Qin et al., 2019).

2. EDE: Mathematical Formulation

EDE parameterizes the surrogate gradient as a dynamically controlled family of functions:

• Surrogate function: $g(x) = k\,\tanh(t\,x)$
• Derivative: $g'(x) = k\,t\,\left[1 - \tanh^2(t\,x)\right]$

Here, $t$ and $k$ are scalar hyperparameters that change as a function of training epoch $i$ in $0,\ldots,N$:

$$t(i) = T_{\min} \cdot 10^{\frac{i}{N}\log_{10}(T_{\max}/T_{\min})}, \qquad k(i) = \max\biggl(\frac{1}{t(i)},\,1\biggr)$$

with $T_{\min}=10^{-1}$, $T_{\max}=10^{1}$.

This yields:

• At $i=0$ ($t=0.1$, $k=10$): $g(x)\approx x$ (Identity-like regime, broad gradient support)
• At $i=N$ ($t=10$, $k=1$): $g(x)\approx \tanh(10x)$ (Sharp approximation of $\mathrm{sign}(x)$, narrow gradient focus near the discontinuity)

During the backward pass through a binarization node, the chain rule applies:

$$\frac{\partial\mathcal{L}}{\partial x} = \frac{\partial\mathcal{L}}{\partial Q(x)} \, g'(x)$$

where $Q(x) = \mathrm{sign}(x)$ is the binarization function used on the forward path (Qin et al., 2019).

3. Dynamic Scheduling and Training Dynamics

The dynamic scheduling of $t$ and $k$ forms the crux of EDE's effectiveness:

• Stage 1 (early epochs, small $t$): $g'(x) \approx 1$ over a wide $x$ range, emulating the Identity STE and preserving gradient flow for large-magnitude weights, which facilitates rapid parameter exploration and update ability.
• Stage 2 (late epochs, large $t$): $g'(x)$ peaks sharply at $x=0$ and decays outside a narrow region, mimicking the true (Dirac delta) derivative of the sign function and reducing gradient mismatch.

By smoothly shifting from Stage 1 to Stage 2, EDE maintains training stability and ultimately improves the representational matching between the surrogate and the discrete binarization function (Qin et al., 2019).

4. Integration in IR-Net and Algorithmic Steps

EDE is deployed as an integral part of the IR-Net training framework, which combines two principal methods: Libra Parameter Binarization (Libra-PB) for forward information retention, and EDE for backward information retention.

Backward pass of IR-Net with EDE:

1. Compute current $t$ and $k$ for the training epoch.
2. Set surrogate derivatives for activations and weights via $g'(a)$, $g'(w)$.
3. Backpropagate gradients:
   • $$\partial\mathcal{L}/\partial a = (\partial\mathcal{L}/\partial Q_a) g'(a)$$
   • $$\partial\mathcal{L}/\partial w = (\partial\mathcal{L}/\partial Q_w)\,g'(w)\,2^s$$
4. Update full-precision weights $w$ using SGD.

The forward pass applies Libra-PB and binarizes activations as $Q_a(a)=\mathrm{sign}(a)$ (Qin et al., 2019).

5. Experimental Evidence and Quantitative Impact

Extensive evaluation on ResNet-20 with CIFAR-10 using various network structures demonstrates the efficacy of EDE. Reported top-1 accuracy figures are:

Variant	Top-1 Accuracy (%)	Relative Gain
Baseline STE	83.8	–
EDE only	85.2	+1.4
Libra-PB only	84.9	+1.1
Libra-PB + EDE (IR-Net)	86.5	+2.7; gap to FP: 4.3%

Visualization of EDE's effect on weight histograms shows initial wide spread and nearly constant gradients, transitioning to weight concentration around $\pm1$ and a sharply peaked surrogate by epoch 200. The area of gradient mismatch shrinks smoothly, contrasting with persistent mismatch under fixed STEs (Qin et al., 2019).

6. Significance and Broader Implications

EDE provides a simple, two-parameter family of surrogate gradients that "decay" from Identity to Sign via a tanh-based interpolation, directly reconciling the competing demands of wide update support in early training and surrogate-target alignment in late training. Integrated into IR-Net, EDE substantially reduces backward information loss, narrowing the accuracy deficit between 1-bit and full-precision models. This suggests that adaptive, schedule-driven surrogates may be broadly advantageous for gradient-based optimization of architectures with non-differentiable quantization operations (Qin et al., 2019).