ReLUDown Methods in Neural Networks

Updated 6 July 2026

ReLUDown is a cluster of methods that modify standard ReLU behavior to enhance plasticity in continual learning and reduce computational overhead in private inference.
In continual learning, ReLUDown introduces a static activation modification that preserves gradient flow and maintains feature sensitivity, improving current-task performance with modest overhead.
For private inference, ReLUDown employs coordinate descent and ReLU sharing techniques to strategically reduce expensive nonlinear operations, cutting communication and runtime costs.

ReLUDown denotes more than one ReLU-centric method in recent machine-learning literature. In continual learning, it is a static, drop-in alternative to ReLU introduced as the plasticity-preserving component of RDBP, where RDBP denotes “ReLUDown + Decreasing Backpropagation” (Künzel et al., 14 Jul 2025). In private inference, the same name has been used for a coordinate-descent method for network linearization under a ReLU budget (Rakhlin et al., 14 Nov 2025). The supplied literature also associates the term with earlier work on ReLU sharing in privacy-preserving CNN inference, where groups of activations share a single ReLU decision to reduce secure-computation cost (Helbitz et al., 2021). This terminological overlap is substantive rather than cosmetic: the cited works target different bottlenecks—loss of plasticity in continual learning, discrete ReLU-budget optimization in private inference, and secure-inference communication overhead—while all treating standard ReLU as the object of intervention.

1. Terminological scope and canonical usages

The literature represented here supports three distinct usages of “ReLUDown” or closely associated ReLU-reduction mechanisms.

Usage	Core mechanism	Primary setting
ReLUDown in RDBP	Static activation modification $f(x)=\max(0,x)-\max(0,-x+d)$	Continual learning
ReLUDown for network linearization	Block coordinate descent over a binary ReLU mask $m$ under a budget $B$	Private inference
ReLU sharing	One ReLU decision is shared by a group of activations via gReLU	Privacy-preserving CNN inference

The continual-learning version modifies the activation function itself and does not optimize a ReLU budget. The private-inference version instead treats ReLU retention as a discrete sparse optimization problem over a binary mask. The earlier ReLU-sharing work reduces the number of secure ReLU evaluations by grouping activations, and the supplied material explicitly notes that this approach is described in the paper as ReLU sharing even though it is associated here with ReLUDown (Helbitz et al., 2021). This suggests that “ReLUDown” is not yet a single standardized technical term, but a label that has been attached to multiple methods for mitigating drawbacks of standard ReLU.

2. ReLUDown as an activation-function modification in continual learning

In “A Simple Baseline for Stable and Plastic Neural Networks,” ReLUDown is defined as a static, drop-in alternative to ReLU: $f(x) = \max(0,x) - \max(0,-x+d), \qquad d<0.$ The method is introduced to address loss of plasticity in continual learning, specifically neuron dormancy and feature sensitivity loss induced by preactivation distribution shift. Standard ReLU has zero gradient for $x<0$ ; ReLUDown instead adds a linear component with a non-zero gradient for the range from the hinge point $d<0$ to $-\infty$ , so units continue to receive gradient information even when preactivations drift negative. The paper describes the intended effect as preserving the nonlinearity of ReLU while preventing the network from entering a regime in which many units output zero indefinitely (Künzel et al., 14 Jul 2025).

The same work positions ReLUDown against both standard ReLU and alternative activation strategies. Compared with standard ReLU, the central difference is the negative-side slope. Compared with adaptive rational activations or Padé Activation Units, ReLUDown is intentionally static and lightweight: it does not learn extra activation parameters, does not add extra modules, and does not change the architecture. Compared with tanh, it remains closer to standard CNN behavior. The paper further states that ReLUDown helps the preactivation distribution converge toward a normal distribution, mitigating the plasticity-loss mechanism identified as linearization and preactivation distribution shift.

Within RDBP, ReLUDown is paired with Decreasing Backpropagation, which addresses the stability side of the stability-plasticity tradeoff via

$\text{bp}_{decrease}(n, l, f, a) = \text{bp}_{standard} \cdot \bigl(1-(l\cdot f) + (l\cdot f)\,a^{-n}\bigr).$

Here $n$ is task number, $l$ is layer index, $m$ 0 is decrease factor, and $m$ 1 is speed factor. The paper’s division of labor is explicit: ReLUDown supports plasticity by keeping units alive and responsive, while DBP reduces catastrophic overwriting by progressively shielding early layers from large updates.

Empirically, the paper reports that standard ReLU shows declining current-task performance over time on Continual ImageNet, while ReLUDown maintains plasticity much better. ReLUDown and PAU both preserve current-task performance over time, but ReLUDown does so with less computational overhead than PAU. The appendix reports relative training-time overhead per task of baseline for ReLU, $m$ 2 for tanh, $m$ 3 for PAU, $m$ 4 for ReLUDown, and $m$ 5 for DBP + ReLUDown. The same paper reports that preactivation distributions with ReLU shift negative and broaden, whereas with ReLUDown they converge toward a near-normal shape with fewer fluctuations (Künzel et al., 14 Jul 2025).

A notable internal ambiguity concerns the hinge parameter. The main text states $m$ 6, the appendix later lists the hinge point parameter as $m$ 7, and the figure appendix says they used $m$ 8 in experiments. The negative-hinge interpretation is the one aligned with the formula and conceptual discussion, but the discrepancy is part of the published presentation and should be kept in view.

3. ReLUDown as coordinate descent for network linearization

In “Coordinate Descent for Network Linearization,” ReLUDown is a private-inference optimization method rather than an activation function. The paper formulates network linearization as a discrete sparse optimization problem over weights $m$ 9 and a binary ReLU mask $B$ 0: $B$ 1 Here $B$ 2 means a ReLU is kept, $B$ 3 means the ReLU is replaced by identity or another non-ReLU operation, and $B$ 4 is the target ReLU budget. The paper follows the definition used by SNL: network linearization means replacing some ReLUs with identity functions so that the network becomes more linear and cheaper for PI; it does not mean making the whole network linear (Rakhlin et al., 14 Nov 2025).

The main methodological claim is that prior selective methods such as SNL and AutoReP solve the discrete problem indirectly through smooth relaxation and a final hard-thresholding step, which often produces a large accuracy drop. ReLUDown instead works directly in the discrete domain by block coordinate descent. At each iteration it removes a block of ReLUs of size DRC, performs up to RT random trials, selects the candidate subset with the smallest accuracy degradation, masks those ReLUs out, and fine-tunes the network. ADT, the Accuracy Degradation Tolerance, is used to accept a candidate early when degradation is small enough. The algorithm therefore produces a sparse solution by design rather than by thresholding a soft mask at deployment time.

The paper gives a coordinate-descent-style bound. Let $B$ 5 be the optimal mask and $B$ 6 the mask after $B$ 7 steps. Then

$B$ 8

with

$B$ 9

With probability larger than $f(x) = \max(0,x) - \max(0,-x+d), \qquad d<0.$ 0,

$f(x) = \max(0,x) - \max(0,-x+d), \qquad d<0.$ 1

The paper uses these expressions to motivate smaller DRC values and a larger number of coordinate-descent iterations.

Experimentally, the method is evaluated on ResNet18 and WideResNet22-8 for CIFAR-10, CIFAR-100, and TinyImageNet, with SGD in some settings, Adam in others, cosine annealing schedules, and fine-tuning after each reduction step. The paper reports that ReLUDown outperforms SNL at every tested ReLU budget. Examples given in the reported tables include CIFAR-100 with ResNet18 at $f(x) = \max(0,x) - \max(0,-x+d), \qquad d<0.$ 2 ReLUs, where SNL attains $f(x) = \max(0,x) - \max(0,-x+d), \qquad d<0.$ 3 and ReLUDown attains $f(x) = \max(0,x) - \max(0,-x+d), \qquad d<0.$ 4; CIFAR-10 with ResNet18 at $f(x) = \max(0,x) - \max(0,-x+d), \qquad d<0.$ 5 ReLUs, where SNL attains $f(x) = \max(0,x) - \max(0,-x+d), \qquad d<0.$ 6 and ReLUDown attains $f(x) = \max(0,x) - \max(0,-x+d), \qquad d<0.$ 7; and TinyImageNet with ResNet18 at $f(x) = \max(0,x) - \max(0,-x+d), \qquad d<0.$ 8 ReLUs, where SNL attains $f(x) = \max(0,x) - \max(0,-x+d), \qquad d<0.$ 9 and ReLUDown attains $x<0$ 0. The paper states that the largest advantage appears in the low-budget regime, that mask overlap across budgets is typically above $x<0$ 1, and that ReLUDown can also be applied on top of AutoReP (Rakhlin et al., 14 Nov 2025).

In “Reducing ReLU Count for Privacy-Preserving CNN Speedup,” the central mechanism is not named ReLUDown in the paper itself, but the supplied material associates it with the term and describes it as ReLU sharing. The method reduces private-inference cost by allowing a group of activations to share one ReLU decision. This is motivated by the observation that linear layers and nonlinear layers are handled by different cryptographic tools, so switching representations at every ReLU is expensive. In the SecureNN/PySyft setting reported by the paper, linear or convolution layers require $x<0$ 2 rounds, DReLU requires $x<0$ 3 rounds, Mul requires $x<0$ 4 rounds, and a full ReLU $x<0$ 5 DReLU + Mul $x<0$ 6 rounds; DReLU communication is $x<0$ 7, while Mul communication is $x<0$ 8 (Helbitz et al., 2021).

The paper generalizes elementwise ReLU to a grouped decision rule: $x<0$ 9 and also defines a patch-based version in which a patch $d<0$ 0 has its own vector $d<0$ 1. Grouping can be uniform and data-agnostic, such as non-overlapping $d<0$ 2, $d<0$ 3, or whole-channel $d<0$ 4 patches, or data-dependent via agglomerative clustering of binary ReLU responses over the training set using Hamming distance. The paper also explores noisy generalized ReLU, where with probability $d<0$ 5 the normal grouped decision is used and otherwise the sign is flipped, thereby acting as a randomized perturbation.

The experimental results are reported on CIFAR-10, SVHN, and Fashion-MNIST. On CIFAR-10, the baseline network has $d<0$ 6 ReLUs per channel across the first two ReLU layers and $d<0$ 7 accuracy. Uniform $d<0$ 8 sharing yields $d<0$ 9 ReLUs and $-\infty$ 0 accuracy, uniform $-\infty$ 1 yields $-\infty$ 2 and $-\infty$ 3, and uniform $-\infty$ 4 yields $-\infty$ 5 and $-\infty$ 6. The same paper reports that “Layer 1 only” sharing $-\infty$ 7 reaches $-\infty$ 8, whereas “Layer 2 only” sharing $-\infty$ 9 falls to $\text{bp}_{decrease}(n, l, f, a) = \text{bp}_{standard} \cdot \bigl(1-(l\cdot f) + (l\cdot f)\,a^{-n}\bigr).$ 0, supporting the interpretation that early layers are easier to compress. With learned fully connected weighting plus $\text{bp}_{decrease}(n, l, f, a) = \text{bp}_{standard} \cdot \bigl(1-(l\cdot f) + (l\cdot f)\,a^{-n}\bigr).$ 1 noise and only one ReLU per channel per layer, the paper reports $\text{bp}_{decrease}(n, l, f, a) = \text{bp}_{standard} \cdot \bigl(1-(l\cdot f) + (l\cdot f)\,a^{-n}\bigr).$ 2 accuracy. For CIFAR-10, runtime drops from $\text{bp}_{decrease}(n, l, f, a) = \text{bp}_{standard} \cdot \bigl(1-(l\cdot f) + (l\cdot f)\,a^{-n}\bigr).$ 3 s and $\text{bp}_{decrease}(n, l, f, a) = \text{bp}_{standard} \cdot \bigl(1-(l\cdot f) + (l\cdot f)\,a^{-n}\bigr).$ 4 MB communication in the original network to $\text{bp}_{decrease}(n, l, f, a) = \text{bp}_{standard} \cdot \bigl(1-(l\cdot f) + (l\cdot f)\,a^{-n}\bigr).$ 5 s and $\text{bp}_{decrease}(n, l, f, a) = \text{bp}_{standard} \cdot \bigl(1-(l\cdot f) + (l\cdot f)\,a^{-n}\bigr).$ 6 MB for uniform $\text{bp}_{decrease}(n, l, f, a) = \text{bp}_{standard} \cdot \bigl(1-(l\cdot f) + (l\cdot f)\,a^{-n}\bigr).$ 7, while FC+Noise attains $\text{bp}_{decrease}(n, l, f, a) = \text{bp}_{standard} \cdot \bigl(1-(l\cdot f) + (l\cdot f)\,a^{-n}\bigr).$ 8 s and $\text{bp}_{decrease}(n, l, f, a) = \text{bp}_{standard} \cdot \bigl(1-(l\cdot f) + (l\cdot f)\,a^{-n}\bigr).$ 9 MB. The paper summarizes the broader outcome as cutting the number of ReLU operations by up to three orders of magnitude and cutting communication bandwidth by more than $n$ 0 (Helbitz et al., 2021).

5. Position within the broader ReLU-reduction literature

The private-inference variants of ReLUDown sit in a broader line of work that treats ReLU count as the principal optimization target because ReLU dominates secure-inference cost. DeepReDuce makes this point explicitly: under DELPHI, ReLUs account for $n$ 1 of ResNet32’s online private inference time. Its pipeline consists of ReLU Culling, ReLU Thinning, and ReLU Reshaping, coupled with knowledge distillation, and it outputs a Pareto frontier of networks that trade off ReLU count and accuracy. The paper reports up to $n$ 2 accuracy improvement at iso-ReLU count and up to $n$ 3 fewer ReLUs at iso-accuracy over the previous state of the art (Jha et al., 2021).

AutoReP advances the same objective with a different mechanism. Rather than simply removing ReLUs or replacing them uniformly, it performs feature-map-level selection between ReLU and polynomial functions under a user-specified ReLU budget $n$ 4: $n$ 5 with a budget-aware objective

$n$ 6

Its distribution-aware polynomial approximation minimizes expected squared approximation error under the actual feature distribution, including a closed-form second-order solution under a Gaussian assumption. AutoReP uses a softplus-based straight-through estimator and a hysteresis rule with a reported good working value around $n$ 7. The paper states that ReLU can dominate latency by up to $n$ 8 relative to convolution + batch normalization, reports $n$ 9 accuracy on CIFAR-100 at a $l$ 0 ReLU budget, $l$ 1 on Tiny-ImageNet at $l$ 2, and $l$ 3 accuracy on EfficientNet-B2 on ImageNet with $l$ 4 times ReLU budget reduction (Peng et al., 2023).

Against this background, the coordinate-descent ReLUDown paper can be read as a direct response to smooth-relaxation approaches. Its core claim is not that ReLU replacement is ineffective, but that solving the discrete binary-mask problem directly avoids the performance loss introduced by a final hard-thresholding step. This places ReLUDown, AutoReP, DeepReDuce, and ReLU sharing in the same design space—reducing expensive nonlinear operations in PI—but with distinct optimization primitives: grouping, heuristic ranking, differentiable mask learning with polynomial surrogates, and discrete coordinate descent.

6. Conceptual distinctions, limitations, and recurring misconceptions

A recurrent misconception is that every method called ReLUDown reduces the number of ReLUs. The continual-learning ReLUDown does not formulate or solve a ReLU-budget problem; it replaces standard ReLU with a static alternative to prevent dormancy and preserve feature sensitivity (Künzel et al., 14 Jul 2025). By contrast, the private-inference versions explicitly target the number of nonlinear secure operations, either by selecting which ReLUs to keep, by replacing some ReLUs with identity or polynomial surrogates, or by sharing one ReLU decision across multiple activations (Rakhlin et al., 14 Nov 2025, Peng et al., 2023, Helbitz et al., 2021).

Another misconception concerns “network linearization.” In the coordinate-descent formulation, linearization means replacing some ReLUs with identity functions so that the network becomes more linear and cheaper for PI; it is not the claim that the entire model is reduced to a linear map (Rakhlin et al., 14 Nov 2025). Similarly, AutoReP’s use of quadratic surrogates is not mere ReLU removal: the paper argues that first-order replacement reduces cost but can damage accuracy because it removes nonlinearity, whereas second-order polynomials provide a better compromise (Peng et al., 2023).

The private-inference papers also make setting-specific assumptions. AutoReP is built for 2PC-style private inference with semi-honest, non-colluding servers, relies on the CrypTen framework, assumes pretrained CNNs, and uses batch-norm statistics or similar estimates of feature distributions for DaPa, so distribution shift can affect approximation quality (Peng et al., 2023). ReLU sharing is evaluated in a simplified environment in which all three parties run on the same machine and no network latency is included, which limits direct interpretation of absolute runtime numbers for distributed deployments (Helbitz et al., 2021). DeepReDuce and the coordinate-descent ReLUDown inherit the same PI-specific cost model in which ReLU count, rather than FLOPs or parameter count, is the dominant practical variable (Jha et al., 2021, Rakhlin et al., 14 Nov 2025).

Taken together, the literature supports a technically precise but narrow conclusion: ReLUDown is best understood not as a single operator but as a cluster of methods that intervene on standard ReLU for different reasons. In continual learning, the objective is to avoid dormant neurons and retain plasticity. In private inference, the objective is to reduce the number or cost of expensive nonlinear cryptographic operations. The shared premise is that standard ReLU can become the limiting factor, but the operative meaning of “down” depends entirely on the surrounding optimization problem.