RReLU: Randomized Negative-Slope Activation

• RReLU is a neural activation function that uses randomized negative slopes during training to improve gradient propagation and add implicit regularization.
• It outperforms deterministic activations like Leaky ReLU and PReLU in reducing overfitting, as demonstrated on benchmarks such as CIFAR and NDSB.
• Practical implementation uses hyperparameters ℓ=0.3 and u=0.8, offering a simple, efficient method to boost generalization in small-data regimes.

Randomized Negative-Slope Activations (RReLU) are a rectified activation function for neural networks, extending the standard Leaky Rectified Linear Unit (Leaky ReLU) by introducing stochasticity in the negative-slope parameter during training. RReLU was introduced in "Empirical Evaluation of Rectified Activations in Convolutional Network" (Xu et al., 2015) (Xu et al., 2015) and is motivated by the dual aims of facilitating gradient flow in all regimes and mitigating overfitting, particularly on small datasets, via implicit regularization.

1. Formal Definition and Theoretical Properties

Let $x_{j,i}$ denote the pre-activation input for the $i$-th channel on the $j$-th example. The RReLU activation is defined during training as:

$$y_{j,i} = \begin{cases} x_{j,i}, & \text{if } x_{j,i} \geq 0 \ a_{j,i} x_{j,i}, & \text{if } x_{j,i} < 0,\quad a_{j,i} \sim \mathcal{U}(\ell, u) \end{cases}$$

where $a_{j,i}$ is sampled independently for each unit and example from the uniform distribution $\mathcal{U}(\ell, u)$ with $0 \leq \ell < u < 1$. During inference, the stochasticity is removed:

$a^{*} = \mathbb{E}[a_{j,i}] = \frac{\ell + u}{2}$

$$y_{j,i} = \begin{cases} x_{j,i}, & x_{j,i} \geq 0 \ a^{*} x_{j,i}, & x_{j,i} < 0 \end{cases}$$

Recommended hyperparameters are $\ell = 0.3$, $i$0, resulting in $i$1 (often rounded to $i$2 for implementation convenience) (Xu et al., 2015).

2. Motivation and Rationale for Randomization

The introduction of randomization in the negative-slope differentiates RReLU from deterministic Leaky ReLU and learnable PReLU. Fixed negative slopes (Leaky ReLU) or optimizable negative slopes (PReLU) can improve training accuracy but tend to overfit, especially with small datasets. Randomizing the negative slope functions as implicit regularization:

• Regularization by Multiplicative Noise: Per-unit, per-example randomness in the activation function forces the network to learn representations robust to families of piecewise linear activations.
• Improved Gradient Flow: Compared to ReLU, where the negative regime is entirely shut off ($i$3), nonzero and random negative slopes ensure nontrivial gradient propagation throughout all units, reducing dead unit phenomena.
• Analogy to Dropout: The effect is analogous to Dropout, in that induced noise disrupts co-adaptation and emphasizes robust features.
• Avoidance of Overfitting: Random negative slopes prevent the model from converging to a potentially poorly-regularizing “optimal” leak, as may occur with PReLU, and instead encourage exploration of activation regimes (Xu et al., 2015).

3. Experimental Evaluation and Results

Datasets

Three primary benchmarks were employed:

• CIFAR-10: 50,000 train / 10,000 test; 10 classes; 32×32 RGB.
• CIFAR-100: 50,000 train / 10,000 test; 100 classes; 32×32 RGB.
• Kaggle NDSB (National Data Science Bowl): 30,336 labeled grayscale images, 121 classes; split into 25,000 train / 5,336 validation; 130,400 private test images.

Architectures

• CIFAR-10/100: Network in Network (Lin et al., 2013): several stacked convolutional blocks ($i$4, $i$5 kernels, dropout, max/avg pooling).
• CIFAR-100: Inception-style subset with BatchNorm and global average pooling.
• NDSB: “AuroraXie” winner architecture with complex convolutional paths, spatial pyramid pooling, and two 1024-unit fully connected layers.

Within each dataset, all hyperparameters and preprocessing were held constant across activation types. No data augmentation or color normalization for CIFAR; standard augmentation for NDSB (Xu et al., 2015).

Comparative Performance

Activation	CIFAR-10 Test Error	CIFAR-100 Test Error	NDSB Val Log-Loss
ReLU	12.45%	42.96%	0.7727
Leaky ReLU ($i$6)	12.66%	42.05%	0.7601
Leaky ReLU ($i$7)	11.20%	40.42%	0.7391
PReLU	11.79%	41.63%	0.7454
RReLU ($i$8, $i$9)	11.19%	40.25%	0.7292

• On CIFAR-10 and CIFAR-100, RReLU matched or slightly outperformed Leaky ReLU and PReLU in test accuracy.
• On the smallest dataset (NDSB), RReLU provided the most marked reduction in validation log-loss, supporting its regularization effect.
• PReLU typically yielded the lowest training loss but displayed more overfitting (train/val performance gap); RReLU reduced overfitting while slightly increasing training error, which is consistent with its role as a noise-based regularizer (Xu et al., 2015).

4. Analysis of Generalization and Regularization Effects

Empirical analyses indicate several core factors for RReLU’s generalization advantage:

• Multiplicative Noise as Regularization: Injecting noise on negative activations (rather than activations or weights as in Dropout) ensures robustness to a spectrum of possible activation regimes, improving ability to generalize beyond the training set.
• Prevention of Slope Saturation: No single learned negative-slope parameter can dominate, contrasting with PReLU, so networks are discouraged from adopting brittle representations.
• Notable Benefits on Small Datasets: The regularization effect of RReLU is most apparent in restricted data regimes (e.g., NDSB), as the benefit of noise outweighs the adverse effect of variance; ablation experiments indicate that very small fixed leaks ($j$0) are functionally equivalent to ReLU, while moderate leaks perform better but still underperform the randomized version (Xu et al., 2015).

5. Practical Implementation Guidelines

Key recommendations for deploying RReLU in convolutional networks:

• Default Hyperparameters: Use $j$1, $j$2, yielding $j$3 for test-time. For simplicity, test-time slope may be rounded to $j$4.
• Backward Pass: During training, treat $j$5 as constant during the backward pass for each forward sample.
• Compatibility: RReLU is fully compatible with standard regularization approaches (Dropout, BatchNorm). On CIFAR-100, an Inception+BatchNorm model incorporating RReLU achieved 75.68% test accuracy (single-model, single-view).
• Avoid Learnable Slopes in Small Data: When aiming to reduce overfitting, prefer RReLU or Leaky ReLU over PReLU on small datasets.
• Compute Considerations: Sampling from $j$6 is computationally inexpensive and vectorizable; test-time complexity matches Leaky ReLU.
• Monitoring Overfitting: Excessive training-accuracy improvements (as with PReLU) coupled with poor validation improvements may signal overfitting; increasing the randomization range (widen $j$7) or deploying RReLU can provide corrective regularization.

6. Comparative Summary and Broader Significance

RReLU is a direct drop-in for ReLU or Leaky ReLU, differing only by randomized negative slopes during training and averaged slope at inference. Across robust experiments with consistent protocols, RReLU consistently improved generalization, especially on small and medium-sized image recognition problems. These findings challenge the assumption that activation sparsity is the main driver of ReLU’s success. Adopting stochastic negative slopes provides a regularization mechanism orthogonal to weight-based dropout, activation dropout, or learnable parameterization, and is particularly effective for small data regimes and non-ensemble models.

RReLU, as analyzed and validated in (Xu et al., 2015), offers a compelling combination of implementation simplicity, negligible computational overhead, and measurable benefits in validation accuracy and loss, making it a practical component of modern convolutional network architectures.