Leaky Exponential Linear Unit (LELU)

• LELU is a family of parametric activation functions that blends ELU's smooth, C¹-continuous characteristics with learnable leakage in the negative regime to mitigate vanishing gradients.
• It employs adaptive parameters (a, b in PELU or β in LELU) to control negative saturation and slope, ensuring improved convergence and bias shift mitigation over fixed nonlinearities.
• Empirical studies demonstrate that LELU/PELU yield faster convergence, lower error rates, and enhanced robustness in deep architectures and nonlinear regression tasks compared to ReLU, ELU, and PReLU.

The Leaky Exponential Linear Unit (LELU) refers to a family of parametric activation functions designed to combine the smoothness and bias-shift mitigation of the Exponential Linear Unit (ELU) with learnable leakage in the negative regime, thereby addressing the vanishing-gradient and saturation limitations of prior nonlinearities. There are two principal lines of LELU research reflected in the literature: (1) the "Parametric ELU" (PELU), widely tested in convolutional vision benchmarks (Trottier et al., 2016), and (2) the "LELU" as a regression-oriented, smooth, C¹-continuous variant with a tunable nonzero negative gradient (Bigarella, 9 Jul 2025). Both approaches yield superior generalization and faster convergence compared to fixed ELU, Leaky ReLU, or PReLU, especially in deep architectures and highly nonlinear regression settings.

1. Mathematical Definitions

PELU (as LELU in (Trottier et al., 2016)): For a preactivation $h\in\mathbb{R}$ and positive parameters $a,\,b>0$, the function is defined as

$f(h) = \begin{cases} \tfrac{a}{b}\,h, & h \geq 0 \[1em] a \left( \exp\left( \frac{h}{b} \right) - 1 \right), & h < 0 \end{cases}$

• $a$: controls the negative saturation floor.
• $b$: tunes the exponential decay rate and positive-side slope.

LELU as in (Bigarella, 9 Jul 2025): Given a preactivation $x\in\mathbb{R}$ and leak parameter $\beta\in\mathbb{R}$,

$\mathrm{LELU}(x;\,\beta) = \begin{cases} x, & x > 0 \[6pt] \exp\!\left( (1-\beta)x \right) - 1 + \beta x, & x \leq 0 \end{cases}$

• When $\beta=0$: recovers ELU with $\alpha=1$.
• As $a,\,b>0$0: function becomes identity on $a,\,b>0$1, with no negative branch curvature.

Both variants ensure $a,\,b>0$2 smoothness (continuity of value and first derivative at $a,\,b>0$3).

2. Parameter Roles, Smoothness, and Comparison

Parametrization enables per-layer adaptation:

• PELU (Trottier et al., 2016):
  • Negative saturation floor set by $a,\,b>0$4; more negative $a,\,b>0$5 increases the activation’s range below zero.
  • The parameter $a,\,b>0$6 tightens or loosens exponential approach to saturation; also controls slope for $a,\,b>0$7 through $a,\,b>0$8.
  • Tied positive slope ensures $a,\,b>0$9 continuity at $f(h) = \begin{cases} \tfrac{a}{b}\,h, & h \geq 0 \[1em] a \left( \exp\left( \frac{h}{b} \right) - 1 \right), & h < 0 \end{cases}$0.
• LELU (Bigarella, 9 Jul 2025):
  • Leakiness parameter $f(h) = \begin{cases} \tfrac{a}{b}\,h, & h \geq 0 \[1em] a \left( \exp\left( \frac{h}{b} \right) - 1 \right), & h < 0 \end{cases}$1 sets the minimal negative-side slope; as $f(h) = \begin{cases} \tfrac{a}{b}\,h, & h \geq 0 \[1em] a \left( \exp\left( \frac{h}{b} \right) - 1 \right), & h < 0 \end{cases}$2, left-branch derivative approaches $f(h) = \begin{cases} \tfrac{a}{b}\,h, & h \geq 0 \[1em] a \left( \exp\left( \frac{h}{b} \right) - 1 \right), & h < 0 \end{cases}$3 rather than zero.
  • The flexibility score $f(h) = \begin{cases} \tfrac{a}{b}\,h, & h \geq 0 \[1em] a \left( \exp\left( \frac{h}{b} \right) - 1 \right), & h < 0 \end{cases}$4 quantitatively captures the deviation of the activation derivative across its domain; smaller $f(h) = \begin{cases} \tfrac{a}{b}\,h, & h \geq 0 \[1em] a \left( \exp\left( \frac{h}{b} \right) - 1 \right), & h < 0 \end{cases}$5 yields higher flexibility but may overfit.
  • $f(h) = \begin{cases} \tfrac{a}{b}\,h, & h \geq 0 \[1em] a \left( \exp\left( \frac{h}{b} \right) - 1 \right), & h < 0 \end{cases}$6 smooth at $f(h) = \begin{cases} \tfrac{a}{b}\,h, & h \geq 0 \[1em] a \left( \exp\left( \frac{h}{b} \right) - 1 \right), & h < 0 \end{cases}$7; derivative equals $f(h) = \begin{cases} \tfrac{a}{b}\,h, & h \geq 0 \[1em] a \left( \exp\left( \frac{h}{b} \right) - 1 \right), & h < 0 \end{cases}$8 from both sides.

Comparison with other nonlinearities:

• ReLU, Leaky ReLU, PReLU: only $f(h) = \begin{cases} \tfrac{a}{b}\,h, & h \geq 0 \[1em] a \left( \exp\left( \frac{h}{b} \right) - 1 \right), & h < 0 \end{cases}$9; possible dead neurons due to zero gradient for $a$0; Leaky ReLU adds constant leakage but non-saturating negative branch.
• ELU: $a$1 but negative-branch gradient vanishes for large $a$2; no learnable adaptation per layer.
• LELU/PELU: combine ELU’s smoothness and finite negative saturation with tunable, nonzero negative gradients via parameterization (Trottier et al., 2016, Bigarella, 9 Jul 2025).

3. Training and Implementation Details

Parameter update regime:

• PELU:
  • $a$3 optimized via standard backpropagation; gradients provided for both parameters across both branches.
  • Initialize $a$4 (match ELU).
  • Constrain $a$5 (via clipping) to preserve monotonicity and continuity.
  • SGD, Adam, RMSProp supported; optional weight decay.
• LELU (Bigarella, 9 Jul 2025):
  • $a$6 is trainable; typically initialized in $a$7 (default $a$8).
  • Gradient-based optimization with precise formula for $a$9.
  • No specific regularization necessary, but optional clipping $b$0 or weak L2 possible.
  • Forward pass can use conditional computation; $b$1 may be global or per-layer.

Practical code (LELU):

$\beta\in\mathbb{R}$2

4. Empirical Performance and Benchmark Results

Dataset/Task	Activation	Best/Test Error/MAE	Relative/Qualitative Result
MNIST autoencoder (Trottier et al., 2016)	PELU	MSE ≈ $b$2	Lower and faster converged than ELU
	ELU	MSE ≈ $b$3
	ReLU+BN	MSE ≈ $b$4
CIFAR-10/ResNet-110 (Trottier et al., 2016)	PELU	5.36% (best)	10.5% rel. gain over ELU
	ELU	5.99% (best)
	BN–ReLU	5.41% (best)
CIFAR-100/ResNet-110 (Trottier et al., 2016)	PELU	24.55% (best)	5.9% rel. gain over ELU
	ELU	26.59% (best)
ImageNet 2012/NiN/All-CNN/Overfeat	PELU	up to –7.3% rel. top-1 error (NiN)	Only +24 params, consistent 3–5% gain
1D/3D regression (Bigarella, 9 Jul 2025)	LELU, $b$5	Lowest diffusion loss, train MAE	Most robust to overfitting
	ELU/SiLU	Higher diffusion loss/MAE	More sensitive to model size
	Leaky ReLU	High diffusion loss	Poor smoothing, prone to overfit

On large-scale convolutional models (NiN, Overfeat, All-CNN, ResNet), replacing ELU or ReLU by PELU consistently reduced test errors with negligible parameter overhead: only two scalars per layer (e.g., +24 for all NiN layers gives –7.3% relative error improvement) (Trottier et al., 2016). In nonlinear regression, LELU was less sensitive to overfitting as model capacity increased and consistently returned the lowest mean absolute and diffusion losses (Bigarella, 9 Jul 2025).

5. Theoretical Characterization and Motivation

The LELU/PELU architecture is motivated by several properties:

• Bias shift mitigation: Per-layer parameterization lets the network fine-tune the balance of mean activations, reducing hidden-layer bias shift for improved learning dynamics (Trottier et al., 2016).
• Avoidance of vanishing gradients and "dead" units: Negative-side leakage in both LELU and PELU ensures gradients do not vanish for large negative activations, preventing the stalling of learning common with ReLU/ELU (Bigarella, 9 Jul 2025).
• Smoothness (C¹ continuity): Ensures that both the value and gradient flow are continuous at the regime boundary ($b$6), minimizing artificial kinks or sharp changes in the learned mapping—a crucial property in high-precision regression (Bigarella, 9 Jul 2025).
• Controlled flexibility: The flexibility metric $b$7 in (Bigarella, 9 Jul 2025) quantifies the trade-off: high $b$8 (low $b$9) permits more nonlinearity but risks overfitting, while higher $x\in\mathbb{R}$0 (less flexible) encourages smoothness and implicit regularization.

6. Practical Guidelines for Use

• Integration:
  • Replace ReLU/ELU activations by LELU/PELU, initializing all parameters at their canonical values ($x\in\mathbb{R}$1 or $x\in\mathbb{R}$2) (Trottier et al., 2016, Bigarella, 9 Jul 2025).
  • BatchNorm: Do not place BatchNorm immediately before the parametric exponential activation; doing so degrades generalization for ELU/PELU (e.g., CIFAR-10: ELU test error 5.99%→10.39%; PELU: 5.36%→5.85%) (Trottier et al., 2016). In pipelines with pre-activation BN→ReLU, remove the intermediate BN when switching to PELU.
  • Enforce parameter constraints during training: $x\in\mathbb{R}$3 for PELU; $x\in\mathbb{R}$4 via clipping or sigmoid parameterization for LELU (Bigarella, 9 Jul 2025).
• Initialization and optimization:
  • HeNormal (for ReLU-like nets), batch sizes 32–64, starting learning rate $x\in\mathbb{R}$5 annealed downward are recommended defaults (Bigarella, 9 Jul 2025).
  • Standard updates by SGD or Adam suffice. Parameters $x\in\mathbb{R}$6, $x\in\mathbb{R}$7, and/or $x\in\mathbb{R}$8 require no special regularization but may benefit from weak weight decay if overfitting is detected.
• Monitoring:
  • Diffusion-loss metric (see below) is recommended in regression to assess overfitting and spurious oscillations (Bigarella, 9 Jul 2025).

7. Novel Metrics and Regression-Specific Considerations

(Bigarella, 9 Jul 2025) introduces the diffusion-loss metric to quantify suppression of spurious oscillations between training nodes:

• 1D case: Measures difference between true output diffusion at sample sites and predicted mid-point diffusion using finite-difference stencils.
• Metric formulas:

$x\in\mathbb{R}$9

$\beta\in\mathbb{R}$0

The mean squared error between predicted and true diffusion, $\beta\in\mathbb{R}$1, provides a sensitive test for overfitting in highly nonlinear regression.

• Application:
  • LELU demonstrates minimal diffusion loss and robust generalization under varying network depths and widths, outperforming ELU, SiLU, and Leaky ReLU on both one- and multi-dimensional regression tasks (Bigarella, 9 Jul 2025).

• "Parametric Exponential Linear Unit for Deep Convolutional Neural Networks" (Trottier et al., 2016)
• "Robust Deep Network Learning of Nonlinear Regression Tasks by Parametric Leaky Exponential Linear Units (LELUs) and a Diffusion Metric" (Bigarella, 9 Jul 2025)