Leaky Exponential Linear Unit

• Leaky Exponential Linear Unit is a family of ELU-derived activation designs featuring nonzero negative slopes and varied saturation properties to improve gradient flow.
• Variants like MPELU, LELU, and xIELU demonstrate diverse implementations that balance leaky linear behavior near the origin with either saturating or non-saturating negative tails.
• Empirical results on CIFAR and ImageNet highlight these activations’ impact on faster convergence, higher accuracy, and reduced overfitting in deep learning models.

Searching arXiv for the cited activation-function papers and variants. Leaky Exponential Linear Unit denotes a family of ELU-derived or ELU-related activation designs in which the negative branch retains a nonzero slope or non-vanishing gradient while preserving, modifying, or replacing the exponential negative-side behavior of the Exponential Linear Unit. The term is not used in a single canonical sense across the cited literature. In the original ELU paper, no “leaky ELU” variant is defined; ELU is explicitly contrasted with leaky linear activations such as Leaky ReLU and Parametric ReLU (Clevert et al., 2015). Later work instantiates the idea in several distinct ways: as a parametric exponential family that approximates leaky linear behavior near the origin while still saturating for large negative inputs (Li et al., 2016), as a smooth non-saturating activation explicitly named the Leaky Exponential Linear Unit or LELU (Bigarella, 9 Jul 2025), and as an ELU-derived integrated activation with a non-vanishing negative-tail gradient, described as realizing “leaky-like” ELU behavior rather than as a separate named Leaky ELU (Huang et al., 2024).

1. Terminology and scope

In the ELU formulation introduced by Clevert, Unterthiner, and Hochreiter, the activation under discussion is the Exponential Linear Unit, not a Leaky Exponential Linear Unit. The paper states that “Leaky ReLU (LReLU)” and “Parametric ReLU (PReLU)” use a linear negative branch with a nonzero slope, whereas ELU uses an exponential negative branch that saturates to a fixed negative value (Clevert et al., 2015). The same source therefore treats “leaky” and “ELU” as contrasting design choices on the negative half-axis rather than as a combined named variant.

Subsequent literature broadens the design space. The MPELU paper presents “Multiple Parametric Exponential Linear Units” as a generalized form that can recover ELU and approximate PReLU or Leaky ReLU behavior via parameter settings on the negative branch (Li et al., 2016). The 2025 regression paper introduces a smooth “Leaky Exponential Linear Unit” as a named activation, abbreviated LELU, with a trainable leakage parameter on the negative side (Bigarella, 9 Jul 2025). The xIELU paper does not introduce a named Leaky ELU, but explicitly characterizes its negative branch as providing the leaky-like property of a nonzero gradient in the negative tail, in contrast to ELU’s vanishing negative-tail gradient (Huang et al., 2024).

This suggests that “Leaky Exponential Linear Unit” is best understood as a descriptive umbrella for several activation constructions rather than as one universally fixed formula.

2. ELU and the original contrast with leaky linear activations

The canonical ELU with $\alpha>0$ is defined by

$$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$

with derivative

$$f'(x)= \begin{cases} 1, & x>0 \ \alpha e^x, & x\le 0. \end{cases}$$

On the negative branch, ELU satisfies the identity $f'(x)=f(x)+\alpha$ for $x\le 0$, and its negative side saturates as $\lim_{x\to-\infty}f(x)=-\alpha$ (Clevert et al., 2015).

The ELU paper emphasizes three linked properties. First, like ReLU, LReLU, and PReLU, ELU alleviates vanishing gradients on the positive side because $f(x)=x$ for $x>0$. Second, unlike ReLU, ELU produces negative outputs, which push mean activations toward zero. The paper argues that this reduces the bias shift effect and makes the normal gradient closer to the unit natural gradient, thereby speeding learning (Clevert et al., 2015). Third, unlike LReLU and PReLU, ELU’s negative branch saturates. The authors identify this saturation as a “noise-robust deactivation state”: sufficiently negative inputs produce little change in output and gradient, thereby decreasing forward propagated variation and information in the deactivated regime (Clevert et al., 2015).

The contrast with leaky activations is therefore structural. LReLU is

$$f(x)= \begin{cases} x, & x>0 \ ax, & x\le 0 \end{cases}$$

with $0Clevert et al., 2015). Because those activations do not saturate on the negative side, large negative inputs continue to propagate linearly varying signals. In the ELU paper’s interpretation, they therefore do not ensure a noise-robust deactivation state.

The reported empirical results in the same paper connect these design choices to optimization and generalization. On CIFAR-100 in an 11-layer CNN, the mean test errors were ELU 28.75% $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$0, SReLU 29.35% $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$1, LReLU 30.59% $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$2, and ReLU 31.56% $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$3, with ELU achieving the lowest training loss and test error and the differences reported as statistically significant by a Wilcoxon signed-rank test with $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$4 (Clevert et al., 2015). On ImageNet, a 15-layer ELU CNN reached 20% top-5 validation error about 40k iterations sooner than the corresponding ReLU network, namely 160k versus 200k iterations, while incurring only slightly higher per-iteration wall-clock time, approximately 5% (Clevert et al., 2015).

3. Parametric exponential units as operational “leaky ELU” constructions

The MPELU paper makes the most explicit connection between ELU-like exponentials and leaky linear behavior. It defines the activation, for input $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$5, as

$$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$6

with learnable parameters $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$7 and $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$8 and $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$9 (Li et al., 2016). Here $$f'(x)= \begin{cases} 1, & x>0 \ \alpha e^x, & x\le 0. \end{cases}$$0 controls the negative saturation value, since $$f'(x)= \begin{cases} 1, & x>0 \ \alpha e^x, & x\le 0. \end{cases}$$1 as $$f'(x)= \begin{cases} 1, & x>0 \ \alpha e^x, & x\le 0. \end{cases}$$2, while $$f'(x)= \begin{cases} 1, & x>0 \ \alpha e^x, & x\le 0. \end{cases}$$3 controls the curvature of the negative branch.

The derivatives used for backpropagation are

$$f'(x)= \begin{cases} 1, & x>0 \ \alpha e^x, & x\le 0. \end{cases}$$4

and

$$f'(x)= \begin{cases} 1, & x>0 \ \alpha e^x, & x\le 0. \end{cases}$$5

The function is continuous at $$f'(x)= \begin{cases} 1, & x>0 \ \alpha e^x, & x\le 0. \end{cases}$$6, and differentiable there only if $$f'(x)= \begin{cases} 1, & x>0 \ \alpha e^x, & x\le 0. \end{cases}$$7 (Li et al., 2016).

Its relevance to the leaky-ELU idea arises from the first-order expansion for small negative inputs:

$$f'(x)= \begin{cases} 1, & x>0 \ \alpha e^x, & x\le 0. \end{cases}$$8

Near the origin, the negative branch therefore behaves like a leaky linear unit with effective slope $$f'(x)= \begin{cases} 1, & x>0 \ \alpha e^x, & x\le 0. \end{cases}$$9 (Li et al., 2016). The paper gives the specific example $f'(x)=f(x)+\alpha$0 and $f'(x)=f(x)+\alpha$1, yielding behavior like PReLU with slope $f'(x)=f(x)+\alpha$2 near the origin, while still saturating exponentially to $f'(x)=f(x)+\alpha$3 for large negative inputs (Li et al., 2016). In that operational sense, MPELU realizes a “leaky ELU”: it combines a controllable nonzero local negative slope with ELU-style negative saturation.

The same paper derives an initialization rule for exponential-type activations by a semi-linear approximation of the negative branch. Under the stated assumptions, stable variance propagation is obtained by enforcing

$f'(x)=f(x)+\alpha$4

leading to Gaussian initialization

$f'(x)=f(x)+\alpha$5

The rule reduces to He/Kaiming initialization for ReLU when $f'(x)=f(x)+\alpha$6, and to $f'(x)=f(x)+\alpha$7 for ELU when $f'(x)=f(x)+\alpha$8 and $f'(x)=f(x)+\alpha$9 (Li et al., 2016).

Empirically, the paper reports that this activation-initialization pair improves both convergence and accuracy. On CIFAR-10 with a NIN architecture and augmentation, the mean errors were PReLU $x\le 0$0, ELU $x\le 0$1, and MPELU $x\le 0$2 at $x\le 0$3; without augmentation, MPELU $x\le 0$4 achieved $x\le 0$5 and MPELU $x\le 0$6 achieved $x\le 0$7 (Li et al., 2016). In the same NIN setup, the convergence comparison to 15% test error was approximately 9k iterations for MPELU, 15k for PReLU, and 25k for ReLU (Li et al., 2016). For a 30-layer ELU/MPELU network without batch normalization, Gaussian initialization failed to converge, whereas the proposed initialization yielded 37.08% for ELU and 36.49% for MPELU on ImageNet (Li et al., 2016).

4. LELU as an explicit named Leaky Exponential Linear Unit

The 2025 paper introduces an activation explicitly called the Leaky Exponential Linear Unit, or LELU, for multidimensional nonlinear regression (Bigarella, 9 Jul 2025). It is defined piecewise by

$x\le 0$8

with $x\le 0$9 (Bigarella, 9 Jul 2025).

This construction differs fundamentally from standard ELU. For $\lim_{x\to-\infty}f(x)=-\alpha$0, the negative branch becomes $\lim_{x\to-\infty}f(x)=-\alpha$1, so LELU recovers ELU with $\lim_{x\to-\infty}f(x)=-\alpha$2 on the negative side. As $\lim_{x\to-\infty}f(x)=-\alpha$3, the activation becomes globally linear, because $\lim_{x\to-\infty}f(x)=-\alpha$4 and the negative branch tends to $\lim_{x\to-\infty}f(x)=-\alpha$5 (Bigarella, 9 Jul 2025). For any $\lim_{x\to-\infty}f(x)=-\alpha$6, however, the negative branch is non-saturating: as $\lim_{x\to-\infty}f(x)=-\alpha$7,

$\lim_{x\to-\infty}f(x)=-\alpha$8

and the derivative tends to $\lim_{x\to-\infty}f(x)=-\alpha$9 rather than to $f(x)=x$0 (Bigarella, 9 Jul 2025).

Its first and second derivatives are

$f(x)=x$1

and

$f(x)=x$2

Accordingly, for $f(x)=x$3 the derivative lies in $f(x)=x$4, so the paper states that LELU has non-vanishing gradients when $f(x)=x$5, is strictly increasing for $f(x)=x$6, is convex, and is 1-Lipschitz because its derivative is bounded above by 1 (Bigarella, 9 Jul 2025). The function is $f(x)=x$7 for all $f(x)=x$8 because the left and right derivatives at $f(x)=x$9 both equal 1.

The paper also defines a flexibility score

$x>0$0

for which LELU satisfies $x>0$1, since $x>0$2 and $x>0$3 (Bigarella, 9 Jul 2025). The authors report that lower flexibility, meaning larger $x>0$4, empirically correlates with reduced overfitting in the regression settings studied.

A concise comparison of the principal formulas appearing in the cited literature is useful here.

Variant	Negative branch	Tail behavior for $x>0$5
ELU (Clevert et al., 2015)	$x>0$6	saturates to $x>0$7
MPELU (Li et al., 2016)	$x>0$8	saturates to $x>0$9
LELU (Bigarella, 9 Jul 2025)	$$f(x)= \begin{cases} x, & x>0 \ ax, & x\le 0 \end{cases}$$0	behaves like $$f(x)= \begin{cases} x, & x>0 \ ax, & x\le 0 \end{cases}$$1
xIELU (Huang et al., 2024)	$$f(x)= \begin{cases} x, & x>0 \ ax, & x\le 0 \end{cases}$$2	asymptotically linear

The table makes clear that “leaky” can mean at least two different negative-tail regimes in ELU-related work: saturating exponential families with a locally nonzero leak near the origin, as in MPELU, and genuinely non-saturating negative branches with a non-vanishing tail slope, as in LELU and xIELU.

5. xIELU and integrated gradient design

The xIELU paper derives an ELU-like activation by specifying a trainable gradient and integrating it (Huang et al., 2024). The gradient is

$$f(x)= \begin{cases} x, & x>0 \ ax, & x\le 0 \end{cases}$$3

With the paper’s main design choices $$f(x)= \begin{cases} x, & x>0 \ ax, & x\le 0 \end{cases}$$4, continuity of the gradient at $$f(x)= \begin{cases} x, & x>0 \ ax, & x\le 0 \end{cases}$$5 is enforced. Integrating the two branches and choosing constants to ensure continuity gives

$$f(x)= \begin{cases} x, & x>0 \ ax, & x\le 0 \end{cases}$$6

The paper constrains $$f(x)= \begin{cases} x, & x>0 \ ax, & x\le 0 \end{cases}$$7 by a softplus reparameterization and enforces $$f(x)= \begin{cases} x, & x>0 \ ax, & x\le 0 \end{cases}$$8 by setting $$f(x)= \begin{cases} x, & x>0 \ ax, & x\le 0 \end{cases}$$9, with initialization $0weight decay on these parameters (Huang et al., 2024).

On the negative side, the gradient range is

$0

so under the paper’s constraint $0Huang et al., 2024). This is the sense in which xIELU realizes a leaky-like ELU behavior: unlike standard ELU, whose negative-side gradient $0

The paper therefore attributes to xIELU a mixed profile: on the positive side it resembles ReLU$0Huang et al., 2024). It is $0

The reported language-model experiments are at 1.1B scale on 126B tokens of FineWeb Edu using Llama-like transformers. At matched compute and parameter count, xIELU achieved lower perplexity than both ReLU$0SwiGLU. After 126B tokens, the losses and perplexities were SwiGLU: loss $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$00, PPL $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$01; ReLU$$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$02: loss $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$03, PPL $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$04; xIELU: loss $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$05, PPL $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$06 (Huang et al., 2024). In a 4B-token ablation, xIELU with the negative component set to 0 had perplexity $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$07, whereas the base xIELU had $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$08, which the paper interprets as evidence that trainable negative gradient flow is beneficial (Huang et al., 2024).

6. Empirical roles, practical considerations, and conceptual distinctions

Across the cited work, three distinct mechanisms recur under the broad “leaky ELU” label.

First, there is the saturating exponential mechanism represented by ELU and its MPELU generalization. Here the negative branch tends to a fixed asymptote, reducing forward propagated variation for sufficiently negative inputs and pushing mean activations toward zero (Clevert et al., 2015). MPELU adds learnable parameters so that the branch can behave approximately linearly near the origin while retaining negative saturation farther left (Li et al., 2016).

Second, there is the non-saturating smooth leakage mechanism represented by LELU. Its negative derivative is bounded below by $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$09, which removes vanishing gradients in the negative regime while preserving $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$10 smoothness and a global Lipschitz constant of 1 (Bigarella, 9 Jul 2025). In the reported regression settings, this was associated with reduced localized overfitting as measured by the paper’s diffusion-loss metric.

Third, there is the integrated non-vanishing negative-tail mechanism represented by xIELU. This does not preserve ELU’s negative saturation; instead it enforces a bounded but nonzero tail gradient through integration of a trainable ELU-derived gradient field (Huang et al., 2024).

These distinctions matter in practice. The original ELU paper states that ELU networks often benefit less from batch normalization because ELU already pushes activations toward zero mean and reduces variance in the deactivated regime (Clevert et al., 2015). By contrast, the MPELU paper reports that MPELU can work with batch normalization and that ELU with batch normalization tended to degrade accuracy, whereas MPELU’s learnable $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$11 alleviated that issue (Li et al., 2016). The LELU paper reports its experiments without batch normalization and without explicit regularization by default, using HeNormal initialization, Adam, and MAE loss (Bigarella, 9 Jul 2025). The xIELU paper similarly relies on architectural and parameterization choices rather than normalization-layer claims, emphasizing fixed $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$12, scalar learned $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$13, and numerical stabilization with expm1 on the negative side (Huang et al., 2024).

A frequent misconception is that “Leaky Exponential Linear Unit” names a single standard activation comparable to ReLU or ELU. The literature summarized here does not support that reading. The 2015 ELU paper explicitly states that no such variant is introduced there (Clevert et al., 2015). Later papers use the phrase either explicitly for a specific new formula, as in LELU (Bigarella, 9 Jul 2025), or descriptively for parameter regimes or design goals, as in MPELU and xIELU (Li et al., 2016, Huang et al., 2024).

A second misconception is that adding a leak to ELU necessarily preserves ELU’s most distinctive property, namely negative saturation. That is true for MPELU, where the branch remains exponential and saturating (Li et al., 2016), but false for LELU and xIELU, whose negative branches are asymptotically linear and therefore non-saturating (Bigarella, 9 Jul 2025, Huang et al., 2024). A plausible implication is that the phrase “leaky ELU” is most precise only when accompanied by the exact formula.

7. Position within activation-function research

The papers collectively place Leaky Exponential Linear Unit variants at the intersection of three activation-function desiderata: preservation of gradient flow, control of negative-side information propagation, and compatibility with large-scale optimization.

In the ELU line, the main objective is to combine ReLU’s identity mapping for positive inputs with negative outputs that reduce mean shift and yield a noise-robust deactivation state (Clevert et al., 2015). MPELU extends that line by showing that exponential activations can be parameterized to cover ELU, approximate PReLU-like leakage near the origin, and admit a variance-preserving initialization suitable for very deep networks (Li et al., 2016).

In the regression-oriented LELU line, the emphasis shifts from negative saturation to smoothness plus non-vanishing negative gradients. The paper argues that smooth but vanishing-gradient activations such as ELU or SiLU have limited performance in the studied nonlinear regression tasks, while non-smooth activations such as ReLU and Leaky ReLU impart discontinuity in the trained model. LELU is positioned as a compromise that is smooth, monotone, convex, 1-Lipschitz, and non-vanishing in gradient for $$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$14 (Bigarella, 9 Jul 2025).

In the xIELU line, activation design is reframed as integration of a chosen gradient field. The resulting activation extends the ELU family toward trainable negative-side gradient flow and a ReLU$$f(x)= \begin{cases} x, & x>0 \ \alpha(\exp(x)-1), & x\le 0 \end{cases}$$15-like positive side, with reported improvements in 1.1B-parameter Llama pretraining at matched compute cost and parameter count (Huang et al., 2024).

Taken together, these works indicate that the research significance of Leaky Exponential Linear Unit lies less in a single settled formula than in a recurring design problem: how to modify ELU’s negative branch so as to retain some combination of smoothness, gradient flow, saturation control, and optimization stability. The different answers offered by ELU, MPELU, LELU, and xIELU define distinct subfamilies within that broader problem space.