--- title: Leaky Exponential Linear Unit url: https://www.emergentmind.com/topics/leaky-exponential-linear-unit type: topic --- # Leaky Exponential Linear Unit Searching arXiv for the cited activation-function papers and variants. Leaky Exponential Linear Unit denotes a family of ELU-derived or ELU-related activation designs in which the negative branch retains a nonzero slope or non-vanishing gradient while preserving, modifying, or replacing the exponential negative-side behavior of the Exponential Linear Unit. The term is not used in a single canonical sense across the cited literature. In the original ELU paper, no “leaky ELU” variant is defined; ELU is explicitly contrasted with leaky linear activations such as Leaky ReLU and Parametric ReLU [1511.07289]. Later work instantiates the idea in several distinct ways: as a parametric exponential family that approximates leaky linear behavior near the origin while still saturating for large negative inputs [1606.00305], as a smooth non-saturating activation explicitly named the Leaky Exponential Linear Unit or LELU [2507.06765], and as an ELU-derived integrated activation with a non-vanishing negative-tail gradient, described as realizing “leaky-like” ELU behavior rather than as a separate named Leaky ELU [2411.13010]. ## 1. Terminology and scope In the ELU formulation introduced by Clevert, Unterthiner, and Hochreiter, the activation under discussion is the Exponential Linear Unit, not a Leaky Exponential Linear Unit. The paper states that “Leaky ReLU (LReLU)” and “Parametric ReLU (PReLU)” use a linear negative branch with a nonzero slope, whereas ELU uses an exponential negative branch that saturates to a fixed negative value [1511.07289]. The same source therefore treats “leaky” and “ELU” as contrasting design choices on the negative half-axis rather than as a combined named variant. Subsequent literature broadens the design space. The MPELU paper presents “Multiple Parametric Exponential Linear Units” as a generalized form that can recover ELU and approximate PReLU or Leaky ReLU behavior via parameter settings on the negative branch [1606.00305]. The 2025 regression paper introduces a smooth “Leaky Exponential Linear Unit” as a named activation, abbreviated LELU, with a trainable leakage parameter on the negative side [2507.06765]. The xIELU paper does not introduce a named Leaky ELU, but explicitly characterizes its negative branch as providing the leaky-like property of a nonzero gradient in the negative tail, in contrast to ELU’s vanishing negative-tail gradient [2411.13010]. This suggests that “Leaky Exponential Linear Unit” is best understood as a descriptive umbrella for several activation constructions rather than as one universally fixed formula. ## 2. ELU and the original contrast with leaky linear activations The canonical ELU with $\alpha>0$ is defined by $$ f(x)= \begin{cases} x, & x>0 \\ \alpha(\exp(x)-1), & x\le 0 \end{cases} $$ with derivative $$ f'(x)= \begin{cases} 1, & x>0 \\ \alpha e^x, & x\le 0. \end{cases} $$ On the negative branch, ELU satisfies the identity $f'(x)=f(x)+\alpha$ for $x\le 0$, and its negative side saturates as $\lim_{x\to-\infty}f(x)=-\alpha$ [1511.07289]. The ELU paper emphasizes three linked properties. First, like ReLU, LReLU, and PReLU, ELU alleviates vanishing gradients on the positive side because $f(x)=x$ for $x>0$. Second, unlike ReLU, ELU produces negative outputs, which push mean activations toward zero. The paper argues that this reduces the bias shift effect and makes the normal gradient closer to the unit natural gradient, thereby speeding learning [1511.07289]. Third, unlike LReLU and PReLU, ELU’s negative branch saturates. The authors identify this saturation as a “noise-robust deactivation state”: sufficiently negative inputs produce little change in output and gradient, thereby decreasing forward propagated variation and information in the deactivated regime [1511.07289]. The contrast with leaky activations is therefore structural. LReLU is $$ f(x)= \begin{cases} x, & x>0 \\ ax, & x\le 0 \end{cases} $$ with $00$ [1606.00305]. Here $\alpha$ controls the negative saturation value, since $f(y)\to-\alpha$ as $y\to-\infty$, while $\beta$ controls the curvature of the negative branch. The derivatives used for backpropagation are $$ \frac{\partial f}{\partial y}= \begin{cases} 1, & y>0 \\ \beta\alpha e^{\beta y}, & y\le 0, \end{cases} \qquad \frac{\partial f}{\partial \alpha}= \begin{cases} 0, & y>0 \\ e^{\beta y}-1, & y\le 0, \end{cases} $$ and $$ \frac{\partial f}{\partial \beta}= \begin{cases} 0, & y>0 \\ y\alpha e^{\beta y}, & y\le 0. \end{cases} $$ The function is continuous at $y=0$, and differentiable there only if $\alpha\beta=1$ [1606.00305]. Its relevance to the leaky-ELU idea arises from the first-order expansion for small negative inputs: $$ e^{\beta x}\approx 1+\beta x, \qquad f(x)\approx \alpha\beta x. $$ Near the origin, the negative branch therefore behaves like a leaky linear unit with effective slope $a_{\text{leak}}=\alpha\beta$ [1606.00305]. The paper gives the specific example $\alpha=25.6302$ and $\beta=0.01$, yielding behavior like PReLU with slope $0.2563$ near the origin, while still saturating exponentially to $-\alpha$ for large negative inputs [1606.00305]. In that operational sense, MPELU realizes a “leaky ELU”: it combines a controllable nonzero local negative slope with ELU-style negative saturation. The same paper derives an initialization rule for exponential-type activations by a semi-linear approximation of the negative branch. Under the stated assumptions, stable variance propagation is obtained by enforcing $$ \frac{1}{2} k_i^2 c_i \left(1+\alpha_i^2\beta_i^2\right)\mathrm{Var}(w_i)=1, $$ leading to Gaussian initialization $$ w_i\sim\mathcal{N}\!\left(0,\sigma_i^2\right), \qquad \sigma_i=\sqrt{\frac{2}{k_i^2 c_i (1+\alpha_i^2\beta_i^2)}}. $$ The rule reduces to He/Kaiming initialization for ReLU when $\alpha=0$, and to $\sigma=\sqrt{1/(k^2 c)}$ for ELU when $\alpha=1$ and $\beta=1$ [1606.00305]. Empirically, the paper reports that this activation-initialization pair improves both convergence and accuracy. On CIFAR-10 with a NIN architecture and augmentation, the mean errors were PReLU $7.49\pm0.14$, ELU $7.83\pm0.05$, and MPELU $\alpha=1,\beta=1$ at $7.57\pm0.16$; without augmentation, MPELU $\alpha=1,\beta=1$ achieved $9.19\pm0.11$ and MPELU $\alpha=0.25,\beta=1$ achieved $9.27\pm0.12$ [1606.00305]. In the same NIN setup, the convergence comparison to 15% test error was approximately 9k iterations for MPELU, 15k for PReLU, and 25k for ReLU [1606.00305]. For a 30-layer ELU/MPELU network without batch normalization, Gaussian initialization failed to converge, whereas the proposed initialization yielded 37.08% for ELU and 36.49% for MPELU on ImageNet [1606.00305]. ## 4. LELU as an explicit named Leaky Exponential Linear Unit The 2025 paper introduces an activation explicitly called the Leaky Exponential Linear Unit, or LELU, for multidimensional nonlinear regression [2507.06765]. It is defined piecewise by $$ \mathrm{LELU}(x;\beta)= \begin{cases} x, & x>0 \\ \exp\!\big((1-\beta)x\big)-1+\beta x, & x\le 0, \end{cases} $$ with $0\le \beta \le 1$ [2507.06765]. This construction differs fundamentally from standard ELU. For $\beta=0$, the negative branch becomes $\exp(x)-1$, so LELU recovers ELU with $\alpha=1$ on the negative side. As $\beta\to 1$, the activation becomes globally linear, because $\exp((1-\beta)x)\approx 1+(1-\beta)x$ and the negative branch tends to $x$ [2507.06765]. For any $\beta>0$, however, the negative branch is non-saturating: as $x\to-\infty$, $$ \mathrm{LELU}(x;\beta)\sim \beta x - 1, $$ and the derivative tends to $\beta$ rather than to $0$ [2507.06765]. Its first and second derivatives are $$ \frac{d}{dx}\mathrm{LELU}(x;\beta)= \begin{cases} 1, & x>0 \\ (1-\beta)\exp\!\big((1-\beta)x\big)+\beta, & x\le 0, \end{cases} $$ and $$ \frac{d^2}{dx^2}\mathrm{LELU}(x;\beta)= \begin{cases} 0, & x>0 \\ (1-\beta)^2\exp\!\big((1-\beta)x\big), & x\le 0. \end{cases} $$ Accordingly, for $x\le 0$ the derivative lies in $(\beta,1]$, so the paper states that LELU has non-vanishing gradients when $\beta>0$, is strictly increasing for $\beta>0$, is convex, and is 1-Lipschitz because its derivative is bounded above by 1 [2507.06765]. The function is $C^1$ for all $\beta\in[0,1]$ because the left and right derivatives at $0$ both equal 1. The paper also defines a flexibility score $$ \eta(\phi)=1-\frac{\min \phi'}{\max \phi'}, $$ for which LELU satisfies $\eta(\mathrm{LELU})=1-\beta$, since $\min \phi'=\beta$ and $\max \phi'=1$ [2507.06765]. The authors report that lower flexibility, meaning larger $\beta$, empirically correlates with reduced overfitting in the regression settings studied. A concise comparison of the principal formulas appearing in the cited literature is useful here. | Variant | Negative branch | Tail behavior for $x\to-\infty$ | |---|---|---| | ELU [1511.07289] | $\alpha(e^x-1)$ | saturates to $-\alpha$ | | MPELU [1606.00305] | $\alpha(e^{\beta x}-1)$ | saturates to $-\alpha$ | | LELU [2507.06765] | $\exp((1-\beta)x)-1+\beta x$ | behaves like $\beta x-1$ | | xIELU [2411.13010] | $\alpha_n(e^x-1)-\alpha_n x+0.5x$ | asymptotically linear | The table makes clear that “leaky” can mean at least two different negative-tail regimes in ELU-related work: saturating exponential families with a locally nonzero leak near the origin, as in MPELU, and genuinely non-saturating negative branches with a non-vanishing tail slope, as in LELU and xIELU. ## 5. xIELU and integrated gradient design The xIELU paper derives an ELU-like activation by specifying a trainable gradient and integrating it [2411.13010]. The gradient is $$ \frac{d}{dx}\text{xIELU}(x)= \begin{cases} 2\alpha_p x+\beta_p, & x>0 \\ \alpha_n(e^x-1)+\beta_n, & x\le 0. \end{cases} $$ With the paper’s main design choices $\beta_p=\beta_n=0.5$, continuity of the gradient at $x=0$ is enforced. Integrating the two branches and choosing constants to ensure continuity gives $$ \text{xIELU}(x)= \begin{cases} \alpha_p x^2+0.5x, & x>0 \\ \alpha_n(e^x-1)-\alpha_n x+0.5x, & x\le 0. \end{cases} $$ The paper constrains $\alpha_p>0$ by a softplus reparameterization and enforces $\alpha_n>\beta_n$ by setting $\alpha_n=\beta_n+\text{softplus}(\theta_n)$, with initialization $\alpha_p=\alpha_n=0.8$ and no weight decay on these parameters [2411.13010]. On the negative side, the gradient range is $$ g_-(x)\in[\beta_n-\alpha_n,\beta_n), $$ so under the paper’s constraint $\alpha_n>\beta_n$, the tail gradient is nonzero and strictly negative as $x\to-\infty$ [2411.13010]. This is the sense in which xIELU realizes a leaky-like ELU behavior: unlike standard ELU, whose negative-side gradient $\alpha e^x$ vanishes in the tail, xIELU has a bounded, non-vanishing negative-tail gradient. Unlike Leaky ReLU, however, that tail slope is not a small positive constant; it is trainable and typically negative under the main parameterization. The paper therefore attributes to xIELU a mixed profile: on the positive side it resembles ReLU$^2$ through a linearly increasing gradient, while on the negative side it replaces ELU’s vanishing-tail behavior with bounded non-vanishing gradient flow [2411.13010]. It is $C^1$ by construction, but generally not $C^2$ at the origin unless $2\alpha_p=\alpha_n$. The reported language-model experiments are at 1.1B scale on 126B tokens of FineWeb Edu using Llama-like transformers. At matched compute and parameter count, xIELU achieved lower perplexity than both ReLU$^2$ and SwiGLU. After 126B tokens, the losses and perplexities were SwiGLU: loss $2.353\pm0.004$, PPL $10.517$; ReLU$^2$: loss $2.337\pm0.004$, PPL $10.352$; xIELU: loss $2.323\pm0.004$, PPL $10.207$ [2411.13010]. In a 4B-token ablation, xIELU with the negative component set to 0 had perplexity $18.399$, whereas the base xIELU had $17.518$, which the paper interprets as evidence that trainable negative gradient flow is beneficial [2411.13010]. ## 6. Empirical roles, practical considerations, and conceptual distinctions Across the cited work, three distinct mechanisms recur under the broad “leaky ELU” label. First, there is the **saturating exponential mechanism** represented by ELU and its MPELU generalization. Here the negative branch tends to a fixed asymptote, reducing forward propagated variation for sufficiently negative inputs and pushing mean activations toward zero [1511.07289]. MPELU adds learnable parameters so that the branch can behave approximately linearly near the origin while retaining negative saturation farther left [1606.00305]. Second, there is the **non-saturating smooth leakage mechanism** represented by LELU. Its negative derivative is bounded below by $\beta>0$, which removes vanishing gradients in the negative regime while preserving $C^1$ smoothness and a global Lipschitz constant of 1 [2507.06765]. In the reported regression settings, this was associated with reduced localized overfitting as measured by the paper’s diffusion-loss metric. Third, there is the **integrated non-vanishing negative-tail mechanism** represented by xIELU. This does not preserve ELU’s negative saturation; instead it enforces a bounded but nonzero tail gradient through integration of a trainable ELU-derived gradient field [2411.13010]. These distinctions matter in practice. The original ELU paper states that ELU networks often benefit less from batch normalization because ELU already pushes activations toward zero mean and reduces variance in the deactivated regime [1511.07289]. By contrast, the MPELU paper reports that MPELU can work with batch normalization and that ELU with batch normalization tended to degrade accuracy, whereas MPELU’s learnable $\alpha,\beta$ alleviated that issue [1606.00305]. The LELU paper reports its experiments without batch normalization and without explicit regularization by default, using HeNormal initialization, Adam, and MAE loss [2507.06765]. The xIELU paper similarly relies on architectural and parameterization choices rather than normalization-layer claims, emphasizing fixed $\beta=0.5$, scalar learned $\alpha_p,\alpha_n$, and numerical stabilization with `expm1` on the negative side [2411.13010]. A frequent misconception is that “Leaky Exponential Linear Unit” names a single standard activation comparable to ReLU or ELU. The literature summarized here does not support that reading. The 2015 ELU paper explicitly states that no such variant is introduced there [1511.07289]. Later papers use the phrase either explicitly for a specific new formula, as in LELU [2507.06765], or descriptively for parameter regimes or design goals, as in MPELU and xIELU [1606.00305; 2411.13010]. A second misconception is that adding a leak to ELU necessarily preserves ELU’s most distinctive property, namely negative saturation. That is true for MPELU, where the branch remains exponential and saturating [1606.00305], but false for LELU and xIELU, whose negative branches are asymptotically linear and therefore non-saturating [2507.06765; 2411.13010]. A plausible implication is that the phrase “leaky ELU” is most precise only when accompanied by the exact formula. ## 7. Position within activation-function research The papers collectively place Leaky Exponential Linear Unit variants at the intersection of three activation-function desiderata: preservation of gradient flow, control of negative-side information propagation, and compatibility with large-scale optimization. In the ELU line, the main objective is to combine ReLU’s identity mapping for positive inputs with negative outputs that reduce mean shift and yield a noise-robust deactivation state [1511.07289]. MPELU extends that line by showing that exponential activations can be parameterized to cover ELU, approximate PReLU-like leakage near the origin, and admit a variance-preserving initialization suitable for very deep networks [1606.00305]. In the regression-oriented LELU line, the emphasis shifts from negative saturation to smoothness plus non-vanishing negative gradients. The paper argues that smooth but vanishing-gradient activations such as ELU or SiLU have limited performance in the studied nonlinear regression tasks, while non-smooth activations such as ReLU and Leaky ReLU impart discontinuity in the trained model. LELU is positioned as a compromise that is smooth, monotone, convex, 1-Lipschitz, and non-vanishing in gradient for $\beta>0$ [2507.06765]. In the xIELU line, activation design is reframed as integration of a chosen gradient field. The resulting activation extends the ELU family toward trainable negative-side gradient flow and a ReLU$^2$-like positive side, with reported improvements in 1.1B-parameter Llama pretraining at matched compute cost and parameter count [2411.13010]. Taken together, these works indicate that the research significance of Leaky Exponential Linear Unit lies less in a single settled formula than in a recurring design problem: how to modify ELU’s negative branch so as to retain some combination of smoothness, gradient flow, saturation control, and optimization stability. The different answers offered by ELU, MPELU, LELU, and xIELU define distinct subfamilies within that broader problem space. Source: https://www.emergentmind.com/topics/leaky-exponential-linear-unit