Continuously Differentiable ELU (CELU)

• CELU is a continuously differentiable activation function that ensures smooth gradients and enhances network optimization.
• It provides bounded derivatives, interpolation between ReLU and identity, and scale-similarity for effective hyperparameter tuning.
• Its theoretical guarantees, including efficient ReLU network approximation without added overhead, offer robust performance in deep architectures.

The Continuously Differentiable Exponential Linear Unit (CELU) is a parametric activation function for neural networks, introduced to improve upon the standard Exponential Linear Unit (ELU) by enforcing continuous differentiability (class $C^1$) with respect to its input for all positive values of its shape parameter $\alpha$. This property eliminates discontinuity in the derivative at the origin present in ELU for $\alpha \neq 1$, facilitating more robust optimization and parameter tuning. CELU retains several advantageous features such as a bounded derivative, the ability to interpolate between $\mathrm{ReLU}$ and the identity function, and scale-similarity with respect to $\alpha$, offering improved stability and interpretability in deep network architectures (Barron, 2017, Zhang et al., 2023).

1. Mathematical Definition and Differentiability

CELU is defined for $x \in \mathbb{R}$ and $\alpha > 0$ as follows: $$\mathrm{CELU}(x,\alpha) = \begin{cases} x, & \text{if } x \geq 0 \ \alpha\left(\exp\left(\frac{x}{\alpha}\right) - 1\right), & \text{if } x < 0 \end{cases}$$

When $\alpha = 1$, CELU coincides with the original ELU. The first derivative with respect to $x$ (denoted $\alpha$0) is

$\alpha$1

Both left and right derivatives at $\alpha$2 equal $\alpha$3, confirming $\alpha$4 continuity on $\alpha$5. Each branch is smooth, and CELU is continuously differentiable for all $\alpha$6 (Barron, 2017, Zhang et al., 2023).

2. Theoretical Properties

CELU exhibits several theoretically significant behaviors:

• Bounded Derivative: For $\alpha$7, $\alpha$8; for $\alpha$9, the derivative is $\alpha \neq 1$0. Thus, $\alpha \neq 1$1 for all $\alpha \neq 1$2, precluding exploding gradients on the negative domain.
• Interpolation of Nonlinearity: As $\alpha \neq 1$3, CELU converges pointwise to $\alpha \neq 1$4, i.e., $\alpha \neq 1$5. As $\alpha \neq 1$6, CELU approaches the identity function for all $\alpha \neq 1$7.
• Scale-Similarity: For any $\alpha \neq 1$8, $\alpha \neq 1$9, an exact property simplifying parameter scaling and weight initialization.
• Special Cases: CELU contains both $\mathrm{ReLU}$0 and the linear transfer function as limiting cases.

These features make CELU flexible for controlling the degree of nonlinearity and ensuring stable gradients.

3. Comparison to ELU and Related Activations

The original ELU is given by: $\mathrm{ReLU}$1 Its derivative at $\mathrm{ReLU}$2 is $\mathrm{ReLU}$3 (for $\mathrm{ReLU}$4), whereas for $\mathrm{ReLU}$5 it is $\mathrm{ReLU}$6, yielding a discontinuity for $\mathrm{ReLU}$7. CELU removes this discontinuity by adopting $\mathrm{ReLU}$8 in the negative branch, ensuring the derivative at the origin matches in both directions.

Unlike standard ELU, CELU's negative-side gradient is always in $\mathrm{ReLU}$9 regardless of $\alpha$0, while ELU's negative slope can become arbitrarily large as $\alpha$1 increases. The continuous first derivative of CELU improves analytical tractability for optimizers sensitive to higher-order smoothness (Barron, 2017, Zhang et al., 2023).

4. Expressive Power and Approximation in Deep Networks

CELU is included in the class of activation functions for which any $\alpha$2 network of width $\alpha$3 and depth $\alpha$4 can be approximated arbitrarily closely (on compact domains) by a CELU-activated network of the same width and depth. For every $\alpha$5 and compact domain $\alpha$6, given a $\alpha$7 network $\alpha$8 (width $\alpha$9, depth $x \in \mathbb{R}$0), there exists a $x \in \mathbb{R}$1 network $x \in \mathbb{R}$2 of identical architecture such that

$x \in \mathbb{R}$3

The constructive proof involves replacing each ReLU unit $x \in \mathbb{R}$4 with $x \in \mathbb{R}$5, where $x \in \mathbb{R}$6 is chosen large enough ($x \in \mathbb{R}$7) to control the approximation error ($x \in \mathbb{R}$8). This direct correspondence enables width-depth scaling factors $x \in \mathbb{R}$9, i.e., no overhead, as opposed to other activation classes which may incur $\alpha > 0$0 overhead. The continuity of CELU's first derivative is instrumental in achieving this result (Zhang et al., 2023).

5. Practical Considerations and Hyperparameter Selection

The parameter $\alpha > 0$1 modulates the activation's shape:

• Small $\alpha > 0$2 $\alpha > 0$3: Activation closely resembles a leaky or standard ReLU, with a sharper hinge at the origin.
• Moderate $\alpha > 0$4 $\alpha > 0$5: Default setting; maintains balanced nonlinearity, zero-mean activations, and avoids large gradients.
• Large $\alpha > 0$6 $\alpha > 0$7: Approximates a linear function, minimizing nonlinearity and the extent of gradient clipping.

In practice, $\alpha > 0$8 can be fixed or learned as a per-layer parameter. Due to bounded negative gradients, large $\alpha > 0$9 values do not introduce gradient instability, contrasting with ELU where large $$\mathrm{CELU}(x,\alpha) = \begin{cases} x, & \text{if } x \geq 0 \ \alpha\left(\exp\left(\frac{x}{\alpha}\right) - 1\right), & \text{if } x < 0 \end{cases}$$0 may cause exploding gradients. Initializing $$\mathrm{CELU}(x,\alpha) = \begin{cases} x, & \text{if } x \geq 0 \ \alpha\left(\exp\left(\frac{x}{\alpha}\right) - 1\right), & \text{if } x < 0 \end{cases}$$1 is standard, and scale-similarity aids in harmonizing $$\mathrm{CELU}(x,\alpha) = \begin{cases} x, & \text{if } x \geq 0 \ \alpha\left(\exp\left(\frac{x}{\alpha}\right) - 1\right), & \text{if } x < 0 \end{cases}$$2 values across layers (Barron, 2017).

6. Empirical and Theoretical Implications

The introduction of CELU targets activation smoothness and gradient stability without empirical disadvantages relative to ELU. CELU inherits ELU's favorable properties (accelerated convergence, improved generalization on benchmarks such as CIFAR-100) and extends them with formal guarantees for smooth optimization landscapes and robust parameter tuning. No new large-scale empirical benchmarks were introduced at inception, but theoretical benefits in stability and ease of analysis are emphasized.

The $$\mathrm{CELU}(x,\alpha) = \begin{cases} x, & \text{if } x \geq 0 \ \alpha\left(\exp\left(\frac{x}{\alpha}\right) - 1\right), & \text{if } x < 0 \end{cases}$$3 continuity is advantageous for employing higher-order optimizers and facilitates convergence proofs. The scale-similarity and interpolation between $$\mathrm{CELU}(x,\alpha) = \begin{cases} x, & \text{if } x \geq 0 \ \alpha\left(\exp\left(\frac{x}{\alpha}\right) - 1\right), & \text{if } x < 0 \end{cases}$$4 and the identity map provide flexibility and interpretability in architectural design and hyperparameterization (Barron, 2017, Zhang et al., 2023).

7. Summary Table: CELU Key Properties

Property	CELU	ELU
$$\mathrm{CELU}(x,\alpha) = \begin{cases} x, & \text{if } x \geq 0 \ \alpha\left(\exp\left(\frac{x}{\alpha}\right) - 1\right), & \text{if } x < 0 \end{cases}$$5 continuity	Yes, for all $$\mathrm{CELU}(x,\alpha) = \begin{cases} x, & \text{if } x \geq 0 \ \alpha\left(\exp\left(\frac{x}{\alpha}\right) - 1\right), & \text{if } x < 0 \end{cases}$$6	Only if $$\mathrm{CELU}(x,\alpha) = \begin{cases} x, & \text{if } x \geq 0 \ \alpha\left(\exp\left(\frac{x}{\alpha}\right) - 1\right), & \text{if } x < 0 \end{cases}$$7
Bounded derivative	$$\mathrm{CELU}(x,\alpha) = \begin{cases} x, & \text{if } x \geq 0 \ \alpha\left(\exp\left(\frac{x}{\alpha}\right) - 1\right), & \text{if } x < 0 \end{cases}$$8	No (unbounded for large $$\mathrm{CELU}(x,\alpha) = \begin{cases} x, & \text{if } x \geq 0 \ \alpha\left(\exp\left(\frac{x}{\alpha}\right) - 1\right), & \text{if } x < 0 \end{cases}$$9)
Special cases	$\alpha = 1$0 $\alpha = 1$1, linear $\alpha = 1$2	$\alpha = 1$3 $\alpha = 1$4
Scale-similarity	Yes: $\alpha = 1$5	No
Depth/width overhead for ReLU approximation	None (scaling $\alpha = 1$6)	Not established

CELU thus provides a flexible, theoretically favored, and robust activation for modern deep learning, with explicit guarantees in function approximation and gradient management (Barron, 2017, Zhang et al., 2023).