---
title: 'α-ReLU: Power Activation in Neural Networks'
url: https://www.emergentmind.com/topics/relu
type: topic
---

# α-ReLU: Power Activation in Neural Networks

The α-ReLU, or power ReLU, refers to a parametric family of activation functions of the form $\varphi_\alpha(x) = [\max\{0, x\}]^\alpha$, where $\alpha > 0$ is the exponent parameter. This class generalizes the standard ReLU ($\alpha=1$) and underpins recent advances in the theoretical analysis and practical design of neural architectures for function approximation, PDE solvers, control barrier functions, and sparse regularized learning. Its properties—homogeneity, smoothness, and approximation-theoretic behavior—are sensitive to the value of $\alpha$, which enables fine-grained control over regularity and functional capacities.

## 1. Formal Definition and Mathematical Properties

The α-ReLU activation is defined as
\[
\varphi_\alpha(x) = [\max\{0, x\}]^\alpha, \quad \alpha > 0.
\]
Key mathematical properties include:

- **Homogeneity**: For any $\lambda \ge 0$,
  \[
  \varphi_\alpha(\lambda x) = \lambda^\alpha \varphi_\alpha(x).
  \]
- **Smoothness**: If $\alpha = k+\gamma$ with $k \in \mathbb{N}_0$, $\gamma \in (0,1)$, then $\varphi_\alpha \in C^{k,\gamma}(\mathbb{R})$ (i.e., $k$-times differentiable, $k$-th derivative $\gamma$-Hölder). For integer $\alpha=k$, $\varphi_k \in C^{k-1,1}(\mathbb{R})$ but not $C^k(\mathbb{R})$.
- **Growth at Infinity**: $\varphi_\alpha(x) = O(x^\alpha)$ as $x \to \infty$, growing sublinearly if $\alpha < 1$.
- **Special Cases**:
  - Standard ReLU: $\alpha = 1$, $\varphi_1(x) = \max\{0, x\}$.
  - Higher-order: integer $\alpha = k > 1$ yields piecewise polynomials.

For variants, such as the two-slope Leaky α-ReLU used in control settings [2603.15286], the function is piecewise linear:
\[
\alpha(s) = 
\begin{cases}
\alpha_{m}\, s, & s \ge 0, \\
\alpha_{1}\, s, & s < 0,
\end{cases}
\]
with $0 < \alpha_1 \leq \alpha_m < \infty$.

## 2. Approximation and Regularity in Shallow α-ReLU Networks

Shallow α-ReLU networks are central to the analysis of PDE solution operators and function approximation in Sobolev/Hölder and Barron-type norms. For the Dirichlet-Laplace (Poisson) problem on half-spaces, solution regularity and approximation rates depend sensitively on $\alpha$ [2412.07728], [2605.18468]:

- **Fractional $\alpha \in (0,1)$**: Network solutions realize fractional Hölder regularity ($C^{0,\alpha}$); the associated Barron norm $B_\alpha$ is compatible with the solution's smoothness.
- **Integer $\alpha = k$**: One obtains $C^{k-1,1}$ regularity, corresponding to one derivative less of Lipschitz continuity (but not $C^k$).
- **Approximation Guarantees**: Given a function $g$ on the boundary with controlled $B_\alpha$ norm, the solution $u$ and its Monte-Carlo approximation $u_N$ in the domain satisfy
  \[
  \|u - u_N\|_{W^{m,q}(\Omega)} \leq C(m, q, \Omega) \|u\|_{B_\alpha} N^{-1/2},
  \]
  under technical conditions on $m, \alpha$, and $q$. For integer $\alpha$, logarithmic penalties in the Barron norm arise due to log-divergences at the boundary [2412.07728].

Approximation of general $f$ in $L^p$ balls or Sobolev spaces with shallow $\alpha$-ReLU networks yields rates that depend polynomially or log-polynomially on the network width $m$, the exponent $\alpha$, and the spatial dimension $d$ [2605.18468].

## 3. Barron, Sobolev, and Spectral Characterizations

The choice of $\alpha$ in α-ReLU directly links the network's functional capacity to analytic regularity scales:

- **Barron Norms**: For $g \in B_\alpha$, defined as the infimum of expected weighted coefficients over representation by α-ReLU ridge superpositions, this norm governs approximation error for PDE boundary data [2412.07728].
- **Sobolev Embedding**: The functional class $B_\alpha(\Omega)$ embeds into $C^{k,\gamma}$ or $W^{k+1,p}$ depending on whether $\alpha = k+\gamma$ or $k$, where $k \in \mathbb{N}_0$ and $\gamma \in (0,1)$.
- **Path-Norm Regularization**: For finite-width networks
  \[
  f_m(x) = \sum_{j=1}^m a_j \sigma_\alpha(w_j \cdot x + b_j),
  \]
  the $\ell^1$ path-norm is
  \[
  \|f_m\|_{P, \alpha} = \sum_{j=1}^m |a_j| ( \|w_j\|_{1} + |b_j| )^\alpha.
  \]
  Minimax-optimal generalization rates are achieved for regression over Barron and Sobolev (fractional) targets, with exponents determined by $\alpha$ [2605.18468].

The critical regularity transition occurs when $\alpha$ crosses an integer: fractional powers yield $C^{k,\gamma}$ for $\gamma \in (0,1)$, while integer exponents only ensure Lipschitz continuity of the $k$-th derivative.

## 4. α-ReLU in Control Barrier Function Synthesis

For control systems with safe set invariance under polytopic input constraints, α-ReLU functions are used as surrogates for extended class-$\mathcal{K}_\infty$ barrier functions [2603.15286]:

- **Two-Slope α-ReLU**: Parameterized by positive slopes $(\alpha_1, \alpha_m)$ on $(-\infty, 0)$ and $[0, \infty)$ respectively, ensuring continuity, piecewise differentiability, radial unboundedness, and strict monotonicity.
- **Convexity in Synthesis**: The two-slope α-ReLU maintains the linearity of control barrier function (CBF) constraints in linear programming synthesis, facilitating tractable certification of safety properties.
- **Conservatism and UIS Construction**: The union of invariant sets (UIS), obtained by max-composing solutions for different slopes, never reduces the certified safe set below that of the optimal linear α; in most cases, it expands it [2603.15286].
  
This surrogate captures the strength of general class-$\mathcal{K}_\infty$ nonlinearities without introducing additional nonconvexity or substantial conservatism in stability certification.

## 5. α-ReLU Variants and Modified Network Architectures

Beyond pointwise power functions, the literature includes sparsifying α–ReLU transforms acting on weights, notably in nonparametric regression [2207.08306]:

- **Sparsifying α**: Defined as
  \[
  \alpha(x) = \begin{cases}
  x+1, & x < -1, \\
  0, & -1 \le x \le 1, \\
  x-1, & x > 1,
  \end{cases}
  \]
  applied entrywise to network weight matrices prior to multiplication and activation, thereby imposing structured sparsity.
- **Statistical Rates**: With $\ell_1$ or $\ell_2^2$-penalized empirical risk minimization, sparsified α-ReLU networks achieve, up to log factors, minimax-optimal $n^{-2\beta/(2\beta+d)}$ prediction rates for $\beta$-Hölder regression under sub-Gaussian noise [2207.08306].

This approach yields scale-invariance of penalty complexity, bypassing the suboptimal covering behavior of conventional penalized ReLU networks.

## 6. Practical Considerations and Trade-offs

Selection of $\alpha$ in ReLU$^\alpha$ activations is a design choice balancing analytic regularity, approximation power, and computational tractability:

- **Regularity Requirements**: Applications like PINNs or strong-form PDE solvers may necessitate $C^k$ regularity, thus motivating integer $\alpha \geq k$.
- **Computational Cost**: For non-integer and especially irrational $\alpha$, the evaluation cost of $x^\alpha$ can be significant.
- **Barron-Norm Growth**: For integer values, an unavoidable logarithmic penalty emerges in the Barron norm near boundaries, impacting the efficiency of representation [2412.07728].
- **Parameter Interpretability**: In two-slope Leaky α-ReLU barrier constructions, tuning $(\alpha_1, \alpha_m)$ modulates the aggressiveness of barrier enforcement for positive and negative violations [2603.15286].

A plausible implication is that the α-ReLU family enables granular matching of network expressivity to analytic and application-driven demands, but judicious tuning is required to balance all competing considerations.

## 7. Comparative Summary

| Variant                | Mathematical Formulation                  | Main Use/Result                                        |
|------------------------|-------------------------------------------|--------------------------------------------------------|
| Standard Power α-ReLU  | $\varphi_\alpha(x) = [\max\{0, x\}]^\alpha$ | PDE solvers, approximation with controlled regularity [2412.07728], [2605.18468] |
| Two-slope Leaky α-ReLU | Piecewise linear with $(\alpha_1, \alpha_m)$ | Barrier certification under control saturation [2603.15286] |
| Sparsifying α-ReLU     | Piecewise constant-linear on weights      | Sparse nonparametric regression at minimax rates [2207.08306] |

## References

- Vaishampayan and Wojtowytsch, "Solving the Poisson Equation with Dirichlet data by shallow ReLU$^\alpha$-networks" [2412.07728].
- Beknazaryan and Sang, "Nonparametric regression with modified ReLU networks" [2207.08306].
- Li, Liu, and Shi, "Shallow ReLU$^s$ Networks in $L^p$-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization" [2605.18468].
- ReLU Barrier Functions (multiple authors), "ReLU Barrier Functions for Nonlinear Systems with Constrained Control: A Union of Invariant Sets Approach" [2603.15286].

Source: https://www.emergentmind.com/topics/relu