α-ReLU: Power Activation in Neural Networks

• α-ReLU is a family of power activation functions defined as [max{0,x}]^α, with α controlling smoothness, homogeneity, and growth behavior.
• It underpins theoretical analysis and practical design in neural networks, benefiting function approximation, PDE solvers, and control barrier synthesis.
• Variants such as two-slope Leaky α-ReLU and sparsifying transformations optimize computational tractability while achieving minimax-optimal rates in regression tasks.

The α-ReLU, or power ReLU, refers to a parametric family of activation functions of the form $\varphi_\alpha(x) = [\max\{0, x\}]^\alpha$, where $\alpha > 0$ is the exponent parameter. This class generalizes the standard ReLU ($\alpha=1$) and underpins recent advances in the theoretical analysis and practical design of neural architectures for function approximation, PDE solvers, control barrier functions, and sparse regularized learning. Its properties—homogeneity, smoothness, and approximation-theoretic behavior—are sensitive to the value of $\alpha$, which enables fine-grained control over regularity and functional capacities.

1. Formal Definition and Mathematical Properties

The α-ReLU activation is defined as

$$\varphi_\alpha(x) = [\max\{0, x\}]^\alpha, \quad \alpha > 0.$$

Key mathematical properties include:

• Homogeneity: For any $\lambda \ge 0$,

$$\varphi_\alpha(\lambda x) = \lambda^\alpha \varphi_\alpha(x).$$

• Smoothness: If $\alpha = k+\gamma$ with $k \in \mathbb{N}_0$, $\gamma \in (0,1)$, then $\alpha > 0$0 (i.e., $\alpha > 0$1-times differentiable, $\alpha > 0$2-th derivative $\alpha > 0$3-Hölder). For integer $\alpha > 0$4, $\alpha > 0$5 but not $\alpha > 0$6.
• Growth at Infinity: $\alpha > 0$7 as $\alpha > 0$8, growing sublinearly if $\alpha > 0$9.
• Special Cases:
  • Standard ReLU: $\alpha=1$0, $\alpha=1$1.
  • Higher-order: integer $\alpha=1$2 yields piecewise polynomials.

For variants, such as the two-slope Leaky α-ReLU used in control settings (Samanipour et al., 16 Mar 2026), the function is piecewise linear: $\alpha=1$3 with $\alpha=1$4.

2. Approximation and Regularity in Shallow α-ReLU Networks

Shallow α-ReLU networks are central to the analysis of PDE solution operators and function approximation in Sobolev/Hölder and Barron-type norms. For the Dirichlet-Laplace (Poisson) problem on half-spaces, solution regularity and approximation rates depend sensitively on $\alpha=1$5 (Vaishampayan et al., 2024, Li et al., 18 May 2026):

• Fractional $\alpha=1$6: Network solutions realize fractional Hölder regularity ($\alpha=1$7); the associated Barron norm $\alpha=1$8 is compatible with the solution's smoothness.
• Integer $\alpha=1$9: One obtains $\alpha$0 regularity, corresponding to one derivative less of Lipschitz continuity (but not $\alpha$1).
• Approximation Guarantees: Given a function $\alpha$2 on the boundary with controlled $\alpha$3 norm, the solution $\alpha$4 and its Monte-Carlo approximation $\alpha$5 in the domain satisfy

$\alpha$6

under technical conditions on $\alpha$7, and $\alpha$8. For integer $\alpha$9, logarithmic penalties in the Barron norm arise due to log-divergences at the boundary (Vaishampayan et al., 2024).

Approximation of general $$\varphi_\alpha(x) = [\max\{0, x\}]^\alpha, \quad \alpha > 0.$$0 in $$\varphi_\alpha(x) = [\max\{0, x\}]^\alpha, \quad \alpha > 0.$$1 balls or Sobolev spaces with shallow $$\varphi_\alpha(x) = [\max\{0, x\}]^\alpha, \quad \alpha > 0.$$2-ReLU networks yields rates that depend polynomially or log-polynomially on the network width $$\varphi_\alpha(x) = [\max\{0, x\}]^\alpha, \quad \alpha > 0.$$3, the exponent $$\varphi_\alpha(x) = [\max\{0, x\}]^\alpha, \quad \alpha > 0.$$4, and the spatial dimension $$\varphi_\alpha(x) = [\max\{0, x\}]^\alpha, \quad \alpha > 0.$$5 (Li et al., 18 May 2026).

3. Barron, Sobolev, and Spectral Characterizations

The choice of $$\varphi_\alpha(x) = [\max\{0, x\}]^\alpha, \quad \alpha > 0.$$6 in α-ReLU directly links the network's functional capacity to analytic regularity scales:

• Barron Norms: For $$\varphi_\alpha(x) = [\max\{0, x\}]^\alpha, \quad \alpha > 0.$$7, defined as the infimum of expected weighted coefficients over representation by α-ReLU ridge superpositions, this norm governs approximation error for PDE boundary data (Vaishampayan et al., 2024).
• Sobolev Embedding: The functional class $$\varphi_\alpha(x) = [\max\{0, x\}]^\alpha, \quad \alpha > 0.$$8 embeds into $$\varphi_\alpha(x) = [\max\{0, x\}]^\alpha, \quad \alpha > 0.$$9 or $\lambda \ge 0$0 depending on whether $\lambda \ge 0$1 or $\lambda \ge 0$2, where $\lambda \ge 0$3 and $\lambda \ge 0$4.
• Path-Norm Regularization: For finite-width networks

$\lambda \ge 0$5

the $\lambda \ge 0$6 path-norm is

$\lambda \ge 0$7

Minimax-optimal generalization rates are achieved for regression over Barron and Sobolev (fractional) targets, with exponents determined by $\lambda \ge 0$8 (Li et al., 18 May 2026).

The critical regularity transition occurs when $\lambda \ge 0$9 crosses an integer: fractional powers yield $$\varphi_\alpha(\lambda x) = \lambda^\alpha \varphi_\alpha(x).$$0 for $$\varphi_\alpha(\lambda x) = \lambda^\alpha \varphi_\alpha(x).$$1, while integer exponents only ensure Lipschitz continuity of the $$\varphi_\alpha(\lambda x) = \lambda^\alpha \varphi_\alpha(x).$$2-th derivative.

4. α-ReLU in Control Barrier Function Synthesis

For control systems with safe set invariance under polytopic input constraints, α-ReLU functions are used as surrogates for extended class-$$\varphi_\alpha(\lambda x) = \lambda^\alpha \varphi_\alpha(x).$$3 barrier functions (Samanipour et al., 16 Mar 2026):

• Two-Slope α-ReLU: Parameterized by positive slopes $$\varphi_\alpha(\lambda x) = \lambda^\alpha \varphi_\alpha(x).$$4 on $$\varphi_\alpha(\lambda x) = \lambda^\alpha \varphi_\alpha(x).$$5 and $$\varphi_\alpha(\lambda x) = \lambda^\alpha \varphi_\alpha(x).$$6 respectively, ensuring continuity, piecewise differentiability, radial unboundedness, and strict monotonicity.
• Convexity in Synthesis: The two-slope α-ReLU maintains the linearity of control barrier function (CBF) constraints in linear programming synthesis, facilitating tractable certification of safety properties.
• Conservatism and UIS Construction: The union of invariant sets (UIS), obtained by max-composing solutions for different slopes, never reduces the certified safe set below that of the optimal linear α; in most cases, it expands it (Samanipour et al., 16 Mar 2026).

This surrogate captures the strength of general class-$$\varphi_\alpha(\lambda x) = \lambda^\alpha \varphi_\alpha(x).$$7 nonlinearities without introducing additional nonconvexity or substantial conservatism in stability certification.

5. α-ReLU Variants and Modified Network Architectures

Beyond pointwise power functions, the literature includes sparsifying α–ReLU transforms acting on weights, notably in nonparametric regression (Beknazaryan et al., 2022):

• Sparsifying α: Defined as

$$\varphi_\alpha(\lambda x) = \lambda^\alpha \varphi_\alpha(x).$$8

applied entrywise to network weight matrices prior to multiplication and activation, thereby imposing structured sparsity.

• Statistical Rates: With $$\varphi_\alpha(\lambda x) = \lambda^\alpha \varphi_\alpha(x).$$9 or $\alpha = k+\gamma$0-penalized empirical risk minimization, sparsified α-ReLU networks achieve, up to log factors, minimax-optimal $\alpha = k+\gamma$1 prediction rates for $\alpha = k+\gamma$2-Hölder regression under sub-Gaussian noise (Beknazaryan et al., 2022).

This approach yields scale-invariance of penalty complexity, bypassing the suboptimal covering behavior of conventional penalized ReLU networks.

6. Practical Considerations and Trade-offs

Selection of $\alpha = k+\gamma$3 in ReLU$\alpha = k+\gamma$4 activations is a design choice balancing analytic regularity, approximation power, and computational tractability:

• Regularity Requirements: Applications like PINNs or strong-form PDE solvers may necessitate $\alpha = k+\gamma$5 regularity, thus motivating integer $\alpha = k+\gamma$6.
• Computational Cost: For non-integer and especially irrational $\alpha = k+\gamma$7, the evaluation cost of $\alpha = k+\gamma$8 can be significant.
• Barron-Norm Growth: For integer values, an unavoidable logarithmic penalty emerges in the Barron norm near boundaries, impacting the efficiency of representation (Vaishampayan et al., 2024).
• Parameter Interpretability: In two-slope Leaky α-ReLU barrier constructions, tuning $\alpha = k+\gamma$9 modulates the aggressiveness of barrier enforcement for positive and negative violations (Samanipour et al., 16 Mar 2026).

A plausible implication is that the α-ReLU family enables granular matching of network expressivity to analytic and application-driven demands, but judicious tuning is required to balance all competing considerations.

7. Comparative Summary

Variant	Mathematical Formulation	Main Use/Result
Standard Power α-ReLU	$k \in \mathbb{N}_0$0	PDE solvers, approximation with controlled regularity (Vaishampayan et al., 2024, Li et al., 18 May 2026)
Two-slope Leaky α-ReLU	Piecewise linear with $k \in \mathbb{N}_0$1	Barrier certification under control saturation (Samanipour et al., 16 Mar 2026)
Sparsifying α-ReLU	Piecewise constant-linear on weights	Sparse nonparametric regression at minimax rates (Beknazaryan et al., 2022)

• Vaishampayan and Wojtowytsch, "Solving the Poisson Equation with Dirichlet data by shallow ReLU$k \in \mathbb{N}_0$2-networks" (Vaishampayan et al., 2024).
• Beknazaryan and Sang, "Nonparametric regression with modified ReLU networks" (Beknazaryan et al., 2022).
• Li, Liu, and Shi, "Shallow ReLU$k \in \mathbb{N}_0$3 Networks in $k \in \mathbb{N}_0$4-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization" (Li et al., 18 May 2026).
• ReLU Barrier Functions (multiple authors), "ReLU Barrier Functions for Nonlinear Systems with Constrained Control: A Union of Invariant Sets Approach" (Samanipour et al., 16 Mar 2026).