---
title: Activation-Integral Theory in Neural Networks
url: https://www.emergentmind.com/topics/activation-integral-theory
type: topic
---

# Activation-Integral Theory in Neural Networks

Activation-Integral Theory (AIT) provides a rigorous mathematical and conceptual framework linking neural network activation functions and integral representations in both finite- and infinite-width regimes. Originating in the context of shallow neural networks with rectified polynomial (ReLU$^k$) activations, AIT unifies integral geometry, function space theory, and neural network expressivity. The theory yields a precise characterization of which functions can be represented by infinite-width neural architectures with bounded norms and provides sharp approximation rates for practical (finite-width) discretizations. Recent developments generalize AIT to arbitrary activation functions, integral transforms, and rule-based logic schemes, allowing principled design and analysis of novel nonlinearities, trainability, and explainability.

## 1. Integral Representation of Functions via Activation Functions

The foundational result of AIT is the exact integral representation of target functions as $L^p$-weighted superpositions of ridge functions parameterized by the activation function. For ReLU$^k$ activations, every $f$ in the Sobolev space $H^{(d+2k+1)/2}(\Omega)$ (for a bounded Lipschitz domain $\Omega\subset\mathbb R^d$) admits the representation
\[
f(x) = \int_{S^d} \sigma_k(\theta\cdot(x,1)) \psi(\theta)\, d\sigma(\theta)
\]
where $\sigma_k(t) = \max(t, 0)^k$ is the ReLU$^k$ activation, $S^d$ is the unit sphere in $\mathbb R^{d+1}$, and $\psi\in L^2(S^d)$ is a suitable coefficient function. Norm equivalence holds:
\[
\|f\|_{H^{(d+2k+1)/2}(\Omega)} \simeq \inf \{\|\psi\|_{L^2(S^d)} : f(x) = \int_{S^d} \sigma_k(\theta\cdot(x,1))\psi(\theta)\, d\sigma(\theta)\}
\]
This mirrors the classical Barron integral representation (ReLU, $k=1$) and identifies a fundamental correspondence between Sobolev regularity and neural network representations [2505.00351].

For the ordinary ReLU (ReLU$^1$), the sphere-based and ridge-based formulations, and explicit inversion formulas involving Radon transforms, provide constructive recipes for infinite- and finite-width representers [1910.02743].

## 2. Function Spaces Representable by Integral Neural Architectures

AIT precisely characterizes the admissible function class for each activation:

- **ReLU/Barron space**: ReLU-shallow networks with bounded $L^1$ outer weights represent functions in Barron space, corresponding to $f$ with finite first-moment Radon transform [1910.02743].
- **ReLU$^k$/Sobolev spaces**: For general $k$, admissible $f$ lie in $H^{(d+2k+1)/2}(\Omega)$, and the integral representation utilizes $L^2$ coefficients [2505.00351].
- **RePU polynomials**: Using rectified power units $\sigma_p(x) = [x]_+^p$, the representable class comprises piecewise polynomials (in 1D) and functions with bounded $(d+p)/2$-th distributional Laplacian (multivariate) [2112.11157].

The minimal integral-norm (e.g., $L^1$ or $L^2$ norm of the coefficient function or measure) provides a natural complexity measure, controlling generalization and approximation error rates.

## 3. Approximation Rates and Linearized Network Constructions

A direct consequence of the integral representation is the derivation of optimal n-width (finite representation) approximation rates using "linearized" (fixed inner parameters, trainable linear coefficients) shallow networks. For ReLU$^k$ networks, the rate is:
\[
O(n^{-1/2 - (2k+1)/(2d)})
\]
for $n$ units, which is optimal for the corresponding Sobolev space. The construction uses random or deterministic quadrature rules to discretize the continuous integral, yielding explicit finite sets of parameters and convex optimization of the linear coefficients [2505.00351].

For RePU, the approximation error rate for uniform approximations on compacts is $O(1/N^{1/p})$ for $N$ atoms, reflecting the degree of the activation's homogeneity [2112.11157].

## 4. Generalizations Beyond ReLU: Activation-Integral Theory for Arbitrary Nonlinearities

AIT extends to arbitrary scalar or vector-valued activations, provided sufficient regularity and polynomial growth conditions. The theory systematically produces new activation functions by integrating desired gradient flows (e.g., choosing $g(x) = f'(x)$ and integrating to obtain $f$). Piecewise or smooth gradient schemes generate a large family of novel nonlinearities, including those which interpolate between ReLU and exponential/sigmoid types [2411.13010].

The "Integral Signatures" framework formalizes the propagation and regularity statistics of arbitrary activations via a 9-dimensional vector of Gaussian moments, asymptotic slopes, and regularity measures, enabling principled taxonomy, stability classification, and kernel conditioning analysis [2510.08456].

## 5. Connection to Rule-Based and Sugeno-Integral Representations

AIT also admits a discrete, rule-based interpretation, especially for binarized neural networks (BNNs). Each BNN neuron's hard threshold can be written as a Sugeno integral over a suitable capacity (fuzzy measure), leading to an explicit, interpretable set-function and equivalent if–then rule set for the neuron's decision logic. The last-layer score in a BNN is similarly a Sugeno integral with a normalized measure. This connection facilitates symbolic reasoning, explainability, and extensions to multicriteria aggregation and verification [2604.17967].

## 6. Extensions: Integral Activation Transform and Complex-Analytic Activations

The Integral Activation Transform (IAT) generalizes classical coordinate-wise nonlinearities to functional transforms based on lifting, nonlinear activation in functional space, and integration against output bases. Specializing IAT to smooth global bases and ReLU yields smoother and more expressive architectures, improving trainability and mitigating vanishing-gradient issues [2312.12578].

Complex-analytic integral theorems (e.g., Cauchy’s integral formula) provide blueprints for new highly regular activation functions, such as the Cauchy activation, supporting universal approximation for analytic functions and yielding well-controlled gradient and locality properties. These structures can be efficiently integrated into modern deep architectures, providing theoretical guarantees for approximation and trainability [2409.19221].

## 7. Implications for Neural Network Theory, Design, and Analysis

AIT recasts the design space of neural activation functions from heuristic experimentation to principled, mathematically grounded construction. It provides:

- Precise characterizations of representable function spaces for various activations and architectures.
- Constructive, sharp approximation rates for finite-width models.
- Taxonomic classification and stability guarantees based on integral signatures and Lyapunov analyses.
- Rule-based, interpretable decision models for binarized and discrete networks.
- Systematic pathways for activation function design via integration of tailored gradient flows.
- Unified perspectives linking classical functional analysis, convex geometry, and deep learning theory.

Recent advances in AIT have led to practical improvements in expressivity, trainability, and interpretability in both standard and novel neural architectures [2505.00351][1910.02743][2112.11157][2411.13010][2510.08456][2604.17967][2312.12578][2409.19221].

Source: https://www.emergentmind.com/topics/activation-integral-theory