FReX: Full-Wave Rectified Exponential

• FReX is an activation function defined as e^(-|x|), uniquely characterized as the fundamental solution of a second-order operator for one-hidden-layer networks.
• The model uses convolution with FReX to achieve exact parametrization and localized learning without needing boundary corrections on the real line.
• Its analytic features—exponential decay, a cusp at zero, and even symmetry—contrast with ReLU, offering precise convergence and stability in gradient descent dynamics.

The Full-Wave Rectified Exponential (FReX) is an activation function introduced in the analysis of one-hidden-layer neural networks with fixed biases as

$$Z(x)=\mathrm{FReX}(x)=e^{-|x|},\qquad x\in\mathbb{R},$$

where it is singled out by an operator-theoretic criterion: it is the unique fundamental solution of the second-order operator $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$ on $\mathbb{R}$. In that setting, FReX yields exact parametrization, explicit convergence analysis for gradient descent, and a localized learning operator. A separate line of work on oscillatory-signal processing does not explicitly define or analyze FReX in this form, but uses the term to situate the family $g_\alpha(x)=\exp(\alpha|x|)-1$ within a framework of full-wave rectified nonlinearities for fundamental-frequency enhancement. The shared motif is full-wave rectification combined with exponential structure, but the mathematical objects and target applications are distinct (Macià et al., 9 Apr 2026, Steinerberger et al., 2021).

1. Definition and operator-theoretic characterization

In the neural-network analysis, FReX is defined exactly by

$Z(x)=e^{-|x|}.$

Its defining structural property is distributional: $\frac12\Bigl\{-\frac{d^2}{dx^2}+1\Bigr\}Z=\delta,$ equivalently $H_0Z=\delta$ and $Z=H_0^{-1}\delta$. This places FReX in the same conceptual class as ReLU, which the paper treats as a fundamental solution of the one-dimensional Laplacian. The guiding principle is that activations which are fundamental solutions of second-order differential operators yield reduced one-layer models with good representability and convergence properties.

This characterization is not merely formal. In the continuous fixed-bias model, convolution with $Z$ realizes the inverse of $H_0$, so the activation kernel is directly tied to the differential operator governing representability. The resulting model is therefore analyzable by PDE and operator-theoretic tools rather than only by generic approximation arguments. A notable consequence is that, unlike the corresponding ReLU model on a finite interval, the FReX model on $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$0 needs no boundary correction term because the activation decays at infinity and is “integrable and almost localized” (Macià et al., 9 Apr 2026).

2. Qualitative analytic properties

From the exact formula $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$1, several properties follow immediately. FReX is continuous on $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$2, even, bounded by $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$3, and satisfies $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$4. It decays exponentially as $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$5, and it is globally Lipschitz with Lipschitz constant $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$6. These features sharply distinguish it from ReLU’s unbounded linear growth while preserving a comparable singularity at the origin.

The activation is not differentiable at $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$7. For $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$8,

$H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$9

and for $\mathbb{R}$0,

$\mathbb{R}$1

Hence the one-sided derivatives satisfy $\mathbb{R}$2 and $\mathbb{R}$3, so the graph has a cusp at the origin. Away from $\mathbb{R}$4, the second derivative satisfies $\mathbb{R}$5, while distributionally

$\mathbb{R}$6

equivalently $\mathbb{R}$7.

Its monotonicity is also asymmetric with respect to sign: FReX is increasing on $\mathbb{R}$8 and decreasing on $\mathbb{R}$9. Globally it is neither convex nor concave; piecewise it is convex on each half-line, but the cusp at $g_\alpha(x)=\exp(\alpha|x|)-1$0 makes the function attain an interior maximum. The paper emphasizes three qualitative features together: FReX is even, “as singular as ReLU at zero,” and exponentially decaying at infinity. The same decay that localizes the learning operator also “raises the possibility of vanishing derivatives,” which is the principal trade-off relative to ReLU in standard feedforward use (Macià et al., 9 Apr 2026).

3. Fixed-bias network model and exact parametrization

The analytical setting is a one-hidden-layer model with fixed biases, derived as a reduction of the usual two-layer architecture. For FReX on the real line, the model is written as

$g_\alpha(x)=\exp(\alpha|x|)-1$1

with $g_\alpha(x)=\exp(\alpha|x|)-1$2. Here $g_\alpha(x)=\exp(\alpha|x|)-1$3 is convolution by the activation kernel, and the operator identity $g_\alpha(x)=\exp(\alpha|x|)-1$4 holds on $g_\alpha(x)=\exp(\alpha|x|)-1$5 with domain $g_\alpha(x)=\exp(\alpha|x|)-1$6.

This identity gives exact parametrization. For any $g_\alpha(x)=\exp(\alpha|x|)-1$7, the unique weight function is

$g_\alpha(x)=\exp(\alpha|x|)-1$8

and then $g_\alpha(x)=\exp(\alpha|x|)-1$9. In contrast with the ReLU model on $Z(x)=e^{-|x|}.$0, where the representation requires an additional affine correction $Z(x)=e^{-|x|}.$1 to compensate for boundary effects, the FReX model on $Z(x)=e^{-|x|}.$2 has no such term.

The paper also develops a discrete analogue on the uniform lattice $Z(x)=e^{-|x|}.$3. With

$Z(x)=e^{-|x|}.$4

and the discrete Laplacian $Z(x)=e^{-|x|}.$5, the activation satisfies a discrete fundamental-solution identity

$Z(x)=e^{-|x|}.$6

where

$Z(x)=e^{-|x|}.$7

Writing $Z(x)=e^{-|x|}.$8, one obtains

$Z(x)=e^{-|x|}.$9

for the discrete operator $\frac12\Bigl\{-\frac{d^2}{dx^2}+1\Bigr\}Z=\delta,$0. Thus the continuous and discrete constructions share the same structural pattern: the activation is the inverse kernel of a second-order operator, and training can be formulated directly in terms of that inverse relation (Macià et al., 9 Apr 2026).

4. Gradient descent dynamics, convergence, and spectral bias

For the continuous FReX model on $\frac12\Bigl\{-\frac{d^2}{dx^2}+1\Bigr\}Z=\delta,$1, full-batch gradient descent has the iteration

$\frac12\Bigl\{-\frac{d^2}{dx^2}+1\Bigr\}Z=\delta,$2

with

$\frac12\Bigl\{-\frac{d^2}{dx^2}+1\Bigr\}Z=\delta,$3

If $\frac12\Bigl\{-\frac{d^2}{dx^2}+1\Bigr\}Z=\delta,$4 denotes convolution by $\frac12\Bigl\{-\frac{d^2}{dx^2}+1\Bigr\}Z=\delta,$5, the prediction error obeys

$\frac12\Bigl\{-\frac{d^2}{dx^2}+1\Bigr\}Z=\delta,$6

The central convergence statement is that if $\frac12\Bigl\{-\frac{d^2}{dx^2}+1\Bigr\}Z=\delta,$7 and $\frac12\Bigl\{-\frac{d^2}{dx^2}+1\Bigr\}Z=\delta,$8, then $\frac12\Bigl\{-\frac{d^2}{dx^2}+1\Bigr\}Z=\delta,$9 as $H_0Z=\delta$0; if in addition $H_0Z=\delta$1, then $H_0Z=\delta$2, where $H_0Z=\delta$3.

The Fourier-space form is explicit. Writing $H_0Z=\delta$4, the paper gives

$H_0Z=\delta$5

Because $H_0Z=\delta$6 is strictly increasing in $H_0Z=\delta$7, low frequencies decay faster than high frequencies. This makes the spectral-bias mechanism transparent: after $H_0Z=\delta$8 iterations, the effectively learned band satisfies the same $H_0Z=\delta$9 scaling as in the companion ReLU analysis. The paper summarizes the analogous ReLU result as frequencies up to index $Z=H_0^{-1}\delta$0 being resolved after $Z=H_0^{-1}\delta$1 iterations.

A major contrast with ReLU appears in the kernel of the induced learning operator. For FReX, the integral kernel of $Z=H_0^{-1}\delta$2 is

$Z=H_0^{-1}\delta$3

which decays rapidly off the diagonal. The paper interprets this as a “fairly localized” learning process. ReLU, by contrast, induces a broadly supported kernel on $Z=H_0^{-1}\delta$4, which complicates both interpretation and analysis. The localized FReX kernel is presented as one reason the continuous-line model becomes especially tractable, and as a possible advantage for analyzing or stabilizing stochastic training (Macià et al., 9 Apr 2026).

5. Relation to full-wave rectification and harmonic enhancement

In the oscillatory-signal literature, the role of a nonlinear activation is different. There the objective is to create or amplify a component at the underlying fundamental angular frequency $Z=H_0^{-1}\delta$5 so as to stabilize ridge detection and improve instantaneous frequency estimation. For a class of signals $Z=H_0^{-1}\delta$6, a nonlinear map $Z=H_0^{-1}\delta$7 has the fundamental enhancement property if there exists $Z=H_0^{-1}\delta$8 such that, for all $Z=H_0^{-1}\delta$9, the transformed signal $Z$0 either strongly enhances the fundamental Fourier coefficient or, when the fundamental is absent or too weak, generically creates a nonzero fundamental whose contribution can be assessed through the fundamental energy ratio

$Z$1

The key mechanism is harmonic mixing. Polynomial or nonanalytic nonlinearities applied to multi-harmonic inputs generate sums and differences of the original frequencies, and the $Z$2 of the harmonic indices can emerge as a new spectral line even when the original fundamental coefficient vanishes. This explains the practical effectiveness of $Z$3, $Z$4, and the paper’s adaptive activation

$Z$5

applied to the normalized signal $Z$6. For $Z$7, the paper proves an asymptotic lower bound under nondegenerate maxima assumptions: $Z$8 so the first Fourier coefficient grows like $Z$9 provided the leading term does not cancel.

Within that framework, the paper does not explicitly define or analyze FReX as $H_0$0. Instead, to situate a full-wave rectified exponential map inside the same mechanism, it considers

$H_0$1

with normalized variant

$H_0$2

Its power series,

$H_0$3

shows that all powers are present but high-order interactions are weighted by factorially decaying coefficients. The paper therefore states that such a FReX-type map still creates $H_0$4-frequency components for generic multi-harmonic signals, yet may be weaker than $H_0$5 because the coefficients $H_0$6 decay rapidly with $H_0$7. It also notes that for a single-tone input, $H_0$8 and $H_0$9 produce only even harmonics, whereas ReLU produces a fundamental because half-wave rectification has a $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$00-periodic Fourier series containing odd terms (Steinerberger et al., 2021).

6. Scope, practical use, and limitations

The main theoretical results for FReX concern a one-dimensional, fixed-bias setting in which the overall map is linear in the trainable parameters. In that regime, the recommended usage pattern is clear: in one-dimensional regression contexts, adopt the convolutional representation $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$01 with $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$02, choose a sufficiently small learning rate, and exploit the fact that no boundary correction is needed on $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$03. The theory requires no special initialization beyond $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$04, and if the target lies in $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$05, parameter convergence to $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$06 follows in addition to prediction convergence.

The same analysis identifies the principal numerical caution. Because FReX decays exponentially away from the origin, its values and derivatives become exponentially small for large $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$07. The paper therefore notes that, in standard feedforward networks, replacing ReLU by FReX may benefit from keeping pre-activations near zero, for example through input normalization or appropriate weight scaling, so as to avoid saturation far from the origin. The localized kernel $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$08 suggests stability and interpretability advantages, but the possibility of vanishing derivatives is the main trade-off.

The empirical evidence reported is deliberately limited. The paper states that, in a simple standard two-layer neural network for MNIST classification, replacing ReLU by FReX yields performance “roughly comparable” to ReLU and “clearly better” than Sigmoid, but it does not elaborate architectures or hyperparameters. The scope limitations are explicit: higher-dimensional models are said to be considerably more complicated; in the continuous case only strong convergence is proved because the operator $H_0=\frac12\{-\frac{d^2}{dx^2}+1\}$09 is compact; and extensions to stochastic gradient descent, deeper networks, and continuum–discrete correspondence are left as future directions. A plausible implication is that FReX is best understood, at present, as a mathematically transparent activation whose strongest guarantees arise in reduced models where the fundamental-solution structure can be fully exploited (Macià et al., 9 Apr 2026).