Periodic Activation Functions

• Periodic activation functions are nonlinearities defined by sine, cosine, or oscillatory components that inject an inductive bias for capturing cyclic and multi-frequency structures in data.
• They are applied in diverse domains such as time-series analysis, implicit neural representations, reinforcement learning, and control, often outperforming traditional activations like ReLU or tanh.
• These functions combine tailored mathematical formulations, adaptive hyperparameters, and gradient control to balance expressivity, efficiency, and training stability in neural networks.

Periodic activation functions are nonlinearities employed in neural networks that are explicitly periodic in their argument, typically adopting sine, cosine, or other oscillatory forms. Unlike standard monotonic activations (e.g. ReLU, sigmoid, tanh), these functions inject an inductive bias toward capturing cyclic, oscillatory, or multi-frequency structure within data. Over the past several years, a diverse taxonomy of periodic and semi-periodic activations has emerged, spanning applications in time-series modeling, implicit neural representations, reinforcement learning, control, and beyond. Periodic activations provide unique advantages in efficiency, expressiveness, and inductive capabilities relative to classical activation mechanisms.

1. Mathematical Formulations and Taxonomy

Periodic activations span a rich mathematical landscape ranging from elementary sines and cosines to more sophisticated forms with tunable amplitude, frequency, phase, or multi-scale distortions. Core families include:

• Pure Sinusoidal: $\sigma(x) = \sin(\omega_0 x)$ as used in SIREN networks for implicit signal modeling, with $\omega_0$ controlling the base frequency (Sitzmann et al., 2020).
• LeakySineLU: A semi-periodic, piecewise activation for time-series tasks,

$$\sigma(x) = \begin{cases} \sin^2(x) + x, & x > 0 \ \tfrac{1}{2}(\sin^2(x)+x), & x \leq 0 \end{cases}$$

with period $\pi$ in the oscillatory component and an unbounded linear term (Júnior et al., 2024).

• Snake: $\sigma(x) = x + \sin^2(ax)/a$, with frequency parameter $a$ and linear tail growth, designed to combine linear extrapolation with periodic bias (Ziyin et al., 2020).
• Periodic Linear Unit (PLU): $$\sigma(x) = x + \frac{\beta_{\text{eff}}}{1+|\beta_{\text{eff}}|}\sin(|\alpha_{\text{eff}}|x)$$, where $\alpha_{\text{eff}}$ and $\beta_{\text{eff}}$ are adaptive via repulsive reparameterization for learnable frequency and amplitude (Kudo, 2 Aug 2025).
• Amplifying Sine Unit (ASU): $\sigma(x) = x\sin(x)$, providing amplitude modulation suitable for nonlinear oscillatory DEs (Rahman et al., 2023).
• HOSC (Hyperbolic Oscillator with Saturation Control): $\sigma(x) = \tanh(\beta\,\sin(\omega_0x))$, merging periodicity with an explicit gradient/saturation bound via $\beta$, with the possibility of interpolation between sine and square-wave limiting cases (Wlodarczyk et al., 10 Jan 2026, Serrano et al., 2024).
• Variable-Periodic/FINER/FINER++: $\sigma(x) = \sin(\omega_0(|x|+1)x)$ and generalized warping for adaptive frequency tiling and spectral-bias tuning (Liu et al., 2023, Zhu et al., 2024).
• Quantum-inspired: $f(\alpha) = 1-\cos(\alpha)$ via quantum circuits encoding linear preactivations as phase angles (Daskin, 2018).
• Other Non-Sinusoidal Periodics: Triangular waves, periodic ReLU, and more (Meronen et al., 2021).

Table: Summary of Representative Periodic Activations

Name	Formula	Key Hyperparameters	Typical Use Case
SIREN	$\sin(\omega_0 x)$	$\omega_0$	INR, PDEs, signal fitting
LeakySineLU	see above	—	Time-series classification
Snake	$x+\sin^2(ax)/a$	$a$	Forecasting, extrapolation
PLU	$x + \frac{\beta}{1+|\beta|}\sin(|\alpha|x)$	$\alpha,\beta,\rho_\alpha,\rho_\beta$	Compact classifiers
ASU	$x\sin(x)$	—	Oscillatory nonlinear ODEs
HOSC	$\tanh(\beta\sin(\omega_0 x))$	$\beta, \omega_0$	INR, image/audio/video fitting
FINER/FINER++	$\sin(\omega_0(|x|+1)x)$ etc.	$\omega_0$, bias range $k$	INR with flexible spectral bias

2. Inductive Bias and Theoretical Motivation

Classical monotonic activations such as ReLU and tanh are fundamentally limited in their ability to extrapolate periodic or oscillatory signals; proofs establish that as input magnitude diverges, these networks either saturate or grow linearly and thus cannot replicate periodic extrapolation (Ziyin et al., 2020). Periodic activations inject a structural inductive bias matching the topology of cyclic signals:

• Fourier Basis Emulation: Periodic nonlinearities, especially sinusoidal, allow hidden units to act as adaptive Fourier bases; linear combinations can approximate arbitrary periodic functions via the universal approximation theorem for periodic activation networks (Ziyin et al., 2020, Kudo, 2 Aug 2025).
• Expressivity and Parameter Efficiency: Learnable frequency and amplitude (PLU, FINER++) enable compact synthesis of highly nonlinear decision boundaries (e.g., “two-spiral” classification via a two-neuron MLP; impossible with ReLU) (Kudo, 2 Aug 2025).
• Spectral Bias Control: Variable-periodic activations (FINER, FINER++) address the spectral-bias and capacity–convergence gap by expanding the supported frequency set, realized by wide-range bias initializations (Liu et al., 2023, Zhu et al., 2024).
• Stationarity and GP Connections: In Bayesian NNs, periodic activations yield translation-invariant, stationary Gaussian process priors, with the spectral measure set by the weight prior (Bochner’s theorem) (Meronen et al., 2021).

3. Empirical Performance Across Domains

Periodic activation functions have established significant empirical benefits across diverse tasks:

• Time Series Classification: LeakySineLU outperforms ReLU, PReLU, and other periodic competitors (e.g. Snake) across 112 UCR datasets, with the best average rank in both MLP and FCN architectures. Gains are attributed to the periodic derivative, which adapts gradients to oscillatory pattern features (Júnior et al., 2024).
• Implicit Neural Representation (INR): SIREN, FINER, HOSC achieve high-fidelity image/audio/video/signal reconstruction, with SIREN providing exact derivatives suited for PDE modeling (Sitzmann et al., 2020, Liu et al., 2023, Wlodarczyk et al., 10 Jan 2026). FINER++ provides state-of-the-art INR fidelity, enabling multi-frequency detail recovery and removing spectral bias limitations (Zhu et al., 2024).
• Reinforcement Learning: Learned Fourier features (LFF) or periodic first-layer critics accelerate learning, doubling sample efficiency over ReLU, but introduce generalization brittleness under noisy observations. Moderate L2 regularization (weight decay) mitigates overfitting to bootstrapped targets (Mavor-Parker et al., 2024).
• Oscillatory Physical Systems: ASU (amplifying sine unit) accelerates convergence and improves accuracy on nonlinear oscillator ODEs (e.g. MEMS beams), outperforming both conventional and oscillatory baselines (sine, GCU, Mish, Tanh) (Rahman et al., 2023).
• Physics-Informed Neural Networks: Substituting sine for tanh in PiNN architectures for multivariate PDEs delivers up to 100× gains in accuracy and 2×—1000× speed-ups in training/inference for solute transport in heterogeneous media (Faroughi et al., 2022).
• Control & Robotics: SIREN-activated G&CNETs converge 2–4× faster and with lower training error than Softplus or ReLU baselines across control domains (drone racing, asteroid landing, interplanetary transfer), often reducing model size by orders of magnitude (Origer et al., 2024).
• Sharp Feature Modeling: HOSC and AdaHOSC activations show superior PSNR on images with sharp edges and higher IoU for SDFs, with the sharpness parameter enabling a trade-off between smooth and abrupt features (Serrano et al., 2024, Wlodarczyk et al., 10 Jan 2026).

4. Gradient Properties, Optimization, and Stability

Gradient and Lipschitz behavior of periodic activations is crucial for stability and expressivity:

• Bounded vs Unbounded Growth: Pure sines are bounded, risking vanishing gradients for deep nets; hybrid activations (Snake, LeakySineLU, ASU, PLU) inject a linear term to maintain gradient flow (Ziyin et al., 2020, Júnior et al., 2024, Kudo, 2 Aug 2025).
• Saturation Control: HOSC introduces an explicit gradient/Lipschitz knob via $\beta$, decoupling frequency content from maximum allowed gradient and supporting safe training even at high frequencies (Wlodarczyk et al., 10 Jan 2026).
• Learnable Frequency/Amplitude: PLU’s repulsive reparametrization prevents collapse to identity, keeping neurons away from degeneracy and ensuring persistent oscillatory structure (Kudo, 2 Aug 2025).
• Gradient Isomorphism for Derivative Learning: For sin-based activations, all derivatives are phase-shifted sines, preserving magnitude and directionality required for PDEs and implicit field learning (Sitzmann et al., 2020).
• Potential Pitfalls: Oscillatory gradients may slow early convergence or create training instability in very small MLPs or data without cyclic structure, though these issues are mitigated by initialization and appropriate learning rates (Júnior et al., 2024, Wlodarczyk et al., 10 Jan 2026).

5. Implementation and Domain-Specific Guidelines

Empirical studies report robust, domain-driven recipes for adopting periodic activations:

• Initialization: SIREN-style weight schemes (Uniform$[-\sqrt{6/n}, \sqrt{6/n}]$) or tailored variance correction for activations with unbounded output (Snake, PLU) (Sitzmann et al., 2020, Ziyin et al., 2020, Kudo, 2 Aug 2025).
• Hyperparameters: First-layer frequencies $\omega_0 \approx 30$ are standard for INR, with per-domain tuning of sharpness parameters (HOSC: $\beta=0.5$–$16$ depending on domain), and frequency set/bias range in FINER/FINER++(Liu et al., 2023, Zhu et al., 2024, Wlodarczyk et al., 10 Jan 2026).
• Gradient Clipping and Learning Rate Schedules: Essential for high-sharpness or variable-periodic activations to avoid oscillatory divergence (Serrano et al., 2024, Wlodarczyk et al., 10 Jan 2026).
• Dropout and Regularization: Moderate dropout ($p=0.1$–$0.3$), L2 decay (esp. for RL) stabilize optimization and generalization (Júnior et al., 2024, Mavor-Parker et al., 2024).
• Architecture Integration: Periodic activations are typically inserted in hidden layers, often replacing only the first hidden layer for RL or all hidden layers for INR. PLU, HOSC, FINER, and LeakySineLU can be deployed as direct ReLU/ELU/SILU swaps in standard blocks (Júnior et al., 2024, Origer et al., 2024).

6. Limitations, Trade-offs, and Future Research Directions

The use of periodic activations is not universally beneficial; several limitations and trade-offs have been elucidated:

• Overfitting Under Noise: High-frequency representations trained with periodic activations can adversely affect out-of-distribution robustness, with LFF-critic RL suffering severe deterioration under observation noise relative to ReLU (Mavor-Parker et al., 2024).
• Task Suitability: Non-monotonic activations (PLU, Snake, HOSC) may be suboptimal for ordinal regression or non-periodic data, where monotonicity is inductively preferred (Kudo, 2 Aug 2025).
• Hardware and Speed: Sine, abs, tanh and associated operations have a higher computational cost than ReLU; optimized hardware implementations are required for large-scale deployment (Kudo, 2 Aug 2025).
• Spectral Bias–Expressivity Trade-off: Tuning parameters (sharpness, frequency, variable warping) can lead to overfitting of high-frequency noise or unstable training if not properly regularized (Liu et al., 2023, Serrano et al., 2024, Wlodarczyk et al., 10 Jan 2026).
• Theoretical Gaps: Deeper understanding of spectral properties (see NTK analysis in FINER/FINER++), extensions to graph neural networks, and further reparameterization schemes (PLU) are open for investigation (Kudo, 2 Aug 2025, Zhu et al., 2024).

A plausible implication is that future work will integrate periodic activations with adaptive frequency learning, normalized spectral control, and hybrid monotonic-periodic architectures tuned for domain-specific data statistics.

---

In summary, periodic activation functions represent a distinct and rigorously characterized toolkit for neural network architectures, conferring spectral expressivity, efficient learning of cyclical structure, and explicit inductive bias toward oscillatory phenomena. They have cemented their role in time-series analysis, physics-informed modeling, INR, reinforcement learning, and beyond, with ongoing research refining their stability, expressivity, and computational scalability (Júnior et al., 2024, Sitzmann et al., 2020, Liu et al., 2023, Kudo, 2 Aug 2025, Wlodarczyk et al., 10 Jan 2026).