Sine-Activated Model in Neural Networks

• Sine-activated models are computational architectures that use periodic sine functions to represent continuous signals and approximate PDEs.
• They integrate periodicity at nodes, edges, or weights to enable harmonic synthesis and improved derivative modeling over traditional activations.
• Applications include implicit neural representations for images, video, audio, and scientific computing, often yielding superior metrics like PSNR.

A sine-activated model is a computational model in which a sine function, or a sine-derived periodic nonlinearity, plays the central representational role. In contemporary machine learning, the term most commonly denotes neural architectures whose hidden computation uses $\sin(\cdot)$ or a closely related periodic transform, as in sinusoidal representation networks for implicit neural representations; more broadly, it also includes edge-activated Kolmogorov–Arnold networks, sine-transformed low-rank parameterizations, and mixed or saturated periodic activations. Outside neural-network practice, the same descriptive idea appears in nonlinear field theories whose dynamics are driven by periodic trigonometric interactions, as in the lattice sine-Gordon model (Sitzmann et al., 2020, Reinhardt et al., 2024, Ji et al., 2024, Flamino et al., 2018).

1. Definition and conceptual scope

The canonical modern sine-activated model is the coordinate-based multilayer perceptron whose hidden layers apply sine elementwise. In the influential SIREN formulation, a continuous signal is represented as a function $$\Phi_\theta:\mathbf{x}\mapsto \Phi_\theta(\mathbf{x})$$, where $\mathbf{x}$ is a coordinate in space or spacetime and the network maps coordinates directly to values such as RGB, amplitude, signed distance, or wavefield components (Sitzmann et al., 2020). This made the phrase strongly associated with implicit neural representations, neural fields, and PDE-constrained function approximation.

The term is not synonymous with a single architecture. Some sine-activated models place periodicity at the node level, as in sine MLPs and SIREN; some place it on edges, as in sinusoidal KANs; some apply sine to a low-rank matrix rather than to activations; and some combine sine with other nonlinearities such as $\tanh$ or residual identity paths (Reinhardt et al., 2024, Ji et al., 2024, Ferreira et al., 2021, Kudo, 2 Aug 2025). This suggests that “sine-activated model” is best understood as a family defined by where periodic trigonometric structure enters the computation, not by one fixed network topology.

A further ambiguity is historical and disciplinary. In field theory, the two-dimensional lattice sine-Gordon model is “sine-activated” in the sense that its essential nonlinearity is produced by a periodic trigonometric potential, with dynamics governed by activation over the infinitely many minima of a cosine landscape (Flamino et al., 2018). In machine learning, by contrast, the term usually refers to trainable models whose inductive bias is periodicity, oscillation, or harmonic synthesis.

2. Mathematical structure and representation

The standard sine-activated MLP replaces monotone hidden nonlinearities with periodic ones. In SIREN, hidden layers are written as

$$\mathbf{x}_{l+1}=\sin(\mathbf{W}_l\mathbf{x}_l+\mathbf{b}_l),$$

and the full model is

$$\Phi(\mathbf{x})=\mathbf{W}_n\left(\phi_{n-1}\circ \phi_{n-2}\circ \cdots \circ \phi_0\right)(\mathbf{x})+\mathbf{b}_n,\qquad \phi_i(\mathbf{x}_i)=\sin(\mathbf{W}_i\mathbf{x}_i+\mathbf{b}_i).$$

This formulation is attractive for signals whose supervision involves not only values but also derivatives, because first- and second-order derivatives remain structured and nontrivial: for a scalar neuron $y=\sin(\mathbf{w}^\top\mathbf{x}+b)$,

$$\nabla_{\mathbf{x}} y=\cos(\mathbf{w}^\top\mathbf{x}+b)\,\mathbf{w},\qquad \nabla_{\mathbf{x}}^2 y=-\sin(\mathbf{w}^\top\mathbf{x}+b)\,\mathbf{w}\mathbf{w}^\top.$$

The SIREN paper argued that this makes sine networks substantially better matched than ReLU networks to losses involving gradients, Laplacians, Hessians, or PDE residuals, since ReLU networks are piecewise linear and have second derivative zero almost everywhere (Sitzmann et al., 2020).

A complementary theoretical description treats sine networks as harmonic generators. For one-dimensional sinusoidal MLPs, each hidden neuron can be expanded as a harmonic sum whose frequencies are integer linear combinations of the first-layer frequencies, with amplitudes governed by products of Bessel functions (Novello, 2022). In that view, the first layer is a harmonic dictionary, while deeper sine composition generates structured frequency families rather than arbitrary spectra. The same work showed that if the input neurons are periodic, then the entire network is periodic with the same period, linking sinusoidal MLPs directly to Fourier-series representations (Novello, 2022).

This frequency-domain interpretation is not confined to continuous regression. On modular addition, two-layer sine MLPs were shown to admit width-$2$ exact realizations for any fixed sequence length, and with bias, width-$2$ exact realizations uniformly over all lengths (Huang et al., 28 Nov 2025). The construction maps residues to phases on the unit circle, so class prediction becomes phase matching rather than piecewise-linear partitioning. A plausible implication is that sine activations are especially natural when the target function itself is periodic in a group-theoretic sense, not only when the observed signal is visually or acoustically oscillatory.

3. Initialization, bandwidth, and optimization

Sine activations are unusually sensitive to parameterization and initialization. SIREN introduced a principled scheme based on activation statistics, together with a first-layer frequency factor $$\Phi_\theta:\mathbf{x}\mapsto \Phi_\theta(\mathbf{x})$$0, using $$\Phi_\theta:\mathbf{x}\mapsto \Phi_\theta(\mathbf{x})$$1 in the first layer and reporting $$\Phi_\theta:\mathbf{x}\mapsto \Phi_\theta(\mathbf{x})$$2 as effective across the paper’s applications (Sitzmann et al., 2020). The same work emphasized that naive random initialization leads to poor convergence, whereas the proposed initialization allows deep sine networks to train robustly with ADAM and often fit a single image in a few hundred iterations (Sitzmann et al., 2020).

A later simplification argued that the essential bandwidth control can be concentrated in the first layer. “Simple sinusoidal networks” retain only a first-layer scale $$\Phi_\theta:\mathbf{x}\mapsto \Phi_\theta(\mathbf{x})$$3,

$$\Phi_\theta:\mathbf{x}\mapsto \Phi_\theta(\mathbf{x})$$4

and use standard He/Kaiming normal initialization throughout (Belbute-Peres et al., 2022). Their neural tangent kernel is approximately a Gaussian low-pass filter whose effective bandwidth is set by $$\Phi_\theta:\mathbf{x}\mapsto \Phi_\theta(\mathbf{x})$$5, and the paper proposed a practical heuristic of choosing $$\Phi_\theta:\mathbf{x}\mapsto \Phi_\theta(\mathbf{x})$$6 at roughly one eighth of the maximum relevant signal frequency or Nyquist frequency (Belbute-Peres et al., 2022). This recast the first-layer sine scale as a bandwidth parameter rather than only an empirical hyperparameter.

Other variants attempted to separate representational frequency from optimization stability more explicitly. HOSC defines

$$\Phi_\theta:\mathbf{x}\mapsto \Phi_\theta(\mathbf{x})$$7

with activation-level Lipschitz constant $$\Phi_\theta:\mathbf{x}\mapsto \Phi_\theta(\mathbf{x})$$8 (Wlodarczyk et al., 10 Jan 2026). The parameter $$\Phi_\theta:\mathbf{x}\mapsto \Phi_\theta(\mathbf{x})$$9 controls saturation strength and gradient magnitudes without discarding the periodic carrier. In that study, recommended values were strongly modality dependent: about $\mathbf{x}$0 for images, $\mathbf{x}$1 for audio, $\mathbf{x}$2 for video, $\mathbf{x}$3 for NeRF, and about $\mathbf{x}$4–$\mathbf{x}$5 for SDFs (Wlodarczyk et al., 10 Jan 2026). The empirical message was that low-dimensional tasks may benefit from sharper periodic gating, whereas higher-dimensional coordinate tasks favor small $\mathbf{x}$6 for stable optimization.

4. Major architectural families

The family of sine-activated models now spans several architectural loci. Some alter the activation itself, some alter only the first layer, some move periodicity from nodes to edges, and some use sine to reparameterize weights rather than hidden states.

Family	Defining form	Functional role
SIREN (Sitzmann et al., 2020)	$\mathbf{x}$7	Canonical sine-activated coordinate MLP for INRs and PDEs
H-SIREN (Gao et al., 2024)	first layer $\mathbf{x}$8, later layers $\mathbf{x}$9	Broaden first-layer frequency support while keeping later SIREN behavior
HOSC (Wlodarczyk et al., 10 Jan 2026)	$\tanh$0	Add saturation control and explicit gradient bound
ASU (Rahman et al., 2023)	$\tanh$1	Oscillatory activation with growing amplitude for nonlinear vibrations
GLN (Ferreira et al., 2021)	$\tanh$2	Trainable global-local mixture between sine and tanh
PLU (Kudo, 2 Aug 2025)	$\tanh$3	Residual sine activation with anti-collapse reparameterization
SineKAN (Reinhardt et al., 2024, Gleyzer et al., 1 Aug 2025)	edge function $\tanh$4 or learnable-frequency sinusoidal inner/outer KAN functions	Move periodicity to learnable univariate edge maps
Sine-activated low-rank models (Ji et al., 2024, 2505.21895)	$\tanh$5 or $\tanh$6	Increase effective rank of low-rank matrices or quantized adapters

These families should not be conflated. H-SIREN remains predominantly a sine-activated coordinate MLP but modifies only the first layer to $\tanh$7 in order to broaden initial frequency coverage (Gao et al., 2024). GLN does not commit to sine alone; it learns whether each hidden layer should become sine-like, tanh-like, or mixed (Ferreira et al., 2021). PLU keeps a residual identity path and introduces repulsive reparameterization because, in that paper’s formulation, naïvely learnable periodic activations tend to collapse toward linearity (Kudo, 2 Aug 2025). SineKAN is not an MLP with node-wise sines at all: it is a KAN in which the learnable univariate edge functions are sinusoidal expansions (Reinhardt et al., 2024, Gleyzer et al., 1 Aug 2025). Low-rank sine models are more distant still, because the sine is applied elementwise to a matrix such as $\tanh$8 rather than to neuron preactivations (Ji et al., 2024).

5. Applications and empirical record

The strongest early empirical record came from implicit neural representations. SIREN represented images, video, audio, wavefields, signed distance functions, and PDE solutions, and reported about 5 dB PSNR improvement over a ReLU MLP on a $\tanh$9, 300-frame video, with average PSNR near 30 dB (Sitzmann et al., 2020). It also supported Poisson image reconstruction and editing, Eikonal/SDF modeling, Helmholtz and wave equations, and full-waveform inversion, with the central comparative claim being superior derivative fidelity rather than only better value fitting (Sitzmann et al., 2020). Later variants refined that picture rather than replacing it outright. H-SIREN reported image-fitting performance of $$\mathbf{x}_{l+1}=\sin(\mathbf{W}_l\mathbf{x}_l+\mathbf{b}_l),$$0 and $$\mathbf{x}_{l+1}=\sin(\mathbf{W}_l\mathbf{x}_l+\mathbf{b}_l),$$1, versus SIREN’s $$\mathbf{x}_{l+1}=\sin(\mathbf{W}_l\mathbf{x}_l+\mathbf{b}_l),$$2 and $$\mathbf{x}_{l+1}=\sin(\mathbf{W}_l\mathbf{x}_l+\mathbf{b}_l),$$3, while also improving several video, NeRF, SDF, and graph-based fluid-flow benchmarks (Gao et al., 2024). HOSC, in turn, was highly competitive on images and NeRF, and particularly strong on audio and video; for video it reported PSNR $$\mathbf{x}_{l+1}=\sin(\mathbf{W}_l\mathbf{x}_l+\mathbf{b}_l),$$4 and MSE $$\mathbf{x}_{l+1}=\sin(\mathbf{W}_l\mathbf{x}_l+\mathbf{b}_l),$$5 versus SIREN’s PSNR $$\mathbf{x}_{l+1}=\sin(\mathbf{W}_l\mathbf{x}_l+\mathbf{b}_l),$$6 and MSE $$\mathbf{x}_{l+1}=\sin(\mathbf{W}_l\mathbf{x}_l+\mathbf{b}_l),$$7 (Wlodarczyk et al., 10 Jan 2026).

Physics and scientific computing have been a second major domain. In three-dimensional Rayleigh–Bénard convection data assimilation, sine-activated PINNs with 5 hidden layers and widths from 32 to 256 typically improved correlation and mean average error relative to tanh and ELU, with the benefit especially evident in PDFs and power spectra of inferred fields (Mommert et al., 2024). In high-frequency 3D photoacoustic tomography, a sine-activated UNET operating on sensor-wise PARF data achieved 2 volumes per second online and, in Gaussian-noise probing, was the only tested model that clearly enhanced high temporal frequencies in the 20–31.25 MHz range at a $$\mathbf{x}_{l+1}=\sin(\mathbf{W}_l\mathbf{x}_l+\mathbf{b}_l),$$8 dB cutoff (Sulistyawan et al., 28 Jul 2025). In nonlinear MEMS vibration modeling, the proposed oscillatory activation ASU, $$\mathbf{x}_{l+1}=\sin(\mathbf{W}_l\mathbf{x}_l+\mathbf{b}_l),$$9, reduced training from 6,000 epochs and 3 minutes 16 seconds for plain sine to 2,000 epochs and 1 minute 14 seconds on the reported benchmark (Rahman et al., 2023).

Sine activation has also proved useful outside continuous fields. For modular addition, the provable width-$$\Phi(\mathbf{x})=\mathbf{W}_n\left(\phi_{n-1}\circ \phi_{n-2}\circ \cdots \circ \phi_0\right)(\mathbf{x})+\mathbf{b}_n,\qquad \phi_i(\mathbf{x}_i)=\sin(\mathbf{W}_i\mathbf{x}_i+\mathbf{b}_i).$$0 constructions for sine MLPs were accompanied by empirical results showing substantially better sample efficiency and length extrapolation than ReLU across underparameterized, overparameterized, and Transformer settings (Huang et al., 28 Nov 2025). In parameter-efficient adaptation, sine-transformed low-rank updates increased expressivity without changing parameter count (Ji et al., 2024), and quantized SineLoRA preserved much of that benefit: in one LLM setup, rank-8 SineLoRA at 5-bit reached 78.1 average accuracy versus 78.0 for full-precision rank-8 LoRA, while using 9.1 MB instead of 27.1 MB (2505.21895). The empirical pattern across these works is that sine activation is not restricted to coordinate regression; it can function as a periodic basis, a rank-enhancing transform, or a task-matched algebraic inductive bias.

6. Limitations, controversies, and outlook

The main controversies concern universality of benefit, choice of parameterization, and the extent of current theory. Standard SIREN has been criticized as potentially “sub-optimal due to their limited supported frequency set as well as their tendency to generate over-smoothed solutions,” which directly motivated H-SIREN’s first-layer hyperbolic-periodic modification (Gao et al., 2024). HOSC argued that pure sine couples carrier frequency and gradient magnitude too tightly, but also showed that its own extra parameter can be hazardous: large $$\Phi(\mathbf{x})=\mathbf{W}_n\left(\phi_{n-1}\circ \phi_{n-2}\circ \cdots \circ \phi_0\right)(\mathbf{x})+\mathbf{b}_n,\qquad \phi_i(\mathbf{x}_i)=\sin(\mathbf{W}_i\mathbf{x}_i+\mathbf{b}_i).$$1 values improved some low-dimensional tasks while causing severe degradation in higher-dimensional ones such as video and NeRF (Wlodarczyk et al., 10 Jan 2026). This suggests that sine-based models do not remove hyperparameter sensitivity; they shift it toward spectral and gradient-control choices.

Another debate concerns whether pure sine is even the right endpoint. The GLN work showed that the learned mixture parameter can move toward nearly pure tanh, nearly pure sine, or remain intermediate depending on the task and layer depth, so a sine-only prescription is not always favored by training (Ferreira et al., 2021). PLU went further and argued that learnable periodic activations tend to collapse toward the identity map unless zero is made repulsive through special reparameterization (Kudo, 2 Aug 2025). By contrast, the SSN work claimed that some of SIREN’s complexity is unnecessary and that first-layer frequency control plus standard He initialization already captures the essential behavior (Belbute-Peres et al., 2022). These positions are not mutually exclusive, but they indicate that “sine-activated model” now denotes a design space rather than a settled recipe.

Theoretical coverage remains uneven. Harmonic expansion and periodicity theorems exist for sinusoidal MLPs (Novello, 2022), NTK-based bandwidth analysis exists for simplified sine networks (Belbute-Peres et al., 2022), and sharp expressivity and sample-complexity results now exist for modular addition (Huang et al., 28 Nov 2025). But several later activation proposals still rely mainly on heuristic spectral arguments or task-specific empirical evidence, and some evaluations remain narrow; ASU, for example, was validated on a single MEMS application class rather than a broad benchmark suite (Rahman et al., 2023). A plausible implication is that the next stage of the subject will be less about demonstrating that sine can work and more about formalizing when periodicity should appear at the node, edge, layer, or weight-parameterization level, and how its frequency content should be matched to data, geometry, and optimization constraints.