---
title: Sine-Activated Model in Neural Networks
url: https://www.emergentmind.com/topics/sine-activated-model
type: topic
---

# Sine-Activated Model in Neural Networks

A sine-activated model is a computational model in which a sine function, or a sine-derived periodic nonlinearity, plays the central representational role. In contemporary machine learning, the term most commonly denotes neural architectures whose hidden computation uses $\sin(\cdot)$ or a closely related periodic transform, as in sinusoidal representation networks for implicit neural representations; more broadly, it also includes edge-activated Kolmogorov–Arnold networks, sine-transformed low-rank parameterizations, and mixed or saturated periodic activations. Outside neural-network practice, the same descriptive idea appears in nonlinear field theories whose dynamics are driven by periodic trigonometric interactions, as in the lattice sine-Gordon model [2006.09661] [2407.04149] [2403.19243] [1811.01219].

## 1. Definition and conceptual scope

The canonical modern sine-activated model is the coordinate-based multilayer perceptron whose hidden layers apply sine elementwise. In the influential SIREN formulation, a continuous signal is represented as a function $\Phi_\theta:\mathbf{x}\mapsto \Phi_\theta(\mathbf{x})$, where $\mathbf{x}$ is a coordinate in space or spacetime and the network maps coordinates directly to values such as RGB, amplitude, signed distance, or wavefield components [2006.09661]. This made the phrase strongly associated with implicit neural representations, neural fields, and PDE-constrained function approximation.

The term is not synonymous with a single architecture. Some sine-activated models place periodicity at the node level, as in sine MLPs and SIREN; some place it on edges, as in sinusoidal KANs; some apply sine to a low-rank matrix rather than to activations; and some combine sine with other nonlinearities such as $\tanh$ or residual identity paths [2407.04149] [2403.19243] [2101.06100] [2508.01175]. This suggests that “sine-activated model” is best understood as a family defined by where periodic trigonometric structure enters the computation, not by one fixed network topology.

A further ambiguity is historical and disciplinary. In field theory, the two-dimensional lattice sine-Gordon model is “sine-activated” in the sense that its essential nonlinearity is produced by a periodic trigonometric potential, with dynamics governed by activation over the infinitely many minima of a cosine landscape [1811.01219]. In machine learning, by contrast, the term usually refers to trainable models whose inductive bias is periodicity, oscillation, or harmonic synthesis.

## 2. Mathematical structure and representation

The standard sine-activated MLP replaces monotone hidden nonlinearities with periodic ones. In SIREN, hidden layers are written as
$$
\mathbf{x}_{l+1}=\sin(\mathbf{W}_l\mathbf{x}_l+\mathbf{b}_l),
$$
and the full model is
$$
\Phi(\mathbf{x})=\mathbf{W}_n\left(\phi_{n-1}\circ \phi_{n-2}\circ \cdots \circ \phi_0\right)(\mathbf{x})+\mathbf{b}_n,\qquad
\phi_i(\mathbf{x}_i)=\sin(\mathbf{W}_i\mathbf{x}_i+\mathbf{b}_i).
$$
This formulation is attractive for signals whose supervision involves not only values but also derivatives, because first- and second-order derivatives remain structured and nontrivial: for a scalar neuron $y=\sin(\mathbf{w}^\top\mathbf{x}+b)$,
$$
\nabla_{\mathbf{x}} y=\cos(\mathbf{w}^\top\mathbf{x}+b)\,\mathbf{w},\qquad
\nabla_{\mathbf{x}}^2 y=-\sin(\mathbf{w}^\top\mathbf{x}+b)\,\mathbf{w}\mathbf{w}^\top.
$$
The SIREN paper argued that this makes sine networks substantially better matched than ReLU networks to losses involving gradients, Laplacians, Hessians, or PDE residuals, since ReLU networks are piecewise linear and have second derivative zero almost everywhere [2006.09661].

A complementary theoretical description treats sine networks as harmonic generators. For one-dimensional sinusoidal MLPs, each hidden neuron can be expanded as a harmonic sum whose frequencies are integer linear combinations of the first-layer frequencies, with amplitudes governed by products of Bessel functions [2212.01833]. In that view, the first layer is a harmonic dictionary, while deeper sine composition generates structured frequency families rather than arbitrary spectra. The same work showed that if the input neurons are periodic, then the entire network is periodic with the same period, linking sinusoidal MLPs directly to Fourier-series representations [2212.01833].

This frequency-domain interpretation is not confined to continuous regression. On modular addition, two-layer sine MLPs were shown to admit width-$2$ exact realizations for any fixed sequence length, and with bias, width-$2$ exact realizations uniformly over all lengths [2511.23443]. The construction maps residues to phases on the unit circle, so class prediction becomes phase matching rather than piecewise-linear partitioning. A plausible implication is that sine activations are especially natural when the target function itself is periodic in a group-theoretic sense, not only when the observed signal is visually or acoustically oscillatory.

## 3. Initialization, bandwidth, and optimization

Sine activations are unusually sensitive to parameterization and initialization. SIREN introduced a principled scheme based on activation statistics, together with a first-layer frequency factor $\omega_0$, using $\sin(\omega_0\mathbf{W}\mathbf{x}+\mathbf{b})$ in the first layer and reporting $\omega_0=30$ as effective across the paper’s applications [2006.09661]. The same work emphasized that naive random initialization leads to poor convergence, whereas the proposed initialization allows deep sine networks to train robustly with ADAM and often fit a single image in a few hundred iterations [2006.09661].

A later simplification argued that the essential bandwidth control can be concentrated in the first layer. “Simple sinusoidal networks” retain only a first-layer scale $\omega$,
$$
f_1(x)=\sin\!\left(\omega(W_1x+b_1)\right),\qquad
f_l(x)=\sin(W_lx+b_l),\ l>1,
$$
and use standard He/Kaiming normal initialization throughout [2211.14503]. Their neural tangent kernel is approximately a Gaussian low-pass filter whose effective bandwidth is set by $\omega$, and the paper proposed a practical heuristic of choosing $\omega$ at roughly one eighth of the maximum relevant signal frequency or Nyquist frequency [2211.14503]. This recast the first-layer sine scale as a bandwidth parameter rather than only an empirical hyperparameter.

Other variants attempted to separate representational frequency from optimization stability more explicitly. HOSC defines
$$
\mathrm{HOSC}(x)=\tanh\bigl(\beta\sin(\omega_0 x)\bigr),
$$
with activation-level Lipschitz constant $L_{\text{HOSC}}=\beta\omega_0$ [2601.07870]. The parameter $\beta$ controls saturation strength and gradient magnitudes without discarding the periodic carrier. In that study, recommended values were strongly modality dependent: about $\beta=14$ for images, $\beta=5$ for audio, $\beta=0.5$ for video, $\beta\in[0.5,1.0]$ for NeRF, and about $\beta=0.8$–$1.0$ for SDFs [2601.07870]. The empirical message was that low-dimensional tasks may benefit from sharper periodic gating, whereas higher-dimensional coordinate tasks favor small $\beta$ for stable optimization.

## 4. Major architectural families

The family of sine-activated models now spans several architectural loci. Some alter the activation itself, some alter only the first layer, some move periodicity from nodes to edges, and some use sine to reparameterize weights rather than hidden states.

| Family | Defining form | Functional role |
|---|---|---|
| **SIREN** [2006.09661] | $\sin(\mathbf{W}\mathbf{x}+\mathbf{b})$ | Canonical sine-activated coordinate MLP for INRs and PDEs |
| **H-SIREN** [2410.04716] | first layer $\sin(\omega_0\sinh(r x))$, later layers $\sin(\omega_0 x)$ | Broaden first-layer frequency support while keeping later SIREN behavior |
| **HOSC** [2601.07870] | $\tanh\bigl(\beta\sin(\omega_0 x)\bigr)$ | Add saturation control and explicit gradient bound |
| **ASU** [2304.09759] | $x\sin(x)$ | Oscillatory activation with growing amplitude for nonlinear vibrations |
| **GLN** [2101.06100] | $\alpha\sin(x)+(1-\alpha)\tanh(x)-\text{Bias}$ | Trainable global-local mixture between sine and tanh |
| **PLU** [2508.01175] | $x+\frac{\beta_{\text{eff}}}{1+|\beta_{\text{eff}}|}\sin(|\alpha_{\text{eff}}|x)$ | Residual sine activation with anti-collapse reparameterization |
| **SineKAN** [2407.04149] [2508.00247] | edge function $\psi_{ij}(x)=\sum_k w_{ijk}\sin(x+\phi_{jk})$ or learnable-frequency sinusoidal inner/outer KAN functions | Move periodicity to learnable univariate edge maps |
| **Sine-activated low-rank models** [2403.19243] [2505.21895] | $\sin(\omega UV^\top)/g$ or $\sin(\omega Q(A)Q(B))/\gamma$ | Increase effective rank of low-rank matrices or quantized adapters |

These families should not be conflated. H-SIREN remains predominantly a sine-activated coordinate MLP but modifies only the first layer to $\sin(\omega_0\sinh(r x))$ in order to broaden initial frequency coverage [2410.04716]. GLN does not commit to sine alone; it learns whether each hidden layer should become sine-like, tanh-like, or mixed [2101.06100]. PLU keeps a residual identity path and introduces repulsive reparameterization because, in that paper’s formulation, naïvely learnable periodic activations tend to collapse toward linearity [2508.01175]. SineKAN is not an MLP with node-wise sines at all: it is a KAN in which the learnable univariate edge functions are sinusoidal expansions [2407.04149] [2508.00247]. Low-rank sine models are more distant still, because the sine is applied elementwise to a matrix such as $UV^\top$ rather than to neuron preactivations [2403.19243].

## 5. Applications and empirical record

The strongest early empirical record came from implicit neural representations. SIREN represented images, video, audio, wavefields, signed distance functions, and PDE solutions, and reported about 5 dB PSNR improvement over a ReLU MLP on a $512\times512$, 300-frame video, with average PSNR near 30 dB [2006.09661]. It also supported Poisson image reconstruction and editing, Eikonal/SDF modeling, Helmholtz and wave equations, and full-waveform inversion, with the central comparative claim being superior derivative fidelity rather than only better value fitting [2006.09661]. Later variants refined that picture rather than replacing it outright. H-SIREN reported image-fitting performance of $\text{PSNR }37.80\pm 3.10$ and $\text{SSIM }0.973\pm0.010$, versus SIREN’s $26.87\pm3.18$ and $0.808\pm0.065$, while also improving several video, NeRF, SDF, and graph-based fluid-flow benchmarks [2410.04716]. HOSC, in turn, was highly competitive on images and NeRF, and particularly strong on audio and video; for video it reported PSNR $38.7137$ and MSE $5.5673\times10^{-4}$ versus SIREN’s PSNR $35.9106$ and MSE $1.0487\times10^{-3}$ [2601.07870].

Physics and scientific computing have been a second major domain. In three-dimensional Rayleigh–Bénard convection data assimilation, sine-activated PINNs with 5 hidden layers and widths from 32 to 256 typically improved correlation and mean average error relative to tanh and ELU, with the benefit especially evident in PDFs and power spectra of inferred fields [2403.02970]. In high-frequency 3D photoacoustic tomography, a sine-activated UNET operating on sensor-wise PARF data achieved 2 volumes per second online and, in Gaussian-noise probing, was the only tested model that clearly enhanced high temporal frequencies in the 20–31.25 MHz range at a $-12$ dB cutoff [2507.20575]. In nonlinear MEMS vibration modeling, the proposed oscillatory activation ASU, $\mathrm{ASU}(x)=x\sin(x)$, reduced training from 6,000 epochs and 3 minutes 16 seconds for plain sine to 2,000 epochs and 1 minute 14 seconds on the reported benchmark [2304.09759].

Sine activation has also proved useful outside continuous fields. For modular addition, the provable width-$2$ constructions for sine MLPs were accompanied by empirical results showing substantially better sample efficiency and length extrapolation than ReLU across underparameterized, overparameterized, and Transformer settings [2511.23443]. In parameter-efficient adaptation, sine-transformed low-rank updates increased expressivity without changing parameter count [2403.19243], and quantized SineLoRA preserved much of that benefit: in one LLM setup, rank-8 SineLoRA at 5-bit reached 78.1 average accuracy versus 78.0 for full-precision rank-8 LoRA, while using 9.1 MB instead of 27.1 MB [2505.21895]. The empirical pattern across these works is that sine activation is not restricted to coordinate regression; it can function as a periodic basis, a rank-enhancing transform, or a task-matched algebraic inductive bias.

## 6. Limitations, controversies, and outlook

The main controversies concern universality of benefit, choice of parameterization, and the extent of current theory. Standard SIREN has been criticized as potentially “sub-optimal due to their limited supported frequency set as well as their tendency to generate over-smoothed solutions,” which directly motivated H-SIREN’s first-layer hyperbolic-periodic modification [2410.04716]. HOSC argued that pure sine couples carrier frequency and gradient magnitude too tightly, but also showed that its own extra parameter can be hazardous: large $\beta$ values improved some low-dimensional tasks while causing severe degradation in higher-dimensional ones such as video and NeRF [2601.07870]. This suggests that sine-based models do not remove hyperparameter sensitivity; they shift it toward spectral and gradient-control choices.

Another debate concerns whether pure sine is even the right endpoint. The GLN work showed that the learned mixture parameter can move toward nearly pure tanh, nearly pure sine, or remain intermediate depending on the task and layer depth, so a sine-only prescription is not always favored by training [2101.06100]. PLU went further and argued that learnable periodic activations tend to collapse toward the identity map unless zero is made repulsive through special reparameterization [2508.01175]. By contrast, the SSN work claimed that some of SIREN’s complexity is unnecessary and that first-layer frequency control plus standard He initialization already captures the essential behavior [2211.14503]. These positions are not mutually exclusive, but they indicate that “sine-activated model” now denotes a design space rather than a settled recipe.

Theoretical coverage remains uneven. Harmonic expansion and periodicity theorems exist for sinusoidal MLPs [2212.01833], NTK-based bandwidth analysis exists for simplified sine networks [2211.14503], and sharp expressivity and sample-complexity results now exist for modular addition [2511.23443]. But several later activation proposals still rely mainly on heuristic spectral arguments or task-specific empirical evidence, and some evaluations remain narrow; ASU, for example, was validated on a single MEMS application class rather than a broad benchmark suite [2304.09759]. A plausible implication is that the next stage of the subject will be less about demonstrating that sine can work and more about formalizing when periodicity should appear at the node, edge, layer, or weight-parameterization level, and how its frequency content should be matched to data, geometry, and optimization constraints.

Source: https://www.emergentmind.com/topics/sine-activated-model