---
title: 'SIREN: Sinusoidal Representation Networks'
url: https://www.emergentmind.com/topics/siren-architecture
type: topic
---

# SIREN: Sinusoidal Representation Networks

A SIREN (Sinusoidal Representation Network) is a multilayer perceptron (MLP) that uses the sinusoidal activation function in each hidden layer to parameterize continuous, differentiable implicit neural representations (INRs) of signals such as images, audio, 3D geometry, and physical fields. SIREN architectures are defined by their characteristic use of sine activation, principled initialization to maintain signal propagation and frequency fidelity, and the ability to directly model signal derivatives necessary for representing physical PDEs. Subsequent advances have introduced neuron-specific frequency multipliers (e.g., FM-SIREN), adaptive weight initialization strategies (WINNER/SIREN²), and specialized variants for diverse tasks such as noise-robust INR, high-fidelity audio compression, and multi-modal registration.

## 1. Mathematical Definition and Architectural Principles

A canonical SIREN consists of a fully connected stack of layers, where each hidden layer applies an affine transformation followed by the elementwise sine nonlinearity. For input $x \in \mathbb{R}^{d_{in}}$, parameters $\theta = \{W_i, b_i\}_{i=0}^n$, and layer input/output dimensions $M_i, N_i$, the mapping is:
$$
\varphi_\theta(x) = W_n(\varphi_{n-1} \circ \ldots \circ \varphi_0)(x) + b_n,
$$
where each hidden layer applies
$$
\varphi_i(u) = \sin(W_i u + b_i).
$$
In practice, SIREN uses a frequency scaling hyperparameter $\omega_0$ for the first layer only:
$$
\varphi_0(x) = \sin(\omega_0 W_0 x + b_0).
$$
In all subsequent layers, $\omega_0 = 1$ (unless using alternative variants). This configuration ensures that the network’s post-activation distribution remains stable across layers and retains sensitivity to both low- and high-frequency signals [2006.09661].

Weight initialization is critical for signal propagation: for layers $i = 1, \ldots, n-1$, set $W_i \sim \text{Uniform}(-\sqrt{6/M_i}, +\sqrt{6/M_i})$; for the first layer, $W_0 \sim \text{Uniform}(-\sqrt{6/M_0}/\omega_0, +\sqrt{6/M_0}/\omega_0)$. This stabilizes the variance of pre-activations and ensures sinusoids do not saturate or become degenerate, preserving the network’s representational bandwidth [2006.09661].

## 2. Theoretical Rationale and Frequency Properties

The use of periodic activation is motivated by the superior capacity of $ \sin(\cdot) $ to represent signals with rich local and global oscillatory content. Unlike ReLU or tanh nonlinearities that suppress or rapidly decay high-frequency components, the sine function admits nonzero derivatives of all orders and can represent arbitrarily oscillatory features.

Deeper SIRENs are theoretically analogous to deep Fourier synthesizers; stacking sine activations constructs increasingly complex superpositions of harmonics. Maintaining pre-activation variance at initialization (via principled weight scaling) is essential to preserve high-frequency expressivity and avoid frequency collapse through depth [2006.09661].

Closed-form derivatives (from automatic differentiation) make SIREN architectures directly suitable for learning solutions and derivatives of PDEs. For example, for
$$
\varphi_i(u) = \sin(W_i u + b_i), \quad \frac{\partial \varphi_i}{\partial u} = \cos(W_i u + b_i) \cdot W_i,
$$
higher-order derivatives follow recursively. This analytic differentiability underpins applications from signal fitting to PDE-solving [2006.09661].

## 3. Variants: Frequency-Multiplied and Adaptive Initialization SIRENs

### FM-SIREN

FM-SIREN addresses the core limitation of original SIRENs: feature redundancy induced by a layer-wide fixed frequency multiplier $\omega_0$, leading to correlated, overlapping frequency responses across neurons. Drawing from Nyquist-Shannon sampling theory and the discrete sine transform (DST), FM-SIREN assigns to each neuron in a layer a neuron-specific multiplier,
$$
\omega_k = \frac{k}{M} f_{Nyquist}, \quad k = 0, \ldots, M-1, \quad f_{Nyquist} = \frac{f_s}{2},
$$
where $M$ is the number of neurons and $f_s$ the domain sampling rate. This set $\{\omega_k\}$ spans $[0, f_{Nyquist}]$, eliminating redundancy and ensuring near-orthogonality across features [2509.23438].

Empirically, this yields a 49–50% reduction in off-diagonal covariance of hidden features (quantified via the Frobenius norm of covariance matrices), outperforms baseline SIREN in PSNR and MSE across audio, image, 3D, and NeRF fitting, and does so at unchanged computational cost [2509.23438]. FM-SIREN matches or exceeds the performance of a deep (5-layer) SIREN with only 2 layers.

### WINNER (SIREN²)

SIREN exhibits spectral bias at initialization: pre-activations have low-frequency spectral support, impeding learning of signals with significant high-frequency content. When the target signal's spectral centroid lies outside the network's support, a "spectral bottleneck" arises: the network collapses to near-zero outputs for all frequencies.

WINNER (Weight Initialization with Noise for Neural Representations; SIREN²) injects adaptive Gaussian noise into the first two layers’ weights, with noise magnitude determined by the spectral centroid $\psi$ of the target signal ($\psi=2\sum_k k|\hat{y}(k)|/\sum_k |\hat{y}(k)|$). This increases pre-activation variance and widens spectral support without increasing trainable parameters:
$$
W^{(l)}_{jk} \leftarrow U(-c, c) + \eta_{jk}^{(l)},\quad \eta_{jk}^{(l)} \sim \mathcal{N}\left(0, \frac{s_l}{\omega_0}\right),
$$
with $s_0, s_1$ (noise scales) computed as functions of $\psi$.

Empirically, SIREN² achieves state-of-the-art PSNR in audio, image, and 3D fitting, outperforming both SIREN and parameter-intensive SIREN+RFF, and also enhances denoising in DIP/Noise2Self setups. The analytic effect on spectral energy is accurately predicted by $ \mathrm{Var} \approx 1 + (d \cdot s^2)/2 $ [2509.12980].

## 4. Key Applications and Performance

SIREN and its variants serve as implicit neural representations for a range of tasks:

- **Signal Fitting:** SIRENs and FM-SIRENs precisely reconstruct 1D audio, 2D images, and 3D shapes from coordinate–value pairs, achieving high PSNR and low MSE compared to non-periodic or PE-based MLPs [2006.09661, 2509.23438].
- **Physical PDE Solving:** SIRENs are directly applied to systems defined via derivatives, such as:
  - Poisson equation: via supervision only on gradients or Laplacians.
  - Eikonal equation: for signed distance function (SDF) learning with norm and normal constraints.
  - Helmholtz/wave equation: for learning complex wavefields and FWI velocity models [2006.09661].
- **Noise-Robust INR:** Appropriately limiting width, depth, or frequency scaling enables SIREN to function as a mesh-free, noise-robust regressor for pressure fields from image velocimetry in challenging, noisy environments—outperforming classical mesh-based alternatives [2501.17987].
- **Audio Compression:** Architecture variants (Siamese SIREN) share backbones but separate output heads, facilitating implicit audio compression and self-noise estimation with substantially reduced parameter counts and improved objective and perceptual metrics after quantization [2306.12957].
- **Registration and Multi-modal Fusion:** SIREN architectures have been extended to semantic-guided registration pipelines for multi-robot Gaussian Splatting maps and as collaborative transformer systems for text-to-audio generation [2502.06519, 2510.04577].

## 5. Training, Initialization, and Implementation Details

SIRENs require careful architectural configuration:

- **Depth/Width:** Depths of 5–10 layers and hidden widths of 256–512 neurons balance representational capacity and runtime [2006.09661].
- **Initialization:** Principled uniform initialization (as outlined above) is crucial to avoid vanishing/exploding gradients and to maintain frequency propagation.
- **Optimization:** Training is via Adam (default $\beta_1, \beta_2, \varepsilon$), with learning rates in $10^{-4}$–$10^{-3}$.
- **Loss Functions:** For regression, use MSE on value or derivatives; for PDE constraints, additional loss terms target gradients, Laplacians, or physical constraints.
- **Gradient Supervision:** For PDE tasks, losses on spatial derivatives or Laplacians are computed via autograd.
- **Sampling:** Coordinate minibatches are directly sampled from the input domain (e.g., spatial, spatiotemporal grids).

The frequency multipliers in FM-SIREN are computed once per layer (e.g., per audio sampling rate, image dimension, or voxel grid size) and stored as an $\mathcal{O}(M)$-sized vector per layer [2509.23438].

## 6. Comparative Empirical Results

Key empirical results for SIREN and variants include:

| Architecture       | Audio PSNR | Image PSNR | 3D IoU/PSNR | NeRF PSNR | Notes                                 |
|--------------------|------------|------------|-------------|-----------|---------------------------------------|
| SIREN              | 13.4–59.4  | 21.3–38.9  | 0.960       | 33.02     | Baseline, fixed $\omega_0$            |
| FM-SIREN           | 62.7       | 32.29      | 0.990       | 33.17     | 10–50% reduction in redundancy, faster|
| SIREN² (WINNER)    | 62.7–95.2  | 36.1–75.2  | 55 dB       | –         | Adaptive freq. support, state of art  |
| Siamese SIREN      | –          | –          | –           | –         | 40% fewer params, best audio metrics  |

SIREN²/WINNER matches or exceeds parameter-heavy SIREN+RFF, without increased optimizer or memory cost [2509.12980]. FM-SIREN matches deep classic SIRENs with only 2 layers, and in NeRF reconstruction yields 14% faster training at unchanged fidelity [2509.23438].

## 7. Limitations and Open Directions

- **Spectral Bottleneck:** Even with appropriate initialization and frequency allocation, SIREN’s ability to learn highest frequencies is fundamentally linked to architecture and initialization. WINNER can underfit extremely low-frequency content if noise scales are excessive [2509.12980].
- **Parameter Sensitivity:** FM-SIREN’s frequency allocation requires knowledge of the signal domain’s effective sampling rate. Incorrect estimation can degrade near-orthogonality of hidden features [2509.23438].
- **Hyperparameter Tuning:** Frequency multipliers in vanilla SIREN ($\omega_0$), initialization constants, or adaptive noise scales ($s_0, s_1$) are task/domain-specific. Porting to new signal domains may require retuning.
- **Generalization to Other Periodic Activations:** Research suggests extending adaptive initialization and neuron-specific frequency selection to broader classes of periodic activations and positional encoding schemes is a promising avenue [2509.12980].
- **Integration with Hypernetworks:** SIREN functions parameterized by hypernetworks support priors over distribution families of signals but inherit any representational limitations of the core SIREN.

---

SIREN architectures and their recent extensions define a mathematically principled and empirically validated toolkit for high-fidelity implicit neural representation, with both theoretical underpinnings grounded in harmonic analysis and practical advances in frequency diversity, initialization, compression, and robustness [2006.09661, 2509.23438, 2509.12980, 2501.17987, 2306.12957].

Source: https://www.emergentmind.com/topics/siren-architecture