Sinusoidal Index Encoding

• Sinusoidal index encoding is a technique that maps spatial or temporal indices into high-dimensional vectors using sinusoids with varying frequencies, phases, and amplitudes.
• It employs principled initialization and spectral sampling to control frequency representation, ensuring stable training and effective reconstruction of structured signals.
• The approach leverages amplitude decay bounds and adaptive clamping to suppress high-frequency artifacts, enabling efficient learning in implicit neural representations.

Sinusoidal index encoding is a principled strategy for mapping continuous or discrete indices (such as spatial positions, temporal steps, or other ordinal variables) into high-dimensional vectors using sinusoids of varying frequencies, phases, and amplitudes. These encodings serve as domain-adapted features that facilitate the learning and reconstruction of structured signals—such as periodic, almost-periodic, or bandlimited functions—in machine learning architectures, particularly in implicit neural representations (INRs) and sequence models. Sinusoidal index encodings generalize the concept of fixed-frequency positional encoding and are distinguished by rigorous theoretical underpinnings regarding spectral expressivity, compositional depth, and robust training protocols.

1. Theoretical Foundation: Amplitude–Phase Expansion

Each neuron in a sinusoidal neural network can be exactly rewritten as a weighted superposition of sinusoids, where the output's frequency content grows exponentially with network depth via integer combinations of the input frequencies. Consider a one-hidden-layer architecture: $f(x) = L \circ S \circ D(x)$ where:

• $D(x) = \sin(\Omega x + \phi) \in \mathbb{R}^m$ is the “input layer” with frequencies $\Omega = [\omega_1, ..., \omega_m]^T$ and phase shifts $\phi \in \mathbb{R}^m$,
• $S(z) = \sin(Wz + b) \in \mathbb{R}^n$ is a hidden sinusoidal layer ($W \in \mathbb{R}^{n \times m}$, $b \in \mathbb{R}^n$),
• $L(y) = Cy + d$ is linear output.

Theorem 1 (Amplitude–Phase Expansion):

For each hidden neuron $h_i(x)$,

$$h_i(x) = \sum_{k \in \mathbb{Z}^m} \alpha_k \sin(\beta_k x + \lambda_k)$$

where the frequency, phase and amplitude are:

• $D(x) = \sin(\Omega x + \phi) \in \mathbb{R}^m$0
• $D(x) = \sin(\Omega x + \phi) \in \mathbb{R}^m$1
• $D(x) = \sin(\Omega x + \phi) \in \mathbb{R}^m$2, with $D(x) = \sin(\Omega x + \phi) \in \mathbb{R}^m$3 the $D(x) = \sin(\Omega x + \phi) \in \mathbb{R}^m$4-th Bessel function of the first kind.

In practical implementations, summation is truncated at a multi-index norm $D(x) = \sin(\Omega x + \phi) \in \mathbb{R}^m$5, but even for moderate weight magnitude, a single neuron can generate a rich spectrum of frequencies through layer composition (Novello et al., 2024).

2. Spectral Structure and Expressivity

Layer stacking does not generate genuinely new “off-grid” frequencies; all output frequencies of an L-layer sinusoidal MLP are integer linear combinations of the original input frequency set $D(x) = \sin(\Omega x + \phi) \in \mathbb{R}^m$6. This spectral support is exactly the $D(x) = \sin(\Omega x + \phi) \in \mathbb{R}^m$7-span of the input frequencies: $D(x) = \sin(\Omega x + \phi) \in \mathbb{R}^m$8 Amplitude at each frequency is a product of Bessel functions, drawing from every depth-wise weight. Thus, spectral design is controlled solely via choice and initialization of the input frequency vectors (Novello et al., 2024).

3. Initialization: Spectral Sampling and Bandlimit Control

Principled initialization of the frequency set is central for both stability and expressivity. The recommended procedure is:

• Fix a global period $D(x) = \sin(\Omega x + \phi) \in \mathbb{R}^m$9 (commonly $\Omega = [\omega_1, ..., \omega_m]^T$0 if the data domain is $\Omega = [\omega_1, ..., \omega_m]^T$1), ensuring $\Omega = [\omega_1, ..., \omega_m]^T$2-periodicity.
• Select input frequencies as $\Omega = [\omega_1, ..., \omega_m]^T$3. Freezing $\Omega = [\omega_1, ..., \omega_m]^T$4 during training maintains strict periodicity.
• Uniformly sample $\Omega = [\omega_1, ..., \omega_m]^T$5 from a cube $\Omega = [\omega_1, ..., \omega_m]^T$6, with $\Omega = [\omega_1, ..., \omega_m]^T$7 and $\Omega = [\omega_1, ..., \omega_m]^T$8 the target Nyquist bandlimit. Empirical recommendations set $\Omega = [\omega_1, ..., \omega_m]^T$9.
• Always include basic frequency vectors (e.g., $\phi \in \mathbb{R}^m$0, $\phi \in \mathbb{R}^m$1, etc.) in $\phi \in \mathbb{R}^m$2 to prevent spurious sub-periodicity.
• Initialize phases as $\phi \in \mathbb{R}^m$3.
• For higher control, split the sampled frequency domain into a low-frequency “core” and a high-frequency “shell,” oversampling the core to ensure reconstruction quality near the origin (Novello et al., 2024).

4. Spectral Bounding During Training

Unchecked parameter growth in deeper layers can induce pathological high-frequency artifacts and overfitting. Amplitudes for high-order harmonics decay super-factorially under bounded weights:

Theorem 2 (Amplitude Decay Bound):

$\phi \in \mathbb{R}^m$4

If $\phi \in \mathbb{R}^m$5, higher-order frequency components are suppressed. Two robust spectral bounding schemes are employed:

• Fixed-clamp: Clamp each hidden weight $\phi \in \mathbb{R}^m$6 at each epoch into an interval $\phi \in \mathbb{R}^m$7 with $\phi \in \mathbb{R}^m$8. Typically use a higher bound $\phi \in \mathbb{R}^m$9 for low-frequency columns and lower $S(z) = \sin(Wz + b) \in \mathbb{R}^n$0 for high-frequency columns.
• Adaptive-bounds: Use parameterized scaling $S(z) = \sin(Wz + b) \in \mathbb{R}^n$1 and penalize the bounds with $S(z) = \sin(Wz + b) \in \mathbb{R}^n$2, letting the network learn only the necessary frequency budget but still ensuring spectral decay (Novello et al., 2024).

5. Construction and Workflow for Sinusoidal Index Encoding

A canonical recipe for sinusoidal index encoding is as follows:

1. Declare the global period $S(z) = \sin(Wz + b) \in \mathbb{R}^n$3 and the Nyquist bandlimit $S(z) = \sin(Wz + b) \in \mathbb{R}^n$4.
2. Partition $S(z) = \sin(Wz + b) \in \mathbb{R}^n$5 into a low-frequency core and outer shell, select $S(z) = \sin(Wz + b) \in \mathbb{R}^n$6.
3. Sample $S(z) = \sin(Wz + b) \in \mathbb{R}^n$7 frequency vectors $S(z) = \sin(Wz + b) \in \mathbb{R}^n$8 uniformly from both core and shell; include standard basis vectors; freeze $S(z) = \sin(Wz + b) \in \mathbb{R}^n$9.
4. Sample $W \in \mathbb{R}^{n \times m}$0 randomly.
5. Initialize hidden-layer weights $W \in \mathbb{R}^{n \times m}$1 from Gaussian $W \in \mathbb{R}^{n \times m}$2, using suitable values for $W \in \mathbb{R}^{n \times m}$3.
6. Train the network with frequent weight clamping or adaptive bounds, maintaining $W \in \mathbb{R}^{n \times m}$4 or per-column thresholds.
7. Optionally apply regularization on $W \in \mathbb{R}^{n \times m}$5 to discourage excessive high-frequency amplification (Novello et al., 2024).

Empirical findings indicate that one hidden layer, leveraging the “frequency factory” property, suffices for high-fidelity reconstruction, reducing parameter count and training time relative to shallow, wide networks.

6. Stability, Convergence, and Generalization

Sinusoidal index encoding demonstrates the following practical properties:

• Stability: Freezing the input frequencies and bounding hidden weights precludes chaotic behavior of deep sinusoidal networks.
• Convergent Training: Small initial weights bias the network to fit lower frequencies first, reflecting the spectral concentration of typical natural signals.
• Noise Suppression and Overfitting Mitigation: Soft spectral filtering via amplitude decay bounds achieves superior control relative to hard cutoffs (filter banks), mitigating ringing artifacts and suppressing spurious harmonics.
• Sample Efficiency: The exponential proliferation of integer-combination frequencies enables full-bandwidth signal recovery even in single-hidden-layer settings (Novello et al., 2024).

7. Relationship to Other Sinusoidal and Frequency-Based Encodings

Sinusoidal index encoding is fundamentally distinct from:

• Standard positional encodings: As used in Transformers, which employ geometrically spaced fixed frequencies. These encodings have a pronounced low-pass bias, leading to information loss in high-frequency ranges when the embedding dimension is finite (Idé et al., 2024, Lopez-Rubio et al., 22 Dec 2025).
• Learnable sinusoidal positional encodings: Which adapt frequency scales via gradient descent, offering greater flexibility at the cost of direct spectral control (Sun et al., 2024, Li et al., 2021).
• Alternative periodic encodings: Triangular, square, and sawtooth waveforms have been explored as replacements for sinusoids, preserving shift-invariance but introducing different inductive biases. All such mechanisms share elements of the spectral control strategy but differ in higher-order smoothness, artifact risk, and empirical performance (Lopez-Rubio et al., 22 Dec 2025).

In summary, sinusoidal index encoding provides a mathematically grounded mechanism for matching network spectral response to the bandlimit of the underlying data, ensuring robust coverage of the signal space and controlled generalization. Adhering to a recipe that fixes the spectrum at the input and clamps amplitude amplification in deeper layers permits stable, noise-resistant, and sample-efficient learning in high-capacity sinusoidal neural architectures (Novello et al., 2024).