Neural Spectral Transport Representation (NSTR)

• NSTR is an implicit neural representation that factors signals into a local spectrum field and a frequency transport PDE to capture spatially varying frequency content.
• It efficiently modulates global sinusoidal bases with local amplitudes, leading to sharper reconstructions and reduced parameter counts.
• Empirical evaluations demonstrate significant improvements in image PSNR, audio SNR, and 3D shape fidelity compared to traditional INR methods.

Neural Spectral Transport Representation (NSTR) is an implicit neural representation (INR) framework that explicitly models spatially varying local frequency content in signals such as images, audio, and implicit 3D geometry. In contrast to conventional INR architectures, which assume a global and stationary spectral basis, NSTR factorizes signal representation into a learnable local spectrum field and a frequency transport partial differential equation (PDE), enabling adaptive, interpretable, and efficient modeling of space-varying frequencies (Versace, 23 Nov 2025).

1. Problem Context and Motivation

Traditional implicit neural representations, such as multi-layer perceptrons (MLPs) with Fourier features, SIREN, and multiresolution hash grids, assume the signal can be universally decomposed onto a fixed global frequency basis applied uniformly across all spatial locations. Specifically, SIREN employs a spatially invariant frequency scale, Fourier-feature embeddings use a static set of frequency vectors applied everywhere, and hash grids provide local features without directly encoding spatial frequency variation. However, real-world signals display complex frequency structure, with high-frequency transitions, localized harmonics, and smooth regions intermixed, leading to a mismatch between signal statistics and a global stationary basis. Fixed-frequency approaches tend to underfit localized high-frequency details or over-parameterize smooth zones, highlighting the need for explicit models of spatial spectral variation (Versace, 23 Nov 2025).

2. Core Formulation and Signal Decomposition

NSTR addresses these deficiencies by representing each spatial coordinate $x \in \mathbb{R}^d$ by a local spectrum field:

• Local Spectrum Field $S(x)$: A vector $S(x) = [S_1(x), ..., S_K(x)]^\top \in \mathbb{R}^K$ encodes the amplitude or activation of $K$ global sinusoidal bases at location $x$. $K$ is kept small (e.g., $K=8$–$16$).
• Signal Decoding: The scalar (or vector) signal at $x$ is computed by spatially modulating the sum of global sinusoidal bases with local amplitude $S(x)$:

$$u(x) = \sum_{i=1}^K S_i(x)\,\sin(\omega_i^\top x + b_i)$$

$f(x) = g_\phi(u(x))$

where $g_\phi$ is a shallow MLP decoder and $\{\omega_i, b_i\}$ are global learnable frequencies and phase offsets, respectively.

Crucially, NSTR enforces structure on $S(x)$ through a frequency transport PDE, constraining its evolution across space.

3. Frequency Transport PDE and Learning Framework

The key innovation in NSTR is a learnable frequency transport law—a neural PDE describing how the local spectrum varies smoothly and flexibly throughout the domain. The PDE is enforced as:

$\nabla S(x) \approx F_\theta(x, S(x))$

Here, $$F_\theta : \mathbb{R}^d \times \mathbb{R}^K \to \mathbb{R}^K$$ is a neural network (the "frequency transport network") predicting the spatial derivative of the spectrum at $x$, conditioning on both $x$ and its local spectrum.

• The corresponding soft constraint loss is:

$$\mathcal{L}_{\mathrm{PDE}} = \mathbb{E}_x \| \nabla S(x) - F_\theta(x, S(x)) \|_2^2$$

• The full loss includes the task loss (e.g., MSE for image or audio signal), PDE loss, and a smoothness regularizer:

$$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda_{\mathrm{PDE}}\,\mathcal{L}_{\mathrm{PDE}} + \lambda_{\text{smooth}}\,\mathbb{E}_x \|\nabla S(x)\|_2^2$$

This formalism enables the local spectrum to drift, stretch, and transition throughout space—capturing edges, texture boundaries, and non-stationary frequency phenomena inherent in real-world data.

4. Architecture and Parameterization

NSTR’s architecture consists of three learnable modules and a set of global frequencies:

Component	Role	Parameterization
Spectrum Field $S(x)$	Encodes local frequency composition	Learnable grid + MLP $H_\psi$
Transport Network	Predicts spatial spectrum change	2-layer MLP, width 64
Decoder $g_\phi$	Maps modulated sum to signal output	2-3 layer MLP, width 64
$\{\omega_i\}$	Global sinusoidal frequencies	Learnable, $K=8$–$16$

• The spectrum field uses a coarse learnable grid $Z$ ($\mathbb{R}^{R \times \ldots \times R \times C}$), tri-linear interpolated to $z(x)$, then fused with $x$ and processed by a small MLP ($H_\psi$).
• The frequency transport network $F_\theta$ takes concatenated $x$ and $S(x)$, predicts the spectrum gradient.
• The signal decoder is typically a shallow MLP suited to the output dimensionality of the target signal.
• Global frequencies are typically initialized log-uniformly, then optimized during training; the number of bases $K$ is far smaller than in baseline methods.

5. Training Protocol

Optimization follows standard regimes:

• Optimizer: Adam with learning rate $1 \times 10^{-4}$.
• Batching: $4$k–$16$k randomly sampled coordinates $x$ per step.
• Iterations: $20$k–$50$k, dataset-dependent.
• Loss Weights: $\lambda_{\mathrm{PDE}} = 0.1$, $\lambda_{\text{smooth}} = 1 \times 10^{-3}$.
• Precision: Automatic mixed precision; no special gradient clipping required.
• Sampling of $x$ is uniform and baseline-matched.

6. Empirical Evaluation and Analysis

6.1 Benchmark Performance

NSTR is evaluated on 2D image regression (including CelebA-HQ, procedural textures), 1D audio reconstruction at 44.1 kHz, implicit signed distance function (SDF) geometry (ShapeNet), and NeRF small scenes. Key baselines include SIREN, Fourier-feature MLPs, Instant-NGP, and dense/factorized NeRF variants.

Summarized Results:

Model	Params	Image PSNR (dB)	Audio SNR Δ	SDF Chamfer ↓	NeRF Params ↓ / Speed ↑
Fourier MLP	1.2M	30.1	—	—	—
SIREN	1.2M	31.4	—	Baseline	—
Instant-NGP	0.5M	33.5	—	—	0.3×
NSTR	0.3M	35.7	+3.5 dB	↓28–42%	2–4× params, 1.5× speed

• Images: Sharper edges, minimal artifacts, superior PSNR at lower parameter count.
• Audio: SNR gain of +3.5 dB over SIREN, clean tracking of pitch sweeps without spectral leakage.
• SDF Geometry: Chamfer distance improved by 28–42% over SIREN-DeepSDF; normals are more consistent at corners and creases.
• NeRF: Parametric and speed advantages, with matched or improved PSNR.

6.2 Qualitative Structure and Visualization

The local spectrum field $S(x)$, its Jacobian $\nabla S(x)$, and the predicted flow $F_\theta(x, S(x))$ give rise to visualizations that reveal coherent, interpretable frequency flows, delineating edges, texture transitions, and smooth signal regions. These quantitative and qualitative indicators confirm that the explicit space-varying spectrum captures non-stationary and heterogeneous signal regions more accurately than all baselines.

7. Interpretability, Ablation, and Limitations

The explicit structure afforded by the learned $S(x)$ and its transport PDE enables direct field visualization: $S(x)$ can be decomposed into $K$ scalar fields, their gradients and flows indicate local frequency modulation, and critical structural loci in signals correspond to magnitude and direction changes. Ablation studies show that omitting the PDE loss ($\lambda_{\mathrm{PDE}} = 0$) induces instability and noisier spectra, while varying $K$ demonstrates competitive results even for small $K$ ($K=8$), with little gain at higher $K$. Decoder depth was found to be non-limiting, as expressive power is concentrated in the spectrum modulation.

Some limitations are identified: the frequency transport PDE is enforced as a residual rather than an explicitly integrated dynamic; the learnable grid component becomes memory-intensive for high ambient dimensions; and it is assumed that local frequencies are represented within the convex hull spanned by the global bases.

8. Extension Opportunities and Research Trajectory

The NSTR paradigm suggests broader applicability and potential avenues for development, including modeling multi-modal or spatio-temporal signals (e.g., video), extensions to learnable anisotropic (directional) transport, and operator-based generalization for instance-level tasks. A plausible implication is that regularizing local frequency evolution through PDEs, rather than constraining to global bases, will supersede the traditional approach and yield a fertile area for new INR research (Versace, 23 Nov 2025).