Implicit Neural Representations (INR)

• Implicit Neural Representations (INR) are neural function approximators that encode images, audio, and 3D geometry as continuous, resolution-agnostic functions.
• They leverage a structured dictionary view, connecting harmonic analysis and the neural tangent kernel to guide architectural tuning and mitigate aliasing artifacts.
• Meta-learning techniques refine the NTK eigenstructure of INRs, enabling accelerated convergence and improved generalization for signal-specific adaptations.

Implicit Neural Representations (INR) are a class of neural function approximators in which data such as images, audio, or 3D geometry are encoded as continuous functions parameterized by neural networks—most commonly as multilayer perceptrons (MLPs). Unlike classical discrete data representations (e.g., pixels, voxels, or point clouds), INRs enable resolution-agnostic, compact, and fully differentiable descriptions of arbitrary signals. The theoretical and empirical paper of INRs has revealed deep connections to harmonic analysis, kernel methods, nonlinear dictionary learning, and advances in network architecture and optimization. Recent works have systematically characterized the expressivity, training behavior, and domain-specific adaptations of INRs, emphasizing both their flexibility and limitations.

1. Structured Dictionary Interpretation and Spectral Properties

A foundational insight is that many INR architectures can be interpreted as parameterizing a structured dictionary of signal atoms, where these atoms are sinusoids at frequencies given by integer harmonics of the initial input mapping. For an INR with an input mapping γ: ℝᴰ → ℝᵀ (e.g., a Fourier feature mapping γ(r) = sin(Ωr + φ)) followed by an MLP with polynomial or sinusoidal activations, compositional nonlinearities generate new frequencies as integer combinations of the mapped basis. The resulting output function can be expressed as

$$f(r) = \sum_{\omega' \in \mathcal{H}(\Omega)} c_{\omega'} \sin(\langle \omega', r \rangle + \phi_{\omega'}),$$

where the set $$\mathcal{H}(\Omega) \subseteq \{\sum_t s_t \Omega_t\,|\,s_t \in \mathbb{Z}, \sum_t |s_t| \leq K^{L-1}\}$$ encodes all integer harmonic combinations up to a complexity governed by network depth $L$ and the nonlinearity order $K$ (Yüce et al., 2021). This structured dictionary view generalizes to networks with e.g. periodic activations (SIREN) or Fourier feature encodings, providing a principled framework for reasoning about the spectral capacity, effective basis support, and reconstruction limits of INRs.

The dictionary structure imposes that only signals compatible with these harmonics, as determined by hyperparameters such as the frequency factor or encoding bandwidth, can be represented without aliasing or artifacts. Consequently, architectural design becomes an exercise in tuning these dictionary atoms to align with the frequency content of the target signal, and failure to do so can induce observable degradation.

2. Neural Tangent Kernel and Inductive Bias

INR training dynamics are closely tied to the empirical neural tangent kernel (NTK), which describes the local linearization of the network about its parameters. The NTK at initialization

$$\Theta(r_1, r_2) = \langle \nabla_\theta f_{\theta_0}(r_1), \nabla_\theta f_{\theta_0}(r_2) \rangle$$

admits a spectral decomposition into eigenfunctions $\{\phi_i\}$ and associated eigenvalues $\{\lambda_i\}$, which function as the ‘atoms’ of the NTK-induced dictionary (Yüce et al., 2021). Learning efficiency and reconstruction quality for a given target $g$ are determined by the alignment $\langle \phi_i, g \rangle$ and the concentration of energy in directions with large $\lambda_i$.

The NTK formalism shows that the initialization and architecture of an INR determines a strong inductive bias, predetermining which frequency components are efficiently learned. This mechanism formalizes the empirical “spectral bias” of neural networks, wherein low-frequency components of a target function tend to be fitted before high-frequency features.

Additionally, $$\|g\|_\Theta^2 = \sum_i \lambda_i^{-1} |\langle \phi_i, g \rangle|^2$$ quantifies how the NTK eigenstructure affects sample complexity and learnability for different signal classes, explaining observed limits and failure modes in INR fitting.

3. Meta-Learning and Adaptation in NTK Dictionary

Meta-learning, particularly via model-agnostic meta-learning (MAML) and related algorithms, adaptively refines the NTK to produce a dictionary that is better aligned with the statistics of a task family. During meta-training, the eigenfunctions of the NTK are “reshaped” to more closely correspond with features shared across the distribution of signals (e.g., shared textures or semantic structures in a dataset) (Yüce et al., 2021). As a result, the initial state of the INR after meta-training already contains atoms that can efficiently encode typical instances from the distribution, resulting in both accelerated convergence and improved generalization.

Formally, this is analogous to classic dictionary learning in signal processing, where an overcomplete basis is adapted to allow sparse and efficient representation of typical signals.

4. Architectural and Practical Consequences

The explicit understanding of INRs as integer-harmonic dictionaries enables techniques for diagnostic analysis and architectural tuning:

• The input mapping (determined by the frequency parameters of the Fourier feature mapping or SIREN scaling factor) must be chosen to adequately cover the frequency spectrum of the target signal. Insufficient spectral support results in missing features, while excessive support can cause aliasing.
• The analysis of the empirical NTK provides a data-driven means to optimize hyperparameters so that the dictionary atoms are well matched to target signals.
• Meta-learning is applicable not only as a tool for fast adaptation but as a principled architectural tuning mechanism—by learning an initialization, one can shape the top NTK eigenfunctions to span the most relevant signal subspaces in practice.
• For a given depth $L$ and nonlinearity order $K$, the effective dictionary is restricted to integer combinations with total weight at most $K^{L-1}$, constraining the expressivity with respect to both width and depth.

Key formulas, such as

$$f(r) = \sum_{\omega' \in \mathcal{H}(\Omega)} c_{\omega'} \sin(\langle \omega', r \rangle + \phi_{\omega'})$$

and

$$\rho(e^{j\omega r}) = \sum_k \alpha_k e^{jk\omega r},$$

make the underlying harmonic structure explicit, directly connecting network composition to signal dictionary construction.

5. Connections to Deep Learning Theory and Harmonic Analysis

The structured dictionary perspective unifies themes from classical signal processing—particularly harmonic analysis, spectral bias, and aliasing theory—with modern deep learning tools. The framework generalizes the analysis of spectral bias to arbitrary input encodings and nonlinearities, providing principled explanations for phenomena such as the tendency of wide MLPs to fit simple, smooth functions before higher-frequency features.

By highlighting the role of NTK eigenstructure even for finite-width networks, it bridges NTK theory in infinite-width regimes and practical expressivity in standard-sized neural architectures. Furthermore, it clarifies the relationship between network design (depth, width, encoding choice) and the resulting sample complexity, convergence rate, and ability to generalize to out-of-distribution signals.

Finally, by recasting meta-learning as a special case of nonlinear dictionary learning, this viewpoint motivates new strategies for rapid adaptation and transfer in INR-based models.

6. Real-World Implications and Design Guidelines

The unified harmonic/dictionary/NTK perspective informs both basic research and engineering of practical INR systems:

• Practitioners can directly analyze the induced dictionary to diagnose and debug fitting problems such as missing high-frequency content or unwanted artifacts.
• Network hyperparameters, such as the number of layers, the width, the nonlinearity complexity, and the frequency content of input encodings, can be tuned based on the required spectral range and expected sample complexity.
• Meta-learning procedures can be explicitly leveraged to shape model inductive biases, providing improved initializations for signal families with shared structure, and reducing both convergence time and the number of examples needed to achieve accurate representation.
• The framework also points toward hybrid design approaches where architectural choices and meta-learned initializations are optimized jointly for a target domain.

In application domains such as 3D scene representation and texture synthesis, the structured dictionary theory enables the construction of INRs that balance model capacity, sample efficiency, and data-driven adaptation.

7. Broader Theoretical Significance

This interpretive framework situates INRs within a lineage of signal representation paradigms—spanning wavelet dictionaries, kernel methods, and sparse coding—and extends their analysis beyond purely neural-centric views. It elucidates convergence properties and inductive biases in terms of familiar mathematical objects (harmonics, atoms, kernels). The approach motivates further paper of NTK dynamics in finite networks, and provides a conceptual roadmap for future methodological advances—such as the development of richer input mappings, principled meta-learning strategies, and architectures that more flexibly control the structure of the induced signal dictionary.

By connecting modern INR practice to classical analysis, these insights pave the way for principled design, tuning, and interpretation of implicit neural representations across domains.