High-Order Implicit Neural Representation

• HOIN is a neural representation paradigm that incorporates high-degree polynomial interactions to overcome limitations of conventional MLPs and enhance detailed signal modeling.
• Architectural realizations such as Poly-INR, high-order blocks, split-layer MLPs, and Sobolev-trained backbones address optimization bottlenecks and reduce spectral bias.
• Empirical results demonstrate that HOINs achieve state-of-the-art performance in generative modeling, superresolution, and inverse problems with improved efficiency and expressivity.

High-Order Implicit Neural Representation (HOIN) generalizes implicit neural representations (INRs) by architecturally and functionally enabling the efficient encoding and learning of high-degree polynomial interactions between coordinates and features. This paradigm provides INRs with substantially greater expressivity, enhanced capacity for fine detail, and improved data efficiency compared to conventional multilayer perceptrons (MLPs) equipped solely with linear or sinusoidal activations. HOINs appear in multiple architectural forms, including polynomial modulated MLPs, explicit interaction blocks, split-layers, and Sobolev-trained backbones, each designed to overcome the representational and optimization bottlenecks of standard INRs. Empirical results demonstrate their state-of-the-art performance on tasks spanning generative modeling, signal superresolution, inverse problems, and geometric representation.

1. Conceptual Basis and Formal Definition

An implicit neural representation is a parameterized function $f_\theta: \mathbb{R}^d \rightarrow \mathbb{R}^c$, mapping continuous input coordinates to signal values. Standard coordinate-based MLPs can approximate low-frequency content but are structurally biased toward overly smooth, low-detail reconstructions. High-Order Implicit Neural Representations extend this classic class by construction: their architectures explicitly realize or learn higher-degree monomials of the input coordinates, enabling, for instance, the realization of terms such as $x^p y^q$ for $p+q$ as high as desired given sufficient depth and feature interaction.

Canonical examples include polynomial functions whose coefficients are parameterized by neural networks: $G(x,y) = \sum_{p+q \leq P} g_{pq}(z) x^p y^q$ where the representational capacity scales with polynomial degree $P$ and the learned coefficients $g_{pq}(z)$ may depend on auxiliary latent variables or conditioning signals.

2. Architectural Realizations

Multiple families of HOIN architectures have been introduced, each instantiating the principle of high-order interaction differently.

a. Polynomial Implicit Neural Representation (Poly-INR)

Poly-INR uses a cascade of fully-connected layers, where after each activation, the feature representation is modulated by an affine-transformed coordinate, raising the degree of polynomial interaction iteratively. Each layer effectively enables the network to realize all monomials up to a degree equal to the network’s depth. The mapping network generates all coordinate modulation parameters, and the synthesis network updates the hidden state $h^{(l)}$ as: $$h^{(l)} \leftarrow h^{(l)} \odot (A^{(l)} x + c^{(l)})$$ Stacking $L$ such layers yields a representation capable of modeling degree $L$ polynomials without convolution, normalization, or self-attention layers, as in Poly-INR (Singh et al., 2023).

b. High-Order Neural Blocks

General HOINs replace basic MLP blocks with blocks that concatenate and linearly combine higher-degree monomials of their previous activations: $$h^{(l)} = \phi \left( W^{(l)} [h^{(l-1)}, (h^{(l-1)})^{\odot2}, ..., (h^{(l-1)})^{\odot k}] + b^{(l)} \right)$$ where $(\cdot)^{\odot k}$ denotes the Hadamard (element-wise) $k$-th power and $k$ controls the interaction order (Chen et al., 2024). Blockwise polynomial expansion accelerates frequency fitting and reduces spectral bias.

c. Split-Layer MLPs

The split-layer architecture divides each MLP layer into $N$ parallel branches whose outputs are fused via the Hadamard product. For layer width $C$, each branch has width $C/\sqrt{N}$, and the output is: $$z^{(l)} = (W_1^{(l)} z^{(l-1)}) \odot \cdots \odot (W_N^{(l)} z^{(l-1)})$$ This construction realizes an $N$-th degree polynomial mapping, expanding the feature space to a combinatorial basis of $\binom{C/\sqrt{N} + N -1}{N}$. This form enables exponentially increased representational power per parameter over naïve MLP widening (Cai et al., 13 Nov 2025).

d. Sobolev-Supervised INRs

Sobolev training augments standard loss with supervision on (finite-difference approximated) derivatives: $$\mathcal{L}_{\rm Sobolev} = \mathcal{L}_{\rm val} + \lambda \mathcal{L}_{\rm der}$$ Concretely, this incorporates both pixelwise and (approximate) gradient constraints into training, thereby encouraging the MLP to represent target signals in a higher-order, derivative-sensitive function space. This approach bridges value-based and high-order INRs at the training level (Yuan et al., 2022).

3. Mathematical Properties and Expressivity

HOINs derive their enhanced expressivity from their ability to represent a polynomial spectrum of monomials in coordinates and features. In $L$-layer, $k$-order HOINs, the function realized is a general homogeneous polynomial of degree up to $k^L$, in contrast to the linear scaling of effective degree with depth in standard MLPs. The derivative structure also differs fundamentally: classic ReLU or sine-MLPs have vanishing higher derivatives almost everywhere, restricting their capacity for high-frequency content. In contrast, each high-order block in a HOIN yields nonzero higher derivatives, saturating the function space with richer variations.

Theoretical analysis (e.g., (Chen et al., 2024)) proves that an HOIN with activation leading term degree $r$ and depth $L$ yields a $(2r)^{L-1}$-dimensional monomial basis, compared to $r^{L-1}$ in plain MLPs.

4. Training Dynamics and Neural Tangent Kernel

HOINs alter the optimization landscape by improving Neural Tangent Kernel (NTK) conditioning. The NTK matrix $K_{\rm NTK}(x, x')$ governs how quickly different signal components are learned during gradient descent. HOINs empirically exhibit a heavier diagonal and larger tail of eigenvalues in their NTK spectra compared to standard architectures, leading to (i) reduced spectral bias, (ii) more rapid, balanced fitting of high- and low-frequency signal constituents, and (iii) improved overall convergence rates (Chen et al., 2024, Cai et al., 13 Nov 2025). For split-layer and polynomial modulated networks, the expansion of feature directions directly maps to a broadened NTK spectrum.

5. Empirical Performance and Benchmark Results

Empirical results establish the effectiveness of HOINs for diverse signal and inverse problems. In large-scale generative modeling, Poly-INR matches or exceeds the quality of state-of-the-art GANs with 3–4× fewer parameters. For instance, on ImageNet $128 \times 128$, Poly-INR achieves $\operatorname{FID}=2.08$, $\operatorname{IS}=179.6$ with $\sim 46$M parameters, compared to StyleGAN-XL's $\operatorname{FID}=1.81$, $\operatorname{IS}=200.6$, and $\sim 158.7$M parameters (Singh et al., 2023). On FFHQ $256\times 256$, Poly-INR closes the gap with StyleGAN-XL while outperforming StyleGAN2.

For image representation, denoising, superresolution, CT reconstruction, and inpainting, HOINs (high-order blocks or split-layers) yield systematic PSNR gains of $1$–$3\,\rm dB$ and 5–30% faster convergence, outperforming SIREN, WIRE, INCODE, and other contemporary INR models (Chen et al., 2024, Cai et al., 13 Nov 2025). Notably, split-layer HOINs provide exponential enlargement of function space per parameter and boost task scores (image fitting PSNR, CT reconstruction PSNR, Chamfer distance for 3D shapes) by substantial factors with compute fixed (Cai et al., 13 Nov 2025).

The following table summarizes selected empirical improvements:

Task	Baseline	HOIN Variant	Performance Gain
ImageNet Gen. (FID)	StyleGAN-XL: 1.81	Poly-INR: 2.08	Comparable at 3× less param. (Singh et al., 2023)
Image Denoising (PSNR)	SIREN: 31.1 dB	HO-SIREN: 32.3 dB	+1.2 dB (Chen et al., 2024)
2D Img Fit (PSNR)	ReLU-MLP: 21.24	Split-Layer: 30.89	+45.4% (Cai et al., 13 Nov 2025)
3D Chamfer Error	ReLU: 1e-4	Split-Layer: 2e-5	–80% (Cai et al., 13 Nov 2025)

Poly-INR also delivers continuous extrapolation outside $[0,1]^2$, arbitrary upsampling, and smooth latent interpolation.

6. Training and Application Considerations

HOINs have both strengths and limitations. Their parameter efficiency enables deployment on large, diverse datasets and inherently supports continuous/extrapolatory sampling in resolution or domain. However, their per-coordinate MLP evaluation incurs computational overhead proportional to output resolution, in contrast to convolutional architectures. The lack of spatial inductive bias may weaken global context encoding in generative adversarial settings and relies on complementary discriminators for compensation (Singh et al., 2023).

Sobolev-trained INRs, an alternative route to HOIN, improve sharpness, data efficiency, and generalization by incorporating derivative supervision, but at the cost of higher memory and compute usage due to the need to compute per-sample gradients (Yuan et al., 2022).

7. Broader Implications and Directions

The HOIN paradigm offers a universal plug-in to existing INR architectures, expanding their algebraic function space and improving both trainability and expressivity. Potential extensions include adaptive selection of polynomial order per block or feature, integration with attention or low-rank structures, and targeted control of high-order interaction depth. A plausible implication is the application of HOINs to 3D neural rendering, neural radiance fields, PDE-based inverse problems, and continuous geometric modeling.

Current research highlights the twin benefits of HOINs: (a) enrichment of the representational hypothesis class, and (b) more favorable optimization properties as mediated by improved NTK spectra. The trade-off between high-order expressivity and amplification of noise or overfitting in certain tasks remains an open area for investigation (Chen et al., 2024). Extensions toward direct learning of higher-order derivatives—beyond first-order Sobolev constraints—represent another avenue grounded in the HOIN principle (Yuan et al., 2022).