---
title: NTK-aware Interpolation in Neural Representations
url: https://www.emergentmind.com/topics/ntk-aware-interpolation
type: topic
---

# NTK-aware Interpolation in Neural Representations

NTK-aware interpolation refers to a principled approach for designing and training implicit neural representations (INRs) with explicit control over the conditioning and eigenvalue spectrum of their Neural Tangent Kernel (NTK). By decomposing the NTK eigenvalue variance into interpretable architectural and parametric factors, NTK-aware methods enable the suppression of spectral bias and acceleration of convergence, particularly for high-frequency signal recovery tasks. This methodology underpins a unified understanding of how positional encoding, spherical normalization, and Hadamard modulation each contribute to improved NTK conditioning in coordinate-based MLPs [2512.15169].

## 1. NTK Formalism and Eigenvalue-Variance Decomposition

For a two-layer coordinate MLP, the output is
$$
f(x; W) = \frac{a}{\sqrt{m}} \sum_{r=1}^m \sigma(w_r \cdot \phi(x)),
$$
with $\phi(x) \in \mathbb{R}^d$ denoting a fixed positional encoding, $\sigma$ a pointwise ReLU, and $a$ the readout scale. The NTK at initialization is the $n\times n$ Gram matrix
$$
H_{ij} = \frac{a^2}{m} \rho_{ij} \langle t_i, t_j \rangle,
$$
where $\rho_{ij} = \phi(x_i)^\top \phi(x_j)$ and $t_i = s_i \odot p_i$, with $s_i$ encoding ReLU gating and spherical normalization and $p_i$ capturing Hadamard modulation.

Defining $\lambda_1, \dots, \lambda_n$ as $H$’s eigenvalues, the mean and variance are
$$
\mu_\lambda = \frac{1}{n} \operatorname{Tr} H,\qquad v_\lambda = \frac{1}{n} \operatorname{Tr}(H^2) - \mu_\lambda^2.
$$
Under mild regularity (bounded input norm $R_x$, hidden energy $\bar{S}$, modulation scale $\bar{P}$, self-similarities ≈ 1), the variance admits the proxy
$$
v_\lambda \approx \frac{a^4 R_x^4 \bar{S}^2 \bar{P}^2}{n m^2} \sum_{i \ne j}\tau_{x,ij} \, \tau_{s,ij} \, \tau_{p,ij} \, \tau_{q,ij},
$$
where the similarity factors are:
- $\tau_{x,ij} = \frac{\rho_{ij}^2}{\rho_{ii} \rho_{jj}}$ (input similarity),
- $\tau_{s,ij} = \frac{||s_i \odot s_j||^2}{||s_i||^2 ||s_j||^2}$ (hidden gating/normalization),
- $\tau_{p,ij} = \frac{||p_i \odot p_j||^2}{||p_i||^2 ||p_j||^2}$ (modulation similarity),
- $\tau_{q,ij} = \cos^2\angle(s_i \odot s_j, p_i \odot p_j)$ (coupling).

This decomposition supports design-time and runtime diagnosis of NTK spectrum flatness and spectral bias in INR architectures [2512.15169].

## 2. Architectural Levers: Impact on NTK Similarity Masses

**Positional Encoding (PE):** Utilizing Fourier-feature encodings $\phi(x) = \sqrt{2/d} [\cos(2\pi B x); \sin(2\pi B x)]$, with $B \sim N(0, \sigma^2)$, reshapes input similarity $\tau_{x,ij}$. Lemma 3.1 shows
$$
\mathbb{E}_B \left[ \frac{1}{n(n-1)} \sum_{i \ne j} \tau_{x,ij} \right] \leq \frac{1}{n(n-1)} \sum_{i \ne j} \tau_{x,ij}^{\mathrm{bl}},
$$
and as $\sigma \to \infty$, this mass approaches $1/d \ll 1$ (baseline). By monotonicity, shrinking off-diagonal $\tau_{x,ij}$ directly lowers $v_\lambda$ (Corollary C.2).

**Spherical Normalization (SP):** Introducing $s_i = \sigma(W y_{i-1}) / ||\sigma(W y_{i-1})||_2$ enforces $||s_i||_2 \equiv 1$ ($\bar{S} \approx 1$ vs $\bar{S}_{\mathrm{bl}} \approx m/2$ for standard ReLU MLP), reducing the energy factor approximately by $\bar{S}^2 / \bar{S}_{\mathrm{bl}}^2 \ll 1$ (Corollary 3.4, C.3). The Top-K variant further contracts energy-weighted hidden similarity almost quadratically in $K/m$ (Theorem D.7).

**Hadamard Modulation:** With coefficient $c_r = a_r p_r$, $a_r = \pm a$, and $p_r$ modulation, the factors $\tau_{p,ij}, \tau_{q,ij} < 1$ off-diagonal. Any nontrivial modulation with $\tau_{p,ij}\tau_{q,ij}<1$ for $i \ne j$ further multiplies down $\sum\tau_{x,ij}\tau_{s,ij}$, delivering additional variance reduction (Corollaries 3.5, C.4).

| Mechanism             | Dominant Factor(s) Reduced | Variance Impact                                      |
|-----------------------|---------------------------|------------------------------------------------------|
| Positional Encoding   | $\tau_{x,ij}$             | Mass approaches $1/d$, reducing $v_\lambda$          |
| Spherical Normalization | $\bar{S}$, $\tau_{s,ij}$   | Energy factor contracts by $\ll 1$ vs baseline       |
| Hadamard Modulation   | $\tau_{p,ij}$, $\tau_{q,ij}$ | Multiplies down variance by $<1$ for $i \neq j$      |

## 3. Unified Interpretation and Spectral Bias Mitigation

Each architectural mechanism shrinks one or more of the similarity and scaling factors, contracting the overall NTK variance multiplicatively:
$$
v_\lambda \propto (\text{input mass}) \times (\text{hidden mass}) \times (\text{modulation mass}) \times (\text{coupling}) \times (\text{energy scale})^2.
$$
When $v_\lambda$ is smaller, the NTK spectrum is flatter, leading to reduced spectral bias and more uniform convergence across frequency modes. Improved NTK conditioning thus facilitates faster, more stable recovery of high-frequency signal components and higher-fidelity interpolation [2512.15169]. This decomposition renders diverse INR architectures commensurable through their impact on NTK eigenvalue dispersion.

## 4. NTK-Aware INR Interpolation: Algorithmic Guidelines

**Network Architecture:**
- Input: Random Fourier features $\phi(x)$ with adjustable bandwidth $\sigma$.
- Hidden layers: linear $\rightarrow$ ReLU $\rightarrow$ spherical-norm (or Top-K norm) $\rightarrow$ Hadamard modulation (elementwise product with $p_\ell$).
- Output: Linear readout $a^T y_L$ with $a \in \{\pm a\}^m$ fixed at initialization.

**Initialization:**
- Weights $W_\ell(0) \sim N(0, \kappa^2 I)$, $a_r \sim \mathrm{Uniform}\{\pm a\}$, $p_r$ bounded (e.g., random $\tanh$).
- Small $a$ and $\kappa$ to ensure NTK regime ($||u(0)|| \approx 0$).
- Width $m$ polynomial in $n, 1/\lambda_0$ for kernel stability: $||H(W(k)) - H_0|| \leq \mathcal{O}(m^{-1/2})$.

**Training:**
- Learning rate $\eta \leq 1/\|H_0\|_2$ for stability; linear convergence at $1-\eta\lambda_{\min}$.
- Gradient descent or small-batch SGD; early stopping leverages uniform mode decay.
- $\ell_2$ weight decay or spectral-norm regularization is optional for extra stability.

## 5. Monitoring NTK Eigenvalue Variance in Practice

The NTK variance $v_\lambda$ can be estimated at initialization and throughout training. For computational efficiency, the NTK is typically computed over $n_0 \ll n$ coordinate subsamples. The following pseudocode, as presented in [2512.15169], details this procedure:

```python
def compute_ntk_eig_stats(model, coords, batch_size):
    # coords: array of shape [n0, d0]
    # returns (mu_lambda, v_lambda)
    n0 = coords.shape[0]
    # 1. compute Jacobians J[i] = ∇_W f(coords[i]) reshaped to vector
    #    For two-layer model only first-layer derivatives needed:
    grads = []
    for x in coords:
        model.zero_grad()
        f_x = model(x.unsqueeze(0))  # shape [1]
        f_x.backward()               # populates .grad
        # flatten all trainable weights into a single vector g = ∂f/∂W
        g = torch.cat([p.grad.flatten() for p in model.parameters()])
        grads.append(g.detach().cpu().numpy())
    G = np.stack(grads, axis=1)  # shape [P, n0], P=total param dim
    H = G.T @ G                  # empirical NTK Gram matrix, shape [n0,n0]
    # 2. spectral stats
    vals = np.linalg.eigvalsh(H)
    mu = vals.mean()
    v  = (vals**2).mean() - mu**2
    return mu, v
```

Tracking $v_\lambda$ across training epochs for various architectures (base MLP, +PE, +Norm, +Hada) empirically validates that each augmentation sequentially lowers NTK variance, flattens the spectrum, and produces faster, more stable, and higher-quality interpolation of continuous functions [2512.15169].

## 6. Empirical Implications and Outlook

Experiments confirm the predicted variance reductions and indicate that each architectural intervention incrementally improves NTK conditioning and convergence properties. A plausible implication is that further architectural innovations or regularization strategies could be systematically evaluated through their effect on the four-factor variance decomposition. NTK-aware interpolation thus supplies both a diagnostic and a design framework for future advances in implicit neural representations [2512.15169].

Source: https://www.emergentmind.com/topics/ntk-aware-interpolation