NTK-Guided Implicit Neural Teaching (NINT)

• The paper introduces NTK-Guided Implicit Neural Teaching (NINT), a method that leverages NTK-augmented loss gradients to dynamically score and select training coordinates for optimal convergence.
• NINT significantly reduces training time and computational cost in high-dimensional tasks, demonstrating up to a 48.7% speed-up and improved metrics in image, audio, and 3D representations.
• The algorithm integrates dynamic NTK-guided sampling with scheduled recomputation, ensuring efficient functional improvement without requiring architectural changes or extra supervision.

NTK-Guided Implicit Neural Teaching (NINT) is a training acceleration method for Implicit Neural Representations (INRs) that leverages the Neural Tangent Kernel (NTK) to dynamically select training coordinates, targeting rapid functional improvement in high-dimensional continuous signal modeling. INRs parameterize signals such as images, audio, and 3D volumes with multilayer perceptrons (MLPs) in a resolution-independent fashion, but standard training incurs significant computational cost due to the sheer volume of coordinates involved. NINT addresses this bottleneck by scoring coordinates according to their NTK-augmented loss gradients—incorporating both fitting error and the heterogeneous leverage each coordinate exerts on the global function—and subsampling points for maximal overall convergence. This approach demonstrates significant reductions in training time, outperforming prior sampling-based strategies without requiring architectural changes or additional supervision (Zhang et al., 19 Nov 2025).

1. Motivation and Problem Formulation

High-resolution INR training involves minimizing

$\theta^\star=\arg\min_\theta \frac1N\sum_{i=1}^N\cL\bigl(f_\theta(x_i),y_i\bigr),$

where $f_\theta$ is an MLP mapping input coordinates $x_i$ to signal values $y_i$, and $N$ often exceeds $10^6$. Full-batch stochastic gradient descent (SGD) over all $N$ points per iteration is computationally prohibitive for images, video, and 3D scenes. Simple subsampling by error magnitude fails to account for a coordinate's global influence. NINT introduces NTK-guided sampling to address this deficiency: it scores each point according to its ability to induce large updates in the entire signal via both self-leverage and cross-coordinate coupling.

2. NTK-Augmented Scoring and Functional Evolution

Training an INR via full-batch updates entails functional evolution at any coordinate $x$ described by

$\frac{\partial f_{\theta^t}(x)}{\partial t} \approx -\frac\eta N \sum_{i=1}^N \langle \nabla_\theta f_{\theta^t}(x_i), \nabla_\theta f_{\theta^t}(x) \rangle \nabla_{f}\cL\bigl(f_{\theta^t}(x_i),y_i\bigr),$

where the central object is the NTK

$$K_{\theta^t}(x_i,x_j) = \langle \nabla_\theta f_{\theta^t}(x_i), \nabla_\theta f_{\theta^t}(x_j) \rangle.$$

For coordinate-wise selection, NINT approximates the global functional update resulting from a single point as

$$\Delta f \propto K_{\theta^t}(x_i,\,\cdot\,)\,g_i^t,$$

and defines the NTK-guided score

$$s(x_i) = \bigl\lVert K_{\theta^t}(x_i,\,\cdot\,)\,g^t\bigr\rVert_2$$

for selecting points that maximize expected global change. This mechanism captures both self-leverage ($K(x_i,x_i)$) and cross-coupling ($K(x_i,x_j)$ for $j\neq i$).

3. Algorithmic Workflow and Complexity

NINT’s core workflow comprises NTK-guided dynamic sampling, decaying over the training process per an exponential schedule, and occasional recomputation of NTK-based scores:

Input: data S={(x_i,y_i)}_{i=1}^N, MLP f_θ, batch size B, learning rate η, total steps T, static fraction ξ∈[0,1], NTK decay λ, recompute interval α Initialize θ ← θ₀ Pre-sample a static set S₀ of size ⌊ξN⌋ (uniformly at random) for t in 0,…,T−1 do 1) Forward: compute predictions f_θ(x_i) and errors g_i = ∇_f L(f_θ(x_i),y_i) 2) Let r = exp(−λ t / α) 3) With probability (1−ξ)·r, do NTK sampling: • If t mod α = 0, compute full NTK-gradient product: For each i: s_i ← ‖K(x_i,·)·g‖₂ • Else reuse previous s_i • Select top B·(1−ξ)·r indices by s_i Else use uniform/error-based sampling 4) Form batch B_t of size B, union of S₀ and selected points 5) Parameter update: θ ← θ − (η/|B_t|)∑_{i∈B_t} ∇_θ L(f_θ(x_i),y_i) end for Output: trained f_θ

NTK row computation scales as $O(N\cdot\dim\theta)$ per recomputation, mitigated by caching and infrequent updates ($\alpha$ steps). Overall amortized per-step cost is

$$O\Bigl(B \cdot \dim\theta + \frac{1}{\alpha} N \cdot \dim\theta\Bigr)$$

with $B\ll N$ and moderate $\alpha$.

4. Theoretical Analysis

In the "infinite-width" regime, the NTK $K_{\theta^t}$ is static, and training corresponds to kernel regression in function space. For practical MLPs, NTK evolves slowly, and linearization remains accurate in early-to-mid training (Lee et al. 2019, Arora et al. 2019). By sampling coordinates according to $s_i$, each NINT iteration maximizes a lower bound on the decrease of the global squared error $\|g\|_2^2$ within the induced RKHS. The key lemma states that selecting the subset maximizing $\sum_{i\in\cB} s_i^2$ yields steepest descent in global risk under mild Lipschitz and positive-definiteness conditions on $K$. NINT thus guarantees at least as rapid convergence as uniform sampling, often substantially faster, up to NTK drift effects late in training.

5. Experimental Results

NINT was evaluated on

• 2D image fitting: Kodak, DIV2K datasets;
• 1D audio fitting: LibriSpeech “test.clean”;
• 3D shape fitting: Stanford 3D Scanning Repository (SDF reconstruction).

Architectures included SIREN (default: 5 layers × 256 units), MLP, FFN (Fourier-encoded), FINER, GAUSS, PEMLP, WIRE, with network widths from 64 to 256 units. Baselines compared were full-batch (Stand.), uniform sampling (Unif.), EGRA, Soft Mining, INT, EVOS, and Expansive Supervision. Metrics included PSNR, SSIM, LPIPS for images; SI-SNR, STOI, PESQ for audio; IoU, Chamfer for 3D. All experiments used NVIDIA RTX-4090, $10^{-4}$ learning rate, and batch size at 20% unless full-batch.

Results substantiate NINT's acceleration:

• Image fitting: at 250 iterations, NINT achieves $>$28.9 dB PSNR, surpassing all baselines; to reach 30 dB PSNR, NINT requires 25 seconds (380 iters) versus 49 seconds (523 iters) for full-batch, a 48.7% speed-up.
• Audio fitting: at 3 seconds, NINT SI-SNR = 3.84 dB vs. INT = 0.97 dB; PESQ 1.20 vs. 1.17.
• 3D shapes: at 5k iterations, NINT IoU = 0.9762 (vs. Stand. 0.9776), Chamfer distance = $6.023\times 10^{-3}$ (vs. $5.975\times 10^{-3}$), with further improvements using step-wise batch scheduling.
• Ablations confirm NINT’s gains are robust across architectures, sizes, and hyperparameters, reaching up to 43.3% reduced training time and up to 11.6% PSNR improvement.

6. Limitations and Enhancement Opportunities

The primary computational bottleneck for NINT is NTK row calculation, which demands $O(N\cdot\dim\theta)$ memory, particularly for large $N$ or deep networks. While infrequent recomputation amortizes this cost, limitations persist for extreme-scale or high-depth scenarios. NTK linearization is accurate only for finite-width MLPs in early and mid-training; late-stage drift may impair functional correspondence. Candidate enhancements include:

• NTK approximations (random features, low-rank factorization) for tractable $K\cdot g$;
• Adaptive batching by tuning batch size $B$ and recompute interval $\alpha$ as convergence progresses;
• Hybrid integration with partitioned or grid-based priors (e.g., hash encoding) for further domain-specific acceleration.

7. Open Questions and Research Directions

Key unresolved challenges include extending NTK-guided teaching to loss functions that are adversarial or perceptual. A plausible implication is that coupling NTK-driven curriculum strategies—initial broad NTK sampling followed by fine error-based selection—may yield fastest convergence. Further exploration of NTK scoring’s applicability across more INR domains and tasks remains a priority.

NINT systematically selects coordinates to maximize the global functional improvement at each training iteration,

$s(x_i)=\|K(x_i,\,\cdot\,)\,g\|_2,$

delivering nearly halved training times on image, audio, and 3D INR tasks compared to prevailing methods, and establishing a benchmark for sampling-based acceleration without modifying network architectures or supervision protocols (Zhang et al., 19 Nov 2025).