Neuron Interpolation in Neural Networks

• Neuron interpolation is a set of methods that enable neural networks to exactly match specified outputs and align internal representations.
• It extends classical interpolation schemes with neural operators such as Taylor-accelerated, RBF-based, and tensor decomposition techniques.
• The approach underpins improvements in function reconstruction, model fusion, and generalization with rigorously established theoretical bounds.

Neuron interpolation refers to a spectrum of theoretical, algorithmic, and architectural constructs in which neural network models achieve or exploit interpolation properties at the level of units, representations, or function approximations. This paradigm encompasses classical interpolation schemes interpreted through the neural-network lens, theoretical results on memorization and universal approximation, neuron-centric fusion and alignment protocols, and advanced operator constructions attaining high-order or structure-preserving interpolation across Euclidean and non-Euclidean domains. Neuron interpolation plays a foundational role in contemporary machine learning theory, scientific computing, and multi-model systems, undergirding advances in representation fusion, function reconstruction, mesh-based simulations, and generalization theory.

1. Definition and Theoretical Landscape

Neuron interpolation formalizes the ability of a neural network, or a set of its constituent neurons, to exactly fit prescribed values at specified sites—be these spatial sample points, activation patterns, or model checkpoints. At the function-approximation level, this requires the network $\mathcal{N}$ with sufficient capacity to construct $f_\theta$ such that $f_\theta(x_i) = y_i$ for each $(x_i, y_i)$ training pair, or (in a structural context) align or interpolate internal activations across models or sub-networks.

Theoretical bounds delineate the sample complexity and network width/depth necessary for interpolation. For instance, deep feedforward networks with real-analytic, nonpolynomial activations attain the optimal $\Theta(\sqrt{nd'})$ neuron count for $n$ generic data points in $\mathbb{R}^d \times \mathbb{R}^{d'}$, independent of the activation’s specific shape as long as it avoids piecewise polynomial restriction (Madden, 2024). In contrast, for ReLU (piecewise-linear) networks, tight lower and upper bounds demonstrate a phase transition: if the minimal pointwise separation $\delta(N)$ between nodes decays exponentially in $N$ ($\delta < \exp(-cN)$), a linear number of parameters $\Omega(N)$ is both necessary and sufficient to guarantee interpolation of arbitrary labels, reflecting an intrinsic combinatorial bottleneck (Siegel, 2023).

2. Neural Network Interpolation Operators

Neural network operators for interpolation generalize classical linear interpolants (Lagrange, piecewise polynomial, splines) by embedding their logic into neural architectures. A canonical case is the Taylor-accelerated neural network interpolation operator $\widetilde{T}_{n,r}$, defined over quasi-uniform grids as

$$\widetilde{T}_{n,r}(g; y) = \frac{\sum_{k=0}^{n} \sigma\left(\frac{2m}{d_1}(y - z_k)\right) P_r(g; y, z_k)} {\sum_{k=0}^{n} \sigma\left(\frac{2m}{d_1}(y - z_k)\right)}$$

where $P_r(g; y, z_k)$ is the degree-$r$ Taylor polynomial centered at $z_k$, and the sigmoidal $\sigma$ enforces localization. This operator exactly interpolates at the nodes, reproduces polynomials up to degree $r$, and attains an $O(n^{-(r+1)})$ error rate for $C^{r+1}$ functions, outperforming Lagrange-type NNOs—especially on nonuniform meshes (Saini, 5 Feb 2026).

Boundary interpolation on non-rectangular domains is addressed via Boolean-sum neural network operators $\mathcal{B}_{n_1, n_2, \xi}$, which combine univariate interpolants in each coordinate direction and their tensor product, guaranteeing exact reproduction on all triangle edges and convergence in mixed moduli of continuity as neuron counts increase (Bhat et al., 2024).

3. Algorithmic Approaches: ELM, RBF-ELM, and Tensor Decompositions

Algorithmic neuron interpolation is achieved predominantly by two classes of constructions: overparameterized shallow architectures and tensorized multilinear networks.

• Extreme Learning Machines (ELM): In single-layer (shallow) networks, neuron weights and biases are randomized and fixed, and only the output weights are learned by solving a linear system to impose interpolation constraints—either square systems $Sw = y$ for equal nodes/neuron regime or least squares for overparameterized settings. This supports guaranteed exact interpolation with high numerical stability, and avoids the Runge phenomenon observed in high-degree polynomial interpolation, achieving Chebyshev-optimal convergence rates even for equispaced or random nodes (Auricchio et al., 2023).
• RBF-augmented ELM: By replacing generic activations with localized radial basis functions, one obtains mesh- and topology-agnostic interpolators with vastly increased robustness, reaching machine-precision errors ($10^{-10}$ to $10^{-14}$) on unstructured and adaptive meshes with computational cost dominated by a single linear solve (Hao et al., 16 Nov 2025).
• Tensor Neural Network (TNN) Interpolation: For high-dimensional inputs, tensor-product neural architectures decompose the function into products of univariate subnetworks, parameterized as $$\Psi(x; \Theta) = \sum_{j=1}^p c_j \prod_{i=1}^d \widehat{\phi}_{i,j}(x_i; \theta_i)$$. This structure reduces theoretical and numerical complexity of high-dimensional interpolation and allows efficient quadrature and PDE solvers in moderate ranks $p$, circumventing the curse of dimensionality present in dense mesh-based interpolators (Li et al., 2024). Similar tensor-decomposition and low-rank expansions are exploited in interpolating neural network (INN) models used in scientific computing and simulation—drastically reducing parameter and computational counts compared to full MLPs while preserving spline-type convergence guarantees (Park et al., 2024).

4. Interpolation in Representation Fusion and Model Alignment

Neuron interpolation serves as the mathematical core of model fusion, representation alignment, and layerwise matching protocols.

• Model Fusion via Neuron Interpolation: When merging several pretrained models with potentially misaligned internal representations, neuron interpolation proceeds by clustering neurons (via weighted K-means or Hungarian assignment), constructing target cluster centers (weighted averages of activations), and fitting the fused model so its neuron activations replicate these interpolation targets. The approach generalizes to any architecture (MLP, CNN, Transformer) and supports arbitrary layer widths. Empirically this yields superior “zero-shot” performance in non-IID or sharded data settings, outperforming ensembling, OT, or “re-basin” methods (Luenam et al., 18 Jun 2025).
• Permutation-Invariant Linear Paths and REPAIR: Interpolating network weights between two independently trained solutions generally requires solving an alignment problem due to permutation invariance of neurons. Even after alignment, naive linear interpolation often exhibits “variance collapse,” resulting in degenerate activations and poor performance. The REPAIR scheme rescales preactivations at every channel to match the convex combination of endpoint statistics, eliminating variance collapse and realizing near-zero loss barriers along alignment-corrected interpolation paths. This enables empirical validation of the conjecture that SGD solutions can be linearly connected (after proper re-normalization), for deep and wide networks across architectures and normalization schemes (Jordan et al., 2022).

5. Geometric, Structural, and Generalization Perspectives

Neuron interpolation is tightly linked to the geometric configuration of data and the architecture of representations.

• Intrinsic vs. Ambient Dimension: Neural networks’ generalization often occurs in a latent space of far lower effective dimension than the input or original feature space. In this space, most test samples lie in the convex hull of training activations—a regime dominated by interpolation rather than extrapolation, defying the “curse of dimensionality” argument. Statistical proximity (nearest-neighbor distances) better predicts classification accuracy than convex-hull membership, linking successful generalization to coverage and local interpolation (Bonnasse-Gahot, 2022).
• Sample Separation and Memorization Barriers: There exists a geometric phase transition in minimal model size as a function of point separation. For well-separated points ($\delta \gtrsim N^{-\alpha}$), the interpolation task can be solved with sublinear parameter counts (e.g., $O(\sqrt{N} poly\log N)$ for certain cases), but as $\delta$ falls below exponential thresholds the minimal model size jumps to $\Omega(N)$ (Siegel, 2023).

6. Advanced Extensions and Specialized Frameworks

Neuron interpolation encompasses a diverse range of specialized formulations:

• Spline-based and Localized Activations: Construction of neuron-specific, adaptive activations via cubic spline interpolation enables local, data-dependent tuning of functional shape—optimizing the representation capacity and regularity properties of the network. The training procedure, based on local cardinal spline coefficients, supports stable learning dynamics, explicit shape regularization, and potential for non-standard activation morphologies (Scardapane et al., 2016).
• Neural Operator-based Interpolation: In boundary value problems, operator-inspired neural network interpolants such as generalized Boolean-sum constructions achieve exact satisfaction of boundary constraints and rapid convergence in the interior, with explicit weights and biases determined by geometric and neuron-count parameters—eliminating the need for data-driven training (Bhat et al., 2024).
• Continuous Coordinate-based Interpolation: In graphics, coordinate-conditioned networks represent mappings from continuous geometry or spatio-temporal coordinates to high-resolution representations (e.g., neural view interpolation in light-field rendering), demonstrating the inherent capacity for continuous, differentiable interpolation across sparse or sampled data while encoding scene or physical priors (Bemana et al., 2019).

7. Summary Table: Illustrative Neuron Interpolation Paradigms

Paradigm	Key Principle	Reference
Taylor-accelerated NNO	Local Taylor + sigmoidal basis	(Saini, 5 Feb 2026)
Model fusion via neuron alignment	K-means/Hungarian interpolation of neurons	(Luenam et al., 18 Jun 2025)
RBF-ELM mesh transfer	Localized RBF activation, closed-form solve	(Hao et al., 16 Nov 2025)
TNN/INN tensor-product methods	Tensorized product of univariate subnetworks	(Li et al., 2024, Park et al., 2024)
Layerwise REPAIR interpolation	Alignment, mean/variance-corrected rescaling	(Jordan et al., 2022)
Spline-based per-neuron AFs	B-spline parameterization, local adaptivity	(Scardapane et al., 2016)

8. Open Problems and Future Directions

Several structural and theoretical challenges remain outstanding:

• Rigorous interpolation guarantees for ReLU (and other piecewise-polynomial activations) in the non-generic or worst-case data regime; closing the gap between lower and upper bounds for practical datasets (Madden, 2024).
• Incorporation of conservation laws and physics-informed priors in neuron-interpolation frameworks for scientific computing (Hao et al., 16 Nov 2025).
• Characterization of the trade-off between minimal-interpolant capacity and generalization under stochastic training, implicit regularization, or data-dependent activation adaptation (Bonnasse-Gahot, 2022, Scardapane et al., 2016).
• Interpolatory operator constructions for unstructured domains, manifolds, or graphs with controlled regularity and fast evaluation, including high-dimension scaling.

Neuron interpolation, therefore, constitutes a multifaceted and unifying construct encompassing the algebraic, geometric, and statistical apparatus underlying the approximation, memorization, and fusion capabilities of neural network models. It bridges classical numerical analysis, modern deep learning theory, and real-world computational science (Saini, 5 Feb 2026, Jordan et al., 2022, Luenam et al., 18 Jun 2025, Hao et al., 16 Nov 2025, Siegel, 2023, Madden, 2024, Li et al., 2024, Park et al., 2024, Auricchio et al., 2023, Bhat et al., 2024, Scardapane et al., 2016, Bemana et al., 2019, Bonnasse-Gahot, 2022).