---
title: InterpoLated Learning (InterpoLL) Overview
url: https://www.emergentmind.com/topics/interpolated-learning-interpoll
type: topic
---

# InterpoLated Learning (InterpoLL) Overview

Searching arXiv for the cited papers to ground the article.
Searching arXiv for "Continual Learning with Weight Interpolation" and related InterpoLL uses.
InterpoLated Learning (InterpoLL) denotes a set of interpolation-centered learning formulations rather than a single universally fixed algorithm. In recent arXiv usage, the label has been applied to at least three prominent settings: representation-space interpolation for mitigating shortcut learning in NLP, weight interpolation for rehearsal-based continual learning, and broader interpolation-driven designs in optimization, nonparametric prediction, instance-based learning, and neural-network architecture [2507.05527] [2404.04002] [1906.05661] [1810.02814] [2108.11530] [2404.10296]. A related but distinct acronym is INTERPOL, an identification framework for de-anonymizing model outputs in pairwise voting settings via interpolated preference learning; despite the similar name, it addresses a different problem class [2603.15220].

## 1. Nomenclature and conceptual scope

The common element across these formulations is that interpolation is treated as an active mechanism of learning rather than as a purely numerical post-processing device. Depending on the paper, the interpolation target may be latent representations, model weights, stochastic step-sizes derived from an interpolation regime, neighbor weights in nonparametric estimation, or trainable interpolation nodes embedded directly in a neural architecture [2507.05527] [2404.04002] [1906.05661] [1810.02814] [2404.10296].

| Formulation | Interpolation locus | Representative paper |
|---|---|---|
| Shortcut mitigation InterpoLL | Latent representations of majority and minority examples | [2507.05527] |
| CLeWI / InterpoLL in continual learning | Old and new model weights after permutation alignment | [2404.04002] |
| ALI-G and interpolation property | Optimization regime where per-sample losses are driven to zero | [1906.05661] |
| Interpolated-NN | Neighbor weights that diverge at zero distance | [1810.02814] |
| ADB interpolation | Coordinate-wise approximation-degree and sum-times-difference synthesis | [2108.11530] |
| INN | Trainable interpolation points and basis-function evaluations | [2404.10296] |

This multiplicity of usage is itself an important feature of the term. It suggests that InterpoLL functions as a broader paradigm in which interpolation is elevated to a structural prior for learning, but the operational meaning of interpolation depends on the application domain.

A recurrent misconception is that “interpolation” necessarily denotes naive averaging or uncontrolled overfitting. The papers associated with InterpoLL argue for more specialized constructions: intra-class latent interpolation with $\lambda \sim \mathrm{Uniform}(0,0.5)$ for shortcut mitigation, permutation-aligned weight interpolation to exploit mode connectivity in continual learning, and clipped stochastic Polyak-style updates when the interpolation property holds in deep optimization [2507.05527] [2404.04002] [1906.05661].

## 2. Representation-space InterpoLL for shortcut mitigation

In "Mitigating Shortcut Learning with InterpoLated Learning" [2507.05527], InterpoLL is introduced as a model-agnostic representation-space augmentation designed to weaken spurious correlations exploited under ERM. The setup decomposes the predictor into an encoder and classifier,
$$
f_\theta(x)=f_{\mathrm{cls}}(f_{\mathrm{enc}}(x)),
$$
with latent representation $z_i=f_{\mathrm{enc}}(x_i)$. Within each class $y$, the method distinguishes majority examples $g_{\mathrm{maj}}(y)$, whose spurious features align with $y$, from minority examples $g_{\mathrm{min}}(y)$, whose features contradict the shortcut. The split is inferred by an under-parameterized auxiliary model $f_\phi$: examples mis-predicted by $f_\phi$ are treated as minority, and correctly predicted examples as majority [2507.05527].

For each majority example $(x_i,y_i)\in g_{\mathrm{maj}}$, a same-class minority example $(x_j,y_j)\in g_{\mathrm{min}}$ with $y_j=y_i$ is sampled, and their representations are interpolated:
$$
z_i'=(1-\lambda)\,f_{\mathrm{enc}}(x_i)+\lambda\,f_{\mathrm{enc}}(x_j), \qquad \lambda\sim \mathrm{Uniform}(0,0.5).
$$
Minority examples use their normal representation. The resulting objective is
$$
J_{\mathrm{InterpoLL}}(\theta)
= \mathbb{E}_{(i,j,\lambda)}[\ell(h(z_i'),y_i)]
+ \mathbb{E}_{(i:y_i\in g_{\min})}[\ell(h(f_{\mathrm{enc}}(x_i)),y_i)],
$$
where $h(z)=f_{\mathrm{cls}}(z)$. All model parameters are updated by back-propagating this loss [2507.05527].

Algorithmically, the method first uses $f_\phi$ to partition the data into $g_{\min}$ and $g_{\maj}$, then processes each minibatch by replacing the latent representation of each majority example with an interpolated representation and leaving minority examples unchanged. The forward pass and loss are then computed on the transformed latent batch [2507.05527].

The empirical scope is broad. The paper reports results on natural language inference tasks such as MNLI, FEVER, and QQP with OOD test sets and stress tests including HANS and PAWS; text-classification tasks including FDCL18, CivilComments-WILDS, and Amazon-WILDS; and domain generalization via GLUE-X. Architectures span encoder-only models such as BERT-base, BERT-large, RoBERTa-large, ELECTRA-large, and XLNet-large, encoder-decoder models such as T5-large and T5-3B, and decoder-only models such as GPT2-medium and GPT2-large [2507.05527].

A concise summary of reported benchmark deltas is as follows.

| Setting | ERM | InterpoLL |
|---|---:|---:|
| NLI OOD / Stress | $62.4 / 62.9$ | $75.6 / 66.5$ |
| FDCL18 minority accuracy | $35.6$ | $61.2$ |
| CivilComments minority accuracy | $63.5$ | $73.9$ |
| GLUE-X average | $69.3$ | $72.4$ |
| BERT-large on NLI $\rightarrow$ HANS | $74.7$ | $80.1$ |

The paper also reports that Uniform$(0,0.5)$ is the best interpolation ratio, whereas Uniform$(0,1)$ and $\mathrm{Beta}(\alpha,\alpha)$ with $\alpha\neq 0.5$ hurt either ID or OOD performance. Inter-class mixing and inverting interpolation underperform intra-class mixing. Minimum description length probing indicates that InterpoLL yields the lowest extractability of known shortcuts. With $5\%$ synthetic label flips, the method degrades much less than ERM and other methods. On MNLI, runtime is reported as $4$ h for ERM and $5$ h for InterpoLL, while most other debiasing methods take at least $6$ h and up to $39$ h [2507.05527].

The paper is explicit about limitations. It notes a small drop in ID accuracy on some tasks, recommends interpolation in upper encoder layers, and warns that mixing may introduce unnatural representations if minority examples are too dissimilar. It also reports that a no-auxiliary variant, in which the learner self-identifies $g_{\min}$ and $g_{\maj}$, still outperforms prior methods [2507.05527].

## 3. Weight interpolation in continual learning

In "Continual Learning with Weight Interpolation" [2404.04002], the relevant InterpoLL formulation is called CLeWI. It is a post-task weight consolidation method for continual learning with rehearsal. After learning task $t$, let $\theta_P$ denote the saved network weights from task $t-1$, and let $\theta$ denote the weights after fine-tuning on task $t$. A permutation $\pi$ is first computed via the REPAIR procedure to align $\theta_P$ to $\theta$ in weight space, and the interpolated weights are then formed as
$$
\theta_{\mathrm{interp}}=(1-\alpha)\,\theta+\alpha\,\pi(\theta_P),
$$
with $\alpha\in[0,1]$. In the formulation given in the paper, $(1-\alpha)$ controls plasticity and $\alpha$ controls stability [2404.04002].

The role of $\alpha$ is directly quantified. On split-CIFAR10, the reported ablation gives:
- $\alpha=0.1$: average accuracy $27.6\%$, last-task accuracy $87.8\%$, forgetting $59.1\%$;
- $\alpha=0.2$: $34.8\%$, $83.9\%$, $47.1\%$;
- $\alpha=0.3$: $40.0\%$, $72.2\%$, $30.7\%$;
- $\alpha=0.4$: $42.0\%$, $44.6\%$, $18.9\%$;
- $\alpha=0.5$: $40.3\%$, $16.3\%$, $12.6\%$ [2404.04002].

This ablation confirms the stability–plasticity trade-off claimed by the method. A small $\alpha$ favors current-task adaptation but increases forgetting; a large $\alpha$ preserves earlier-task performance but restricts adaptation to new tasks. In practice, $\alpha$ is selected by lightweight grid search on a held-out stream or by inspecting interpolation plots of test accuracy versus $\alpha$ using the replay buffer [2404.04002].

CLeWI is designed as a plug-in over rehearsal-based training. The pseudocode given in the paper applies standard replay-based SGD with a replay buffer $\mathcal{M}$, then, for tasks $t>1$, computes a permutation $\pi \leftarrow \mathrm{calc\_permutation}(\theta,\theta_P,\mathcal{M})$, interpolates the weights, and recomputes BatchNorm statistics through `update_batchnorm(θ,𝓜)`. The buffer is used both for feature-map correlation estimates in the permutation step and for re-estimating BatchNorm statistics to avoid variance collapse. `calc_permutation` solves a bipartite matching on feature-map correlations, with explicit reference to Jordan et al. 2022 [2404.04002].

The theoretical intuition is framed via the cumulative test loss
$$
\mathcal{L}_D(\theta)=\sum_{i=1}^T \mathcal{L}(\theta,D_i),
$$
and the forgetting increment
$$
\Delta \mathcal{L}_F=\sum_{i=1}^{T-1}\bigl(\mathcal{L}(\theta_T,D_i)-\mathcal{L}(\theta_i,D_i)\bigr).
$$
The paper argues that minimizing cumulative loss requires both low current-task loss and low forgetting, and that SGD solutions for consecutive tasks often lie in mode-connected, or near-convex, regions of the loss landscape once aligned by permutation. A linear interpolation in this aligned space is therefore expected to remain in a low-loss valley that balances old and new performance [2404.04002].

Empirically, the method is evaluated on class-incremental splits of CIFAR-10 with $5$ tasks, CIFAR-100 with $10$ tasks, and Tiny-ImageNet with $20$ tasks, using a replay buffer of $500$ examples and ResNet-18 as the primary architecture, with WideResNet used in ablations [2404.04002]. The paper reports the following average accuracy and forgetting improvements:

| Method pair | CIFAR-10 | CIFAR-100 | Tiny-ImageNet |
|---|---|---|---|
| ER $\rightarrow$ CLeWI+ER | $53.2\%\!\to 62.8\%$, $44.0\%\!\to 31.8\%$ FM | $22.5\%\!\to 40.3\%$, $65.6\%\!\to 12.8\%$ FM | $6.4\%\!\to 11.7\%$, $75.9\%\!\to 66.8\%$ FM |
| MIR $\rightarrow$ CLeWI+MIR | $48.2\%\!\to 73.1\%$, $49.0\%\!\to 6.7\%$ FM | $22.0\%\!\to 40.1\%$, $66.1\%\!\to 13.5\%$ FM | $6.3\%\!\to 19.8\%$, $76.1\%\!\to 25.5\%$ FM |
| DER++ $\rightarrow$ CLeWI+DER++ | $70.1\%\!\to 71.8\%$, $21.1\%\!\to 11.2\%$ FM | $36.6\%\!\to 38.2\%$, $48.1\%\!\to 14.3\%$ FM | $13.5\%\!\to 16.6\%$, $55.7\%\!\to 25.2\%$ FM |

The paper further states that the method can be plugged into ER, aGEM, ER-ACE, MIR, BIC, and DER++ with no further algorithmic changes, and that without any replay, pure interpolated SGD remains poor. That result is important because it circumscribes the method’s scope: the evidence supports CLeWI as a complement to rehearsal rather than as a replacement for it [2404.04002].

## 4. Interpolation as an optimization regime: ALI-G

A different, earlier interpolation-centered usage appears in "Training Neural Networks for and by Interpolation" [1906.05661]. Here the central object is the interpolation property of supervised learning: for a parametric model $w\in\mathbb{R}^p$, per-sample loss $\ell_z(w)\ge 0$, and expected risk
$$
f(w)=\mathbb{E}_{z\sim Z}[\ell_z(w)], \qquad f_*=\min_{w\in\Omega} f(w),
$$
the problem satisfies interpolation if there exists $w_*$ such that
$$
\forall z\in Z:\ \ell_z(w_*)=0 \Longleftrightarrow f_*=0.
$$
The paper then derives Adaptive Learning-rates for Interpolation with Gradients (ALI-G), a stochastic Polyak-style method that exploits this property [1906.05661].

Starting from Polyak’s update
$$
w_{t+1}=\Pi_\Omega\!\bigl(w_t-\gamma_t \nabla f(w_t)\bigr), \qquad
\gamma_t=\frac{f(w_t)-f_*}{\|\nabla f(w_t)\|^2},
$$
the interpolation setting with $f_*=0$ motivates the stochastic raw stepsize
$$
\gamma_t^{\mathrm{raw}}=\frac{\ell_{z_t}(w_t)}{\|\nabla \ell_{z_t}(w_t)\|^2+\delta},
$$
with a small $\delta>0$ to avoid division by zero. ALI-G clips this value at a user-chosen maximum $\eta$:
$$
\gamma_t=\min\!\left\{\frac{\ell_{z_t}(w_t)}{\|\nabla \ell_{z_t}(w_t)\|^2+\delta},\eta\right\},
\qquad
w_{t+1}=\Pi_\Omega\!\bigl(w_t-\gamma_t \nabla \ell_{z_t}(w_t)\bigr).
$$
Clipping is presented as crucial in nonconvex settings; the paper gives the one-dimensional counterexample $f(w)=w^2-|w|^3$, which oscillates without clipping [1906.05661].

The convergence guarantees are stratified by assumption class. For convex and $C$-Lipschitz losses under interpolation, ALI-G attains an $\mathcal{O}(1/\sqrt{T})$ rate. With additional $\beta$-smoothness, the expected suboptimality improves to $\mathcal{O}(1/T)$. In the strongly convex and smooth setting, the method achieves an exponential rate,
$$
\mathbb{E}[f(w_{T+1})]-f_*
\le \frac{\beta}{2}\exp\!\bigl(-\tfrac{\alpha T}{4\beta}\bigr)\,\|w_0-w_*\|^2.
$$
For certain nonconvex objectives satisfying the Restricted Secant Inequality,
$$
\nabla f(w)^\top(w-w_*)\ge \alpha \|w-w_*\|^2,
$$
the paper gives an exponential convergence statement when $\eta\in[1/(2\beta),2\alpha/\beta^2]$ and smoothness holds [1906.05661].

The empirical evaluation covers a differentiable neural computer copy task, WRN-16-4 on SVHN, a Bi-LSTM on SNLI, WRN-40-4 and DenseNet-40-40 on CIFAR-10/100, and ResNet-18 on ImageNet without data augmentation [1906.05661]. Representative figures include:
- on SVHN after $160$ epochs, ALI-G reaches $98.1\%$ test accuracy, compared with $98.3\%$ for hand-scheduled SGD;
- on SNLI, ALI-G reaches $84.8/85.2$ for CE/SVM loss, matching SGD’s $84.7/85.2$;
- on CIFAR-10 WRN, ALI-G achieves $95.2\pm 0.1$ versus SGD’s $95.3\pm 0.3$;
- on CIFAR-100 DenseNet, ALI-G achieves $76.3\pm 0.1$, equal to SGD’s $76.3\pm 0.2$;
- on ImageNet, with batch-size $1024$ over $4$ GPUs and fixed $\eta=10$, ALI-G reaches $99\%$ top-5 train accuracy in $12$ epochs [1906.05661].

In this usage, InterpoLL does not denote interpolation between examples or weights. Instead, interpolation is a property of the data-fitting regime that enables a closed-form adaptive learning rate. This is a materially different interpretation of the term from the shortcut-mitigation and continual-learning formulations.

## 5. Nonparametric and instance-based interpolation

Two additional lines of work extend the InterpoLL idea into nonparametric estimation and instance-based learning. In "Statistical Optimality of Interpolated Nearest Neighbor Algorithms" [1810.02814], the predictor is an interpolated nearest-neighbor rule. For ordered Euclidean distances $d_{(1)}\le \cdots \le d_{(n)}$ from a query $x$, neighborhood size $k<n$, and a positive function $\phi:[0,1]\to (0,\infty)$ with $\lim_{t\to 0}\phi(t)=\infty$, the estimate is
$$
\hat\eta(x)=\sum_{i=1}^k w_i\,y_{(i)},
$$
with weights
$$
w_i=\frac{\phi(d_{(i)}/d_{(k+1)})}{\sum_{j=1}^k \phi(d_{(j)}/d_{(k+1)})}
$$
when $d_{(i)}>0$, and $w_i=1$ when the query coincides with a training point. The classification rule is $\hat g(x)=1\{\hat \eta(x)>1/2\}$ [1810.02814].

The paper proves that, under Hölder smoothness, support regularity, a compact-support density assumption, and a Tsybakov margin condition for classification, the method attains minimax-optimal rates in both regression and classification. For regression, choosing $k\asymp n^{2\alpha/(2\alpha+d)}$ yields
$$
\mathbb{E}[\hat\eta(x)-\eta(x)]^2 = \mathcal{O}(n^{-2\alpha/(2\alpha+d)}).
$$
For classification, the excess-risk rate becomes
$$
\mathbb{E}[R_{n,k}-R^*]=\mathcal{O}(n^{-\alpha(\beta+1)/(2\alpha+d)}).
$$
The analysis emphasizes the usual bias–variance decomposition: interpolated-NN reduces bias by heavily down-weighting distant neighbors but incurs a variance penalty through unequal weights. Empirically, the paper reports slightly lower excess risk than classical $k$NN across simulated settings and consistently lower classification error on the HTRU2 pulsar dataset [1810.02814].

In "A New Interpolation Approach and Corresponding Instance-Based Learning" [2108.11530], the relevant formulation is ADB interpolation, based on the approximation-degree between numerical values. For a center $x_0$ in an interval and approximation region $[a_0,b_0]$, the approximation-degree is piecewise linear, equals $1$ at $x_0$, and vanishes on the region boundary. This is extended to a strict approximation region in $\mathbb{R}^n$ via an axis-aligned box [2108.11530].

The one-dimensional ADB interpolant reconstructs $g(x)\approx f(x)$ locally from adjacent sample points and yields piecewise linear formulas such as
$$
g_i^-(x)=y_{i-1}+(y_i-y_{i-1})\frac{x-x_{i-1}}{x_i-x_{i-1}},
$$
and
$$
g_i^+(x)=y_{i+1}-(y_{i+1}-y_i)\frac{x-x_i}{x_{i+1}-x_i}.
$$
In higher dimensions, the paper introduces the sum-times-difference synthesis
$$
\hat y=\sum_{k=1}^n y_k-(n-1)\,y_0,
$$
with coordinate-wise partial interpolations $y_k$ around a center value $y_0$ [2108.11530].

The induced instance-based learner includes both regression and classification procedures. The ADB regressor first finds the closest training point in a Chebyshev sense, performs one-dimensional ADB interpolation along each coordinate axis, and returns the sum-times-difference estimate. The ADB classifier searches for a training point whose strict approximation region contains the query; if found, it returns that class label, otherwise it abstains or returns “unknown” [2108.11530].

The computational analysis states a nearest-neighbor lookup cost of $\mathcal{O}(\log M)$ with a kd-tree or $\mathcal{O}(1)$ with hash grids under assumptions, plus $\mathcal{O}(n)$ for one-dimensional interpolation along each axis, for total cost $\mathcal{O}(\log M+n)$. The paper reports empirical results including maximum absolute error on the order of $1\times 10^{-3}$ for $\sin x$ on $[0,2\pi]$, RMSE $0.37$ versus approximately $0.35$ for bilinear interpolation on $z=x^2+y^2$, RMSE approximately $0.015$ versus $0.013$ for trilinear interpolation on $u=e^{-x^3-y^3-z^3}$, and $100\%$ classification accuracy on test points inside strict regions for a synthetic two-class dataset [2108.11530].

These two works are conceptually related but methodologically distinct. Interpolated-NN is a statistical learning rule with explicit minimax analysis, whereas ADB is framed as an interpolation-first analogue-based learner with structured abstention in classification [1810.02814] [2108.11530].

## 6. Interpolating neural networks and broader synthesis

A further extension of the InterpoLL idea appears in "Interpolating neural network: A novel unification of machine learning and interpolation theory" [2404.10296]. This paper defines InterpoLated Learning as the construction of end-to-end differentiable models by embedding classical interpolation theory into the architecture itself. Instead of dense unstructured weight matrices, the model introduces trainable interpolation points with coordinates $\{p_j\}$ and associated values $\{u_j\}$, and defines the forward map through interpolation basis functions [2404.10296].

In one dimension, a segment network partitions the input interval into contiguous segments bounded by interpolation nodes, computes local interpolant basis functions $\{\tilde N_j(p)\}$, and outputs
$$
u^h(p)=\sum_{j=j_0-s+1}^{j_0+s} \tilde N_j(p)\,u_j.
$$
In multiple dimensions, the paper avoids a full tensor of size $J^I$ through tensor decomposition, writing
$$
u^h(p_1,\ldots,p_I)=\sum_{m=1}^M \prod_{i=1}^I \bigl[\tilde N_{p_i}^{(m)}(p_i)\cdot u_{p_i}^{(m)}\bigr].
$$
The corresponding forward pass for mode $m$ uses
$$
\alpha_i^{(m)}=\tilde{\mathbf N}_{p_i}(p_i)\cdot u_{p_i}^{(m)},
\qquad
u^h(\mathbf p)=\sum_{m=1}^M \prod_{i=1}^I \alpha_i^{(m)}.
$$
The paper interprets this construction as a CP-style tensor factorization of the weight tensor [2404.10296].

The claimed advantages are parameter efficiency, training speed, compact memory footprint, and interpretability of learned node values as values at physical interpolation points. Parameter growth is characterized as $\mathcal{O}(M\cdot I\cdot J\cdot L)$ for node values, with optional node coordinates and analytic-network weights typically negligible relative to that count [2404.10296].

The main application is Laser Powder Bed Fusion heat conduction, with an $8$-dimensional input consisting of $3$ spatial coordinates, $1$ temporal coordinate, and $4$ uncertain parameters. The paper reports that with $M=20$ modes and $J=10$ nodes in each dimension, the model uses approximately $20\cdot 8\cdot 10=1{,}600$ node values. Reported performance includes speed-up versus an FEM-plus-surrogate pipeline of approximately $10^5\times$ faster inference for a $10$ mm tool-path at $10\,\mu$m resolution, a model size of tens of megabytes rather than gigabytes, sub-$10\,\mu$m temperature-field error over a $10$ mm scan length, calibration from only $7$ FEM-generated melt-pool profiles with $6.4\%$ error relative to experimental melt-pool temperatures, and real-time online control in which a comparable FFNN uses only $18\%$ of the parameters and trains $18$–$31\times$ faster to the same loss [2404.10296].

The paper also states explicit limitations: selecting segment count $J$, modes $M$, and basis type still requires user guidance; mathematical proof of convergence and generalization bounds for InterpoLL remains open; and graph-structured data or non-rectangular domains may require specialized basis constructions [2404.10296].

Taken together, these formulations suggest that InterpoLL is best understood as a research direction organized around where interpolation is inserted into the learning system. In some works it modifies latent representations to suppress shortcuts; in others it merges models across tasks, exploits the interpolation regime of overparameterized training, reweights nearest neighbors, encodes instance-based approximation rules, or replaces dense layers with interpolation-theoretic operators [2507.05527] [2404.04002] [1906.05661] [1810.02814] [2108.11530] [2404.10296]. A separate but related direction, INTERPOL, uses interpolated preference data and a staged triplet-based curriculum to identify anonymous LLM outputs and quantify ranking-manipulation threats in Arena-style leaderboards, underscoring that interpolation has also become a design primitive in preference learning [2603.15220].

Source: https://www.emergentmind.com/topics/interpolated-learning-interpoll