- The paper proves that, under infinite-width NTK training, standard RND errors have the same expected uncertainty as deep ensemble predictive variance, including an exact multi-head distributional equivalence.
- The paper introduces a target-engineering method that aligns the RND prior with the NTK dynamics kernel, making each resulting error an independent sample from the centered Bayesian posterior predictive distribution.
- The paper validates variance calibration on synthetic regression as network width increases, while leaving finite-width feature learning, stochastic optimization, online data, and real-world applications as open challenges.
Random network distillation (RND) is a widely used uncertainty quantification technique in which a predictor network is trained to match the output of a fixed, randomly initialized target network, with the squared prediction error serving as a novelty or epistemic uncertainty signal. Despite its empirical success in exploration, out-of-distribution detection, and continual learning, the theoretical status of the RND error signal has remained unclear. This paper by Zanger et al. (2602.19964) supplies that missing theory: working within the neural tangent kernel (NTK) framework in the limit of infinite network width, it proves that RND errors coincide with deep ensemble variance, and that a deliberately engineered target function makes the RND error distribution identical to the centered posterior predictive distribution of Bayesian inference with infinitely wide networks. The analysis thereby places a computationally cheap single-model method within the same theoretical framework as ensembles and exact Bayesian inference.
Background: RND and NTK learning dynamics
RND trains a predictor u(x;θt​) against a fixed random target g(x;θ0​) under a squared loss, and uses the mean squared error across K output heads as an uncertainty score. The paper analyzes fully connected networks under NTK parametrization trained by gradient flow on square loss. In the infinite-width limit, the neural tangent kernel becomes deterministic at initialization and constant during training, so the function-space dynamics are linear and converge to the kernel regression solution; moreover, both initializations and converged functions are Gaussian processes, governed by the NNGP kernel κ and the training dynamics kernel Θ respectively.
A key structural observation is that the distribution over converged network functions given in prior work is not directly interpretable as a Bayesian posterior predictive distribution: its covariance involves two distinct kernels (κ and Θ), whereas a posterior predictive covariance takes the form of a prior kernel conditioned through the data. Resolving this mismatch is the central technical device of the paper.
Equivalence between standard RND and deep ensembles
The first main result characterizes the law of the post-convergence RND error ϵ(x;θ∞​,θ0​)=u(x;θ∞​)−g(x;θ0​). Because gradient-flow dynamics are linear, the converged predictor is an affine transformation of its initialization, which follows the NNGP; since this transformation is independent of the target draw, the error is itself a zero-mean GP whose covariance is built from the combined initialization kernel κxx′ϵ​=κxx′u​+κxx′g​ propagated through the conditioning structure. Taking architectural equivalence between predictor, target, and an ensemble member, the expected squared RND error equals the predictive variance of the corresponding infinite deep ensemble exactly:
Eθ0​,ϕ0​​[ϵ2(x)]=Vθ0​​[f(x;θ∞​)].
This is the first formal characterization of RND in the NTK regime, and it provides a direct theoretical justification for the algorithm's empirical behavior: expected self-predictive error quantifies ensemble variance for every input.
To connect to practical multi-headed RND implementations, the paper proves that output heads of a wide network with shared hidden layers are mutually independent GPs throughout training—an artifact of lazy training, since without feature learning the heads remain statistically independent despite sharing a body. Consequently, a g(x;θ0​)0-headed RND model produces i.i.d. errors, and the sample mean squared error obeys a distributional equality with the sample variance of a g(x;θ0​)1-member finite deep ensemble: half the mean RND error distributes as a scaled g(x;θ0​)2 random variable whose scale is the analytical ensemble variance. The practical implication is that a multi-headed RND model is, in the infinite-width limit, statistically indistinguishable from a finite deep ensemble at a fraction of the memory and compute cost.
Posterior equivalence via target engineering
The second contribution shows the framework is strictly more expressive than the ensemble equivalence suggests. The mismatch between the NNGP kernel g(x;θ0​)3 and dynamics kernel g(x;θ0​)4 can be eliminated by engineering the RND target. The construction exploits the fact that the last-layer NTK component equals the last-layer NNGP component, g(x;θ0​)5, because the output is linear in last-layer parameters. Defining the modified target as
g(x;θ0​)6
where g(x;θ0​)7 copies g(x;θ0​)8 with last-layer weights set to zero, yields a target GP with kernel g(x;θ0​)9, independent of K0. The resulting error prior kernel is then K1, aligning prior and dynamics kernels. Under this "Bayesian RND" construction, the post-convergence error distribution has covariance
K2
which is exactly the centered posterior predictive covariance of a Bayesian model with NTK prior K3 conditioned on K4. The error signal is therefore not merely a heuristic distance measure but a genuine sample from the Bayesian posterior predictive distribution of an infinitely wide network. This result builds on the Bayesian deep ensembles of He et al., transferring their kernel-alignment idea into the distillation setting.
The multi-headed version yields an immediately actionable corollary: each head's error is an independent draw from the centered posterior predictive, so adding i.i.d. RND errors to any estimate of the posterior mean generates up to K5 i.i.d. posterior samples. This constitutes a posterior sampling algorithm requiring only a single model and a Jacobian-vector product to construct the target—substantially cheaper than MCMC, variational inference, or maintaining an explicit ensemble.
Numerical validation
The empirical section is limited in scope but consistent with the theory. On synthetic regression tasks with single-layer SiLU networks, per-test-sample squared Bayesian RND errors are compared against Monte Carlo estimates of the true GP variance computed from 1024 independent models, using a 1024-head Bayesian RND model. Predictive variances become increasingly correlated and correctly calibrated in scale as network width grows, matching the asymptotic prediction. The authors note that residual discrepancy at finite width is to be expected from the finite Monte Carlo approximation and finite head count. Notably, no experiments involve realistic architectures, datasets, or downstream tasks such as exploration or OOD detection, so the practical benefit of Bayesian RND over standard RND remains untested empirically.
Limitations and open questions
All equivalences hold under strong assumptions that the paper states plainly. First, the analysis is confined to the infinite-width NTK regime, where networks train lazily as kernel machines without feature learning; whether the ensemble and posterior equivalences survive at finite width with feature learning is explicitly left open. Second, training is assumed to be full-batch gradient flow, excluding stochastic minibatch optimizers used universally in practice, although NTK extensions to stochastic noise exist in principle. Third, the dataset is fixed, whereas RND is most prominently applied in online reinforcement learning with non-stationary data streams; characterizing how these equivalences degrade under distribution shift remains an open problem. Finally, the numerical study validates only variance calibration on toy regression, leaving the downstream utility of Bayesian RND unexamined. A constructive direction the authors identify is that observed deviations between RND, ensembles, and Bayesian posteriors in practice must stem precisely from departures from the NTK regime, so quantifying these deviations could inform principled uncertainty methods outside the kernelized limit; they also propose "target engineering" as a cheap lever for studying priors in Bayesian deep learning.
Conclusion
This work supplies the first rigorous account of what RND measures: in the infinite-width NTK limit, standard RND errors recover deep ensemble predictive variance exactly, and with a suitably engineered target, the error distribution coincides with the centered posterior predictive of exact Bayesian inference, enabling i.i.d. posterior sampling from a single distillation model. The results unify three otherwise disparate uncertainty quantification approaches under one theoretical lens and give a principled explanation for RND's empirical effectiveness. The reach of these conclusions beyond lazy-training regimes—and their value in applications such as exploration—remain open questions that follow directly from the stated assumptions.