Papers
Topics
Authors
Recent
Search
2000 character limit reached

On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inference

Published 23 Feb 2026 in cs.LG, cs.AI, math.PR, and stat.ML | (2602.19964v1)

Abstract: Uncertainty quantification is central to safe and efficient deployments of deep learning models, yet many computationally practical methods lack lacking rigorous theoretical motivation. Random network distillation (RND) is a lightweight technique that measures novelty via prediction errors against a fixed random target. While empirically effective, it has remained unclear what uncertainties RND measures and how its estimates relate to other approaches, e.g. Bayesian inference or deep ensembles. This paper establishes these missing theoretical connections by analyzing RND within the neural tangent kernel framework in the limit of infinite network width. Our analysis reveals two central findings in this limit: (1) The uncertainty signal from RND -- its squared self-predictive error -- is equivalent to the predictive variance of a deep ensemble. (2) By constructing a specific RND target function, we show that the RND error distribution can be made to mirror the centered posterior predictive distribution of Bayesian inference with wide neural networks. Based on this equivalence, we moreover devise a posterior sampling algorithm that generates i.i.d. samples from an exact Bayesian posterior predictive distribution using this modified \textit{Bayesian RND} model. Collectively, our findings provide a unified theoretical perspective that places RND within the principled frameworks of deep ensembles and Bayesian inference, and offer new avenues for efficient yet theoretically grounded uncertainty quantification methods.

Summary

  • The paper proves that, under infinite-width NTK training, standard RND errors have the same expected uncertainty as deep ensemble predictive variance, including an exact multi-head distributional equivalence.
  • The paper introduces a target-engineering method that aligns the RND prior with the NTK dynamics kernel, making each resulting error an independent sample from the centered Bayesian posterior predictive distribution.
  • The paper validates variance calibration on synthetic regression as network width increases, while leaving finite-width feature learning, stochastic optimization, online data, and real-world applications as open challenges.

Random network distillation (RND) is a widely used uncertainty quantification technique in which a predictor network is trained to match the output of a fixed, randomly initialized target network, with the squared prediction error serving as a novelty or epistemic uncertainty signal. Despite its empirical success in exploration, out-of-distribution detection, and continual learning, the theoretical status of the RND error signal has remained unclear. This paper by Zanger et al. (2602.19964) supplies that missing theory: working within the neural tangent kernel (NTK) framework in the limit of infinite network width, it proves that RND errors coincide with deep ensemble variance, and that a deliberately engineered target function makes the RND error distribution identical to the centered posterior predictive distribution of Bayesian inference with infinitely wide networks. The analysis thereby places a computationally cheap single-model method within the same theoretical framework as ensembles and exact Bayesian inference.

Background: RND and NTK learning dynamics

RND trains a predictor u(x;θt)u(x;\theta_t) against a fixed random target g(x;θ0)g(x;\theta_0) under a squared loss, and uses the mean squared error across KK output heads as an uncertainty score. The paper analyzes fully connected networks under NTK parametrization trained by gradient flow on square loss. In the infinite-width limit, the neural tangent kernel becomes deterministic at initialization and constant during training, so the function-space dynamics are linear and converge to the kernel regression solution; moreover, both initializations and converged functions are Gaussian processes, governed by the NNGP kernel κ\kappa and the training dynamics kernel Θ\Theta respectively.

A key structural observation is that the distribution over converged network functions given in prior work is not directly interpretable as a Bayesian posterior predictive distribution: its covariance involves two distinct kernels (κ\kappa and Θ\Theta), whereas a posterior predictive covariance takes the form of a prior kernel conditioned through the data. Resolving this mismatch is the central technical device of the paper.

Equivalence between standard RND and deep ensembles

The first main result characterizes the law of the post-convergence RND error ϵ(x;θ∞,θ0)=u(x;θ∞)−g(x;θ0)\epsilon(x;\theta_\infty,\theta_0) = u(x;\theta_\infty) - g(x;\theta_0). Because gradient-flow dynamics are linear, the converged predictor is an affine transformation of its initialization, which follows the NNGP; since this transformation is independent of the target draw, the error is itself a zero-mean GP whose covariance is built from the combined initialization kernel κxx′ϵ=κxx′u+κxx′g\kappa^\epsilon_{xx'} = \kappa^u_{xx'} + \kappa^g_{xx'} propagated through the conditioning structure. Taking architectural equivalence between predictor, target, and an ensemble member, the expected squared RND error equals the predictive variance of the corresponding infinite deep ensemble exactly:

Eθ0,ϕ0[ϵ2(x)]=Vθ0[f(x;θ∞)].\mathbb{E}_{\theta_0,\phi_0}\big[\epsilon^2(x)\big] = \mathbb{V}_{\theta_0}\big[f(x;\theta_\infty)\big].

This is the first formal characterization of RND in the NTK regime, and it provides a direct theoretical justification for the algorithm's empirical behavior: expected self-predictive error quantifies ensemble variance for every input.

To connect to practical multi-headed RND implementations, the paper proves that output heads of a wide network with shared hidden layers are mutually independent GPs throughout training—an artifact of lazy training, since without feature learning the heads remain statistically independent despite sharing a body. Consequently, a g(x;θ0)g(x;\theta_0)0-headed RND model produces i.i.d. errors, and the sample mean squared error obeys a distributional equality with the sample variance of a g(x;θ0)g(x;\theta_0)1-member finite deep ensemble: half the mean RND error distributes as a scaled g(x;θ0)g(x;\theta_0)2 random variable whose scale is the analytical ensemble variance. The practical implication is that a multi-headed RND model is, in the infinite-width limit, statistically indistinguishable from a finite deep ensemble at a fraction of the memory and compute cost.

Posterior equivalence via target engineering

The second contribution shows the framework is strictly more expressive than the ensemble equivalence suggests. The mismatch between the NNGP kernel g(x;θ0)g(x;\theta_0)3 and dynamics kernel g(x;θ0)g(x;\theta_0)4 can be eliminated by engineering the RND target. The construction exploits the fact that the last-layer NTK component equals the last-layer NNGP component, g(x;θ0)g(x;\theta_0)5, because the output is linear in last-layer parameters. Defining the modified target as

g(x;θ0)g(x;\theta_0)6

where g(x;θ0)g(x;\theta_0)7 copies g(x;θ0)g(x;\theta_0)8 with last-layer weights set to zero, yields a target GP with kernel g(x;θ0)g(x;\theta_0)9, independent of KK0. The resulting error prior kernel is then KK1, aligning prior and dynamics kernels. Under this "Bayesian RND" construction, the post-convergence error distribution has covariance

KK2

which is exactly the centered posterior predictive covariance of a Bayesian model with NTK prior KK3 conditioned on KK4. The error signal is therefore not merely a heuristic distance measure but a genuine sample from the Bayesian posterior predictive distribution of an infinitely wide network. This result builds on the Bayesian deep ensembles of He et al., transferring their kernel-alignment idea into the distillation setting.

The multi-headed version yields an immediately actionable corollary: each head's error is an independent draw from the centered posterior predictive, so adding i.i.d. RND errors to any estimate of the posterior mean generates up to KK5 i.i.d. posterior samples. This constitutes a posterior sampling algorithm requiring only a single model and a Jacobian-vector product to construct the target—substantially cheaper than MCMC, variational inference, or maintaining an explicit ensemble.

Numerical validation

The empirical section is limited in scope but consistent with the theory. On synthetic regression tasks with single-layer SiLU networks, per-test-sample squared Bayesian RND errors are compared against Monte Carlo estimates of the true GP variance computed from 1024 independent models, using a 1024-head Bayesian RND model. Predictive variances become increasingly correlated and correctly calibrated in scale as network width grows, matching the asymptotic prediction. The authors note that residual discrepancy at finite width is to be expected from the finite Monte Carlo approximation and finite head count. Notably, no experiments involve realistic architectures, datasets, or downstream tasks such as exploration or OOD detection, so the practical benefit of Bayesian RND over standard RND remains untested empirically.

Limitations and open questions

All equivalences hold under strong assumptions that the paper states plainly. First, the analysis is confined to the infinite-width NTK regime, where networks train lazily as kernel machines without feature learning; whether the ensemble and posterior equivalences survive at finite width with feature learning is explicitly left open. Second, training is assumed to be full-batch gradient flow, excluding stochastic minibatch optimizers used universally in practice, although NTK extensions to stochastic noise exist in principle. Third, the dataset is fixed, whereas RND is most prominently applied in online reinforcement learning with non-stationary data streams; characterizing how these equivalences degrade under distribution shift remains an open problem. Finally, the numerical study validates only variance calibration on toy regression, leaving the downstream utility of Bayesian RND unexamined. A constructive direction the authors identify is that observed deviations between RND, ensembles, and Bayesian posteriors in practice must stem precisely from departures from the NTK regime, so quantifying these deviations could inform principled uncertainty methods outside the kernelized limit; they also propose "target engineering" as a cheap lever for studying priors in Bayesian deep learning.

Conclusion

This work supplies the first rigorous account of what RND measures: in the infinite-width NTK limit, standard RND errors recover deep ensemble predictive variance exactly, and with a suitably engineered target, the error distribution coincides with the centered posterior predictive of exact Bayesian inference, enabling i.i.d. posterior sampling from a single distillation model. The results unify three otherwise disparate uncertainty quantification approaches under one theoretical lens and give a principled explanation for RND's empirical effectiveness. The reach of these conclusions beyond lazy-training regimes—and their value in applications such as exploration—remain open questions that follow directly from the stated assumptions.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.