---
title: Ensemble Distribution Distillation (EDD)
url: https://www.emergentmind.com/topics/ensemble-distribution-distillation-edd
type: topic
---

# Ensemble Distribution Distillation (EDD)

Ensemble Distribution Distillation (EDD), originally introduced as EnD$^2$, is a family of distillation methods in which a single student model is trained to reproduce not merely the mean prediction of an ensemble, but a distribution over the ensemble’s predictions. In the canonical classification setting, the student outputs parameters of a Dirichlet distribution over class-probability vectors; later variants move to logistic-normal, Gaussian, Laplace, or function-space distributions over logits or predictive parameters. The defining aim is to retain the ensemble’s predictive accuracy together with its uncertainty structure—especially the separation between aleatoric and epistemic uncertainty—while replacing the ensemble’s $M$ forward passes by single-model inference or by inexpensive sampling from a learned student distribution [1905.00076] [2002.11531] [2305.10384].

## 1. Historical emergence and conceptual scope

EDD arose from the observation that ordinary ensemble distillation and standard knowledge distillation preserve the ensemble’s average soft target but discard its diversity. The original EnD$^2$ formulation treated the per-model predictive distributions of a deep ensemble as samples from a Dirichlet distribution and trained a single Prior Network student to fit that distribution. This allowed the student to preserve both the ensemble mean and its dispersion, so that uncertainty decomposition remained available after compression [1905.00076].

Subsequent work generalized the idea beyond the original Dirichlet-prior-network setting. A general framework reformulated EDD as learning a distribution over ensemble predictive parameters $z$, applicable to both regression and classification, with Gaussian or logistic-normal students as natural choices [2002.11531]. Later work addressed failure modes of the original maximum-likelihood Dirichlet objective when the number of classes is very large, introduced logit-based formulations for machine translation and other autoregressive sequence tasks, and extended EDD to federated learning, self-distillation, function-space distillation, Gaussian latent-factor models, and self-supervised human activity recognition [2105.06987] [2305.10384].

The resulting literature uses several closely related names—EnD$^2$, EnDD, EDD, distribution distillation, and predictive distribution distillation—but the common principle is consistent: the student is trained to model the ensemble’s output distribution rather than only its averaged prediction.

## 2. Probabilistic formulation and uncertainty decomposition

For an ensemble of $M$ teachers, the ensemble predictive distribution is approximated by averaging member predictions. In the sequence case this is written per timestep $t$ as
$$
p(y_t \mid x,\mathcal D)\approx \frac{1}{M}\sum_{m=1}^M p_m(y_t\mid x).
$$
In classification, total predictive uncertainty is the entropy of this averaged predictive distribution, while the aleatoric component is the mean per-model entropy and the epistemic component is the residual mutual information:
$$
H[y_t\mid x,\mathcal D]
=
\mathbb E_{\theta}\!\left[H[y_t\mid x,\theta]\right]
+
I[y_t,\theta\mid x,\mathcal D].
$$
Equivalently,
$$
I[y_t,\theta\mid x,\mathcal D]
=
H[y_t\mid x,\mathcal D]
-
\mathbb E_{\theta}\!\left[H[y_t\mid x,\theta]\right].
$$
This is the central decomposition EDD attempts to preserve after compressing the ensemble [1905.00076] [2305.10384].

In the original Dirichlet-based formulation, the student predicts concentration parameters $\alpha(x)$ over the simplex. The mean categorical probabilities are
$$
\mathbb E[p_k]=\frac{\alpha_k}{\alpha_0},
\qquad
\alpha_0=\sum_{k=1}^K \alpha_k.
$$
The concentration $\alpha_0$ acts as evidence: large $\alpha_0$ corresponds to a sharp distribution over class-probability vectors and thus low epistemic uncertainty, while small $\alpha_0$ corresponds to diffuse distributions and high epistemic uncertainty [1905.00076].

The same decomposition extends to regression through variance rather than entropy. For an ensemble of predictive distributions,
$$
\mathrm{Var}(y\mid x,D)
=
\mathbb E_w[\mathrm{Var}(y\mid x,w)]
+
\mathrm{Var}_w(\mathbb E[y\mid x,w]).
$$
Here the first term is aleatoric and the second epistemic. The general EDD framework preserves this by fitting a student distribution over the ensemble’s predictive parameters rather than distilling only a single Gaussian with total variance [2002.11531].

A central conceptual distinction therefore separates EDD from standard knowledge distillation. KD matches the mean teacher prediction; EDD matches the distribution of teacher predictions. This is why KD can preserve accuracy yet erase epistemic uncertainty, whereas EDD can reconstruct entropy- and mutual-information-based uncertainty measures from a single student [2305.10384].

## 3. Student distributions and training objectives

EDD is not tied to a single student family. What remains fixed is the target: the empirical distribution of ensemble outputs for a given input.

| Student family | Support | Representative use |
|---|---|---|
| Dirichlet | Probability simplex | Original EnD$^2$, proxy-target EDD, HAR |
| Logistic-normal / Gaussian | Logits or predictive parameters | General classification/regression EDD |
| Diagonal Laplace over logits | Normalized token logits | Large-vocabulary translation |
| Function-space or GP-like students | Joint outputs across inputs | Functional and Gaussian distillation |

The canonical EnD$^2$ objective assumes that ensemble member probabilities $\{p^{(m)}(x)\}_{m=1}^M$ are i.i.d. samples from a Dirichlet distribution and fits the student by maximum likelihood:
$$
\mathcal L_{\mathrm{EnD}^2}(x)
=
-
\sum_{m=1}^M
\log \mathrm{Dir}\!\big(p^{(m)}(x)\mid \alpha(x)\big).
$$
This directly fits both the ensemble mean and dispersion. Practical parameterizations either predict each $\alpha_k(x)$ directly via a positive transform such as softplus, or separate the mean $\pi(x)$ from a scalar evidence term and set $\alpha_k(x)=e(x)\,\pi_k(x)$ [1905.00076].

A broader EDD formulation replaces the Dirichlet by a learned distribution over ensemble predictive parameters,
$$
v\!\left(z; g_\varphi(x)\right),
$$
and trains the student by maximizing the likelihood of teacher outputs $z_j=f_{w_j}(x)$:
$$
L_{\mathrm{DD}}(\varphi)
=
-
\mathbb E_{p(x)}
\left[
\frac{1}{M}\sum_{j=1}^M
\log v\!\left(z_j; g_\varphi(x)\right)
\right].
$$
This generalization admits Gaussian students for regression and logistic-normal students for classification, and it preserves the aleatoric/epistemic split because the student explicitly models variability across ensemble predictive parameters [2002.11531].

For many-class problems, however, the original Dirichlet maximum-likelihood loss becomes poorly behaved. The core pathology identified in the scaling study is that classes with very low probability induce larger gradients than high-probability classes, forcing optimization to focus on tail-class probabilities. The proposed remedy is a Proxy-Dirichlet target constructed from ensemble mean and variance and trained with reverse KL,
$$
\mathcal L_{\mathrm{RKL}}(x)
=
\mathrm{KL}\!\big(
\mathrm{Dir}(\alpha^{\mathrm{proxy}}(x))
\;\|\;
\mathrm{Dir}(\alpha^{\mathrm{pred}}(x))
\big),
$$
often with evidence clipping, temperature smoothing, and top-$K$ truncation for stability [2105.06987].

This line of work also clarified a modeling limitation of the original approach. A single Dirichlet is unimodal on the simplex. When ensemble predictions are genuinely multimodal, a single Dirichlet can only be a compact approximation rather than an exact representation. Later work therefore explored logistic-normal, Gaussian, Laplace, and function-space alternatives rather than treating the Dirichlet as universally sufficient [1905.00076] [2002.11531].

## 4. Autoregressive sequence modeling and large vocabularies

Sequence-to-sequence applications introduced a distinct difficulty: EDD must be performed at each decoder step, conditioned on $(x,y_{<l})$, over very large vocabularies. In grammatical error correction, this led to token-level EnDD students that output Dirichlet parameters $\alpha_l$ for each step and are trained either by direct Dirichlet negative log-likelihood over teacher probabilities or by a two-stage objective that first fits a per-token teacher Dirichlet and then minimizes Dirichlet KL. The study also reported that temperature annealing from $T:10.0\to 3.0$ was important for stability, while $T=1.0$ was unstable. Because teacher forcing at training and free-run decoding at test time create a mismatch, a Guided Uncertainty Approach (GUA) was introduced: an EnD student performs decoding, while an EnDD student computes uncertainties on the decoded prefix [2012.07535].

Large-vocabulary machine translation motivated a more radical change. Probability-space EDD was found hard to scale because Dirichlet-based optimization becomes problematic when vocabularies are large and distributions are sparse and concentrated. The logit-based translation study therefore modeled the ensemble logits directly, using diagonal Laplace or diagonal Gaussian students over normalized token logits. This retained $O(V)$ per-timestep complexity, avoided full covariance $O(V^2)$ parameterization, decoded deterministically via $\mathrm{softmax}(\mu)$, and estimated uncertainty by sampling logits from the student distribution and pushing them through softmax [2305.10384].

The empirical results of this logit-space formulation were unusually strong. On WMT’16 En-De with a base Transformer, standard training yielded $25.85 \pm 0.17$ BLEU, the deep ensemble $26.72$, KD from the snapshot ensemble $27.02 \pm 0.19$, and Laplace L-EDD $27.08 \pm 0.20$. On the same setting, out-of-distribution detection improved sharply: for Khresmoi, deep-ensemble total uncertainty (TU) and knowledge uncertainty (KU) AUROC were $48.0$ and $61.9$, whereas Laplace L-EDD reached $65.1 \pm 1.8$ and $73.1 \pm 1.7$; on WMT’20 En-Ru with a big Transformer, L-EDD also improved Khresmoi TU/KU from $39.3/53.2$ for the deep ensemble to $51.0 \pm 0.9/63.4 \pm 1.2$ [2305.10384].

These sequence studies established two distinct lessons. First, EDD can be made compatible with autoregressive factorization by distilling per-step predictive distributions. Second, large-vocabulary settings often favor logit-space students over probability-space Dirichlet students, especially when robustness and OOD scoring are central objectives.

## 5. Extensions across learning settings

EDD has been adapted to distributed, self-distilled, function-space, and Gaussian-process-like settings. In federated learning, FedDF instantiates distribution-level distillation by treating client models as teachers, averaging their logits on unlabeled server-side data, and training a server student with KL divergence to the aggregated predictive distribution. Because aggregation occurs through predictions rather than parameter alignment, heterogeneous client architectures, sizes, and precisions are supported. Empirically, on CIFAR-10 with ResNet-8 under moderate non-IID conditions ($\alpha=1$, $C=0.4$), FedDF reached $80\%$ accuracy in $28\pm10$ rounds with $20$ local epochs, versus $97\pm29$ for FedAvg; under strong non-IID conditions ($\alpha=0.1$), it reached $75\%$ in $51\pm4$ rounds versus $445\pm8$ for FedAvg [2006.07242].

Self-Distribution Distillation (S2D) internalized EDD within a single network. A stochastic teacher branch generates multiple predictions per input using multiplicative Gaussian noise, while a student branch outputs a Dirichlet distribution trained by KL to a proxy Dirichlet fitted to those stochastic predictions. Hierarchical Distribution Distillation (H2D) then distills ensembles of such S2D models either into a single Dirichlet student (H2D-Dir) or into a Gaussian over log-concentrations (H2D-Gauss). On CIFAR-100, H2D-Gauss reached accuracy/NLL/ECE of $77.5/0.77/1.39$, compared with $77.9/0.91/10.36$ for the EnD baseline, and single-pass H2D students often rivaled or exceeded MC/SWAG ensembles on OOD detection [2203.08295].

Functional Ensemble Distillation (FED) criticized a structural limitation of classical EDD: per-input distributions over predictions do not provide cross-input covariance. FED therefore introduced a student generator with shared noise $\epsilon$ and matched the ensemble in function space using MMD over batchwise function representations. In its CIFAR-100 comparison, classical EnDD achieved accuracy $71.16$, agreement $100.0$, and ECE $14.51$, while FED reached $74.48 \pm 0.10$, agreement $87.82 \pm 0.60$, and ECE $1.64 \pm 0.17$. The same study also argued that distilling only on training data can collapse diversity when ensemble members are nearly identical on the training set, and found mixup-based auxiliary data crucial in this regime [2206.02183].

Gaussian distillation via the Deep Latent Factor (DLF) model moved EDD toward Gaussian-process-like students over functions or logits. Each teacher is treated as a realization of a latent-factor Gaussian process, and the student learns mean and covariance structure by an EM algorithm. This formulation applies to both regression and classification. On CIFAR-10, the reported DLF student achieved $93.40\%$ accuracy, NLL $0.2246$, and ECE $2.79\%$; for OOD detection using predictive mutual information, AUROC reached $0.9359$ on SVHN, $0.8357$ on CIFAR-100, and $0.8291$ on Tiny ImageNet [2510.19290].

Together these variants show that EDD is not a single architecture but a methodological principle. The student may be a Prior Network, a logistic-normal model, a Laplace logit model, a function generator with shared noise, or a latent-factor Gaussian model, provided that it is trained to match ensemble diversity rather than only ensemble means.

## 6. Empirical profile, limitations, and recurring misconceptions

Across image-classification benchmarks, the original EnD$^2$ study showed that a single EDD student could approach ensemble classification performance while outperforming standard DNNs and standard ensemble distillation on misclassification and out-of-distribution detection on CIFAR-10, CIFAR-100, and TinyImageNet [1905.00076]. Later domain-specific work reported similar benefits in self-supervised human activity recognition. On HHAR, a single model achieved $67.84 \pm 1.64$ clean accuracy and $37.31 \pm 2.48$ under FGSM $\epsilon=0.1$, while EDD reached $82.61 \pm 2.51$ clean and $63.05 \pm 3.83$ adversarial accuracy; on the same dataset, uncertainty-as-discriminator AUC-ROC improved from $0.758$ to $0.854$ on clean data and from $0.444$ to $0.732$ under attack [2509.08225].

The main limitations are recurrent and technically specific. Dirichlet maximum-likelihood training scales poorly to very large label spaces because gradient magnitude is dominated by tail probabilities [2105.06987]. A single Dirichlet is unimodal and may misrepresent multimodal ensembles [1905.00076]. Per-input EDD formulations do not recover cross-input covariance, which motivated FED’s function-space alternative [2206.02183]. In seq2seq tasks, teacher-forcing/free-run mismatch makes EnDD harder to optimize than EnD, and the GEC study found that EnDD alone lagged EnD on GLEU, whereas GUA recovered EnD accuracy while preserving ensemble-like uncertainty [2012.07535]. In large-vocabulary translation, diagonal logit distributions ignore vocabulary correlations, decoding uses the approximation $\mathbb E[\mathrm{softmax}(z)]\approx \mathrm{softmax}(\mu)$, and consistent logit normalization is required because softmax is shift-invariant [2305.10384].

Several misconceptions recur in the literature. EDD is not equivalent to ordinary KD: KD matches the ensemble mean and therefore cannot represent epistemic uncertainty; EDD learns a distribution over teacher outputs and can reconstruct mutual-information-based uncertainty [2305.10384]. Nor does “single-pass uncertainty” always mean fully deterministic uncertainty evaluation: logit-based and Gaussian EDD variants often decode from a deterministic mean but estimate entropy or mutual information by Monte Carlo sampling from the student distribution [2305.10384] [2510.19290]. Finally, EDD is not guaranteed to dominate ensembles in every regime. Some studies reported that its performance depends strongly on teacher diversity, auxiliary or distillation data, augmentation coverage, and the chosen student family; overconfident or insufficiently diverse teachers can collapse the very signal EDD is meant to preserve [2203.08295] [2206.02183].

In this sense, EDD is best understood as a distribution-matching framework for ensemble compression rather than a single algorithm. Its distinctive contribution is to turn ensemble diversity into an explicit student output object—Dirichlet, logistic-normal, Laplace, Gaussian, or functional—so that uncertainty decomposition survives compression and can be deployed under single-model or near-single-model computational budgets.

Source: https://www.emergentmind.com/topics/ensemble-distribution-distillation-edd