Ensemble Distillation: Efficient Model Compression

• Ensemble distillation is a technique that compresses multiple predictive models into one efficient student model while striving to maintain the ensemble’s accuracy, uncertainty, and diversity.
• It employs various strategies such as mean-matching, multi-head architectures, and distribution alignment to overcome the loss of model diversity and preserve calibration.
• Applications span machine translation, autonomous driving, medical imaging, and speech processing, offering reduced inference costs while retaining robust performance.

Ensemble distillation is the compression of an ensemble of predictive models into a cheaper student model while attempting to preserve the ensemble’s predictive behavior, and in some variants its uncertainty structure, diversity, calibration, or robustness. In the literature covered here, the object being distilled ranges from the ensemble’s mean predictive distribution, to full distributions over predictions, to sequence-level outputs, trajectory distributions, and even distributions over functions. The topic therefore spans classical mean-matching objectives, uncertainty-preserving distribution distillation, multi-head and multi-branch students, adaptive teacher aggregation, and domain-specific formulations for structured prediction, speech, medical imaging, autonomous driving, and robustness-sensitive settings (Freitag et al., 2017, Malinin et al., 2019, Tran et al., 2020, Penso et al., 2022, Ettinger et al., 2024, Kenfack et al., 2024).

1. Foundational formulation

A standard starting point is to treat the ensemble as a teacher that defines an averaged predictive distribution and to train a single student against that average. For classification, one recurring notation is

$$\bar{\boldsymbol{\pi}}(x)=\frac1M\sum_{m=1}^M \boldsymbol{\pi}^{(m)}(x),$$

which preserves the ensemble mean but not the disagreement among ensemble members. In a temperature-scaled formulation, one representative objective is

$$\mathcal{L}_{\text{ensKD}} = \tau^2 \cdot KL\!\left(\sigma\!\left(\frac{z^{s}}{\tau}\right),\frac{1}{M}\sum_{m=1}^{M}\sigma\!\left(\frac{z^{m}}{\tau}\right)\right),$$

and structured-prediction work often uses a weighted mixture of gold-label supervision and soft-target supervision from the ensemble (Kenfack et al., 2024, Ryabinin et al., 2021, Reich et al., 2020).

Within the papers cited here, early sequence-level work in neural machine translation already showed that ensemble distillation need not be limited to token-level soft targets. One influential formulation translates the full training corpus with a stronger teacher—especially a 6-model ensemble teacher—and trains the student on teacher-generated target sequences, either alone or together with the original references. In that setting, distillation is implemented as ordinary sequence training on modified target data rather than explicit token-level KL matching, and teacher-based data filtering with $\operatorname{TER} \le 0.8$ is used to prune noisy sentence pairs (Freitag et al., 2017).

This basic mean-matching picture remains the canonical baseline across later work, but the same literature repeatedly identifies its central limitation: averaging compresses the ensemble into a single first moment. That is sufficient for some accuracy gains, yet it removes member-wise diversity, covariances between predictions, and uncertainty decompositions that motivate ensembles in the first place (Malinin et al., 2019, Tran et al., 2020, Penso et al., 2022).

2. Student architectures and internal ensemble structure

The student in ensemble distillation is not always a plain monolithic network. One line of work uses explicitly ensemble-like students. “Ensemble Knowledge Distillation for Learning Improved and Efficient Networks” trains a compact multi-branch student, CompNet, against a heterogeneous teacher ensemble. Each branch predicts logits $Q_i^s$, the student combines them by

$P_s = \sum_{i=1}^{N_s} Q_i^s,$

and distillation is applied both at the branch level and at the ensemble level. The paper’s central claim is that branch-specific supervision from different teachers promotes heterogeneous branch learning, while summation of branch logits reduces variance in the final prediction (Asif et al., 2019).

Hydra pushes this idea in a different direction by replacing a single distilled predictor with a shared body and one head per teacher ensemble member. Its central loss is

$$\mathcal{L}(x,y;\theta_{\mathrm{hydra}},\theta_{\mathrm{ens}}) = \frac{1}{M}\sum_{m=1}^M \mathrm{KL}\!\big( p(y\mid x;\theta_{\mathrm{ens},m}) \parallel p(y\mid x;\theta_{\mathrm{body}},\theta_{\mathrm{head},m}) \big),$$

which is deliberately different from standard mean-only distillation. The average over teacher members is pulled outside the KL, so each head is trained to imitate a distinct ensemble member. This preserves member-level diversity more faithfully and allows the distilled model to approximate disagreement-based uncertainty with lower cost than a full ensemble, though at higher cost than a single-head student (Tran et al., 2020).

A related architectural theme appears in progressive ensemble distillation. B-DISTIL does not distill an ensemble into one student; it decomposes a large teacher $g$ into an ordered set of small students $f_1,f_2,\dots$ whose average

$F_T(x) = \frac{1}{T}\sum_{t=1}^T f_t(x)$

progressively approaches the teacher as more members are evaluated. Later students can reuse intermediate activations from earlier ones, which the paper uses to support runtime-adaptive inference under changing compute budgets (Dennis et al., 2023).

Representation-space distillation provides another architectural variant. For sentence embeddings, “Sentence Embeddings by Ensemble Distillation” uses a student with the same architecture as the teachers and trains it by mean-squared error to reproduce the average sentence embedding

$S_E = \frac{1}{N} \sum_{i=1}^{N} M_i(S).$

This replaces logit-space KD with direct embedding-space regression and produces a single encoder that is close to the full ensemble while being much more robust across runs (Sahlgren, 2021).

3. Distilling uncertainty, diversity, and predictive distributions

A major branch of ensemble distillation argues that the correct target is not the ensemble mean but the full distribution of ensemble predictions. “Ensemble Distribution Distillation” formalizes this by treating the teacher outputs $$\mathcal{L}_{\text{ensKD}} = \tau^2 \cdot KL\!\left(\sigma\!\left(\frac{z^{s}}{\tau}\right),\frac{1}{M}\sum_{m=1}^{M}\sigma\!\left(\frac{z^{m}}{\tau}\right)\right),$$0 as samples from an empirical distribution

$$\mathcal{L}_{\text{ensKD}} = \tau^2 \cdot KL\!\left(\sigma\!\left(\frac{z^{s}}{\tau}\right),\frac{1}{M}\sum_{m=1}^{M}\sigma\!\left(\frac{z^{m}}{\tau}\right)\right),$$1

and fitting a Prior Network that outputs Dirichlet parameters $$\mathcal{L}_{\text{ensKD}} = \tau^2 \cdot KL\!\left(\sigma\!\left(\frac{z^{s}}{\tau}\right),\frac{1}{M}\sum_{m=1}^{M}\sigma\!\left(\frac{z^{m}}{\tau}\right)\right),$$2. The distilled student then exposes the predictive mean

$$\mathcal{L}_{\text{ensKD}} = \tau^2 \cdot KL\!\left(\sigma\!\left(\frac{z^{s}}{\tau}\right),\frac{1}{M}\sum_{m=1}^{M}\sigma\!\left(\frac{z^{m}}{\tau}\right)\right),$$3

while also retaining concentration information that supports entropy, expected entropy, mutual-information-style uncertainty, and variance over class probabilities (Malinin et al., 2019).

“A general framework for ensemble distribution distillation” generalizes this beyond classification. Instead of distilling only output probabilities, it learns a distribution

$$\mathcal{L}_{\text{ensKD}} = \tau^2 \cdot KL\!\left(\sigma\!\left(\frac{z^{s}}{\tau}\right),\frac{1}{M}\sum_{m=1}^{M}\sigma\!\left(\frac{z^{m}}{\tau}\right)\right),$$4

over ensemble predictive parameters $$\mathcal{L}_{\text{ensKD}} = \tau^2 \cdot KL\!\left(\sigma\!\left(\frac{z^{s}}{\tau}\right),\frac{1}{M}\sum_{m=1}^{M}\sigma\!\left(\frac{z^{m}}{\tau}\right)\right),$$5, with training objective

$$\mathcal{L}_{\text{ensKD}} = \tau^2 \cdot KL\!\left(\sigma\!\left(\frac{z^{s}}{\tau}\right),\frac{1}{M}\sum_{m=1}^{M}\sigma\!\left(\frac{z^{m}}{\tau}\right)\right),$$6

Predictions are then recovered by marginalization,

$$\mathcal{L}_{\text{ensKD}} = \tau^2 \cdot KL\!\left(\sigma\!\left(\frac{z^{s}}{\tau}\right),\frac{1}{M}\sum_{m=1}^{M}\sigma\!\left(\frac{z^{m}}{\tau}\right)\right),$$7

which preserves the separation between aleatoric uncertainty inside $$\mathcal{L}_{\text{ensKD}} = \tau^2 \cdot KL\!\left(\sigma\!\left(\frac{z^{s}}{\tau}\right),\frac{1}{M}\sum_{m=1}^{M}\sigma\!\left(\frac{z^{m}}{\tau}\right)\right),$$8 and epistemic uncertainty in the spread of $$\mathcal{L}_{\text{ensKD}} = \tau^2 \cdot KL\!\left(\sigma\!\left(\frac{z^{s}}{\tau}\right),\frac{1}{M}\sum_{m=1}^{M}\sigma\!\left(\frac{z^{m}}{\tau}\right)\right),$$9 (Lindqvist et al., 2020).

The many-class regime exposes a limitation of the original Dirichlet-likelihood formulation. “Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targets” shows that the standard Dirichlet log-likelihood has gradients dominated by the ensemble’s tiny tail probabilities when the class count is very large. The paper therefore replaces direct likelihood fitting with a Proxy-Dirichlet target and optimizes

$\operatorname{TER} \le 0.8$0

thereby stabilizing ImageNet- and WMT17-scale distillation (Ryabinin et al., 2021).

Functional Ensemble Distillation pushes the same argument one step further. Rather than returning only a distribution over predictions at one input, FED trains a stochastic student $\operatorname{TER} \le 0.8$1 and matches batches of teacher function samples and student function samples with MMD. Because one sampled noise variable $\operatorname{TER} \le 0.8$2 induces one coherent predictor across multiple inputs, FED can preserve predictive covariance between inputs, not only marginal uncertainty at each point. The paper also shows that mixup-generated distillation inputs are crucial in low-data settings where all teacher ensemble members interpolate the training set almost perfectly (Penso et al., 2022).

4. Adaptive aggregation, robustness, and nonuniform teacher weighting

A recurrent finding across the literature is that the arithmetic mean of teacher outputs is often an inadequate aggregation rule. UniKD addresses this directly by distinguishing labeled and unlabeled distillation. On labeled data, the teacher ensemble is weighted according to teacher correctness, and the student’s distillation loss is attenuated when average teacher loss is high: $\operatorname{TER} \le 0.8$3 On unlabeled data, teacher predictions are averaged, but the KD term is amplified by teacher disagreement measured with average pairwise KL divergence,

$\operatorname{TER} \le 0.8$4

This makes sample importance depend on disagreement when ground truth is unavailable (Wu et al., 2022).

Experience Ensemble Knowledge Distillation changes the source of the ensemble itself. Instead of training $\operatorname{TER} \le 0.8$5 independent teachers, it samples $\operatorname{TER} \le 0.8$6 intermediate checkpoints from a single teacher’s trajectory, combines them as

$\operatorname{TER} \le 0.8$7

and learns the weights with an attention mechanism. A striking conclusion reported in that paper is that strong ensemble teachers do not necessarily produce strong students, which it attributes to “cognitive conflicts” created by excessive diversity among teachers (Wang et al., 2022).

LatentBE addresses a different trade-off: how to absorb ensemble diversity during training while keeping single-model inference at test time. It trains a BatchEnsemble-style student with multiple subnetworks, distills each subnetwork one-to-one from a teacher, and then merges the subnetworks by weight averaging,

$\operatorname{TER} \le 0.8$8

Its perturbation strategy explicitly targets points where teacher diversity is high and student diversity is low,

$\operatorname{TER} \le 0.8$9

which improves calibration and corruption robustness while preserving single-model inference cost (Nam et al., 2022).

Robustness-sensitive aggregation becomes especially important when the ensemble is used to mitigate spurious correlations. “Adaptive Group Robust Ensemble Knowledge Distillation” shows that standard ensemble KD can substantially degrade worst-group accuracy even when the teacher ensemble itself is debiased. AGRE-KD introduces an additional biased reference model and weights teacher $Q_i^s$0 on sample $Q_i^s$1 by

$Q_i^s$2

Teachers whose gradients align with the biased shortcut direction are downweighted, while those whose gradients deviate from it are upweighted. The paper reports that this can substantially improve worst-group accuracy relative to AVER, Random, and AE-KD, and on CelebA can even surpass both the teacher ensemble and the majority-vote deep ensemble in worst-group accuracy (Kenfack et al., 2024).

5. Structured outputs and domain-specific realizations

The topic becomes especially rich in structured prediction, where the target is not a single label. In NER and machine translation, token-level ensemble distillation has been studied as a way to preserve both calibration and predictive quality. One formulation uses

$Q_i^s$3

with $Q_i^s$4 the averaged token-level teacher distribution. The paper shows that, across NER and NMT, distilled students retain much of the calibration and accuracy gains of ensembles while requiring only a single model during test-time (Reich et al., 2020).

For grammatical error correction, the same mean-preserving versus distribution-preserving distinction reappears in seq2seq form. The paper distinguishes EnD, which matches the average token-level teacher predictive distribution under teacher forcing, from EnDD, which fits a Dirichlet distribution over token-level predictive distributions and estimates sequence-level total, data, and knowledge uncertainty by Monte Carlo. It finds that pure sequence EnDD is substantially harder to optimize than EnD, and proposes a hybrid Guided Uncertainty Approach in which a standard distilled model is used for prediction and a distribution-distilled model is used for uncertainty estimation (Fathullah et al., 2020).

Speech and representation learning introduce yet أخرى forms of the problem. For self-supervised speech models, EKD over HuBERT, RobustHuBERT, and WavLM is performed at the level of hidden representations rather than logits. The best-performing variant uses multiple sets of prediction heads, one set per teacher, and improves a 23M student on Phoneme Recognition, Speaker Identification, Emotion Recognition, and Automatic Speech Recognition in the hidden-set track of SUPERB (Huang et al., 2023). For sentence embeddings, the student is trained to approximate the average embedding space from multiple teacher encoders, which improves unsupervised and zero-shot STS robustness without requiring an ensemble at inference time (Sahlgren, 2021).

In multimodal structured prediction for autonomous driving, the teacher is a mixture over trajectory distributions rather than labels. “Scaling Motion Forecasting Models with Ensemble Distillation” defines the ensemble forecast as

$Q_i^s$5

samples trajectories $Q_i^s$6 from that ensemble, and trains the student with

$Q_i^s$7

On WOMD, the reported student achieves $Q_i^s$8 mAP and $Q_i^s$9 soft-mAP at $P_s = \sum_{i=1}^{N_s} Q_i^s,$0 relative FLOPs, versus $P_s = \sum_{i=1}^{N_s} Q_i^s,$1 mAP and $P_s = \sum_{i=1}^{N_s} Q_i^s,$2 soft-mAP for a 20-model ensemble at $P_s = \sum_{i=1}^{N_s} Q_i^s,$3 FLOPs, illustrating the core deployment trade-off (Ettinger et al., 2024).

In retinal vessel segmentation, ensemble distillation is used as a calibration-preserving alternative to deep ensembles. The paper distills the ensemble-averaged predictive distribution of five FR-UNet models into a single student with

$P_s = \sum_{i=1}^{N_s} Q_i^s,$4

and evaluates calibration with ECE, Brier Score, NLL, and reliability diagrams on DRIVE and FIVES. The central empirical message is that KL-based output-distribution matching preserves calibration much better than the representation-distillation variant EnD-CRD while reducing inference from five forward passes to one (Fadugba et al., 15 Sep 2025).

6. Limitations, failure modes, and theoretical outlook

Several recurring limitations are explicit across the literature. Mean-only distillation often erases ensemble diversity and therefore weakens uncertainty estimation, covariance structure, or epistemic signals (Malinin et al., 2019, Penso et al., 2022). In many-class problems, the original Dirichlet likelihood overemphasizes tail probabilities and converges poorly (Ryabinin et al., 2021). In subgroup-robust classification, naively distilling even debiased teachers can re-amplify shortcut features in a smaller student (Kenfack et al., 2024). In sequence generation, teacher-forcing distillation can mismatch free-running inference, which is one reason sequence EnDD remains difficult in GEC (Fathullah et al., 2020). Even when the student is accurate, the full training pipeline still requires querying or storing the teacher ensemble, so the main savings are usually at deployment rather than end-to-end training (Freitag et al., 2017, Fadugba et al., 15 Sep 2025).

A further limitation is that stronger teachers are not uniformly better distillers. EEKD explicitly reports that stronger ensemble teachers do not necessarily produce stronger students, and AGRE-KD shows that all-biased teacher ensembles leave little useful diversity to exploit (Wang et al., 2022, Kenfack et al., 2024). This suggests that teacher diversity, teacher quality, and student-friendliness are distinct design variables rather than one scalar notion of “teacher strength.”

The most explicit recent theoretical synthesis frames multi-teacher ensemble distillation as probability-domain aggregation by an operator

$P_s = \sum_{i=1}^{N_s} Q_i^s,$5

The framework imposes five axioms—convexity preservation, positivity inheritance, weight monotonicity, continuity, and temperature coherence—and proves both existence and non-uniqueness of operator families satisfying them. It also gives a Jensen-type comparison between aggregated-target and sum-of-KLs training: $P_s = \sum_{i=1}^{N_s} Q_i^s,$6 with

$P_s = \sum_{i=1}^{N_s} Q_i^s,$7

when $P_s = \sum_{i=1}^{N_s} Q_i^s,$8. It also recovers classical variance-reduction results for aggregation operators linear in teacher weights and formalizes safety attenuation for convex moderation of extreme teacher probabilities (Flouro et al., 14 Jan 2026).

A plausible implication is that ensemble distillation is moving from fixed averaging heuristics toward operator design under explicit constraints: predictive accuracy, calibration, uncertainty preservation, structured multimodality, subgroup robustness, safety moderation, and deployment cost. The corpus summarized here suggests that no single target or aggregation rule dominates across all settings. Instead, the correct distilled object depends on what the ensemble was valuable for in the first place: average accuracy, member diversity, uncertainty decomposition, robustness to spurious correlations, structured calibration, or controllable inference under resource constraints.