---
title: Multi-Objective Knowledge Distillation (MOKD)
url: https://www.emergentmind.com/topics/multi-objective-knowledge-distillation-mokd
type: topic
---

# Multi-Objective Knowledge Distillation (MOKD)

Searching arXiv for relevant MOKD and related knowledge distillation papers to ground the article.
Multi-Objective Knowledge Distillation (MOKD) denotes knowledge-distillation regimes in which a student is optimized against multiple objectives rather than a single teacher–student loss. In the current literature, those objectives may originate from multiple tasks, multiple teacher models, multiple distillation paths, multiple knowledge levels or granularities, or joint prediction–uncertainty targets. The formulation is explicit in marketplace ranking, where objective-specific teachers are aggregated into soft labels [2407.07181], and in recent vision work that treats task loss and distillation loss as a two-objective optimization problem [2505.08170]. Closely related formulations also appear under adjacent names such as multi-task KD [1911.03588], multi-level KD [2012.00573], multi-granularity distillation [2108.06681], adaptive multi-path distillation [2110.09674], and “Multi-Mode Online Knowledge Distillation,” where the acronym MOKD refers to two distinct distillation modes whose optimization is explicitly multi-objective in effect [2304.06461].

## 1. Scope and terminology

The term is not fully standardized. In some papers, “multi-objective” refers to multiple business or task objectives; in others it refers to multiple KD signals, multiple branches, or multiple learning modes. What unifies these formulations is that the student is shaped by more than one supervisory criterion, and the criteria are not reducible to a single conventional soft-target loss.

| Formulation | Source of multiple objectives | Optimization pattern |
|---|---|---|
| Marketplace ranking MOKD [2407.07181] | Primary objective plus per-objective teacher soft labels | Constrained optimization relaxed into scalarized distillation |
| MoKD [2505.08170] | Task loss plus distillation loss | Explicit two-objective optimization with adaptive combination |
| MKD [1911.03588] | Per-task distillation across GLUE tasks | Unweighted sum over task losses |
| MLKD [2012.00573] | Alignment, correlation, supervised contrastive, and CE terms | Weighted-sum scalarization |
| MetaMixer [2303.07951] | Classification, mixed-logit KD, feature KD, endpoint KD | Weighted-sum online KD |
| DAUM distillation [2511.05582] | Supervised multi-task learning, predictive KD, uncertainty KD | Weighted sum with uncertainty regression |

A recurrent source of confusion is that “multi-objective” does not always imply Pareto-front optimization. Several MOKD systems are explicitly scalarized weighted sums, including MKD, MLKD, MetaMixer, the marketplace ranking formulation, and the smart-agriculture hybrid KD framework [1911.03588] [2012.00573] [2303.07951] [2407.07181] [2512.22239]. By contrast, MoKD formulates KD as a genuine multi-objective optimization with an adaptive update direction, and Adaptive Distillation includes an MGDA-style multi-objective baseline over KD paths [2505.08170] [2110.09674].

## 2. Formal optimization patterns

A canonical MOKD pattern is scalarization. In the ranking formulation deployed at Airbnb, multi-objective learning-to-rank is written first as an $\epsilon$-constraint problem over objective costs $C_k(X)$ and then relaxed into a distillation objective. Under the paper’s derivation, the student minimizes cross-entropy against the primary hard label and an aggregated teacher soft label,
$$
\underset{\theta}{\min}\; CE(f(\theta,X), L_1) + CE\!\left(f(\theta,X), \sum_{k=1}^{K}\omega_k f_k(\theta_k^*,X)\right),
$$
with aggregated soft label
$$
\sum_{k=1}^{K}\omega_k f_k(\theta_k^*,X),
$$
and a hard–soft mixture
$$
\mathrm{Loss} = -\alpha \sum_i l_i \log f(\theta, X_i)\;-\;(1-\alpha)\sum_i \hat{l}_i \log f(\theta, X_i).
$$
This converts constrained multi-objective ranking into single-model optimization while preserving closeness to objective-specific optima through teacher predictions [2407.07181].

A second scalarization pattern appears in multi-task KD. MKD distills only teacher logits and aggregates per-task losses by simple summation,
$$
L_{distill}^{t} = \lVert z_{T}^{t} - z_{S}^{t} \rVert^{2}_{2}, \qquad
L_{distill} = L_{distill}^{1}+L_{distill}^{2}+\cdots+L_{distill}^{t},
$$
followed by supervised fine-tuning per task [1911.03588]. MLKD extends the same principle to multiple knowledge types, using
$$
\mathcal{L}_{\mathrm{MLKD}}
=
\lambda_{1}\mathcal{L}_{\mathrm{Align}}
+
\lambda_{2}\mathcal{L}_{\mathrm{Corr}}
+
\lambda_{3}\mathcal{L}_{\mathrm{Sup}}
+
\lambda_{\mathrm{CE}}\mathcal{L}_{\mathrm{CE}},
$$
with fixed weights $\lambda_{1}=10$, $\lambda_{2}=20$, $\lambda_{3}=0.5$, and $\lambda_{\mathrm{CE}}=1.0$ in the supervised setting [2012.00573]. The smart-agriculture hybrid KD framework is similarly explicit that its “multi-objective” training is a scalarized weighted sum and “not Pareto-based multi-objective optimization” [2512.22239].

A stricter multi-objective view is represented by MoKD. That method treats KD as a vector-valued objective
$$
L(\theta) = [L_{distill}(\theta;\theta_T), L_{task}(\theta)]^\top,
$$
and constructs an adaptive descent direction
$$
g^* = \pi_1^* \nabla_\theta \log L_{distill} + \pi_2^* \nabla_\theta \log L_{task},
$$
where
$$
\pi_1^* = \frac{g_{22}-g_{12}}{g_{11}+g_{22}-2g_{12}}, \qquad
\pi_2^* = \frac{g_{11}-g_{12}}{g_{11}+g_{22}-2g_{12}},
$$
with $g_{11}=\|g_1\|^2$, $g_{22}=\|g_2\|^2$, and $g_{12}=\langle g_1,g_2\rangle$. The stated goal is to address both gradient conflict and gradient dominance between task learning and distillation [2505.08170].

Adaptive Distillation generalizes this idea from two losses to multiple KD paths. It writes the objective as the main supervised loss plus a weighted sum of $K$ KD losses and learns positive path weights via $v_i=e^{-z_i}$ with regularization,
$$
L(T;w^S)+\alpha\left(\sum_{i=1}^{K} e^{-z_i}\mathcal{L}_i + \sum_{i=1}^{K} z_i\right).
$$
The same paper also instantiates an MGDA-style baseline that solves a simplex-constrained quadratic program over path gradients [2110.09674].

## 3. Distillation signals and objective composition

MOKD differs from classical KD not merely in the number of loss terms, but in the heterogeneity of the distilled signals. One branch of the literature couples per-sample and relational objectives. MLKD defines alignment as
$$
\mathcal{L}_{\mathrm{Align}}
=
\mathbb{E}_{x}\Big[\|h_{\varphi}(f^{S}_{\theta}(x)) - f^{T}_{\eta}(x)\|_{2}^{2}\Big],
$$
and correlation as a KL divergence between teacher and student relational distributions over batchwise cosine similarities. The result is a task-agnostic and model-agnostic framework that transfers both knowledge alignment and knowledge correlation, rather than only one of them [2012.00573].

Another branch emphasizes multi-level or multi-granularity signals inside a single teacher. The multi-granularity distillation mechanism uses a frozen self-analyzed teacher $T_{SA}$ and a trainable student $S$. In the available description, the teacher exposes $F_{AK}^{(T_{SA})}$, $F_{NK}^{(T_{SA})}$, and $F_{DK}^{(T_{SA})}$, while the student produces the corresponding $F_{AK}^{(S)}$, $F_{NK}^{(S)}$, and $F_{DK}^{(S)}$. A second mode, “Stable excitation distillation,” additionally computes $F_{AKB}^{T_{(SA)}}$ and $F_{DKB}^{T_{(SA)}}$ and ensembles them with $F_{NK}^{(T_{SA})}$ via Equation 9. The description explicitly frames these signals as different granularities of supervision and different teaching patterns [2108.06681].

Online MOKD in self-supervised learning distributes objectives across both models and feature spaces. The method called MOKD in that context contains self-distillation and cross-distillation, each applied in an MLP space and a transformer-head space. Its total losses are
$$
\mathcal{L}_1=\mathcal{L}_{\mathrm{self}1}+\lambda_1\,\mathcal{L}_{\mathrm{cross}1},\qquad
\mathcal{L}_2=\mathcal{L}_{\mathrm{self}2}+\lambda_2\,\mathcal{L}_{\mathrm{cross}2},
$$
and cross-distillation in the transformer head is augmented by a cross-attention feature search that lets a query token from one model attend to local tokens from the other model’s momentum teacher [2304.06461].

MetaMixer makes the multi-objective structure explicit at the level of low-level and high-level knowledge. Its student loss is
$$
\mathcal{L}^s = \mathcal{L}^s_{cls} + \beta\,\mathcal{L}^s_{m\_logit} + \gamma\,\mathcal{L}^s_{fea} + \delta\,\mathcal{L}^s_{e\_logit},
$$
where $\mathcal{L}^s_{cls}$ is supervised CE, $\mathcal{L}^s_{m\_logit}$ is KD on globally mixed features, $\mathcal{L}^s_{fea}$ is feature-map distillation, and $\mathcal{L}^s_{e\_logit}$ is endpoint logit KD with confidence weighting. Its two-stage mixing combines local CutMix-like input-space mixing and manifold-mixup-style feature-space mixing [2303.07951].

The signal space can also include uncertainty. In DAUM, the teacher produces both predictive means $\mu_t$ and epistemic variances $\sigma_t^2$ for four objectives, and the student adds a per-task uncertainty head. The student is then trained with supervised BCE, predictive distillation, and uncertainty regression,
$$
\mathcal{L}_{\mathrm{total}}
=
\alpha \,\mathcal{L}_{\mathrm{sup}}
+
\beta \,\mathcal{L}_{\mathrm{KD}^{\mathrm{pred}}}
+
\gamma \,\mathcal{L}_{\mathrm{KD}^{\mathrm{unc}}}
+
\lambda \|\theta\|_2^2.
$$
This is a distinct form of MOKD in which the distilled target is not only a probability distribution but also a calibrated estimate of predictive uncertainty [2511.05582].

## 4. Architectures and training regimes

Architecturally, MOKD spans offline teacher–student pipelines, online peer learning, multi-task shared-backbone systems, and task-specific deployment simplifications. In the Airbnb ranking system, each objective has a pre-trained teacher $f_k(\theta_k^*,X)$, the student is a single MLP ranker with listwise softmax outputs, and serving uses only the student. No online score fusion or teacher inference is required at serving time, and self-distillation further simplifies training by replacing the teachers with the previous student’s predictions [2407.07181].

In MKD for pretrained language models, the shared student backbone is distilled jointly across GLUE tasks, each with its own head. The design is explicitly architecture-agnostic because it distills only logits, making the framework applicable to both an LSTM-based bi-attentive student and a three-layer Transformer student. Distillation training merges and shuffles task-specific batches, after which the distilled student is fine-tuned separately on each task [1911.03588].

Online MOKD in SSL dispenses with a static pre-trained teacher. Two online encoders $f_1$ and $f_2$ are trained simultaneously, each with a momentum teacher, an MLP-Head, and a T-Head. The EMA update is
$$
\theta_i' \leftarrow \mu\,\theta_i' + (1-\mu)\,\theta_i,
$$
and the framework uses two global views and eight local views under multi-crop augmentation. The stronger model is typically assigned a smaller cross-distillation weight than the weaker one, or equal weights when the backbones are the same [2304.06461].

Feature alignment can also be heavily engineered. The smart-agriculture hybrid KD framework uses a ResNet18 teacher and a customized student that combines MobileNetV2-style inverted residual blocks with DenseNet-style dense connectivity, and both teacher and student have main and auxiliary branches. Feature-level distillation is performed on GAP vectors rather than full spatial maps, and an auxiliary alignment branch maps teacher Stage-3 output to the student’s third downsampling resolution before GAP. Teacher training and student distillation are sequential within each mini-batch [2512.22239].

Adaptive Distillation treats architecture as a set of distillation paths. The paths may include soft targets, hint layers such as Attention Transfer, Neural Selective Transfer, segmentation logits, detector backbone or pyramid features, and localization distillation for bounding-box generators. Each path enters as a distinct $\mathcal{L}_{KD}^{jk}$ and receives an adaptive weight, making the topology of the student–teacher interaction itself part of the optimization problem [2110.09674].

## 5. Empirical performance and application domains

The empirical record is broad and application-specific rather than tied to a single benchmark family. In online marketplace ranking, MOKD improved offline NDCG by $+1.1\%$ versus the production multi-task baseline on a 7-day holdout, improved online CVR by $+0.37\%$ with $p\_val = 0.02$, maintained neutral changes on secondary objectives, and reduced serving latency by $-1.6\%$. The same system also reduced the Kendall’s $\tau$ SxS change rate by $53\%$ and reduced Relative Prediction Difference from $0.407$ to $0.363$ [2407.07181].

In image classification and detection, MoKD reports state-of-the-art performance relative to the cited baselines. On ImageNet-1K, DeiT-Tiny improved from $72.2$ Top-1 as the no-distillation baseline to $79.7$ under MoKD, exceeding VkD at $78.3$; DeiT-Small improved from $79.8$ to $83.1$, exceeding VkD at $82.3$. On COCO detection with a ViDT-Base teacher, MoKD-nano reached AP $43.7$ versus $43.0$ for VkD and $40.4$ without distillation, while MoKD-small reached AP $49.6$ and surpassed a Swin-base model trained from scratch at AP $49.4$ [2505.08170].

Multi-level and multi-signal distillation also show consistent gains. MLKD improved CIFAR-100 performance across similar and cross-architecture teacher–student pairs; for example, $\text{resnet32}\times4 \rightarrow \text{resnet8}\times4$ reached $77.11\%$ Top-1 versus $75.90\%$ for CRD, and on ImageNet $\text{ResNet34} \rightarrow \text{ResNet18}$ reduced Top-1/Top-5 error to $27.88/9.30$ versus $28.83/9.87$ for CRD. On Cityscapes semantic segmentation, MLKD raised mIoU from $69.10$ for the student to $75.73$, exceeding IFVD at $74.54$ [2012.00573].

Online MOKD for SSL reports strong mutual gains for heterogeneous backbones. For the R50–ViT-B pair, linear probing improved from $72.1\%\rightarrow75.6\%$ for R50 and $77.0\%\rightarrow78.0\%$ for ViT-B, while k-NN improved from $62.8\%\rightarrow70.6\%$ for R50 and $73.6\%\rightarrow75.2\%$ for ViT-B. Against prior SSL-KD baselines, the paper reports that for R50–R34 the student linear-probing score improved from $62.5\%$ under DisCo to $67.0\%$ under MOKD, while the teacher improved from $67.4\%$ to $72.3\%$ [2304.06461].

MetaMixer reports gains across CIFAR and ImageNet in online KD. On CIFAR-100 with two ResNet-110 peers, MetaMixer achieved average Top-1 accuracy $78.58$ versus $75.42$ for DML and $77.97$ for ONE; on ImageNet with two ResNet-18 peers, average accuracy reached $70.88$ versus $69.72$ for the baseline and $70.32$ for MCL. The same paper states that performance gains grow with depth and width, and that combining local and global mixing outperforms either stage alone [2303.07951].

Domain-specific MOKD variants also exhibit strong efficiency–accuracy trade-offs. The smart-agriculture hybrid KD student reached $98.56\%$ accuracy on rice seed variety identification versus $98.65\%$ for the ResNet18 teacher while using $0.68$ GFLOPs and approximately $1.07$ million parameters, corresponding to about $2.7\times$ lower computational cost and more than $10\times$ smaller model size than the teacher. On the hardest TBR-45 variety, the full configuration improved the main-branch accuracy from $94.23\%$ for Hard + Response KD to $97.22\%$ after adding feature distillation and self-distillation [2512.22239].

A more compact but still representative result comes from multi-granularity distillation, which reports that the mechanism improves accuracy by $0.58\%$ on average and by $1.08\%$ in the best case over the baselines, and further improves student fine-tuning ability and robustness to noisy inputs [2108.06681].

## 6. Limitations, misconceptions, and open directions

A central misconception is that every MOKD method solves a formal multi-objective problem in the Pareto sense. Much of the literature instead uses scalarized weighted sums with static coefficients. That includes the marketplace ranking system’s mixing weights $\omega_k$ and $\alpha$, MLKD’s fixed $\lambda$ weights, MetaMixer’s fixed $\beta,\gamma,\delta$, MKD’s implicit uniform task weights, and the smart-agriculture framework’s dataset-specific $\lambda_1,\lambda_2,\lambda_3,\lambda_4$ [2407.07181] [2012.00573] [2303.07951] [1911.03588] [2512.22239]. The practical implication is that weight selection remains a critical design variable even when the method is called “multi-objective.”

Weight sensitivity and signal conflict are recurrent issues. MoKD identifies both gradient conflicts and gradient dominance as core KD pathologies [2505.08170]. Adaptive Distillation states that some paths may have negative effects on student generalization and therefore should be down-weighted adaptively rather than aggregated with fixed importance [2110.09674]. In online SSL MOKD, improper cross-distillation weighting can hurt the stronger model, and the T-Head introduces additional cost [2304.06461].

Teacher quality and objective imbalance also limit transfer. The ranking formulation explicitly notes that if teachers are biased or outdated, the aggregated soft label can steer the student incorrectly, and that large distribution shifts can reduce soft-label relevance [2407.07181]. DAUM shows that naive uncertainty can fail under severe imbalance because the usual “maximum uncertainty near $0.5$” intuition becomes unreliable for extremely sparse tasks; its remedy is uncertainty sharing from more balanced correlated objectives [2511.05582]. This suggests that MOKD is often most effective when the multiple objectives are not only numerous but also structurally informative.

Open directions in the cited literature concentrate on automatic weighting, richer objective geometry, and improved uncertainty modeling. The ranking paper proposes learning $\omega_k$ through a Mixture-of-Experts layer and suggests Pareto-aware distillation and uncertainty-aware KD as extensions [2407.07181]. DAUM proposes deeper modeling of interrelationships among multiple uncertainties in the multi-objective setting [2511.05582]. MoKD identifies data-free KD as a non-trivial extension, particularly when the formulation relies on multi-objective updates and learned subspace alignment [2505.08170]. Across these works, the long-term trend is clear: MOKD is moving from hand-tuned loss aggregation toward adaptive, structure-aware coordination of heterogeneous supervisory signals.

Source: https://www.emergentmind.com/topics/multi-objective-knowledge-distillation-mokd