Bidirectional Distillation in Knowledge Transfer

• Bidirectional distillation is a framework that employs dual-direction approaches—across context, tasks, or models—to address limitations of unidirectional knowledge transfer.
• It is applied in diverse areas such as language modeling, speech translation, video generation, and recommendation systems, each adapting the dual mechanism to specific challenges.
• The method leverages techniques like dual KL divergence, mutual refinement, and gradient-feature matching to improve performance and mitigate modality gaps.

Searching arXiv for recent and foundational papers on bidirectional distillation across domains. Bidirectional distillation is a family of knowledge-transfer formulations in which the “two directions” are defined by the structure of the task rather than by a single canonical recipe. In the surveyed literature, the term denotes at least four distinct patterns: distilling distributions conditioned on both left and right context into bidirectional encoders, combining forward and backward translation or temporal generation teachers, organizing mutual or consensus-based transfer between co-evolving models, and jointly using forward and reverse alignment objectives such as forward/reverse KL or teacher-top-$K$/student-top-$K$ logit matching (Kuncoro et al., 2020, Inaguma et al., 2021, Feng et al., 2021, Wang et al., 7 Jan 2025, Zhong et al., 16 Apr 2026, Zhang et al., 17 Jun 2026, Liu et al., 29 Jun 2026). The common departure from conventional distillation is that a fixed one-way teacher$\rightarrow$student pathway is no longer sufficient: the second direction is introduced to reduce modality gaps, control teacher–student mismatch, preserve complementary modes, or make a single model usable under forward, backward, or mixed conditioning.

1. Conceptual scope and terminology

The literature does not use the expression “bidirectional distillation” uniformly. In some papers, “bidirectional” refers to context direction, as in constructing a teacher distribution over words in full left-and-right context for BERT pretraining (Kuncoro et al., 2020). In others, it refers to task direction, such as combining source$\rightarrow$target and target$\rightarrow$source sequence-level knowledge distillation for end-to-end speech translation (Inaguma et al., 2021). A third usage emphasizes model interaction, where multiple students or heterogeneous recommenders exchange knowledge and co-evolve rather than obeying a frozen teacher (Feng et al., 2021, Wu et al., 23 May 2025). A fourth usage emphasizes objective direction, such as combining $\mathrm{KL}(p_T\parallel p_S)$ and $\mathrm{KL}(p_S\parallel p_T)$ or supervising both teacher-selected and student-selected top-$K$ logits (Liu et al., 29 Jun 2026, Zhong et al., 16 Apr 2026).

Setting	Meaning of “bidirectional”	Representative paper
Bidirectional encoders	Teacher distribution over left and right context	(Kuncoro et al., 2020)
Speech translation	Forward and backward SeqKD from two NMT teachers	(Inaguma et al., 2021)
Dialogue / recommendation	Mutual or consensus-based model exchange	(Feng et al., 2021, Wu et al., 23 May 2025)
Missing multimodal sentiment	Distillation on temporally bidirectional representations	(Wang et al., 7 Jan 2025)
Any-order video generation	Forward and backward autoregressive rollouts	(Zhang et al., 17 Jun 2026)
Text generation / unlearning	Forward/reverse KL or dual top-$K$ logit matching	(Liu et al., 29 Jun 2026, Zhong et al., 16 Apr 2026)

This diversity matters because the design questions differ accordingly. Some methods seek a better teacher distribution for a bidirectional student, some seek mutual refinement between peers, and some seek symmetric optimization pressure on head and tail modes. Consequently, empirical gains are not attributable to a single mechanism called bidirectional distillation, but to a set of related strategies that introduce a second direction where one-way transfer was inadequate.

2. Sequence, context, and temporal bidirectionality

A foundational formulation appears in syntactic pretraining for bidirectional encoders. “Syntactic Structure Distillation Pretraining For Bidirectional Encoders” constructs an approximate teacher distribution

$$\tilde{t}_{\boldsymbol\phi,\boldsymbol\omega}(x_i \mid \mathbf{x}_{<i}, \mathbf{x}_{>i}) = \frac{\frac{t_{\boldsymbol\phi}(x_i \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(x_i \mid \mathbf{x}_{>i})}{q(x_i)}} {\sum_{w\in\Sigma}\frac{t_{\boldsymbol\phi}(w \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(w \mid \mathbf{x}_{>i})}{q(w)}},$$

by combining left-to-right and right-to-left RNNGs in a product-of-experts–like form, and distills that distribution into BERT’s masked LM objective (Kuncoro et al., 2020). The reported effect is a reduction of relative error by $K$0-$K$1 on structured prediction tasks, with especially strong gains on constituent parsing, CCG supertagging, and low-resource fine-tuning, while GLUE results are mixed (Kuncoro et al., 2020). Here bidirectionality is not mutual learning but a correction of the mismatch between a directional teacher and a bidirectional student.

Speech translation adopts a different sequence-level meaning. “Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation” uses two text NMT teachers: a forward source$K$2target teacher producing distilled translations $K$3, and a backward target$K$4source teacher producing paraphrased transcriptions $K$5 via back-translation (Inaguma et al., 2021). A single bilingual speech translation model is trained with both

$K$6

optionally mixed with original references. On MuST-C, forward SeqKD alone raises an autoregressive En–De baseline from $K$7 to $K$8 BLEU, while bidirectional SeqKD reaches $K$9, and a 2-reference configuration reaches $\rightarrow$0; analogous gains appear on En–Fr and on non-autoregressive models (Inaguma et al., 2021). The second direction contributes not by reversing gradients but by simplifying the source-side textual target and improving semantic alignment.

Temporal bidirectionality is extended further in generative video modeling. “UniTemp: Unlocking Video Generation in Any Temporal Order via Bidirectional Distillation” trains a single autoregressive student and a single fake critic with forward and backward self-rollouts under Self-Forcing + DMD, while introducing blockwise anchor latents to compensate for the forward-causal 3D VAE during backward generation (Zhang et al., 17 Jun 2026). The anchor mechanism reduces backward inter-block Flickering Ratio from $\rightarrow$1 to $\rightarrow$2 at anchor size $\rightarrow$3, close to the forward value, and enables forward extension, backward extension, and inbetween generation in one model (Zhang et al., 17 Jun 2026). In this case, bidirectionality denotes temporal generation order, not symmetric teacher–student learning.

Across these cases, the second direction is introduced because a one-sided factorization is structurally mismatched to the target problem: bidirectional context for BERT, dual translation directions for speech translation, and any-order conditioning for autoregressive video generation.

3. Mutual, consensus-based, and co-evolving model distillation

A distinct line of work treats bidirectional distillation as co-evolution rather than directional context. “Multi-View Feature Representation for Dialogue Generation with Bidirectional Distillation” partitions the dialogue dataset into $\rightarrow$4 subtasks, assigns one student to each subtask, aggregates peer predictions with a Bernoulli imitation gate, and replaces one-way KL with a Jensen–Shannon objective between each student and its peer aggregate (Feng et al., 2021). The total objective is

$\rightarrow$5

On DailyDialog, MRBD improves Dist-1/Dist-2 to $\rightarrow$6 and lowers Dis-1/Dis-2 to $\rightarrow$7, outperforming Seq2Seq+Att, KD, co-teaching, and DML; similar gains appear on PersonaChat (Feng et al., 2021). The paper’s central claim is that general knowledge should be found through consensus among diversified learning partners, not imposed by a fixed teacher trained on the same data.

Sequential recommendation introduces a heterogeneous version of the same idea. “Bidirectional Knowledge Distillation for Enhancing Sequential Recommendation with LLMs” defines CRM$\rightarrow$8LLM “Downward Enhancement” and LLM$\rightarrow$9CRM “Upward Semantics Distillation,” alternates the two optimization phases, and uses sample-wise adaptive weighting based on the rank difference of the ground-truth item under the two models (Wu et al., 23 May 2025). The distributions are matched in the common item space rather than in hidden layers. On Beauty, for example, the CRM-side NDCG@10 rises from the best CRM-side baseline $\rightarrow$0 to $\rightarrow$1, and the LLM-side HR@10 rises from E4SRec’s $\rightarrow$2 to $\rightarrow$3 (Wu et al., 23 May 2025). Bidirectionality here is neither temporal nor symmetric KL; it is an alternating loop between two heterogeneous recommenders with different inductive biases.

“Biologically Plausible Learning via Bidirectional Spike-Based Distillation” generalizes the mutual view to spike-based networks (Lv et al., 24 Sep 2025). A feedforward spiking network and a backward spiking network align basal and apical voltages layer by layer with a Relaxed Contrastive loss,

$\rightarrow$4

and each side acts as teacher and student for the other (Lv et al., 24 Sep 2025). BSD reaches $\rightarrow$5 on MNIST, $\rightarrow$6 on CIFAR-10, and $\rightarrow$7 on CIFAR-100 with CNNs, while spike alignment rises above $\rightarrow$8 Hamming similarity (Lv et al., 24 Sep 2025). The “bidirectional” element is explicitly local and reciprocal: opposite mappings are learned simultaneously and mutually constrain each other.

These methods share a common departure from orthodox KD: the teacher is not a static oracle but part of an interaction topology. Bidirectionality then means that the second direction protects against obedience to a possibly inferior or incomplete teacher, and turns distillation into a process of negotiated agreement.

4. Modality, domain, and system-level bidirectional transfer

In multimodal learning, the term often denotes transfer across missing inputs, sensor branches, or distributed systems. “Modality-Invariant Bidirectional Temporal Representation Distillation Network for Missing Multimodal Sentiment Analysis” defines bidirectionality at two levels: forward and backward temporal modeling within each modality,

$\rightarrow$9

and teacher$\rightarrow$0student distillation on these bidirectional representations under random temporal masking (Wang et al., 7 Jan 2025). Distillation operates at the MIB-TRL level, at the fusion level, and through student self-distillation, while SimSiam provides a bidirectional similarity loss between teacher and student representations (Wang et al., 7 Jan 2025). On CH-SIMS under incomplete modalities, MITR-DNet reports MAE $\rightarrow$1, Corr $\rightarrow$2, Acc-3 $\rightarrow$3, Acc-2 $\rightarrow$4, and F1 $\rightarrow$5, outperforming EMT-DLFR on all metrics except an almost-equal MAE; the best depth is $\rightarrow$6 $\rightarrow$7 modules (Wang et al., 7 Jan 2025).

LiDAR semantic segmentation uses a different modular interpretation. “CMDFusion: Bidirectional Fusion Network with Cross-modality Knowledge Distillation for LIDAR Semantic Segmentation” combines one-way cross-modality distillation from a frozen ResNet-101 camera branch to a 3D SPVCNN “2D knowledge branch” with bidirectional fusion between that branch and a separate 3D LiDAR branch (Cen et al., 2023). The training loss is

$\rightarrow$8

where $\rightarrow$9 is an L2 feature-matching loss on points in the camera FOV (Cen et al., 2023). CMDFusion achieves $\mathrm{KL}(p_T\parallel p_S)$0 mIoU on SemanticKITTI-O and $\mathrm{KL}(p_T\parallel p_S)$1 mIoU / $\mathrm{KL}(p_T\parallel p_S)$2 fwIoU on nuScenes, while requiring only LiDAR at inference (Cen et al., 2023). Strictly speaking, the distillation itself is one-way; the bidirectional element is the feature fusion between two LiDAR-based branches.

Federated learning turns bidirectionality into a system-level pipeline. “BiFedKD: Bidirectional Federated Knowledge Distillation Framework for Non-IID and Long-Tailed ECG Monitoring” organizes bottom-up client$\mathrm{KL}(p_T\parallel p_S)$3server logit upload on a public proxy dataset and top-down server$\mathrm{KL}(p_T\parallel p_S)$4client soft-target broadcast after temperature-scaled aggregation and server-side teacher refinement (Shu et al., 14 May 2026). The server objective is

$\mathrm{KL}(p_T\parallel p_S)$5

with $\mathrm{KL}(p_T\parallel p_S)$6 and $\mathrm{KL}(p_T\parallel p_S)$7 in the experiments (Shu et al., 14 May 2026). On MIT-BIH Arrhythmia, BiFedKD improves accuracy and Macro-F1 by $\mathrm{KL}(p_T\parallel p_S)$8 and $\mathrm{KL}(p_T\parallel p_S)$9, and for the same Macro-F1 reduces communication overhead by $\mathrm{KL}(p_S\parallel p_T)$0 and computation cost by $\mathrm{KL}(p_S\parallel p_T)$1 (Shu et al., 14 May 2026). Here the two directions are not peer-to-peer but client↔server.

These domain-specific variants reveal a consistent pattern: the second direction is introduced where a one-way interface leaves an important blind spot—missing modalities, unavailable cameras at inference, or non-IID client drift.

5. Logit-space, divergence-space, and parameter-space bidirectionality

Some recent work defines bidirectional distillation directly at the level of divergences, logits, or parameter mappings. In text generation, “ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation” combines forward and reverse KL,

$\mathrm{KL}(p_S\parallel p_T)$2

with $\mathrm{KL}(p_S\parallel p_T)$3 chosen by a small policy network from a 6-dimensional state vector of entropies, variances, and current KL values (Liu et al., 29 Jun 2026). On DollyEval, GPT-2 $\mathrm{KL}(p_S\parallel p_T)$4B $\mathrm{KL}(p_S\parallel p_T)$5 $\mathrm{KL}(p_S\parallel p_T)$6M rises from static FKL+RKL $\mathrm{KL}(p_S\parallel p_T)$7 to ARKD $\mathrm{KL}(p_S\parallel p_T)$8 in ROUGE-L/BERTScore, and the learned policy shifts from FKL-heavier exploration to RKL-dominant stability, settling around $\mathrm{KL}(p_S\parallel p_T)$9 in late training (Liu et al., 29 Jun 2026). Bidirectionality here means explicit use of both $K$0 and $K$1 to cover principal and long-tail modes.

In LLM unlearning, “Harmonizing Multi-Objective LLM Unlearning via Unified Domain Representation and Bidirectional Logit Distillation” defines a dual top-$K$2 logit loss

$K$3

where the first term imitates teacher top-$K$4 logits and the second suppresses the student’s own high-confidence hazardous logits (Zhong et al., 16 Apr 2026). On MUSE-Book, the method reports Forget $K$5, General retain $K$6, Neighbor retain acc $K$7, ASR $K$8, MMLU $K$9, and Overall performance $K$0, decisively outperforming DUET and BiLD in the reported trade-off (Zhong et al., 16 Apr 2026). The “bi-directional” qualifier therefore refers not to mutual teachers, but to supervision on two different logit sets.

Parameter-space bidirectionality appears in “Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher” (Guo et al., 2024). The method builds a larger dynamic teacher from the student by Inverse Reparameterization (IR), then repeatedly extracts the current student from the teacher via Channel-Branch Reparameterization (CBR), while the dynamic teacher stays slightly ahead in accuracy (Guo et al., 2024). In ImageNet KD settings, DKD on ResNet34$K$1ResNet18 rises from $K$2 to $K$3, and DKD on ResNet50$K$4MobileNet rises from $K$5 to $K$6; GPD also improves training from scratch on ResNet18 from $K$7 to $K$8 (Guo et al., 2024). Here “bidirectional mappings” are explicit teacher←student initialization and teacher→student extraction in parameter space rather than symmetric output-level distillation.

Dataset distillation presents yet another objective-level meaning. “DREAM+: Efficient Dataset Distillation by Bidirectional Representative Matching” matches both backward information (gradients) and forward information (feature embeddings) between synthetic and representative real images selected by clustering (Liu et al., 2023). On CIFAR-10, 10 IPC, ConvNet-3, DREAM+ reaches $K$9 versus IDC’s $$\tilde{t}_{\boldsymbol\phi,\boldsymbol\omega}(x_i \mid \mathbf{x}_{<i}, \mathbf{x}_{>i}) = \frac{\frac{t_{\boldsymbol\phi}(x_i \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(x_i \mid \mathbf{x}_{>i})}{q(x_i)}} {\sum_{w\in\Sigma}\frac{t_{\boldsymbol\phi}(w \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(w \mid \mathbf{x}_{>i})}{q(w)}},$$0, while cutting the number of iterations needed to reach IDC-level performance by roughly $$\tilde{t}_{\boldsymbol\phi,\boldsymbol\omega}(x_i \mid \mathbf{x}_{<i}, \mathbf{x}_{>i}) = \frac{\frac{t_{\boldsymbol\phi}(x_i \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(x_i \mid \mathbf{x}_{>i})}{q(x_i)}} {\sum_{w\in\Sigma}\frac{t_{\boldsymbol\phi}(w \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(w \mid \mathbf{x}_{>i})}{q(w)}},$$1 (Liu et al., 2023). In this usage, the two directions are gradient space and feature space, not two models.

Taken together, these methods show that “bidirectional” can be attached to losses themselves: two divergences, two logit sets, two information channels, or two parameter transforms.

6. Specialized application forms, empirical regularities, and limitations

Several application-specific systems combine the above motifs in domain-tailored ways. “DS_FusionNet: Dynamic Dual-Stream Fusion with Bidirectional Knowledge Distillation for Plant Disease Recognition” trains EfficientNet-B4 and ConvNeXt-Tiny jointly with a fused student, using KL divergence between the ensemble teacher and the student while the backbones co-evolve (Song et al., 29 Apr 2025). The full system reaches $$\tilde{t}_{\boldsymbol\phi,\boldsymbol\omega}(x_i \mid \mathbf{x}_{<i}, \mathbf{x}_{>i}) = \frac{\frac{t_{\boldsymbol\phi}(x_i \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(x_i \mid \mathbf{x}_{>i})}{q(x_i)}} {\sum_{w\in\Sigma}\frac{t_{\boldsymbol\phi}(w \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(w \mid \mathbf{x}_{>i})}{q(w)}},$$2 on PlantDisease with full data, $$\tilde{t}_{\boldsymbol\phi,\boldsymbol\omega}(x_i \mid \mathbf{x}_{<i}, \mathbf{x}_{>i}) = \frac{\frac{t_{\boldsymbol\phi}(x_i \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(x_i \mid \mathbf{x}_{>i})}{q(x_i)}} {\sum_{w\in\Sigma}\frac{t_{\boldsymbol\phi}(w \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(w \mid \mathbf{x}_{>i})}{q(w)}},$$3 on CIFAR-10 using only $$\tilde{t}_{\boldsymbol\phi,\boldsymbol\omega}(x_i \mid \mathbf{x}_{<i}, \mathbf{x}_{>i}) = \frac{\frac{t_{\boldsymbol\phi}(x_i \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(x_i \mid \mathbf{x}_{>i})}{q(x_i)}} {\sum_{w\in\Sigma}\frac{t_{\boldsymbol\phi}(w \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(w \mid \mathbf{x}_{>i})}{q(w)}},$$4 of labeled data, and $$\tilde{t}_{\boldsymbol\phi,\boldsymbol\omega}(x_i \mid \mathbf{x}_{<i}, \mathbf{x}_{>i}) = \frac{\frac{t_{\boldsymbol\phi}(x_i \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(x_i \mid \mathbf{x}_{>i})}{q(x_i)}} {\sum_{w\in\Sigma}\frac{t_{\boldsymbol\phi}(w \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(w \mid \mathbf{x}_{>i})}{q(w)}},$$5 on PlantWild at $$\tilde{t}_{\boldsymbol\phi,\boldsymbol\omega}(x_i \mid \mathbf{x}_{<i}, \mathbf{x}_{>i}) = \frac{\frac{t_{\boldsymbol\phi}(x_i \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(x_i \mid \mathbf{x}_{>i})}{q(x_i)}} {\sum_{w\in\Sigma}\frac{t_{\boldsymbol\phi}(w \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(w \mid \mathbf{x}_{>i})}{q(w)}},$$6 data, while the authors note feature conflicts on the complex PlantWild dataset and the absence of a direct “with vs without KD” ablation (Song et al., 29 Apr 2025). “EchoDistill: Bidirectional Concept Distillation for One-Step Diffusion Personalization” makes the teacher$$\tilde{t}_{\boldsymbol\phi,\boldsymbol\omega}(x_i \mid \mathbf{x}_{<i}, \mathbf{x}_{>i}) = \frac{\frac{t_{\boldsymbol\phi}(x_i \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(x_i \mid \mathbf{x}_{>i})}{q(x_i)}} {\sum_{w\in\Sigma}\frac{t_{\boldsymbol\phi}(w \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(w \mid \mathbf{x}_{>i})}{q(w)}},$$7student direction explicit through alignment losses and the student$$\tilde{t}_{\boldsymbol\phi,\boldsymbol\omega}(x_i \mid \mathbf{x}_{<i}, \mathbf{x}_{>i}) = \frac{\frac{t_{\boldsymbol\phi}(x_i \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(x_i \mid \mathbf{x}_{>i})}{q(x_i)}} {\sum_{w\in\Sigma}\frac{t_{\boldsymbol\phi}(w \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(w \mid \mathbf{x}_{>i})}{q(w)}},$$8teacher direction explicit through an echo stage in which student-generated samples replace real images in the teacher’s DDPM loss (Yang et al., 23 Oct 2025). On DreamBench-style evaluation, the one-step student reports CLIP-I $$\tilde{t}_{\boldsymbol\phi,\boldsymbol\omega}(x_i \mid \mathbf{x}_{<i}, \mathbf{x}_{>i}) = \frac{\frac{t_{\boldsymbol\phi}(x_i \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(x_i \mid \mathbf{x}_{>i})}{q(x_i)}} {\sum_{w\in\Sigma}\frac{t_{\boldsymbol\phi}(w \mid \mathbf{x}_{<i})\, r_{\boldsymbol\omega}(w \mid \mathbf{x}_{>i})}{q(w)}},$$9 and DINO $K$00, and echoing improves teacher CLIP-I from $K$01 to $K$02 and DINO from $K$03 to $K$04 (Yang et al., 23 Oct 2025). “Cross-Paradigm Knowledge Distillation” extends the term to RF$K$05DNN transfer with progressive multi-stage distillation, multi-teacher tree ensembles, and uncertainty-aware transfer, reporting NN-COMPACT at $K$06 classification accuracy and NN-WIDE at $K$07 $K$08 in regression (Moghadasi, 19 May 2026).

Despite the diversity of mechanisms, several regularities recur. First, the second direction is typically introduced to handle a failure mode of unidirectional KD: teacher–student mismatch in GPD, loss of tail information in ARKD, robustness to hazardous logits in unlearning, inability to use missing future context in UniTemp, or inability to use image information without cameras at inference in CMDFusion. Second, many gains come from making the bidirectional signal selective rather than fully symmetric: stop-gradient SimSiam in MITR-DNet, sample-wise weighting in LLMD4Rec, top-$K$09 logit suppression in unlearning, or time-dependent alignment weights in EchoDistill. Third, the literature repeatedly distinguishes two-way interaction from full reciprocity. CMDFusion, for example, is bidirectional in fusion but not in teacher–student distillation (Cen et al., 2023); MITR-DNet is teacher$K$10student distillation on temporally bidirectional representations rather than cross-modal distillation in both directions (Wang et al., 7 Jan 2025).

The limitations are equally patterned. Several methods are teacher-quality dependent: unsafe or confused teachers remain a risk in unlearning (Zhong et al., 16 Apr 2026), and strong NMT teachers are assumed in speech translation (Inaguma et al., 2021). Domain-specific constraints remain prominent: data imbalance in missing-modality sentiment analysis (Wang et al., 7 Jan 2025), dependence on a public proxy dataset in federated ECG monitoring (Shu et al., 14 May 2026), forward bias from a causal VAE in UniTemp (Zhang et al., 17 Jun 2026), and stylized biological assumptions in BSD (Lv et al., 24 Sep 2025). Other systems trade performance for cost or complexity: DS_FusionNet roughly doubles parameters and FLOPs relative to a single backbone (Song et al., 29 Apr 2025), and GPD increases per-epoch time in return for higher ImageNet accuracy (Guo et al., 2024).

A plausible synthesis is that bidirectional distillation is best understood not as a single algorithmic family but as a design principle: when a one-way supervisory path discards essential complementary structure, a second direction can be introduced in the most relevant space—context, models, modalities, logits, gradients, or parameters. The surveyed literature shows that the effectiveness of the idea depends less on the label “bidirectional” than on whether the chosen second direction directly addresses the bottleneck of the underlying task.