Hierarchical Multi-to-Single-Modal Distillation

Updated 7 July 2026

Hierarchical multi-to-single-modal knowledge distillation is a training strategy that transfers multi-level internal representations from a rich, multimodal teacher to a deployable unimodal student.
It leverages diverse architectural patterns—such as multi-output heads, contrastive losses, and attention-map distillation—to effectively align cross-modal representations and improve performance in tasks like ASR and emotion recognition.
The approach enables practical deployment by preserving efficiency, achieving up to 9% relative error reduction and significant computational savings, while benefiting from richer training signals.

Searching arXiv for the specified topic and cited papers. Searching arXiv for "hierarchical multi-to-single-modal knowledge distillation". Searching arXiv for "Knowledge distillation from LLM to acoustic model: a hierarchical multi-task learning approach". Hierarchical multi-to-single-modal knowledge distillation denotes a class of training strategies in which a student model operating on a single modality is supervised by a richer teacher through multiple levels of transfer rather than only through final logits. In the literature represented by speech recognition, multimodal emotion recognition, and tokamak disruption prediction, the central objective is consistent: exploit text-only or multimodal supervision during training, but retain only a single-modal student at inference so that the system benefits from external linguistic or multimodal structure without the full inference-time cost of the teacher (Lee et al., 2021, Han et al., 2023, Agarwal et al., 2021, Chen et al., 5 Jul 2026).

1. Problem formulation and scope

The topic arises when the most informative training configuration is not the most deployable one. In speech recognition, a LLM is trained on large amounts of text while the acoustic model is trained on speech-text pairs, creating a cross-modal transfer problem between text and speech (Lee et al., 2021). In CIF-based end-to-end ASR, a pre-trained LLM such as BERT contains linguistic knowledge that is difficult to inject into an ASR model because the acoustic and linguistic representations are structurally different (Han et al., 2023). In multimodal transformers for emotion recognition, the teacher can use video, audio, and language, whereas the deployed student may need to operate with only video because of missing modalities or computational constraints (Agarwal et al., 2021). In disruption prediction for EAST, visible images provide complementary spatial cues, but the deployed predictor may need to use only time-series diagnostics for efficient online warning (Chen et al., 5 Jul 2026).

A concise way to characterize the field is that it combines multimodal training or cross-modal supervision with unimodal inference. The “hierarchical” qualifier does not refer to a single fixed mechanism across all papers. Instead, it denotes multi-level transfer: different linguistic units, different internal student representations, different transformer stages, or different levels such as graph structure, representation, and decision.

Paper	Teacher $\rightarrow$ student	Hierarchy used
(Lee et al., 2021)	LLM $\rightarrow$ acoustic model	senones, monophones, subwords; auxiliary output heads
(Han et al., 2023)	pre-trained LLM $\rightarrow$ CIF ASR	acoustic-level and linguistic-level distillation
(Agarwal et al., 2021)	multimodal transformer $\rightarrow$ video-only transformer	final layer, penultimate layer, post-attention layers, attention maps
(Chen et al., 5 Jul 2026)	image+signal teacher $\rightarrow$ time-series student	graph-structure, representation, decision

2. Architectural patterns

A recurring architectural pattern is a high-capacity teacher trained with richer inputs or richer symbolic supervision, coupled with a student that preserves only the deployable modality. In the cross-modal ASR framework, the acoustic model uses shared encoding layers consisting of a 2D-CNN front-end and a Transformer stack, followed by two linear layers for supervised learning and two auxiliary linear layers for distillation, so the model has multiple outputs (Lee et al., 2021). This design separates the hard-label objective from the distillation objective while keeping the representation learning shared.

In CIF-based ASR, the student consists of an acoustic encoder, a CIF module, and an autoregressive decoder. The encoder outputs a lower-level acoustic sequence $\mathbf{H}$ , the CIF module transforms it into high-level acoustic/token-level representations $\mathbf{C}$ , and the decoder maps $\mathbf{C}$ into the final linguistic output sequence $\mathbf{S}$ . The teacher is a BERT-like PLM, using bert-base-chinese for AISHELL-1 and bert-base-uncased for LibriSpeech, which produces contextual token representations $\mathbf{E}$ (Han et al., 2023).

In multimodal transformers for CMU-MOSEI, the teacher follows a MulT-style cross-attention architecture. Each modality first passes through a Conv1D layer along the temporal dimension followed by sinusoidal positional embeddings, and the teacher builds 6 cross-attention transformers corresponding to combinations of query, key, and value across video, audio, and language. The student is unimodal at inference: in the main setup, only video is real input, and that video is passed three times in parallel through different Conv1D layers so that the outputs of the second and third paths act as proxies for missing audio and language (Agarwal et al., 2021).

In EAST disruption prediction, the teacher is built in two stages: Transformer-based modality encoders and a prototype-guided spatiotemporal hypergraph fusion module. The image branch uses 12 visible frames per window, a 3D convolution for local spatiotemporal tokenization, and a Transformer for global image features. The signal branch takes an $\rightarrow$ 0 window, uses a 1D convolution to form tokens, and then applies a Transformer to model dependencies among channels and time. The student keeps only the time-series branch (Chen et al., 5 Jul 2026).

3. Meanings of hierarchy

The literature uses “hierarchical” in several technically distinct senses. In the language-model-to-acoustic-model setting, the hierarchy lies in the teacher units: sentence, word/subword, phoneme/phone, and senone/state. The authors train three LMs with different output vocabularies—4,160 senones, 41 phones, and 5,000 subwords—and distill all of them into a single DNN-HMM acoustic model predicting senones (Lee et al., 2021). The hierarchy is therefore in the linguistic abstraction level of the teachers rather than in a stacked chain of student models.

In CIF-based ASR, the hierarchy is internal to the student pipeline: speech input is processed into low-level acoustic features $\rightarrow$ 1, then into high-level acoustic/token-aligned features $\rightarrow$ 2, and then into linguistic representations $\rightarrow$ 3. Hierarchical distillation is placed at the latter two levels, with acoustic contrastive distillation on $\rightarrow$ 4 and linguistic regression distillation on $\rightarrow$ 5 (Han et al., 2023).

In multimodal transformers for emotion recognition, the hierarchy is defined by the stage of the transformer at which transfer occurs. Distillation is explored at four levels: the final linear layer, the penultimate linear layer, the post-attention linear layers, and the attention maps. The paper’s main conclusion is that lower-level distillation, especially attention-map distillation, is more effective than output-only distillation in this multimodal-to-unimodal setting (Agarwal et al., 2021).

In EAST disruption prediction, the hierarchy is tripartite: graph-structure-level distillation transfers the teacher’s sparse node-to-hyperedge preferences, representation-level distillation aligns intermediate latent features, and decision-level distillation transfers the teacher’s soft prediction boundary (Chen et al., 5 Jul 2026). A plausible implication is that “hierarchical” in this area is best understood as multi-level supervision under modality mismatch, not as a single canonical architecture.

4. Distillation objectives and alignment mechanisms

A central methodological distinction concerns whether hard labels and teacher outputs are mixed into a single target distribution or kept as separate objectives. The ASR paper on language-model-to-acoustic-model transfer reviews standard knowledge distillation and the label-interpolation method LST. For LST, the loss is

$\rightarrow$ 6

with $\rightarrow$ 7. The paper identifies three practical issues: the same-output-unit constraint, calibration degradation, and hyperparameter sensitivity. Its proposed alternative is a multi-task learning formulation in which one output branch is trained with ground-truth labels and one auxiliary branch is trained to mimic the teacher, avoiding interpolation into a single target distribution (Lee et al., 2021).

In CIF-based ASR, the total objective is

$\rightarrow$ 8

At the acoustic level, the paper uses an InfoNCE-style contrastive loss on token-aligned CIF outputs $\rightarrow$ 9 and PLM token representations $\rightarrow$ 0. At the linguistic level, it uses MSE regression between projected decoder outputs $\rightarrow$ 1 and $\rightarrow$ 2. It also evaluates MSE and cosine acoustic losses, but reports that contrastive loss outperforms MSE and cosine in the cross-modal acoustic setting (Han et al., 2023).

In multimodal-to-unimodal transformers, the total training loss is

$\rightarrow$ 3

The study compares Contrastive Representation Distillation and Entropy Distillation in Attention Maps. CRD is applied either to intermediate transformer outputs or attention maps, whereas EDAM uses cross-entropy directly on attention maps because attention maps are probability distributions. The best configuration uses EDAM-S↓ on attention maps (Agarwal et al., 2021).

In EAST disruption prediction, the student objective is

$\rightarrow$ 4

with $\rightarrow$ 5, $\rightarrow$ 6, and $\rightarrow$ 7. Graph-structure distillation uses the teacher’s top- $\rightarrow$ 8 hyperedges and a cross-entropy-like objective over association matrices $\rightarrow$ 9 and $\rightarrow$ 0, representation distillation uses SmoothL1 on $\rightarrow$ 1 and $\rightarrow$ 2, and decision distillation uses temperature-scaled KL divergence between teacher and student logits (Chen et al., 5 Jul 2026).

Alignment is equally important. In DNN-HMM ASR, forced alignment from a GMM-HMM system, deduplication to remove repeated consecutive labels, and rearrangement are used so that LM posterior sequences can be aligned with frame-level acoustic targets (Lee et al., 2021). In CIF-based ASR, the CIF module produces token-level acoustic representations strictly aligned with the text sequence during training, making acoustic-level distillation feasible (Han et al., 2023). In CMU-MOSEI, the student constructs proxy modalities from the same video stream to retain a transformer scaffold compatible with the teacher (Agarwal et al., 2021). In EAST, image and signal streams are temporally aligned by compensating an approximately 18 ms image lag and resampling both modalities to a 1 kHz time base (Chen et al., 5 Jul 2026).

5. Empirical results and ablations

Across the four papers, the reported gains are tied not only to accuracy but also to stability, calibration, or efficiency. In the seq2seq ASR comparison between the proposed multi-task distillation and LST, the label-interpolation method is described as highly sensitive to $\rightarrow$ 3, and in some settings it performs worse than supervised learning alone. The proposed method is reported to be more stable across hyperparameters and consistently better than interpolation-based distillation. In the hybrid DNN-HMM ablation, using no LM gives 83.23 ACC and 7.98 WER; senone only gives 84.26 ACC and 7.85 WER; phone only gives 84.41 ACC and 7.51 WER; subword only gives 84.60 ACC and 7.38 WER; senone + phone gives 84.44 ACC and 7.43 WER; and senone + phone + subword gives 84.85 ACC and 7.26 WER, summarized as about a 9% relative WER reduction on the LibriSpeech clean test set (Lee et al., 2021).

For CIF-based ASR, the reported main results are a reduction from about 4.9 CER to 4.2 CER on AISHELL-1 test without LM and from 4.8 CER to 4.1 CER with LM, corresponding to about 15% relative error reduction over the original CIF baseline. On LibriSpeech, HKD improves test-clean from 3.3 to 3.0 WER and test-other from 7.7 to 7.0 WER, about 9% relative WER reduction. The ablations further show that ACD alone improves over baseline, LRD alone improves more than ACD in many cases, and HKD combining both is best (Han et al., 2023).

For CMU-MOSEI, the baseline student without KD reports 41.296% accuracy and 32.142 F1. The best result is obtained with the Complete Teacher Network, a simplified 3-transformer student, and EDAM-S↓ on attention maps, giving 44.231% accuracy and 36.332 F1-score. The paper also reports that the best student has 0.675 million parameters versus 1.802 million for the complete teacher, and inference time drops to 29 ms, about 22% faster than the complete teacher (Agarwal et al., 2021).

For EAST disruption prediction on the 640-discharge dataset, the time-series-only baseline reports TPR 95.83%, FPR 8.21%, F1 86.79%, AUC 96.86%, average alarm 754.78 ms, and inference 3.75 ms. The multimodal teacher reports TPR 100.00%, FPR 2.73%, F1 96.00%, AUC 99.00%, average alarm 199.20 ms, inference 8.09 ms, FLOPs 31.35G, and parameters 58.03M. The distilled time-series student reports TPR 91.66%, FPR 2.73%, F1 91.66%, AUC 97.88%, average alarm 713.36 ms, inference 3.75 ms, FLOPs 9.75G, and parameters 30.26M, corresponding to a 2.16× inference speedup, about 68.90% fewer FLOPs, and about 47.85% fewer parameters relative to the teacher (Chen et al., 5 Jul 2026).

6. Interpretation, limitations, and recurrent misconceptions

One recurrent misconception is that multi-to-single-modal knowledge distillation is simply output-logit matching. The surveyed works consistently place transfer at internal structures: auxiliary heads in acoustic models, token-aligned CIF states, attention maps in cross-attention transformers, or hypergraph associations and intermediate features in disruption prediction. This suggests that modality mismatch often requires supervision of representational geometry or relational structure rather than only supervision of final predictions (Lee et al., 2021, Han et al., 2023, Agarwal et al., 2021, Chen et al., 5 Jul 2026).

A second misconception is that “hierarchical” has a uniform meaning. In fact, the term spans at least four distinct formulations in the cited literature: hierarchy over teacher output units, hierarchy over student representation levels, hierarchy over transformer stages, and hierarchy over structure/representation/decision. The underlying commonality is the use of multiple transfer interfaces between teacher and student, not a single standardized decomposition.

A third issue concerns what is preserved and what is sacrificed. The papers emphasize that the student can inherit substantial multimodal or text-derived benefit without inference-time cost, but the student does not necessarily equal the full teacher. In EAST, for example, the multimodal teacher remains best overall on the main test set and the generalization dataset, while the distilled student improves over the time-series baseline and preserves the efficiency of the single-modal pathway (Chen et al., 5 Jul 2026). In the multimodal transformer study, the student improves substantially over the no-KD student and reduces parameters and inference time, but the setting remains one of approximation under missing modalities rather than exact recovery of teacher behavior (Agarwal et al., 2021).

The main methodological debate, as reflected in these works, concerns where and how to distill under modality and unit mismatch. The evidence reported in these papers favors separate-task supervision over label interpolation in LM-to-AM transfer, contrastive objectives for acoustically grounded cross-modal alignment, attention-map distillation in multimodal transformers, and explicit structure-level transfer when the teacher’s advantage lies in higher-order relations. A plausible implication is that hierarchical multi-to-single-modal distillation is best viewed as a design principle for selecting transfer loci that respect the mismatch between the teacher’s information sources and the student’s deployable input space.