- The paper introduces DASH, a dual-view self-distillation framework using EMA teacher-student dynamics to boost ASR robustness.
- It employs multi-layer prototype-based KL alignment to capture noise-invariant features and achieve lower WERs in noisy conditions.
- The method decouples pre-training from fine-tuning, preserving clean speech accuracy while significantly enhancing performance under noise.
DASH: Dual-View Self-Distillation with Multi-Layer Hidden Representations for Robust Speech Recognition
Motivation and Background
Robustness in ASR systems remains a substantial challenge, particularly when deployed in real-world noisy environments. Conventional supervised approaches leveraging noise-augmented fine-tuning often introduce a trade-off between robustness and performance on clean data, resulting in models that overfit to specific corruption types and degrade in clean scenarios. The need for methods capable of extracting consistent features under varied acoustic perturbations is acute, especially as speech foundation models increasingly adopt SSL paradigms.
Prior works such as Speech-SimCLR, CR-CTC, and HuBERT-VIC have explored cross-view consistency regularization and prototype alignment to boost robustness, but these methods either rely heavily on labeled data, suffer from gradient interference under joint objectives, or insufficiently utilize intermediate encoder representations. The present work addresses these pitfalls by proposing DASH—a self-distillation framework rooted in clean-noisy dual-view consistency, leveraging multi-layer hidden representations and prototype-based distribution alignment.
Methodology
Dual-View Architecture and EMA Teacher-Student Dynamic
DASH adopts a dual-branch encoder architecture comprising a clean (teacher) network and a noisy (student) network, both processing respective clean and noise-augmented views of utterances. The teacher network's parameters are updated using an EMA of the student network's weights, ensuring gradually evolving, stable targets and preventing parameter oscillations typical in rapid step-wise updates.
A critical aspect is the decoupled two-stage training paradigm. DASH pre-trains the encoder via self-distillation with unlabeled speech, followed by supervised fine-tuning for ASR. This approach circumvents delayed convergence and gradient interference common in hybrid joint-training, while enabling scalability with large unlabeled datasets.
Prototype-Based KL Distillation Across Multi-Layer Representations
Direct minimization of distance between teacher and student logits can lead to shortcut learning, where the model focuses on trivial correlations rather than robust phonetic features. DASH addresses this by quantizing encoder outputs into categorical prototypes via k-means clustering, conducted independently for multiple intermediate layers (e.g., layers 6, 11, 17). Frame-prototype assignment distributions from clean and noisy views are aligned using KL divergence, with gradients flowing only through the student (noisy) branch.
Prototype-based distillation, anchored by softmax normalization at each layer, enforces consistency not just on high-level semantic features but also on low-level acoustic cues, robustly capturing hierarchical invariance to noise.
Implementation
The DASH pipeline employs NVIDIA Parakeet TDT-CTC-110M as its baseline, with encoder-only self-distillation (∼5,000 steps, ∼3,230 hours of unlabeled speech) preceding supervised fine-tuning (100,000 steps) using LibriSpeech. Augmentation methods include SpecAugment, MUSAN noise mixing (SNRs: -5 to 15 dB), and reverberation with DNS Challenge RIRs. K-means clustering constructs 512 prototypes from sampled encoder representations, and KL divergence alignment uses an EMA decay rate of 0.999 and temperature of 3.5. The additional pre-training overhead is marginal (<4% of fine-tuning time).
Experimental Results
DASH achieves consistent reductions in WER under noisy conditions, outperforming both clean and noise-augmented fine-tuning baselines across LibriSpeech test-clean, test-other, and noise-mixed settings (white, pink, babble noises). The model preserves or improves clean baseline accuracy, fully mitigating the classic robustness–clean trade-off. Notably, even when fine-tuning is conducted solely on clean data, the DASH-pretrained model exhibits substantial robustness to previously unseen noise, confirming the efficacy of the label-free, prototype-aligned encoder pre-training.
Ablation Analysis: EMA and Layer Selection
Ablation studies reveal the necessity of step-wise EMA updates and multi-layer distillation. Freezing the teacher or reducing EMA frequency leads to consistent accuracy degradation, with frozen teachers and final-layer-only distillation resulting in the most severe performance collapse, even lagging behind standard fine-tuning. This underscores that hierarchical guidance across intermediate encoder layers and dynamic teacher evolution are indispensable for acquiring noise-invariant representations.
Representation Visualization
t-SNE visualizations validate DASH's capacity for aligning noisy examples with clean counterparts in acoustic and semantic spaces. Clusters formed in both low-level (Layer 6) and high-level (Layer 17) representations are notably tighter and less susceptible to noise entanglement under DASH compared to fine-tuning-only baselines, indicating robust noise-invariant feature learning.
Theoretical and Practical Implications
DASH demonstrates that dual-view prototype-based self-distillation with hierarchical encoder guidance and decoupled training can robustly mitigate noise sensitivity in ASR models without compromising clean recognition. This mechanism sidesteps shortcut learning modes prevalent in direct logit alignment and is highly scalable to large unlabeled corpora.
Practically, DASH provides a lightweight, modular pre-training improvement for foundation ASR models, facilitating deployment in real-world scenarios with variable acoustic conditions. The methodological framework is compatible with a range of encoder architectures and augmentation strategies, supporting extensibility to multilingual, multi-speaker, or cross-domain speech tasks.
On the theoretical front, DASH offers evidence that noise robustness can be substantially decoupled from supervised task optimization, provided representations are hierarchically distilled via stable prototype alignment. The success of multi-layer guidance suggests further research directions in leveraging heterogenous intermediate embeddings for unsupervised domain adaptation.
Future Directions
Potential improvements include adaptive prototype selection, dynamic layer weighting, integration with external LLMs for joint representation learning, and extension to end-to-end ASR pipelines with transduction objectives. There is scope for exploring the impact of DASH-style pre-training on transfer learning, cross-accent robustness, and few-shot adaptation scenarios. Further analysis of representation dynamics and shortcut avoidance mechanisms will illuminate generalization boundaries in SSL-based ASR.
Conclusion
DASH advances ASR robustness by combining dual-view, multi-layer self-distillation with prototype-based KL alignment and EMA-driven teacher evolution in a decoupled training pipeline. It achieves superior WER under diverse noisy conditions, preserves clean speech recognition, and eliminates the trade-offs inherent in standard noise-augmented fine-tuning. Step-wise EMA and hierarchical layer distillation are critical for the framework's stability and generalization, setting the foundation for scalable, noise-invariant ASR systems capable of reliable deployment in challenging acoustic environments.
