- The paper introduces BanglaRobustNet, a hybrid ASR architecture that integrates a diffusion-based denoising module and a contextual cross-attention module to Wav2Vec-BERT, improving Bangla ASR performance by reducing WER from 35% to 10% in various noise conditions.
- BanglaRobustNet emphasizes preserving phonetic content by using phonetic consistency loss, making it particularly effective in handling Bangla speech's unique phonological features and dialectal variations in noisy environments.
- The architecture lowers computational demands to a 2.1GB footprint and real-time factor of 016 with optimization projecting for seamless deployment in mobile devices.
- Questions
BanglaRobustNet is a hybrid automatic speech recognition (ASR) architecture that augments a Wav2Vec-BERT backbone with two purpose-built modules: a diffusion-based denoising module (DBDM) and a contextual cross-attention module (CCAM) conditioned on speaker attributes. The work targets Bangla ASR under noisy conditions and speaker diversity, a setting in which prior multilingual systems such as Whisper exhibit word error rates (WER) exceeding 30% on noisy Bangla speech (2601.17679).
Motivation and problem setting
Bangla has roughly 500 hours of labeled ASR data versus more than 10,000 hours for English, and its phonology— aspirated consonants (/pʰ/, /tʰ/, /kʰ/), fricatives, nasal vowels—and pronounced dialectal variation compound the data scarcity. The paper positions these two failure modes, noise-induced phonetic degradation and speaker/dialect variability, as the central obstacles for robust Bangla ASR, and argues that generic feature masking approaches such as SpecAugment can distort phoneme integrity, motivating a denoising approach that explicitly preserves linguistic content.
Architecture
The system is built on a Wav2Vec-BERT backbone: a seven-layer convolutional feature encoder producing 512-dimensional frame features at 1/320 of the original sample rate, followed by a 24-layer transformer encoder (hidden size 1024, 16 heads, feed-forward dimension 4096, pre-norm, sinusoidal positional encoding).
Diffusion-Based Denoising Module (DBDM): The DBDM applies a denoising diffusion probabilistic model over acoustic features using a five-level U-Net (channels 512→32 down, mirrored up, SiLU activations, sinusoidal time embeddings). Its distinguishing element is a phonetic consistency loss computed against a pre-trained Bangla phoneme classifier Φ, penalizing divergence between phoneme posteriors of clean and denoised features. The training objective combines the standard DDPM noise-prediction term with this consistency constraint, intended to prevent over-denoising that erases aspiration or vowel-length distinctions.
Contextual Cross-Attention Module (CCAM): A multi-task classifier head over globally pooled acoustic features predicts gender (2-way), age group (4-way), and dialect (6-way), yielding a 128-dimensional speaker embedding. This embedding is projected into query space and combined with transformer queries via element-wise multiplication before softmax attention, producing speaker-conditioned representations with a residual connection. A speaker consistency loss (cross-entropy on the three attribute tasks) plus a KL-divergence consistency term between speaker posteriors from clean and denoised features regularize the module.
Training: The model is trained end-to-end with a composite objective combining CTC loss over a 52-symbol Bangla grapheme vocabulary, phonetic consistency, speaker alignment, and clean/denoised consistency terms. Training proceeds in three stages: contrastive pre-training on LibriSpeech plus OpenSLR (100 epochs), diffusion training on noised audio at SNRs from −5 to 20 dB (50 epochs), and joint fine-tuning on Common Voice Bangla plus BengaliSR (30 epochs). Augmentation includes speed perturbation, volume scaling, additive noise, room impulse response simulation, and Bangla-specific phonetic augmentations (consonant cluster modification, vowel length adjustment, aspiration intensity variation).
Experimental results
Evaluation uses Common Voice Bangla v20, OpenSLR SLR53, and Bhasha-bichitra dialectal data as clean test sets, plus 20 hours of augmented noisy speech across SNR levels of 0–10 dB and four noise types. Baselines are Whisper-Small, Whisper-Large-v2, and vanilla Wav2Vec-BERT, all fine-tuned identically; significance is assessed via paired t-tests.
On clean speech, BanglaRobustNet reports WER 12.3%, PER 8.9%, and BLEU 76.4, versus 14.42% WER for Wav2Vec-BERT and 28.86–32.17% for Whisper variants—a 12% relative WER reduction over its own backbone. Under noise the gains are larger: at 5 dB SNR the model achieves 24.3% WER versus 65.0% for Whisper-Small, and at 0 dB it reaches 42.4% where Whisper-Small degrades to 100%. Relative improvements of roughly 26–36% are reported across SNR levels, with traffic noise showing the largest gain (35%). The paper claims 18% relative WER reduction under noise and 15% across dialects overall, and states all differences are significant at p<0.001.
The ablation study reveals an important nuance: adding DBDM alone slightly hurts clean-condition WER (14.8% vs. 14.42%) and PER (10.7% vs. 9.37%), while improving noisy WER modestly (28.4% vs. 30.1% at 5 dB). Only the full configuration achieves the headline numbers (12.3% clean, 24.3% at 5 dB), which the authors attribute to synergy between modules. This suggests the denoiser's benefit is contingent on joint optimization with speaker conditioning rather than intrinsic to denoising itself—an assumption the paper asserts but does not isolate further.
Qualitative analysis highlights correct preservation of aspiration (/pÊ°/ vs /p/), vowel length distinctions, and complex conjunct graphemes such as "tRo" in "matrRbhaSha", where baselines introduce substitutions. On computational efficiency, the model attains a real-time factor of 0.16 using a reduced 10-step diffusion schedule, with INT8 quantization yielding a 2.1 GB footprint suitable for mobile deployment at under 5% performance loss.
Limitations and open questions
Several caveats bear directly on the reported results. First, the ablation table omits a CCAM-only configuration, so the claimed "synergistic effect" cannot be decomposed into individual module contributions beyond the DBDM-only row. Second, the Bhasha-bichitra dialectal dataset lacks publicly reported hour counts, making the claimed 15% cross-dialect improvement difficult to verify independently. Third, the DBDM-only degradation on clean speech raises the question of whether the phonetic consistency constraint alone suffices to prevent over-denoising, or whether its benefit depends entirely on co-training with CCAM. Fourth, the paper does not report results for Whisper-Large under noise despite including it among baselines, leaving the comparison against the strongest baseline incomplete. Finally, whether the speaker-attribute classifier generalizes to dialects beyond the six covered, and how the model behaves on unseen noise types, remain untested.
Conclusion
BanglaRobustNet demonstrates that coupling a phonetically constrained diffusion denoiser with speaker-conditioned cross-attention on a self-supervised backbone yields substantial WER reductions for Bangla ASR, particularly in low-SNR conditions where baseline models approach complete failure. The strongest evidence lies in the noisy-condition results (42.4% WER at 0 dB versus 100% for Whisper-Small) and the qualitative preservation of Bangla-specific phonetic distinctions. The open questions center on module attribution, dialectal evaluation rigor, and generalization beyond the tested noise distributions.