Time-Frequency Weighted Losses for Phoneme Reconstruction in DNN-Based Speech Enhancement

Abstract: Conventional training losses for speech enhancement based on the signal-to-distortion ratio (SDR) treat all time-frequency (TF) regions uniformly, overlooking the fine-grained spectral cues that are relevant to specific phoneme intelligibility. We propose a TF weighting framework that modulates the SDR objective based on local speech presence, speech-to-interference ratio (SIR), and spectral flux. By integrating these factors into a differentiable objective, the framework emphasizes TF bins with high speech-noise competition while also accounting for transient cues such as consonant bursts. Experimental results show that our approach improves objective frequency-weighted enhancement metrics, as well as phoneme recognition accuracy, particularly for consonants. Spectral analysis shows better reconstruction of mid-frequency structures at less adverse SIR.

• The paper introduces a novel TF-weighted SDR loss that modulates optimization priorities using SIR, speech presence, and spectral flux.
• Experimental evaluations demonstrate improved phoneme-level performance and robust SIR/FW-SIR gains, especially for transient plosive events.
• The proposed framework combines explicit perceptual modeling with adaptive spectral weighting to enhance speech intelligibility in assistive applications.

Time-Frequency Weighted Losses for Phoneme Reconstruction in DNN-Based Speech Enhancement

Motivation and Contributions

Conventional MCSE systems optimize losses like SDR which apply uniform emphasis across the time-frequency (TF) domain, neglecting the non-uniform perceptual importance of spectral regions and localized phoneme-specific cues. Speech intelligibility, particularly in challenging acoustic environments, is strongly influenced by transient, mid-frequency phoneme events—consonantal bursts, plosive transitions, or formant trajectories—which are susceptible to masking when local speech-to-noise ratios are low. This paper introduces a structured TF-weighting framework that modulates the SDR objective by local speech presence, SIR, and spectral flux. In addition, a learnable spectral weighting baseline is proposed to contrast explicit perceptual modeling with data-driven approaches.

TF-Weighted SDR Formulation

The core methodological innovation is a differentiable time-frequency SDR loss in which local bin weights $w(f, t)$ are designed to modulate optimization priorities:

• SIR-speech presence weighting: Sigmoid gates control bin selection, emphasizing regions where speech activity persists despite low SIR.
• SIR-speech presence-spectral flux weighting: Further integrates spectral flux, amplifying temporal sensitivity to transient speech events (e.g., plosives), which are typically underestimated by magnitude-based weighting alone.
• Learnable spectral weights: Frequency-dependent weights initialized from ANSI SII band importance, but allowed to adapt during training, providing a benchmark for the effectiveness of inductive bias versus learning.

These schemes collectively allow precise shaping of the loss landscape to favor phoneme-level reconstructions under adverse conditions, particularly relevant for hearing assistive scenarios.

Evaluation Protocol and Metrics

A rigorous evaluation is performed using LibriSpeech-derived mixtures subjected to WN and SSN, with reverberant multichannel overlays simulating practical binaural hearing-aid configurations. The enhancement algorithm employs FaSNet, a time-domain adaptive beamformer. Performance is quantified at both utterance and phoneme levels, using classical (SDR, SIR, SAR), intelligibility-weighted (FW-SDR), and recognition-based metrics (WER, PA). Statistical significance is assessed via Wilcoxon signed-rank tests.

Numerical Findings

Experimental results demonstrate several key points:

• Utterance-level performance: TF weighting schemes incorporating both SIR-awareness and spectral flux ($$\mathcal{L}_{\textrm{SIR}\cdot\textrm{SP}\cdot\textrm{SF}}$$) consistently yield stronger SIR/FW-SIR improvements than uniform or SIR-only weighting. Artifact/distortion metrics (SAR, SDR) exhibit modest degradation, especially under SSN, but these are outweighed by gains in interference reduction and intelligibility-weighted metrics.
• Phoneme-level reconstruction: PA for both consonants and vowels increases as SIR-aware modulation and spectral flux are integrated. Notably, plosive PA is maximally benefited at moderate/high SIR levels, reflecting better transient reconstruction.
• Spectral analysis: Enhanced spectral tracking of mid-frequency plosive bands is observed in $$\mathcal{L}_{\textrm{SIR}\cdot\textrm{SP}\cdot\textrm{SF}}$$ relative to time-domain baselines, especially in less adverse masking conditions.
• Recognition metrics: WER improvements are robust for WN, with SIR-flux-weighted losses outperforming both time-domain and learnable weighting baselines.

The paper asserts that neither SIR-based modulation alone nor learned spectral weighting is sufficient; the combination of explicit perceptual modeling and transient detection via spectral flux achieves superior and more stable phoneme-level performance, especially for consonants and plosives.

Practical and Theoretical Implications

The proposed framework aligns model training with perceptual priorities by dynamically emphasizing TF bins where speech-noise competition is strongest, and transient phonetic cues are most critical. This holds significance for:

• Hearing aids and assistive communication: TF weighting improves intelligibility under competitive masking, particularly for phoneme regimes sensitive to adverse conditions.
• DNN training objectives: Integration of perceptual and phonetic phenomena into loss functions leads to enhanced speech enhancement systems without compromising artifact/distortion stability.
• Phoneme-aware speech processing: Explicit modeling of transient and mid-frequency structures is shown to materially affect ASR and phoneme recognition.

Future directions are suggested toward modeling additional perceptual effects, such as frequency-dependent SRTs, band-specific masking, and intelligibility-weighted filtering—a trajectory poised to further refine AI-driven hearing and speech enhancement technologies.

Conclusion

The paper delivers a systematic approach for TF-weighted loss design in DNN-based speech enhancement, demonstrating that explicit speech presence, interference, and transient weighting mechanisms substantially improve phoneme reconstruction. Theoretical and practical implications extend to the development of more intelligent, phoneme-aware objectives for both research and real-world hearing-assistive applications, with room for further exploration of perceptual modeling and integration with end-to-end ASR pipelines.