Determine the contribution of prosodic features to low-pass speech-to-text reconstruction

Determine whether the lexical reconstruction performance of the fine-tuned Whisper-Medium model operating on low-pass-filtered speech can be attributed directly to prosodic features, particularly fundamental frequency and energy.

Background

The study fine-tunes Whisper-Medium using only the 12 lowest Mel-frequency bins of speech, retaining low-frequency information while removing most segmental content. The resulting model achieves substantially better transcription accuracy than a prosody-only pipeline based on fundamental frequency and energy, but the experiments do not establish which properties of the low-pass signal account for this performance.

To investigate the issue, the authors train a Conformer to generate low-frequency Mel spectrograms from fundamental frequency and energy, then use these synthetic spectrograms as input to Whisper. The considerable performance degradation relative to directly low-pass-filtered speech leaves unresolved whether the successful reconstruction is driven by prosodic information, by additional low-frequency spectral cues, or by limitations of the two-stage prosody-generation pipeline.