Word-Level Modeling with Alignment-Aware Acoustic Fusion for Text-Assisted Intelligibility Prediction in Listeners with Hearing Loss

Abstract: We address text-assisted speech intelligibility prediction for hearing-impaired listeners in CPC3. Although the target is a sentence-level percentage, it is determined by reference-word recognition outcomes. We formulate prediction as reference-conditioned word-level correctness modeling: a frozen Whisper encoder analyzes degraded speech, a teacher-forced decoder conditions on the canonical transcript, and sentence intelligibility is obtained by averaging predicted correctness probabilities over valid reference words. To complement transcript-conditioned decoder states, we add a word-aligned local acoustic branch based on character-level cross-attention alignment and an utterance-level global acoustic branch for calibration. On the official evaluation set, the decoder baseline obtains RMSE 24.92 and correlation 0.795, while joint fusion improves to incorrect-word F1 0.778, MCC 0.626, correlation 0.806, and RMSE 24.39. A similar trend with Whisper medium suggests that the gain comes from prediction granularity and alignment-aware fusion.

• The paper introduces a novel word-level modeling approach that reframes intelligibility prediction as a reference-conditioned correctness task.
• The method fuses word-aligned local acoustic features with global calibration signals using a frozen Whisper encoder and character-level teacher forcing, leading to statistically significant improvements.
• The approach enhances intelligibility assessment for hearing-impaired listeners by leveraging canonical transcripts for fine-grained diagnostics, despite additional computational complexity.

Alignment-Aware Word-Level Modeling for Text-Assisted Intelligibility Prediction

Overview

The paper "Word-Level Modeling with Alignment-Aware Acoustic Fusion for Text-Assisted Intelligibility Prediction in Listeners with Hearing Loss" (2605.23604) investigates the problem of predicting speech intelligibility for hearing-impaired listeners, particularly within the Clarity Prediction Challenge (CPC3) context. The authors propose a word-level modeling approach, leveraging reference-conditioned correctness prediction informed by both transcript-conditioned Whisper decoder states and alignment-aware local/global acoustic features. This formulation directly addresses the granularity mismatch inherent in traditional utterance-level regression approaches and exploits the availability of canonical transcripts at inference time as a structured prior for intelligibility prediction.

Methodological Contributions

The central methodological advancement is the reframing of text-assisted intelligibility prediction as a reference-conditioned word-level correctness modeling task. Specifically, the model operates as follows:

• A frozen Whisper encoder processes degraded speech, while a teacher-forced decoder receives the canonical transcript.
• Decoder states are aggregated by reference-word spans, forming a semantic representation for each reference word.
• Word-aligned local acoustic features are extracted via character-level cross-attention alignment, utilizing dynamically selected attention heads determined by sharpness metrics.
• Utterance-level global acoustic features are pooled from encoder states, serving to calibrate overall difficulty.
• All features, including a severity embedding, are concatenated and input to a lightweight classifier, yielding a correctness probability for each reference word.
• Sentence-level intelligibility is computed as the masked mean of word-level correctness probabilities.

The explicit separation of reference-conditioned semantic features, word-aligned local acoustic summaries, and global calibration signals is technically justified and empirically validated. The alignment extraction exploits internal cross-attention mechanisms within Whisper, refined through an auxiliary character-level teacher-forced decoder pass.

Experimental Results

On the official CPC3 evaluation set, the proposed joint local/global fusion model demonstrates statistically significant improvements across relevant metrics:

• Word-level incorrect F1: increases from 0.760 (decoder-only baseline) to 0.778 (joint fusion).
• MCC: rises from 0.601 to 0.626.
• Sentence-level correlation: improves from 0.795 to 0.806.
• Sentence-level RMSE: decreases from 24.92 to 24.39.

These gains persist across listener severity groups, with the largest absolute RMSE reduction for moderately severe listeners, indicating heightened benefit from acoustic evidence aggregation under challenging acoustic conditions. Character-level dynamic head selection outperforms subword-BPE all-head alignment in the local fusion branch, supporting the authors' claim regarding sharper alignment extraction. The teacher-forced reference-conditioned formulation yields substantially higher performance than hypothesis-derived ASR alignments, establishing the necessity of reference anchoring in this context.

Implications

Practically, the model's architecture enables more informative and interpretable intelligibility estimation for applications where canonical transcripts are available, such as controlled speech tests and hearing-aid evaluation. The word-level correctness probabilities offer finer granularity for subsequent diagnostic and enhancement-system assessment. The explicit fusion of local acoustic evidence and global calibration mirrors the structure of human perception and audiological assessment.

Theoretically, the paper delineates the distinction between ASR confidence estimation and perceptual correctness prediction for canonical word anchors—a crucial difference for tasks where perceptual intelligibility is the target. This approach opens pathways to improved reliability estimation in hybrid speech assessment frameworks and indicates the value of leveraging internal alignment signals in foundation-model architectures.

Limitations and Directions for Future Research

The computational complexity associated with the auxiliary character-level decoder is nontrivial, prompting the need for distillation or alternative lightweight alignment extraction. The method assumes access to canonical transcripts at inference, limiting its applicability in fully open-world settings. Metadata conditioning is presently coarse; extending listener representation to include audiograms, cognitive factors, and lexical familiarity could further improve calibration. The uniform weighting of reference words in sentence scoring omits linguistic or phonetic differentiation—a direction ripe for future analysis, which could enhance diagnostic applications.

Further experimental refinement should include stricter controls, such as no-audio and shuffled-audio baselines, as well as direct sentence-regression comparisons with the same backbone to isolate the contributions of word-level aggregation and reference priors.

Conclusion

This work establishes a robust framework for text-assisted intelligibility prediction using reference-conditioned word-level modeling and alignment-aware multi-granular acoustic fusion, validated in CPC3. The reference anchoring, word-aligned local evidence, and global calibration collectively yield superior discrimination and calibration compared to decoder-only and utterance-level baselines. The approach is technically sound, experimentally controlled, and forms a foundation for future enhancement in both diagnostic intelligibility assessment and the development of more reliable speech foundation models in assistive listening contexts (2605.23604).