Frame-Aligned Fusion of Canary and WavLM for Non-Intrusive Intelligibility Prediction of Hearing-Aid-Processed Speech

Abstract: Non-intrusive intelligibility prediction estimates how well hearing-impaired listeners understand hearing-aid-processed speech without a clean reference. We study this task in the 3rd Clarity Prediction Challenge using two frozen speech encoders, Canary and WavLM. The central question is not only whether complementary pretrained representations should be combined, but where their interaction should occur. We compare single-backbone baselines, uniform score averaging, pool-late fusion, cross-attention, frame-aligned fusion, and reverse alignment under a shared left/right-preserving binaural framework. Among the compared systems, the best model temporally prepares WavLM with a learnable strided convolution and fuses it with Canary on the coarser Canary timeline before pooling, reaching Eval RMSE 24.96$\pm$0.06 and Eval Corr 0.796$\pm$0.001. Severity, enhancement-system, layer-window, and temporal-shift analyses indicate that coarse local temporal correspondence before pooling is a useful inductive bias for this task.

• The paper introduces a novel frame-aligned fusion technique that integrates high-temporal WavLM features with Canary’s abstract linguistic cues to improve intelligibility prediction.
• Experiments on the CPC3 dataset show that the proposed fusion with learnable convolution outperforms single-backbone and naïve ensembling methods in both RMSE and correlation.
• The method achieves robust performance across varied hearing loss groups and enhancement systems while using fewer trainable parameters than baseline models.

Frame-Aligned Fusion of Canary and WavLM for Non-Intrusive Intelligibility Prediction of Hearing-Aid-Processed Speech

Introduction and Motivation

Non-intrusive intelligibility prediction for hearing-aid-processed speech entails estimating how comprehensible speech is for hearing-impaired listeners based solely on the processed audio, without access to a clean reference. This capability is essential for enabling objective evaluation and tuning of hearing-aid algorithms in realistic, uncontrolled listening conditions, as formalized in the Clarity Prediction Challenge (CPC3). The intersection of recent advances in speech representation learning—especially large frozen encoders—and the fusion of complementary features in low-resource target tasks, raises an acute question: at which processing stage should representations from different backbones be merged to maximize performance in the non-intrusive scenario?

Architecture Overview and Fusion Strategies

The study conducts a systematic comparison of fusion strategies for two frozen encoders: WavLM (self-supervised, high temporal resolution, acoustic-phonetic focus) and Canary (ASR-oriented, lower frame rate, more linguistic abstraction). Architecture variants are benchmarked under a shared left/right-preserving binaural framework, deferring inter-ear fusion until just prior to regression.

Figure 1: Architectural layouts for feature extraction and fusion. Pool-late fusion combines utterance-level summaries, frame-aligned fusion exploits Canary's coarser time axis for local correspondence, while cross-attention enables unrestricted cross-stream interactions.

Key fusion strategies are:

• Single-backbone baselines: Isolate performance of Canary and WavLM individually.
• Uniform score averaging: Scalar-level ensembling of independent single-backbone predictors.
• Pool-late fusion: Post-pooling concatenation and fusion of utterance-level features.
• Frame-aligned fusion: Temporal preparation (average pooling or learnable convolution) downscales WavLM features, aligning them on Canary's time axis before concatenation and pooling.
• Cross-attention and reverse alignment: Sequence-level interaction using either stream as query, and fusion where the temporal reference is inverted.

Frame-aligned fusion leverages temporal downsampling of the high-rate WavLM features via either fixed average pooling or a learnable convolution (kernel size 4, stride 4), concatenating these prepared representations with Canary features before any global pooling. This design hypothesis is that local, contextually-aware frame-level interactions (prior to sequence summarization) reveal complementary cues for intelligibility that are lost in utterance-summarizing approaches.

Experimental Setting and Protocol

All experiments use the CPC3 dataset, preserving the official train/dev/test splits. Listener-severity is modeled by late-stage embedding, and five-fold cross-validation is employed with seed-level ensembling. Crucially, all encoder weights are frozen, ensuring that improvements stem from architectural and fusion choices rather than overfitting large, trainable towers.

Evaluation hinges on root-mean-square error (RMSE) and Pearson correlation (Corr) between predicted and ground-truth word-level intelligibility percentages, allowing for robust comparison across listener subgroups and enhancement systems.

Main Results

Figure 2: Eval RMSE with error bars over five seed-level ensembles, showing frame-aligned fusion with convolutional preparation as the superior approach.

The principal findings are as follows:

• Canary-only outperforms WavLM-only (Eval RMSE 25.64 vs. 26.62), highlighting the strength of higher-level linguistic abstractions for this task.
• Uniform score averaging provides only marginal improvement over Canary alone (25.53 RMSE), and does not enhance correlation—demonstrating limited benefit from naive scalar ensembling.
• Frame-aligned fusion, especially with learnable convolutional preparation, yields the lowest Eval RMSE (24.96 ± 0.06) and highest Corr (0.796 ± 0.001). The performance advantage is robust to within-family seed-level variance.
• Alternative fusion schemes (pool-late, cross-attention, reverse alignment) consistently underperform the proposed method, supporting the central design hypothesis regarding the locus of interaction.

Notably, the best-performing dual-backbone model uses fewer trainable parameters than any single-backbone baseline (1.30M vs 1.60M for Canary-only), ruling out trivial explanations based on increased model capacity.

Analytical Ablations and Robustness

How the gain arises is interrogated via additional controls:

• Reverse frame-aligned fusion (mapping Canary up to WavLM frame rate) is beneficial but consistently weaker than aligning on Canary's timeline, confirming the inductive superiority of the coarser, linguistically-oriented time axis as an anchor.
• Layer sweeps for WavLM show that upper layers (17–24) are optimal, suggesting that features closer to sub-lexical or phonetic structure, rather than raw acoustics, provide the best auxiliary evidence.
• Temporal shift experiments reveal that the frame-aligned approach is robust to moderate offsets (±160 ms), implying that coarse local temporal correspondence suffices; precise sample-level synchrony is not essential, and locality is the critical regularizer.

Group and System-Level Consistency

Listener subgroup analyses show consistent improvements across mild, moderate, and moderately severe hearing loss categories, with the most pronounced RMSE reduction for the moderately severe group. Across nine diverse enhancement systems (beamforming, denoising, etc.), frame-aligned fusion improves RMSE and correlation in every case, and most frequently also reduces MAE, which simple score averaging fails to do.

These findings demonstrate that the proposed fusion approach is robust to variability in both processing algorithm and listener characteristics, and does not derive its strength from any narrow cohort.

Theoretical and Practical Implications

This work provides conclusive evidence that interaction between complementary frozen speech encoders should occur on a local temporal basis, aligned to the axis of the higher-level, more abstract representation (here, Canary), rather than after global utterance summarization or via unconstrained sequence-level cross-attention. The findings clarify that the gain is not merely due to shallow form of ensemble calibration or increased downstream capacity, but from structural regularization that encourages context-dependent integration of acoustic-phonetic evidence into the linguistic scaffold.

Practically, the method offers a sample-efficient, compute-conscious template for future speech quality/intelligibility regression pipelines: pretrain large encoders, freeze them, and maximize the information synergies via carefully designed local fusion rather than brute-force scalar stacking or deep trainable heads.

Future developments may extend this framework to tri-stream setups including enhancement-centric backbones, incorporate richer listener metadata, or generalize the frame-aligned inductive bias to multimodal and audio-language settings. The principle that coarser timelines serve as the optimal locus for integrating higher-resolution evidence can steer further work in non-intrusive assessment and, more generally, in ensemble architectures for speech and audio understanding.

Conclusion

This study delivers a controlled, reproducible analysis of fusion design for non-intrusive hearing aid intelligibility prediction. Frame-aligned fusion of WavLM-prepared features onto the Canary timeline, followed by pre-pooling interaction, yields the strongest, most robust performance, outstripping both post-pooling and naïve ensemble approaches. This inductive bias—fusing fine-rate evidence into coarser, linguistically ordered states before global aggregation—emerges as the preferred architectural paradigm for combining complementary frozen encoders in speech regression settings. The results have broad implications for both the theory and practice of frozen-backbone ensemble methods in spoken language processing.

For all technical details, results, and referenced benchmarks, see "Frame-Aligned Fusion of Canary and WavLM for Non-Intrusive Intelligibility Prediction of Hearing-Aid-Processed Speech" (2605.23619).