CPC2: Advancing Non-Intrusive Speech Intelligibility

• CPC2 is a benchmark that defines non-intrusive speech intelligibility prediction using binaural, hearing-aid processed recordings and listener audiograms.
• It standardizes regression-based prediction by integrating auditory-model features, frozen speech foundation models, and explicit listener conditioning.
• Key insights highlight that lightweight prediction heads with cross-channel attention and ensemble methods yield measurable improvements over traditional baselines.

Searching arXiv for recent and foundational papers on CPC2 and related non-intrusive intelligibility prediction. Clarity Prediction Challenge 2 (CPC2) is a benchmark for non-intrusive speech intelligibility prediction for hearing-impaired listeners. In its canonical formulation, the task is to predict the percentage of words correctly perceived by a listener from a binaural speech-in-noise recording that has undergone hearing-aid processing, together with the listener’s audiogram, without access to a clean speech reference. The challenge has become a focal point for comparing auditory-model baselines, speech foundation model (SFM) backbones, ASR-derived predictors, and later efficiency-oriented sequence models, all under a common emphasis on hearing-aid outputs, listener conditioning, and utterance-level regression (Cuervo et al., 2024).

1. Definition of the task

CPC2 frames speech intelligibility prediction for hearing-impaired listeners as a non-intrusive regression problem. The predictor receives a binaural speech-in-noise recording after hearing-aid processing, together with listener information, and estimates the listener’s intelligibility score as percentage of words correctly perceived. “Non-intrusive” denotes the absence of a clean speech reference at inference time; the system must operate on the processed mixture alone, augmented only with listener information such as the audiogram (Cuervo et al., 2024).

The target variable is the listener’s intelligibility score $y \in [0,100]$. In the SFM benchmark, the scalar model output is bounded to the target range by a sigmoid:

$$\hat{y} = 100 \cdot \sigma(z), \qquad z = w^\top \Big(\tfrac{1}{2}(h_\mathrm{L}+h_\mathrm{R})\Big)+b.$$

The official leaderboard metric in that study is root mean squared error (RMSE) on the test set,

$$\mathrm{RMSE} = \sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2},$$

and statistical comparison across backbones is carried out with a paired Wilcoxon signed-rank test using the hypothesis that a candidate model’s RMSE is greater than the RMSE of a reference backbone, wav2vec 2.0 FT (Cuervo et al., 2024).

A persistent technical distinction in the CPC2 literature is between systems that predict subjective listener outcomes and systems that predict an auxiliary intrusive metric. This distinction is central because CPC2 is motivated by listener-based intelligibility, whereas some later non-intrusive models use CPC2 signals to approximate objective scores such as GESI rather than human listening-test outcomes (Luo et al., 22 Dec 2025).

2. Data, partitions, and listener information

The CPC2 organizers provide audio, listener audiograms, and listener responses expressed as word-correct percentages. In the SFM study, the dataset is partitioned into three independent train/test splits, and CPC2 requires training a separate prediction system for each train split. Across these partitions there are 24,630 training examples and 897 test examples in total. Audio is binaural at 32 kHz, and the SFM systems resample it to 16 kHz for encoder compatibility. For model selection, a development set is constructed by sampling complex scenes from a partition different from the one used for training (Cuervo et al., 2024).

Audiograms in that study are represented as 8-dimensional, non-negative vectors, one threshold per audiometric frequency, and are embedded within the prediction head. The paper notes that further corpus specifics, including numbers of scenes, talkers, noise types, SNR distributions, and the precise hearing-loss model used in the hearing-aid processing, are not detailed there (Cuervo et al., 2024).

Other CPC2 papers describe closely related dataset views. One challenge-oriented study describes the 2023 Clarity Prediction Challenge dataset as comprising scenes with 6 talkers, 10 enhancement methods mapped to 10 hearing-aid front-end systems, and 25 listeners who rated intelligibility. It reports three tracks with 2779, 2796, and 2772 utterances, and test sets of 305, 294, and 298 utterances respectively, with unseen listeners and unseen hearing-aid systems at test time (Zezario et al., 2023). A later Mamba-based study uses the CPC2 dataset together with earlier CEC1 train partitions, specifically CEC2.train.$k$ plus CEC1.train.$k$ for training and a cyclic validation protocol using the next partition, again under a binaural, hearing-aid–processed, listener-conditioned setting (Yamamoto et al., 8 Jul 2025).

These descriptions collectively indicate that CPC2 is less a single monolithic tensor benchmark than a family of fixed challenge partitions with strong cross-system and cross-listener generalization requirements. A plausible implication is that reported performance depends not only on architecture but also on the exact partition protocol and whether earlier Clarity corpora are incorporated.

3. Baselines and early non-intrusive systems

CPC2 includes a baseline based on logistic regression on HASPI, the Hearing-Aid Speech Perception Index. HASPI is an intrusive intelligibility metric derived from an auditory model that compares test and reference signals; in CPC2’s non-intrusive setting it is used as a feature source rather than as a direct metric. That baseline achieved test RMSE 28.70 (Cuervo et al., 2024).

Before the systematic SFM benchmark, CPC2 already included strong Whisper-based submissions. The SFM study references E023 with RMSE 26.34 and E002 with RMSE 25.30. It contrasts those systems with an architecture that keeps the speech backbone frozen, trains only a lightweight task-specific head, explicitly conditions on the listener’s audiogram, and models binaural interactions via cross-attention (Cuervo et al., 2024).

Another competitive non-intrusive system is MBI-Net+, which extends the multi-branched intelligibility predictor from the first Clarity Prediction Challenge. MBI-Net+ uses Whisper-medium embeddings from the final encoder layer, a 257-dimensional spectral feature stream, a CNN-BLSTM with attention in each left/right branch, a 10-class System Classifier that predicts the enhancement method, and a multi-task objective that adds HASPI as a complementary target during training. On the CPC2 test set, its full model reports All-track performance of RMSE 26.1, LCC 0.76, and SRCC 0.767, compared with RMSE 28.1, LCC 0.724, and SRCC 0.729 for the original MBI-Net. The same paper reports that E011 and E002 remain ahead at RMSE 25.1 and 25.3 respectively (Zezario et al., 2023).

A distinct line uses a pretrained noise-robust Transformer ASR model. In that work, the intrusive predictor compares decoder hidden representations between clean reference and processed speech, while the non-intrusive predictor derives utterance-level uncertainty from beam-based negative entropy and maps it with a two-parameter logistic function,

$f(x)=\frac{1}{1+\exp(ax+b)}.$

The ASR backbone is trained on simulated noisy LibriSpeech with room impulse responses and ESC noise, not on CPC2 data. The non-intrusive system reports subset-wise performance of RMSE 0.303, 0.274, and 0.256; NCC 0.660, 0.715, and 0.773; and Kendall’s Tau 0.500, 0.531, and 0.607 on the three CPC2 partitions (Tu et al., 2023).

These baselines establish the core methodological axes that continue throughout CPC2 research: auditory-model features, large pretrained speech representations, explicit metadata or listener conditioning, and ASR-derived uncertainty.

4. Frozen SFM architectures and the winning formulation

The most systematic CPC2 architecture study evaluates ten frozen SFMs as backbones: wav2vec 2.0 Large, wav2vec 2.0 Large FT, wav2vec 2.0 robust Large, wav2vec 2.0 robust Large FT, HuBERT Large, HuBERT Large FT, HuBERT X-Large, HuBERT X-Large FT, WavLM Large, and the Whisper Large-v2 encoder. For each backbone, layer-wise features are extracted from all transformer layers. If the backbone produces an $L \times t \times d$ tensor, frames are downsampled in time by average pooling with factor 20, and a learned linear projection maps the feature dimension $d$ to 384 at each time step (Cuervo et al., 2024).

The prediction head is lightweight and intelligibility-specific. A bidirectional transformer over the time axis performs temporal attention pooling with a prepended CLS token, yielding an $L \times 384$ representation. The listener audiogram $a \in \mathbb{R}^{+8}$ is projected to $$\hat{y} = 100 \cdot \sigma(z), \qquad z = w^\top \Big(\tfrac{1}{2}(h_\mathrm{L}+h_\mathrm{R})\Big)+b.$$0 and appended along the layer axis, producing an $$\hat{y} = 100 \cdot \sigma(z), \qquad z = w^\top \Big(\tfrac{1}{2}(h_\mathrm{L}+h_\mathrm{R})\Big)+b.$$1 sequence. A second transformer with CLS pooling performs attention across layers and collapses this sequence into a single 384-dimensional vector per channel. To model binaural interactions, the temporal and layer transformers include a cross-attention block between left and right channels, allowing each channel to attend to the other. The final left and right embeddings are averaged, linearly projected to $$\hat{y} = 100 \cdot \sigma(z), \qquad z = w^\top \Big(\tfrac{1}{2}(h_\mathrm{L}+h_\mathrm{R})\Big)+b.$$2, and passed through the sigmoid mapping to produce a bounded percentage score (Cuervo et al., 2024).

Training keeps the backbone fully frozen and optimizes only the prediction head with Huber loss:

$$\hat{y} = 100 \cdot \sigma(z), \qquad z = w^\top \Big(\tfrac{1}{2}(h_\mathrm{L}+h_\mathrm{R})\Big)+b.$$3

Optimization uses Adam with learning rate $$\hat{y} = 100 \cdot \sigma(z), \qquad z = w^\top \Big(\tfrac{1}{2}(h_\mathrm{L}+h_\mathrm{R})\Big)+b.$$4, $$\hat{y} = 100 \cdot \sigma(z), \qquad z = w^\top \Big(\tfrac{1}{2}(h_\mathrm{L}+h_\mathrm{R})\Big)+b.$$5, $$\hat{y} = 100 \cdot \sigma(z), \qquad z = w^\top \Big(\tfrac{1}{2}(h_\mathrm{L}+h_\mathrm{R})\Big)+b.$$6, a cosine schedule with 2000-step linear warm-up, batch size 160, 60k training steps, and dropout $$\hat{y} = 100 \cdot \sigma(z), \qquad z = w^\top \Big(\tfrac{1}{2}(h_\mathrm{L}+h_\mathrm{R})\Big)+b.$$7 in all transformer layers. No data augmentation beyond CPC2’s hearing-aid processing is reported. Public checkpoints are taken from fairseq, microsoft/unilm, and openai/whisper. A single training run takes approximately 9 hours on one NVIDIA A100-80GB GPU, with peak memory about 18.4 GB for HuBERT X-Large or Whisper and about 14.4 GB for the other backbones (Cuervo et al., 2024).

This formulation is important because it makes CPC2 a probe of pretrained speech representations under strict adaptation constraints. Rather than fine-tuning the encoders, it asks how much perceptually relevant structure is already present in frozen SFM features once binaural and listener-aware aggregation is added.

5. Performance, ablations, and statistical interpretation

The SFM benchmark reports that all tested backbones outperform the CPC2 HASPI baseline of 28.70 RMSE. The strongest single-backbone results are summarized below (Cuervo et al., 2024).

System	Test RMSE	Brief note
HASPI logistic baseline	28.70	CPC2 baseline
HuBERT Large	25.05	Best single backbone in the benchmark
WavLM Large	25.28	Top-tier single backbone
wav2vec 2.0 FT	26.74	Reference backbone for statistical tests
E011	25.10	Winning CPC2 submission

The complete benchmark further reports HuBERT Large FT at 26.92, wav2vec 2.0 robust FT at 27.36, Whisper Large-v2 encoder at 27.83, wav2vec 2.0 at 28.27, HuBERT X-Large at 26.85, HuBERT X-Large FT at 28.50, and wav2vec 2.0 robust at 28.74. The winning submission, E011, achieved RMSE 25.10, improving over the HASPI logistic baseline by 3.6 points and over the next-best Whisper entry, E002, by 0.20 points (Cuervo et al., 2024).

Statistical testing is reported relative to wav2vec 2.0 FT. WavLM ($$\hat{y} = 100 \cdot \sigma(z), \qquad z = w^\top \Big(\tfrac{1}{2}(h_\mathrm{L}+h_\mathrm{R})\Big)+b.$$8), HuBERT Large FT ($$\hat{y} = 100 \cdot \sigma(z), \qquad z = w^\top \Big(\tfrac{1}{2}(h_\mathrm{L}+h_\mathrm{R})\Big)+b.$$9), and HuBERT Large ($$\mathrm{RMSE} = \sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2},$$0) are not significantly worse, whereas the remaining backbones have $$\mathrm{RMSE} = \sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2},$$1, indicating significantly greater RMSE than wav2vec 2.0 FT. The study therefore identifies a top tier consisting of wav2vec 2.0 FT, HuBERT Large, HuBERT Large FT, and WavLM. It also reports that, contrary to common trends in other tasks, larger models such as HuBERT X-Large and Whisper underperform, and suggests that they may overfit to variance that is not predictive of intelligibility in hearing-aid–processed mixtures (Cuervo et al., 2024).

The ablations are equally consequential. Removing binaural cross-attention consistently degrades performance: for WavLM, test RMSE increases from $$\mathrm{RMSE} = \sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2},$$2 to $$\mathrm{RMSE} = \sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2},$$3 with $$\mathrm{RMSE} = \sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2},$$4; for Whisper, from $$\mathrm{RMSE} = \sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2},$$5 to $$\mathrm{RMSE} = \sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2},$$6 with $$\mathrm{RMSE} = \sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2},$$7. The study interprets this as support for modeling non-linear binaural interactions rather than relying on independent channel processing or simple averaging (Cuervo et al., 2024).

Ensembling produces the largest gains. Averaging predictions from two different backbones yields RMSE 23.86 for HuBERT Large plus wav2vec 2.0 robust FT and 23.88 for HuBERT Large plus WavLM. Even the worst ensembles examined, such as wav2vec 2.0 robust plus HuBERT X-Large at 25.24 and wav2vec 2.0 robust plus wav2vec 2.0 robust FT at 25.25, outperform all single models except the very best. The study reports that over 60% of ensemble combinations beat the best single model, indicating complementarity among SFMs (Cuervo et al., 2024).

6. Post-challenge developments and methodological diversification

Subsequent CPC2 work has focused on two complementary directions: reducing temporal-modeling cost while preserving binaural performance, and broadening the class of non-intrusive proxy predictors.

A 2025 follow-up replaces transformer temporal blocks with Mamba state-space models while keeping the CPC2-proven Whisper Large-v2 feature pipeline. In that architecture, 1280-dimensional Whisper features over 32 encoder layers are pooled with size $$\mathrm{RMSE} = \sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2},$$8, projected to 384 dimensions, processed by a temporal transform block, globally pooled over time, concatenated with 384-dimensional audiogram embeddings, and then passed through a layer-directional transformer before scalar prediction. In the binaural setting, the transformer version uses self-attention plus cross-attention between channels, whereas the Mamba version uses per-channel Mamba blocks with skip connection and GELU, averaging channel latents after layer pooling. The paper emphasizes the theoretical complexity contrast between self-attention, $$\mathrm{RMSE} = \sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2},$$9 in time and memory, and Mamba, $k$0 with constant memory in sequence length (Yamamoto et al., 8 Jul 2025).

Empirically, that work reports for binaural speech intelligibility prediction an average RMSE/NCC of 27.49 / 0.74 for the baseline transformer temporal block, 27.87 / 0.75 for unidirectional Mamba, and 27.34 / 0.75 for bidirectional Mamba, with Wilcoxon signed-rank testing showing no significant differences between models. It therefore presents bidirectional Mamba as competitive with the transformer baseline while using 5.01M parameters versus 5.23M for the binaural transformer. The same paper explicitly notes that runtime, latency, and power were not directly measured, so its efficiency claims remain theoretical rather than empirical (Yamamoto et al., 8 Jul 2025).

Another strand does not predict CPC2’s subjective target directly. DeepGESI is trained on CPC2 hearing-aid output signals to approximate the intrusive GESIv123 metric computed with audiogram-dependent parameters simulating an 80-year-old hearing loss across 125 Hz–8 kHz. The model combines STFT spectral features and a SincNet learnable filterbank, uses a Transformer-style attention layer with Maxout activation and Rotary Position Embedding, produces frame-level GESI estimates, and aggregates them with global average pooling. On a random 80%/10%/10% split of 5,946 CPC2 training signals, it reports seen-test MSE 0.0011, LCC 0.9613, and SRCC 0.9561; on the official unseen CPC2 evaluation set of 897 signals, it reports MSE 0.0034, LCC 0.9289, and SRCC 0.9212. It also reports 0.005 s per utterance, compared with 9.27 s for GESI and 1.26 s for HASPI v2 on the tested hardware. However, the paper explicitly states that the model was not fine-tuned on subjective listening tests, so its alignment with CPC2’s listener-based end point remains to be established (Luo et al., 22 Dec 2025).

Taken together, these studies suggest that CPC2 has evolved into a testbed not only for accuracy but also for architectural inductive bias: transformer cross-attention for explicit binaural fusion, state-space models for linear-time temporal processing, and fast proxy metrics for large-scale evaluation.

7. Reproducibility, limitations, and research significance

CPC2 work has generally been reproducible at the configuration level, though not uniformly at the code-release level. The frozen-SFM benchmark specifies resampling from 32 kHz to 16 kHz, separate systems per CPC2 train partition, fixed optimizer and schedule settings, and public backbone checkpoints, and states that code will be released at a public repository. The Mamba follow-up identifies its use of the CPC2 GitHub repository, Whisper-AT codebase, and official Mamba implementation. By contrast, DeepGESI does not provide code or checkpoints, though it specifies its front-end, loss, optimizer, batch size, and hardware (Cuervo et al., 2024, Yamamoto et al., 8 Jul 2025, Luo et al., 22 Dec 2025).

Several limitations recur across the literature. The SFM benchmark reports that larger models such as HuBERT X-Large and Whisper underperform on this task and suggests a mismatch between capacity and perceptual relevance; it also notes uncertainty about whether cross-attention gains arise solely from better binaural modeling or partly from increased head capacity, and raises the possibility that pooling by factor 20 may obscure temporal fine structure. The same study identifies unexplored sensitivity to noise types, SNR ranges, and hearing-aid processing variations, and proposes future work on tailored pretraining objectives, layer selection, multi-scale representations, personalized calibration, and monotonicity constraints (Cuervo et al., 2024).

Other papers reveal additional scope conditions. The ASR-based non-intrusive system does not model hearing loss explicitly and uses only a logistic rescaling fitted on CPC2 training sets, which limits listener-specific adaptation. DeepGESI does not ingest per-listener audiograms and is trained on objective GESI rather than subjective listener responses. MBI-Net+ shows that metadata about the enhancement method and an auxiliary HASPI objective can improve aggregate performance, but its gains are not uniform across all tracks and the paper does not report system-classifier accuracy or broader metadata such as SNR, noise type, or room parameters (Tu et al., 2023, Luo et al., 22 Dec 2025, Zezario et al., 2023).

Within speech perception research, CPC2 is significant because it operationalizes non-intrusive, listener-aware, binaural intelligibility prediction under hearing-aid processing constraints. The strongest results to date indicate that frozen SFMs combined with lightweight audiogram-conditioned heads and binaural interaction modeling can outperform challenge baselines and prior submissions, while ensembles reveal substantial complementarity across pretrained representations. This suggests that CPC2 is not merely a leaderboard problem but a structured probe of how well current speech representations encode perceptually relevant information for hearing-impaired listening under realistic noise and amplification conditions (Cuervo et al., 2024).