Interlanguage Speech Intelligibility Benefit in ASR

• Interlanguage Speech Intelligibility Benefit (ISIB) is a phenomenon where non-native listeners with a matched L₁ outperform native L₂ listeners in understanding accented speech.
• Empirical studies demonstrate that ASR systems using discrete token-based architectures and k-means clustering on native L₁ data achieve lower word error rates compared to those trained on L₂ data.
• Advanced methods like differentiable k-means and L₁–L₂ multi-task learning further refine ISIB by jointly optimizing phonetic regularities, enabling robust accent adaptation in ASR.

The Interlanguage Speech Intelligibility Benefit (ISIB) refers to a counter-intuitive and empirically substantiated phenomenon in speech perception wherein non-native listeners whose first language (L₁) matches the speaker’s L₁ demonstrate superior intelligibility for the speaker’s foreign-accented speech compared to native listeners of the spoken language (L₂), and even compared to non-native listeners whose L₁ differs from the speaker’s. ISIB is crucial in accent-robust automatic speech recognition (ASR), particularly for systems processing foreign-accented utterances with only native speech data available for model training. Recent research operationalizes ISIB in ASR through controlled manipulation of tokenization schemes derived from self-supervised learning (SSL) representations, substantiating both the theoretical and quantitative aspects of ISIB using discrete token-based architectures (Onda et al., 22 May 2025, Onda et al., 27 Jan 2026).

1. Psycholinguistic Foundations of ISIB

ISIB is delineated in psycholinguistics as an emergent benefit whereby speech intelligibility crosses native-language boundaries in unexpected ways. Bent and Bradlow (2003) classify ISIB effects into two experimental measures:

• ISIB-L (“listener benefit”): L₁-matched non-native listeners outperform native listeners of L₂ on transcription or comprehension tasks involving foreign-accented speech.
• ISIB-T (“talker benefit”): L₁-matched non-native listeners attain higher accuracy transcribing L₂-accented speech than when transcribing comparable L₂-native speech.

ISIB is compelling both as evidence of L₁-induced perceptual recoding and as motivation for advancing computational models that more closely emulate human categorical perception mechanisms when confronted with L₂-accented L₂ speech produced by L₁ speakers (Onda et al., 22 May 2025).

2. Technical Realization in Discrete Token-Based ASR

Operationalizing ISIB within ASR leverages a two-stage framework predicated on SSL. High-level workflow steps are:

1. SSL Feature Extraction: A HuBERT-base model is used to extract frame-wise feature vectors, $\mathbf{h}_t$, from input waveforms. Layer choice (6, 9, or 12) modulates the abstraction level captured by representations.
2. Clustering/Tokenization: K-means clustering ($K \in \{100, 500, 2000\}$) is trained exclusively on native L₁ speech, producing centroids $C^{(L)}$. Each frame is discretized to the nearest centroid's index $c_t$. Deduplication is applied to collapse consecutive identical tokens.
3. ASR Model: Discrete token sequences feed into a joint CTC/attention encoder-decoder (e.g., ESPnet architecture), with training performed on native L₂ data only.

The ISIB is explicitly measured by: $$\Delta_{\mathrm{ISIB}} = \mathrm{WER}_{\text{kmeans}=L_2} - \mathrm{WER}_{\text{kmeans}=L_1}$$ A positive $\Delta_{\mathrm{ISIB}}$ indicates that L₁-trained tokenization yields lower WER on L₂-accented speech than L₂-trained tokenization, thus validating the ISIB hypothesis quantitatively (Onda et al., 22 May 2025).

3. Empirical Evidence and Quantitative Evaluation

Experiments using English (L₂) read by American-English (native), Japanese learners (ERJ corpus), and speakers with six L₁s in L2-ARCTIC, consistently demonstrate ISIB:

• For Japanese-accented English evaluated on the JE_all set (LibriSpeech-960, $K=2000$, layer 12): k-means trained on Japanese yields WERs of 53.3% (vs. 55.7% for English-trained). The effect intensifies for strong accents (JE_w10: 68.0% vs. 70.8%).
• The benefit generalizes across SSL layer choices and smaller-scale experiments (LibriSpeech-100 pilot).
• For each L₂ accent in L2-ARCTIC, the optimal WER is achieved when k-means clustering is performed on matching L₁ data.

A related phenomenon, mismatched ISIB, is observed, where even a non-matching but phonetically similar L₁ improves WER over L₂-trained clusters, suggesting a nuanced interaction between phonetic inventories and model robustness (Onda et al., 22 May 2025).

4. Theoretical Explanation: Phonetic Manifold Alignment

ISIB emerges in discrete token-based ASR due to how SSL-k-means centroids partition the feature space in a manner reflecting L₁-dependent categorical perception:

• L₂-trained centroids are sensitive to native L₂ contrasts (e.g., /r/–/l/ in English), while L₁-trained centroids (e.g., Japanese) may collapse these contrasts, thus absorbing typical L₁→L₂ substitutions in accented speech.
• A formal perspective models the mapping as $$f_{C^{(L)}}: \mathbf{h}_t \mapsto \arg\min_{c \in C^{(L)}} \|\mathbf{h}_t - c\|$$, where alignment between $C^{(L)}$ and the phonetic manifold of L₁-accented L₂ speech reduces “out-of-vocabulary” clusters and decoding errors.

As a paradigmatic example, k-means-EN clusters misrecognize the word “appreciated” as “appliciated” due to /r/→/l/ confusion, whereas k-means-JP clusters correctly preserve the original token sequence (Onda et al., 22 May 2025).

5. Advanced Modeling: Differentiable K-Means and Multi-Task Learning

Recent advancements extend ISIB implementation via differentiable k-means (DiffKM) and L₁–L₂ multi-task learning:

• DiffKM: Cluster centroids $M = [m_1, ..., m_K]$ are trained via soft assignments $$p_{t,k} = \frac{\exp(-\|h_t - m_k\|^2/\epsilon)}{\sum_j \exp(-\|h_t - m_j\|^2/\epsilon)}$$. The quantized output $v_t = \sum_k p_{t,k} m_k$ is fed into the ASR encoder. Both centroids and SSL encoders are optimized by downstream ASR loss (Onda et al., 27 Jan 2026).
• L₁–L₂ Multi-Task Objective: A composite loss

$$\mathcal{L} = (1-\alpha)\,\mathcal{L}^{\mathrm{asr-l2}} + \alpha\,\mathcal{L}^{\mathrm{asr-l1}}, \quad \alpha \in [0,1]$$

enables joint optimization for both L₂ and L₁ ASR tasks, leading to centroids capturing the intersection of L₁ and L₂ phonetic regularities.

• Joint training and fine-tuning of all modules enable discrete tokens to encode aspects of both native and accented phonology, modeling the dual-layered perceptual space posited in psycholinguistics.

Quantitative results demonstrate that in adaptation scenarios with limited accented data, the DiffKM+MTL model achieves approximately a 20% relative WER reduction compared to the strongest standard k-means baseline (e.g., from 43.0% to 34.7% WER with only 2 hours of accented English training) (Onda et al., 27 Jan 2026).

6. Implications for Accent-Robust ASR and Generalizability

The ISIB framework fundamentally transforms accent adaptation and data requirements for ASR:

• Native-only data sufficiency: Accent-robust systems can be constructed using only native speech for k-means training and ASR, sidestepping the scarcity of accented corpora.
• Scalability: The approach generalizes across L₁–L₂ pairs and, when low-resource L₁ data is unavailable, proxy L₁s with similar phonetics can approximate ISIB effects (“mismatched ISIB”).
• Practicality: The framework enables ASR adaptation to a combinatorial number of accent pairs via existing native corpora, facilitating large-scale deployment in multilingual and multicultural environments (Onda et al., 22 May 2025).

A plausible implication is that integration with speech-language foundation models and automated selection of hyperparameters such as clustering temperature $\epsilon$ and MTL weighting $\alpha$ could further broaden the robustness and applicability of ISIB-based architectures (Onda et al., 27 Jan 2026).

7. Current Limitations and Prospective Directions

Present research on ISIB is centered on specific language pairs (notably Japanese$\rightarrow$English). Extending empirical evaluations to diverse L₁–L₂ configurations (e.g., Mandarin–English, Spanish–French) is needed to confirm universality. Open research areas include:

• Automated or meta-learned selection of cluster parameters and task-weighting ($K$, $\epsilon$, $\alpha$).
• Generalization to unknown or mixed-L₁ scenarios characteristic of spontaneous code-switching or unlabeled corpora.
• Incorporation of cluster-usage regularizers or phonetically informed priors to promote centroid stability.
• Harmonization with large-scale pre-trained speech-LLMs for enhanced accent robustness.

Continued exploration of ISIB may yield foundational insights not only for ASR, but also for the broader design of language technologies sensitive to the nuances of human cross-linguistic perception (Onda et al., 22 May 2025, Onda et al., 27 Jan 2026).