Mispronunciation Detection and Diagnosis

• Mispronunciation Detection and Diagnosis (MDD) is the computational process of identifying and classifying pronunciation errors to support automated feedback in language learning.
• Modern MDD systems employ end-to-end neural architectures, anti-phone modeling, and self-supervised features to enhance phoneme recognition and error diagnosis.
• By integrating acoustic analysis, articulatory feature prediction, and advanced training losses, MDD drives improvements in CAPT systems and personalized language training.

Mispronunciation Detection and Diagnosis (MDD) is the computational process of identifying, localizing, and characterizing pronunciation errors made by language learners with respect to canonical forms, typically at the phoneme or sub-phonemic level. MDD is a fundamental component of Computer-Assisted Pronunciation Training (CAPT) systems, supporting both automatic feedback in second language (L2) learning and the development of formative, actionable guidance for users. This field synthesizes approaches from Automatic Speech Recognition (ASR), acoustic-phonetic modeling, representation learning, and diagnostic interface design.

1. Taxonomy of Pronunciation Errors and Diagnostic Objectives

MDD frameworks generally categorize errors into several types:

• Categorical errors: One canonical phone is substituted, inserted, or deleted (e.g., /dh/→/d/), with clear reference to a phone inventory.
• Non-categorical (distortion) errors: Intermediate or distorted pronunciations, such as L2 phones approximated by similar L1 phones or realizations lying between canonical phones in acoustic space.
• Attribute-based errors: Errors reflect specific articulatory feature violations (e.g., failure in voicing or tongue placement), moving beyond simple phone substitution (Shahin et al., 2023).

Diagnosis refers not only to detecting the occurrence of an error but also to specifying its type (substitution, deletion, insertion) and, in advanced systems, the underlying articulatory/phonological cause.

2. End-to-End Neural MDD Architectures

Recent systems overwhelmingly favor end-to-end (E2E) neural modeling, offering unified acoustic modeling, alignment, and diagnostic capabilities (Yan et al., 2020, Yan et al., 2021, Shahin et al., 2023, Cao et al., 18 Jul 2025). Core architectural motifs include:

• Hybrid CTC–Attention Models: An encoder (typically BLSTM or Transformer) maps input acoustic features to high-level representations. Parallel CTC (Connectionist Temporal Classification) and attention-based decoders provide both monotonic alignment and flexible decoding, interpolated in a joint loss:

$L = \lambda L_\text{ctc} + (1-\lambda)L_\text{att}$

CTC is critical for enforcing proper monotonic alignments, while attention layers improve sequence modeling and robustness to phone length variability (Yan et al., 2020, Yan et al., 2021).

• Anti-Phone Modeling: The canonical phone set is augmented with “anti-phone” symbols (e.g., “#p”), allowing systems to explicitly model distorted/non-categorical pronunciations. Training targets anti-phone labels for known mispronunciations, enabling the system to signal both categorical and subtle distortions without hand-crafted phonological rules (Yan et al., 2020).
• Self-Supervised Feature Encoders: Advanced E2E MDD models leverage self-supervised speech representations (wav2vec2.0, HuBERT, XLS-R) as input features, improving robustness and data efficiency in low-resource L2 settings (Kheir et al., 2023, Yang et al., 2022, Wang et al., 7 Jun 2024). These representations can be directly fine-tuned for phone recognition or mispronunciation detection, bypassing the need for hand-crafted acoustic features.
• Raw Waveform Front Ends: SincNet and related architectures operate directly on raw waveforms, enabling the network to learn task-relevant filterbanks, offering both improved performance and interpretability—especially when adapting to L1-specific acoustic profiles (Yan et al., 2021).
• Hierarchical Architectures: Multi-level structures such as Branchformer or Mamba-based SSM stacks enable the fusion of phoneme-, word-, and utterance-level representations, supporting both localized MDD and global pronunciation assessment (APA) within unified models (Yan et al., 6 Oct 2025, Yang et al., 24 Jun 2025).

3. Training Objectives and Data Regimes

MDD models are trained under several distinctive supervisory approaches and loss designs:

• CTC and Attention Loss: Standard for sequence-to-sequence phoneme recognition. Alignment-free supervision via CTC allows flexible association between phone strings and variable-length acoustic sequences (Yan et al., 2021, Wang et al., 2021).
• Anti-Phone and Augmentation Losses: By augmenting canonical phone targets with anti-phones (or via data-driven label shuffling), models learn to explicitly anticipate mispronunciations and distorted variants (Yan et al., 2020, Fu et al., 2021).
• Multi-Task and Attribute Losses: Models are increasingly supervised with multiple aligned tasks, such as simultaneous phone recognition and articulatory feature prediction via multi-label CTC (Kheir et al., 2023, Shahin et al., 2023). This supports richer diagnostics and improved generalization in data-scarce settings.
• Contrastive and F1-Optimized Losses: To address the mismatch between cross-entropy training objectives and F1-based evaluation in MDD, several works employ contrastive loss (margin between correct and erroneous phone sequences) or directly optimize expected F1 scores using an M-best list approximation (Yan et al., 2021, Peng et al., 2022). These methods incentivize true positive/negative separation and improved error localization.
• Semi-Supervised and Pseudo-Labeling: Scarcity of L2 phoneme labels is addressed using momentum pseudo-labeling, combining labeled and pseudo-labeled L2 data generated on-the-fly by a “teacher” model (EMA of student). This strategy yields measurable gains in both MDD F1 and PER (Yang et al., 2022).
• Decoupled Cross-Entropy (deXent): Weighted CE loss splits terms for correct vs. mispronounced phones, enabling targeted supervision that strongly improves rare-event recall without sacrificing precision (Chao et al., 11 Feb 2025).
• No Training Approaches: Retrieval-based frameworks bypass MDD-specific model training, performing nearest neighbor search in the embedding space of a pretrained ASR encoder. Such approaches achieve state-of-the-art F1 scores (e.g., 69.6% on L2-ARCTIC) without the need for model fine-tuning or additional supervision (Tu et al., 25 Nov 2025).

4. Diagnostic Inference, Feedback, and Evaluation

MDD systems output per-phoneme or per-frame decisions, typically organized as:

• Detection: Determining if a given segment is canonical or erroneous (binary or multi-class labeling).
• Diagnosis: Assigning a type (substitution, deletion, insertion) and, where models allow, specifying the realized phone or underlying attribute error (Shahin et al., 2023).
• Fine-Grained Articulatory Attributes: Low-level attribute prediction enables feedback targeted to specific articulatory properties (e.g., voicing, place), allowing CAPT systems to produce formative, user-actionable guidance (Kheir et al., 2023, Shahin et al., 2023).

Standard performance metrics include:

Metric	Formula / Description
Phone Error Rate (PER)	$(S+D+I)/N$, where S, D, I are substitutions, deletions, insertions
Precision	$TP/(TP+FP)$ (typically per-phone)
Recall	$TP/(TP+FN)$
F1-score	$$2\cdot(\text{Precision}\cdot\text{Recall})/(\text{Precision}+\text{Recall})$$
Diagnostic Accuracy Rate (DAR)	$$\#\text{Correct Diagnosis} / (\#\text{Correct + Incorrect})$$

Advanced works evaluate additional metrics, such as attribute-wise FAR/FRR/DER, error localization accuracy, or subjective ratings’ correlation (Shahin et al., 2023, Cao et al., 18 Jul 2025, Yan et al., 6 Oct 2025).

5. Augmentation, Accent Robustness, and Language Generalization

Several key strategies address variability and resource constraints:

• Data Augmentation: Perturbing input text or phone sequences (VC, CP schemes) balances class frequencies and simulates error patterns, improving error sensitivity and recall (Fu et al., 2021).
• Accent Modulation: Accent-aware embeddings (either “hard,” via known accent labels, or “soft,” inferred via auxiliary classifiers) are injected at the encoder or per-layer level, boosting both F1 and diagnosis accuracy when handling multi-accented learners (Jiang et al., 2021).
• Attribute-Driven, Cross-lingual, and Zero-Shot Robustness: By operating at the phonological/attribute level, or using multilingual self-supervised models, MDD systems generalize to under-resourced L2s, unseen accents, and novel sounds without requiring explicit mispronunciation labels (Shahin et al., 2023, Wang et al., 7 Jun 2024, Kheir et al., 9 Jun 2025).

Recent research has extended MDD paradigm to Mandarin (using pitch-aware RNN-T with HuBERT features and explicit F0 fusion (Wang et al., 7 Jun 2024)), Arabic (specialized phoneme sets; mispronunciation benchmarking for Qur’anic recitation (Kheir et al., 9 Jun 2025)), and other L2 speech domains.

6. Integration with Pronunciation Assessment and Multi-Facet Modeling

The frontiers of MDD research lie in its integration with broader automatic pronunciation assessment (APA):

• Joint APA–MDD Modeling: CAPT systems are increasingly designed to output both detailed error diagnosis (MDD) and holistic/linguistic-level assessment (APA) in a single, parameter-shared architecture (Yan et al., 6 Oct 2025, Yang et al., 24 Jun 2025, Ahn et al., 3 Sep 2025). Hierarchical SSMs and multi-task losses facilitate this joint optimization.
• Modular vs. Unified Frameworks: APA and MDD can be unified within modular pipelines (e.g., LoRA-fine-tuned MLLMs) or trained jointly, trading off model simplicity for mutual performance gains (Ahn et al., 3 Sep 2025, Yang et al., 24 Jun 2025).

7. Limitations, Key Benchmarks, and Future Directions

Current limitations include:

• Diagnosis accuracy ceiling: Despite recent advances, diagnosis accuracy rates (DAR) remain moderate (< 60% in many benchmarks), attributed to omission of prosodic/suprasegmental errors, rare error types, and label taxonomic bottlenecks (Yan et al., 2020, Yan et al., 6 Oct 2025).
• Data scarcity & label noise: Many systems are still limited by the sparsity and noisiness of phoneme-level L2 annotation, despite advances in pseudo-labeling and augmentation (Yang et al., 2022).
• Benchmarks and evaluation: Key datasets include L2-ARCTIC, Speechocean762, CMU Kids, and QuranMB.v1; consistent reporting of F1, PER, DAR, and attribute-wise DER is crucial for comparison.

Future research directions as identified in multiple works:

• Learning continuous manifolds for mispronunciation, replacing discrete anti-phone or attribute schemes.
• Extension to prosody-level diagnostics (rhythm, intonation, lexical stress) and multi-modal feedback (Yang et al., 2022, Wang et al., 7 Jun 2024).
• Open-set/unsupervised and non-native data exploitation for increased generalizability to unseen L1 backgrounds (Tu et al., 25 Nov 2025, Kheir et al., 9 Jun 2025).
• Integration of formative, action-guiding feedback systems for finer learner intervention (Shahin et al., 2023).
• Direct optimization of task-aligned performance criteria (F1, DER) and reinforcement learning for sequence-level correction (Yan et al., 2021, Peng et al., 2022).

In aggregate, current trends in MDD research are toward hierarchical, attribute-informed, and multi-task architectures, immunity to low-resource and accent variability, and increased diagnostic granularity—driven by improvements both in neural modeling and in understanding of linguistic error typology.