Automatic Pronunciation Assessment

• Automatic Pronunciation Assessment (APA) is a method that evaluates second-language speakers’ pronunciation at phoneme, word, and utterance levels using human-like, multi-aspect scoring.
• APA leverages multi-view features such as GOP, SSL-derived cues, and handcrafted prosodic measures to capture both segmental and suprasegmental speech characteristics.
• Modern approaches integrate hierarchical Transformer models and multi-task regression techniques to enhance feedback precision and effectively address score imbalances.

Automatic Pronunciation Assessment (APA) evaluates the pronunciation proficiency of a second-language (L2) learner in a target language and is a major component of computer-assisted pronunciation training (CAPT) and computer-assisted language learning (CALL). In current research, APA is typically formulated as a scoring problem in which speech, often together with a text prompt or canonical phoneme sequence, is mapped to pronunciation scores across multiple linguistic granularities and aspects. Modern work consistently distinguishes APA from mispronunciation detection and diagnosis (MDD): MDD localizes and diagnoses phonetic errors, whereas APA predicts human-like proficiency scores for dimensions such as accuracy, stress, fluency, completeness, prosody, and total score at phoneme, word, and utterance levels (Yan et al., 2024, Chao et al., 2022, Yan et al., 2023).

1. Task formulation and assessment scope

A common formalization writes APA as a mapping

$\mathbf{s} = f(X, T; \theta),$

where $X$ is the learner’s speech, $T$ is the reference text prompt, and $\mathbf{s}$ is a proficiency-score vector containing aggregate and multi-aspect scores at the phone, word, and utterance levels (Li et al., 21 Sep 2025). In multi-granularity settings, the target space is explicitly hierarchical: phoneme-level assessment targets local segmental quality, word-level assessment targets lexical realization and stress, and utterance-level assessment targets broader speaking qualities such as fluency and prosody (Do et al., 2022, Chao et al., 11 Feb 2025).

The target layout used by much of the recent literature is highly standardized on speechocean762-style supervision (Chao et al., 2022, Gong et al., 2022).

Granularity	Typical aspects	Score handling
Phoneme level	accuracy	$[0,2]$
Word level	accuracy, stress, total	$[0,10]$, often normalized to $[0,2]$
Utterance level	accuracy, completeness, fluency, prosody, total	$[0,10]$, often normalized to $[0,2]$

This formulation is broader than mispronunciation detection. Several papers explicitly note that APA is not limited to deciding whether a phoneme is correct or incorrect; it is intended to provide fine-grained, multi-aspect feedback aligned with human judgment, including segmental and suprasegmental properties (Chao et al., 2022, Kim et al., 2022). A recurring implication is that effective APA systems must model both local articulatory evidence and longer-range temporal or prosodic structure.

2. Canonical supervised formulation and benchmark practice

The public benchmark most frequently used in this literature is speechocean762, described as an open-source non-native English pronunciation assessment corpus with 5,000 utterances or recordings from 250 speakers or learners, commonly split into 2,500 training and 2,500 test utterances (Chao et al., 2022, Yan et al., 2023, Chao et al., 11 Feb 2025). Most studies evaluate with Pearson correlation coefficient (PCC), while phoneme-level accuracy is often also reported with mean squared error (MSE) (Chao et al., 2022, Do et al., 2022, Chao et al., 11 Feb 2025).

A foundational supervised backbone is GOPT, a Goodness Of Pronunciation feature-based Transformer that jointly predicts phoneme-, word-, and utterance-level scores with multi-task learning (Gong et al., 2022). Its input pipeline derives 84-dimensional GOP features from ASR posteriors. Using the standard notation,

$$LPP(p) \approx \frac{1}{t_e-t_s+1} \sum_{t=t_s}^{t_e}\log p(p|o_t),$$

and

$X$0

so that with 42 pure phones the GOP vector concatenates 42 $X$1 values and 42 $X$2 values (Gong et al., 2022). In the corresponding Transformer formulation, projected GOP features are combined with canonical phoneme embeddings and positional embeddings, and trainable $X$3 tokens are prepended for utterance-level attributes (Gong et al., 2022).

This baseline established several conventions that later work retains even when the architecture changes: forced alignment against canonical text, phone-level acoustic evidence derived from GOP or related posteriors, multi-task regression with MSE, and evaluation by PCC on word- and utterance-level scores (Gong et al., 2022). It also made clear that a compact model can be effective on this benchmark; the reported best setting in GOPT used a 3-layer Transformer with 24-dimensional projected features (Gong et al., 2022).

3. Representation learning beyond segmental GOP

A major development in APA has been the move from purely segmental inputs to richer multi-view feature sets. The 3M framework explicitly identifies a granularity mismatch in segmental-only systems: GOP is useful for phone scoring, but it is not ideal for suprasegmental properties such as fluency, stress, and intonation (Chao et al., 2022). To address this, 3M augments GOP with duration, root-mean-square energy (RMSE) statistics, self-supervised learning (SSL) features from wav2vec 2.0, HuBERT, and WavLM, and a vowel/consonant positional embedding intended to be more phonology-aware (Chao et al., 2022). Its multi-view fusion is summarized as

$X$4

followed by a dense projection into a shared embedding space (Chao et al., 2022).

Parallel work on hierarchical context-aware APA extends this feature-centric line by concatenating

$X$5

projecting the result, and then introducing sup-phonemes, depth-wise separable convolution, and score-restraint attention pooling (Chao et al., 2023). In that framework, sup-phonemes are intermediate phone-group units learned with a Byte-Pair Encoding-style merging process over phone sequences, intended to capture sub-word pronunciation regularities beyond isolated canonical phones (Chao et al., 2023).

Another line of work treats the encoder itself as the object of pronunciation pretraining. Multi-task pretraining (MTP) masks 15% of phonetic features, with 90% replaced by a mask token and 10% left unchanged, and trains the encoder to reconstruct masked phonetic and prosodic features from surrounding context (Li et al., 21 Sep 2025). The subtasks include phoneme prediction, articulation trait prediction, vowel/consonant prediction, phoneme duration prediction, pitch prediction, and energy prediction (Li et al., 21 Sep 2025). The same framework also imports handcrafted features (HCFs) from automated speaking assessment (ASA), including confidence, F0, RMS energy, rhythm measures, silence, long silence, duration, and error-rate features, to support more interpretable utterance-level scoring (Li et al., 21 Sep 2025).

This progression suggests a consistent research direction: APA systems increasingly treat pronunciation as a fusion problem over segmental, prosodic, phonological, SSL-derived, and sometimes handcrafted descriptors, rather than as a narrow function of phoneme posterior confidence alone.

4. Architectural paradigms: flat Transformers, hierarchies, and joint CAPT models

Although GOPT uses a flat Transformer with task-specific heads, a large fraction of subsequent work argues that pronunciation is inherently hierarchical. HiPAMA makes this point explicit by modeling phoneme, word, and utterance levels sequentially and by adding multi-aspect attention so that each aspect can attend to other aspects at the same level (Do et al., 2022). Its central claim is that parallel prediction neglects both the phoneme $X$6 word $X$7 utterance hierarchy and the relations among aspects such as word accuracy and stress or utterance accuracy and prosody (Do et al., 2022).

Later models deepen this hierarchical view. The context-aware approach with sup-phonemes inserts an intermediate sub-word level and uses depth-wise separable convolution to model local context within words, while score-restraint attention pooling makes utterance scoring depend on lower-level scores (Chao et al., 2023). HIA, a residual hierarchical interactive framework, goes further by arguing that prior methods mostly model unidirectional adjacent dependencies. Its Interactive Attention Module performs bidirectional interaction across phoneme, word, and utterance queries, and its residual hierarchical design reintroduces the original encoder output at each level to alleviate feature forgetting (Han et al., 5 Jan 2026).

A representative cross-section of these architectural trends is shown below.

Model family	Defining property	Citation
GOPT	GOP-based Transformer with multi-task learning	(Gong et al., 2022)
HiPAMA	Hierarchical modeling with multi-aspect attention	(Do et al., 2022)
3M	Multi-view, multi-granularity, multi-aspect fusion	(Chao et al., 2022)
Hierarchical context-aware APA	Sup-phonemes, DS-Conv, score-restraint pooling	(Chao et al., 2023)
HIA	Bidirectional cross-granularity interaction with residual hierarchy	(Han et al., 5 Jan 2026)
HMamba / JCAPT	Joint APA–MDD modeling with Mamba-based sequence encoders	(Chao et al., 11 Feb 2025, Yang et al., 24 Jun 2025)
HiPPO	Free-speaking APA via ASR and G2P proxies	(Yan et al., 4 Dec 2025)

Joint CAPT models treat APA and MDD as complementary rather than separate tasks. HMamba uses a hierarchical selective state space model with bidirectional Mamba blocks and predicts phone-, word-, and utterance-level APA scores in parallel with MDD outputs (Chao et al., 11 Feb 2025). JCAPT also uses a bi-directional Mamba encoder, but emphasizes phonological attribute vectors and appended think tokens for fine-grained temporal reasoning; utterance-level APA is produced with aspect-specific attention pooling (Yang et al., 24 Jun 2025). In both cases, the shared claim is that phoneme-level diagnosis and multi-level scoring should improve each other.

HiPPO extends the architectural discussion to unscripted or free-speaking settings. It uses Whisper to transcribe the learner’s speech, G2P to derive a perceived phone sequence, CTC-based GOP features at phone level, SSL features at utterance level, and Conv-LLaMA blocks to support phone-, word-, and utterance-level scoring without reference text at inference time (Yan et al., 4 Dec 2025). This is a substantive shift away from the read-aloud assumption that dominates earlier APA work.

5. Objective functions, ordinal geometry, and imbalance-aware training

The standard APA objective is MSE over one or more regression heads, and several papers explicitly defend this choice on the grounds that pronunciation targets are continuous or ordinal scores assigned by human experts (Yan et al., 2023, Do et al., 2022). At the same time, recent work increasingly argues that plain MSE is insufficient because it captures score order but not the structure of phoneme categories or the imbalance of score distributions.

The PCO loss addresses the first issue by augmenting MSE with a phoneme-distinct regularizer (Yan et al., 2023). In that formulation,

$X$8

where $X$9 encourages feature centers of different phoneme categories to be far apart, and $T$0 imposes score-weighted within-category tightness so that ordinal score relationships are preserved (Yan et al., 2023). The underlying claim is that regression-trained representations otherwise tend to collapse by score, causing different phoneme categories with similar proficiency values to cluster together.

ConPCO extends this line at the abstract level by first aligning phoneme representations and textual embeddings of phonetic transcriptions via contrastive learning, then retaining phoneme characteristics by regulating distances between inter- and intra-phoneme categories while allowing ordinal relationships among the output targets (Yan et al., 2024). The same abstract states that the method is tailored for regression-based APA and is evaluated with a hierarchical APA model (Yan et al., 2024).

A second optimization problem is score imbalance. Score-balanced loss reweights MSE so that predictions in minority score regions receive higher cost, with two variants: $T$1, based on the effective number of samples, and $T$2, based on class rank (Do et al., 2023). A notable design choice is that the weighting is computed from the predicted score class rather than only the ground-truth class, making the reweighting dynamic during training (Do et al., 2023). This work argues that heavily imbalanced aspects such as word Stress and utterance Completeness are otherwise biased toward majority score regions (Do et al., 2023).

Acoustic Feature Mixup addresses both scarcity and imbalance in feature space rather than label space (Do et al., 2024). It proposes two strategies, static linear interpolation and dynamic non-linear interpolation with the in-batch averaged feature, primarily over GOP-based inputs, and optionally concatenates ASR-derived error-rate features such as CER and MER before final prediction (Do et al., 2024). The paper’s analysis ties the method to improved coverage of rare or unseen distortions, especially for difficult aspects like Completeness and Stress (Do et al., 2024).

Taken together, these objectives reframe APA training as more than scalar regression. The common theme is that the embedding geometry should respect phoneme identity, ordinal score structure, and long-tail score distributions simultaneously.

6. Emerging directions, limitations, and open research questions

Several recent directions relax the assumptions of fully supervised, transcript-aligned APA. One is SSL-centric utterance-level scoring. An early example fine-tunes wav2vec 2.0 and HuBERT with CTC on learner speech, averages layer-wise contextual representations across Transformer layers, and feeds them, together with the corresponding text, to a BLSTM-based scoring module (Kim et al., 2022). Another is zero-shot APA with HuBERT masked token recovery: the score is derived from the average number of mis-recovered tokens under repeated masking, producing the aMRT criterion without any APA training labels (Liu et al., 2023).

A more radical low-resource direction is native-only training. The discrete token surprisal framework trains an SSL encoder plus K-means tokenizer and a token LLM on native speech only, computes surprisal over learner token sequences, and optionally adds transcript-guided Text2DUnit--DTW alignment features (Sara et al., 18 Jun 2026). This work argues that pronunciation errors can be detected as deviations from native phonotactic regularities, without forced alignment, phoneme inventories, or large labeled non-native corpora (Sara et al., 18 Jun 2026).

Another emerging idea is to generate a personalized reference rather than comparing directly to a generic native template. A zero-shot text-to-speech framework synthesizes learner-specific golden speech with YourTTS and uses dynamic time warping (DTW) cost between learner speech and golden speech as an APA signal; the same work also fuses golden-speech representations into 3M and reports small but consistent gains (Lo et al., 2024).

Large multimodal models (LMMs) and multimodal LLMs (MLLMs) form a newer frontier. One line adapts Phi-4-multimodal-instruct with LoRA and task-control tokens so that a single model can perform both APA and MDD; the paper reports strong PCC values for several APA dimensions and argues that LoRA-only fine-tuning is already sufficient for joint pronunciation evaluation (Ahn et al., 3 Sep 2025). Another line fine-tunes Qwen2-Audio-7B-Instruct with LoRA, and optionally SimPO-based preference optimization, for phoneme-, word-, and sentence-level scoring (Wang et al., 19 Sep 2025). That study reports that fine-tuning substantially outperforms zero-shot prompting, that phoneme-level assessment remains challenging for the LMM, and that PCC can be high while Spearman’s rank correlation coefficient (SCC) stays much lower, which it interprets as evidence that APA is not only a regression problem but also an ordinal ranking problem (Wang et al., 19 Sep 2025).

Across these directions, several limitations recur. Many papers emphasize that the field is still dominated by read-aloud benchmarks and that free-speaking assessment remains underexplored (Yan et al., 4 Dec 2025). Multiple studies identify score imbalance as a source of weak performance for Stress and Completeness (Chao et al., 2022, Do et al., 2023). Free-speaking and transcript-guided systems remain dependent on ASR quality, and several joint CAPT papers note limited accent diversity because evaluation is often confined to Mandarin L2 English speech in speechocean762 (Yan et al., 4 Dec 2025, Chao et al., 11 Feb 2025). A plausible implication is that future APA systems will need to combine hierarchical multi-granularity scoring, stronger prosody-aware pretraining, rank-aware objectives or evaluation, and broader cross-accent validation if they are to generalize beyond the current benchmark regime.