A study on weakly-supervised training approaches for phoneme-level pronunciation scoring

Abstract: Phoneme-level computer-assisted pronunciation training systems typically rely on phoneme-level annotations, which are costly and scarce. In this work, we investigate whether phoneme-level mispronunciation information can be learned without phoneme-level supervision by exploiting higher-level pronunciation labels. Specifically, we study a weakly supervised setting in which models are trained using only utterance- or word-level pronunciation labels and analyze whether this supervision induces useful phoneme-level score predictions. We further consider a two-stage training scenario in which a model trained only with utterance-level labels is finetuned using a limited number of carefully-selected phoneme-level labeled utterances. We find that, using our proposed architecture and selection process, the two-stage process leads to comparable results to those obtained with full phoneme-level supervision, requiring only a small fraction of phoneme-level labels.

• The paper demonstrates that weakly-supervised transformer models can achieve phoneme-level scoring performance nearly on par with fully supervised methods, significantly reducing annotation costs.
• It introduces novel pooling strategies, including attention pooling, to effectively propagate higher-level supervision down to phoneme-level predictions.
• The study shows that a two-stage training process with limited fine-grained labels dramatically enhances performance while minimizing the need for extensive annotation.

Weakly-Supervised Approaches for Phoneme-Level Pronunciation Scoring

Introduction

The paper "A study on weakly-supervised training approaches for phoneme-level pronunciation scoring" (2605.23593) addresses the challenge of developing effective phoneme-level mispronunciation detection models with limited phoneme-level supervision. Manual fine-grained phoneme-level annotation is expensive and scarce, motivating methods that leverage higher-level (word or utterance) supervision for training. The study systematically examines the potential of weakly supervised strategies using adaptations of transformer-based models, analyzes their learning dynamics with different granularity of supervision, and highlights the potential of two-stage finetuning for efficient use of limited annotation resources.

Model Architectures and Supervision Strategies

The study evaluates baseline unsupervised methods, notably the Goodness of Pronunciation (GOP) algorithm and support vector regressor (SVR) models trained on GOP-derived features. For neural methods, it utilizes the GOPT transformer backbone, designed for multi-level pronunciation scoring via phoneme-aligned embeddings and multi-headed regressors. A key methodological contribution is the introduction of novel architectural pooling schemes that allow phoneme-level prediction heads to be trained only with higher-level word or utterance supervision. The principal pooling designs are:

• BASE: The original GOPT uses a [CLS] token for utterance-level prediction and trains phoneme/word heads only with corresponding level labels.
• MEAN: Higher-level scores are computed by mean pooling across phoneme-level outputs, enabling supervision signals at word/utterance level to shape phoneme-level predictor parameters.
• ATTN: Higher-level scores are attention-weighted combinations of phoneme-level outputs, with attention weights derived from the transformer's hidden states.

With these pooling strategies, the model can propagate utterance- or word-level supervision down to phoneme-level prediction heads, enabling weakly- or unsupervised phoneme scoring.

Figure 1: Proposed GOPT-based architecture: phoneme, word, and utterance-level scores are produced by aggregating phoneme-level outputs using mean or attention pooling.

Experimental Design

Experiments employ the Speechocean762 corpus, containing multi-granularity pronunciation scores for non-native English speech by Mandarin L1 speakers, annotated at phoneme, word, and utterance levels. Supervision regimes include:

• Fully Supervised (UWP/P): All training utterances have phoneme-level labels.
• Weakly Supervised (W/UW/U): Labels are available only at word or utterance level.
• Two-Stage: Initial training on a large utterance-labeled set, followed by finetuning with a limited sample labeled at the word or phoneme level.

Models are evaluated via Pearson Correlation Coefficient (PCC) and Mean Squared Error (MSE) at each granularity, with results aggregated across seeds and bootstrap confidence intervals to ensure statistical validity.

Key Results

Phoneme-Level Performance with Weak Supervision

All transformer-based models—trained even with only utterance-level supervision—substantially outperform unsupervised GOP, which attains phoneme-level PCC of 0.34. With only utterance supervision, the attention-pooled (ATTN) model achieves phoneme-level PCC of 0.46, increasing to 0.56 with word-level supervision, and up to 0.61 with full phoneme-level supervision. Notably, attention pooling consistently yields better phoneme- and word-level PCC than mean pooling or the original BASE architecture under equivalent supervision. Attention pooling brings word-level PCC gains across all regimes without diminishing utterance-level performance.

The performance gap between weakly- and fully-supervised approaches is modest, with weakly supervised models (ATTN, word-level supervision) achieving within 10% PCC of full supervision. However, MSE shows less consistent alignment with PCC, indicating calibration discrepancies in the score ranges across models. Post-hoc calibration is suggested for further alignment.

Efficient Two-Stage Annotation Schemes

The two-stage procedure is demonstrated to significantly enhance phoneme-level scoring with minimal additional annotation. Starting from a model trained with 2500 utterance-level labels, finetuning with as few as 100 phoneme-labeled utterances brings phoneme-level PCC close to that of models trained from scratch with substantially more fine-grained data.

Figure 2: Learning curves show strong phoneme-level PCC gains as a function of fine-grained annotation budget in two-stage (finetuning or training from scratch) setups.

Balanced selection of samples for fine-grained annotation over the utterance score range (as opposed to simple random selection) is consistently more effective, but there is no statistically significant difference between random and lowest error-based sample selection.

Test Set Generalization and Comparative Analysis

On the held-out test set, several weakly supervised transformer models approach the performance of the full supervision topline. SVR models trained on GOP features also perform surprisingly well and are competitive with transformer models trained with full phoneme-level supervision (narrower confidence intervals for SVR are noted).

Crucially, combining large-scale utterance-level supervised pretraining and limited finetuning with word or phoneme-labeled samples suffices to achieve phoneme-level PCC within 5–10% of full supervision, reducing the need for fine-grained annotation by a factor of up to 5 for comparable outcomes.

Figure 3: Phoneme-level PCC by supervision type, showing unsupervised (red), weakly supervised (orange), and fully supervised (green) systems on the test set.

Implications and Future Directions

The study demonstrates the feasibility of constructing competitive phoneme-level pronunciation scoring models with weak or mixed supervision, using transformer architectures that propagate supervision through pooling across granularities. These findings have immediate practical significance for CAPT system development, enabling the deployment of high-quality mispronunciation detection without incurring high annotation costs. The results suggest that attention pooling architectures should be preferred in low-resource annotation settings.

The observation that SVR models perform on par with deep transformers under data-constrained scenarios indicates potential for simpler, more interpretable models in practical settings and warrants further investigation. Additionally, the decoupling of scoring calibration from ranking (PCC vs MSE) raises important questions for downstream CAPT usability.

In terms of future research, the potential of active or curriculum-based annotation selection, domain adaptation to varied L1 backgrounds, and the extension to prosody or suprasegmental scoring with weak supervision remain important open problems. Further exploration of post-hoc calibration and joint modeling across languages and tasks could maximize the annotation efficiency gains shown here.

Conclusion

This study rigorously establishes that phoneme-level mispronunciation scoring models can be trained effectively with only higher-level supervision, using appropriately-designed transformer pooling architectures. The combination of weak supervision and targeted fine-grained annotation delivers near-parity with full phoneme-level supervision, dramatically lowering annotation costs with minimal compromise in scoring accuracy. These findings have direct importance for scalable and economically viable CAPT system development and motivate further research into weakly- and minimally-supervised architectures for fine-grained speech assessment.