Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representations

Abstract: Speech representations from self-supervised speech models (S3Ms) are known to be sensitive to phonemic contrasts, but their sensitivity to prosodic contrasts has not been directly measured. The ABX discrimination task has been used to measure phonemic contrast in S3M representations via minimal pairs. We introduce prosodic ABX, an extension of this framework to evaluate prosodic contrast with only a handful of examples and no explicit labels. Also, we build and release a dataset of English and Japanese minimal pairs and use it along with a Mandarin dataset to evaluate contrast in English stress, Japanese pitch accent, and Mandarin tone. Finally, we show that model and layer rankings are often preserved across several experimental conditions, making it practical for low-resource settings.

• The paper presents the prosodic ABX framework to evaluate self-supervised speech models' ability to capture prosodic contrasts without needing labeled data.
• It leverages dynamic time warping on minimal pairs from English, Japanese, and Mandarin, showing measurable correlations with human perceptual judgments.
• Results indicate that while S3Ms excel in capturing English lexical stress, they underperform in recognizing Japanese pitch accent and Mandarin tone contrasts.

Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representations

Introduction

The paper "Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representations" presents an innovative approach to evaluating prosodic contrasts within self-supervised speech models (S3Ms). The sensitivity of S3Ms to phonemic contrasts is well-documented. However, their ability to capture prosodic elements—such as stress, intonation, and rhythm—had not been quantitatively assessed until now. This research extends the ABX discrimination framework, traditionally used for phonemic contrast, to measure prosodic contrast without reliance on explicit labels or extensive examples.

Speech prosody plays a crucial role in functions ranging from word distinction to syntactic ambiguity resolution and focus determination, thus making it imperative to understand its representation in S3M models. The authors introduce "prosodic ABX," leveraging minimal pairs from English, Japanese, and Mandarin datasets to scrutinize various prosodic systems, including English lexical stress, Japanese pitch accent, and Mandarin tone.

Figure 1: Conceptual overview. A, B, and X have the same phonemic sequence, but B has a different prosodic pattern. This is illustrated by the colors of the representations, R_A, R_B, and R_X. We align R_A and R_B with R_X using dynamic warping and check whether the alignment cost (or distance) d to R_X is smaller for R_A than for R_B.

Prosodic ABX Framework

Prosodic ABX employs a triplet formation akin to the phonemic ABX task, where samples A, B, and X share phonemic sequences but vary in prosodic patterns to form minimal pairs. The test uses dynamic time warping (DTW) to compare representation sequences, preserving and aligning temporal features that are crucial for capturing prosody. By assessing the alignment cost, the framework measures whether phonemic representations effectively distinguish prosodic contrasts, thereby enabling its application even in data-constrained environments.

The study also illustrates how prosodic ABX bypasses the need for labeled datasets and model training, which are often cost-prohibitive for languages with complex prosodic patterns. This method complements categorical probing by facilitating naturalistic evaluations and drawing direct comparisons with human perceptual judgments.

Dataset and Experimentation

The authors curated datasets reflecting prosodic systems in English, Japanese, and Mandarin, including both natural recordings and synthesized speech corpora. These selections embraced minimal pairs reflecting distinctive prosodic contrasts to enable cross-linguistic evaluations. Notably, the study incorporated synthesized speech from platforms like Google Cloud Text-to-Speech to provide cost-efficient proxies for natural speech, revealing correlations in ABX error rates and model ranking across synthesized and natural contexts.

Figure 2: ABX error rates (downarrow) across different prosodic tasks. We compare the best layer of each S3M (orange box plots, n=17), acoustic baselines (violet dots), and humans (pink 95\% confidence intervals).

The experiments evaluated multiple S3Ms by examining the performance of models such as wav2vec 2.0 and HuBERT. Human ABX tests provided benchmarks, revealing that self-supervised models often match or exceed human performance in recognizing English stress while trailing behind in Japanese pitch accent and Mandarin tone distinctions.

Results and Implications

The study results highlight the variations in S3Ms' sensitivity across different prosodic systems. English lexical stress, with its complex semantic ties, often resulted in models outperforming humans. By contrast, the intricate prosodic features of Japanese pitch accent and Mandarin tone were better captured by human perceptual mechanisms.

Correlations between ABX scores in different tasks suggest that representations sensitive to one prosodic type tend to perform well across other types, possibly due to shared acoustic correlates. Furthermore, the improved performance in contextually rich settings implies that S3Ms benefit from broader linguistic context, a finding aligned with human prosodic cognition.

Figure 3: Word-level ABX error rates for English lexical stress: human participants vs S3Ms.

Conclusion

This study advances the prosodic analysis in self-supervised models through the prosodic ABX framework, providing a language-agnostic, label-free method to appraise prosodic contrasts comprehensively. The findings illustrate the inherent potential of S3Ms in tasks reliant on detailed prosodic evaluations, suggesting their utility in clustering, tokenization, and pronunciation feedback applications. The robustness of prosodic ABX across varied conditions supports its adoption as a reliable mechanism for future prosody-sensitive inquiries in speech representation studies.

Figure 4: Error-rate correlation across prosodic tasks. For all S3Ms, each layer is plotted according to its error rates.