Rapid Prosody Transcription (RPT) Overview

• Rapid Prosody Transcription (RPT) is a real-time method for annotating word-level prosodic errors in synthesized speech using untrained listeners.
• It employs a click-based transcript interface and Gaussian kernel density estimation to precisely map and categorize prosodic anomalies.
• Aggregated error metrics and visual heatmaps complement traditional MOS ratings, providing actionable insights for improving TTS prosody.

Rapid Prosody Transcription (RPT) is a real-time, error-annotation paradigm developed to provide fine-grained, word-level diagnostics of prosodic errors in synthesized speech. Unlike traditional holistic evaluations such as Mean Opinion Score (MOS) tests, RPT enables untrained listeners to annotate spatial and temporal locations within an utterance where prosody is judged to be contextually inappropriate. This approach generates a probabilistic error distribution mapped to the speech signal, offering a direct measure of boundary realization, prominence failures, and other prosodic anomalies, and complementing the global nature of MOS with precise localization and error typology (Gutierrez et al., 2021).

1. Conceptual Framework and Objectives

RPT originates from psycholinguistic methodologies focusing on prosody perception, specifically adapting protocols from works such as Mo et al. (2008) and Cole & Shattuck-Hufnagel (2016). The paradigm instructs untrained listeners to identify at the word level the points where synthesized prosody deviates from contextual appropriateness. Its primary objectives are threefold:

• To attain detailed spatial and temporal annotation of perceived prosodic errors,
• To represent prosodic divergences probabilistically across a listening cohort,
• To enrich and complement MOS testing by providing localized diagnostic information, particularly regarding error localization, boundary realization, and prominence placement (Gutierrez et al., 2021).

2. RPT Annotation Protocol

The annotation procedure within RPT is implemented in a web interface, specifically using LMEDS (Mahrt 2016), and is structured as follows:

• Familiarization Phase: Participants are exposed to example stimuli with annotated prosodic errors and explanations, ensuring consistent task understanding.
• Transcript-Click Interface: The stimulus transcript appears as horizontally aligned word buttons.
• Real-Time Marking: While listening (up to three replays permitted), listeners click any word judged to have “incorrect” intonation; these clicks are color-coded and timestamped at word onset.
• Error Types Survey: Post-annotation, listeners indicate error categories, such as “Abrupt change in pitch,” “Awkward pause,” “Unexpected intonation,” or “Lacking intonation.”
• Prosodic MOS (PMOS) Rating: A global rating of “How natural is the speaker’s intonation?” is collected using a 5-point Likert scale (Gutierrez et al., 2021).

3. Aggregation and Probabilistic Error Mapping

Aggregating RPT listener annotations entails two stages:

3.1. Temporal Density Estimation

Each listener $i$ provides a set of error word-onset times $\{t_{ik}\}$. These are pooled across $M$ listeners to construct a continuous error-probability function via Gaussian kernel density estimation:

• $x_i(t) = \sum_k \delta(t - t_{ik})$
• $$p(t) = \frac{1}{M}\sum_{i=1}^{M} [x_i * G_{\sigma}](t)$$, where $$G_{\sigma}(t) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{t^2}{2\sigma^2}\right)$$

Here, $*$ denotes time convolution and $\sigma$ is typically set to 50–100 ms to smooth discrete clicks into a continuous contour.

3.2. Per-Word Aggregation

At the word level, aggregate error probability is calculated as:

• $p_{w} = \frac{1}{M}\sum_{i=1}^{M} I_{i}(w)$

where $I_{i}(w) = 1$ if listener $\{t_{ik}\}$0 marked word $\{t_{ik}\}$1, and otherwise $\{t_{ik}\}$2 (Gutierrez et al., 2021).

4. Visualization, Interpretation, and Diagnostic Capability

Visualization tools developed for RPT represent error distributions as follows:

• Heatmaps Over Waveforms: $\{t_{ik}\}$3 is overlaid on the audio waveform or spectrogram, with color intensity reflecting the density of error annotations.
• Word-Aligned Error Bars: The transcript is shaded for each word according to $\{t_{ik}\}$4.
• Boundary Clustering and Punctuation Analysis: Error peaks frequently align with words before punctuation in standard audiobook stimuli (LibriTTS). Analytical metrics, such as the proportion of most-annotated words preceding punctuation, distinguish the boundary realization capabilities across systems (Gutierrez et al., 2021).

For question–answer (QA) style stimuli, where information structure is controlled, error peaks reveal misplaced or missing prominence, enabling diagnostic comparisons between TTS systems regarding context-driven prosodic expectations.

5. Comparative Analysis: RPT vs. MOS Paradigms

RPT extends beyond the global assessment offered by MOS by providing:

• Global and Local Metrics: Each stimulus receives both a global PMOS score and localized, word-level error profiles.
• Error Rate Metric: Error rate is defined as $\{t_{ik}\}$5 (with $\{t_{ik}\}$6 the stimulus word count). PMOS and $\{t_{ik}\}$7 generally correlate, preserving system rankings.
• Statistical Correlations: Across all experiments, the correlation between PMOS and $\{t_{ik}\}$8 is Pearson’s $\{t_{ik}\}$9 ($M$0).
• System Differentiation: RPT-based paired t-tests replicate the MOS-based ordering (FastPitch > Ophelia > Festival in E1), but effect sizes decrease when analyses isolate prosody (E2/E3).
• Inter-Annotator Agreement: Krippendorff’s $M$1 and $M$2 (filtered for all-zero annotations) positively track system quality, with higher scores for more natural TTS outputs, indicating convergent error localization across listeners (Gutierrez et al., 2021).

6. RPT Experimental Findings and System Diagnostics

RPT was evaluated under two experimental conditions:

(a) Standard Audiobook Stimuli

• Stimuli: 30 sentences (≤15 words) from LibriTTS.
• Systems: Festival (unit selection), Ophelia, FastPitch (neural).
• Findings: Error heatmaps cluster around major prosodic boundaries (e.g., preceding punctuation). FastPitch demonstrates over-expressive pitch accents before pauses, resulting in high unexpected-intonation marks and a greater proportion of most-error words before punctuation (Gutierrez et al., 2021).

(b) Question–Answer (QA) Stimuli

• Stimuli: 60 context–question pairs designed for informational or corrective focus.
• Design: Encourages strong prosodic expectations regarding givenness and newness.
• Findings: Festival frequently exhibits “lacking intonation” on focus words; FastPitch generates “unexpected intonation” via misplaced focus accents or emphasis on context-given items; Ophelia, despite a lower MOS, sometimes better matches QA-induced intonational structure, reversing MOS system order under these constraints (Gutierrez et al., 2021).

7. Advantages, Significance, and Interpretative Insights

Key advantages of RPT include:

• Fine-Grained Localization: Precise identification of anomalous words and time points, directly guiding TTS prosody system improvements.
• Contextual Resolution: When combined with prosodically diagnostic stimuli (e.g., QA structures), RPT uncovers system distinctions inaccessible to conventional MOS ratings.
• Scalability: Relies on non-expert listeners without specialized phonetic annotation training (e.g., ToBI).
• Complementarity: PMOS and error-rate measures augment global scores with error typology and boundary/prominence-specific diagnostics.
• Interpretation of Prosodic Variation: Error distribution patterns reveal that excessive expressiveness reduces naturalness when contextually misapplied, highlighting the import of context sensitivity for neural TTS evaluation (Gutierrez et al., 2021).

A plausible implication is that RPT’s fine-grained diagnostic capability will catalyze more targeted research into adaptive prosody modeling and error correction within neural TTS systems.