Pitch Accent Detection Overview

• Pitch accent detection is the automatic classification of prosodic prominence in speech, spanning binary word-level to frame-level and contour-based formulations across multiple languages.
• Techniques include CNN-LSTM architectures with acoustic-prosodic features, integration of lexical embeddings, and self-supervised models like wav2vec 2.0, yielding significant performance gains.
• Evaluation protocols vary widely—from precision-based frame-level metrics to quality assessment in TTS—highlighting both the promise and complexity of applying these methods.

Pitch accent detection is the automatic identification or classification of prosodic prominence patterns from speech or closely related representations. In the literature summarized here, the task ranges from binary word-level labeling of accented versus non-accented words in English, to frame-level detection of accented regions for joint automatic speech recognition, to mora-, phrase-, and contour-level classification in Japanese and Seoul Korean, and to adjacent restoration or assessment tasks in accent-marked text and synthetic speech (Nielsen et al., 2020, Sasu et al., 6 Aug 2025, Joo et al., 21 Apr 2026, Kawamura et al., 18 Jun 2026, P et al., 28 Nov 2025). Across these settings, pitch accent is treated as a supra-segmental cue tied to prominence, accent nucleus position, discourse structure, or discrete tonal categories, and the principal methodological question is how to map variable acoustic or textual evidence onto stable accentual labels.

1. Scope and task formulations

The most established formulation in English is binary pitch accent detection at the word level. In this setting, each word is classified as accented or not accented, often after collapsing richer ToBI annotations into two classes. One paper defines the task as deciding, for each word in an utterance, whether it carries a pitch accent, and describes pitch accent as a word-level prosodic prominence signaled by deviations in pitch, intensity, and/or duration from the speaker’s baseline (Nielsen et al., 2020). Another English study similarly treats the labels as accented versus none, with uncertain accent events also grouped into none (Stehwien et al., 2018).

A second formulation treats pitch accent as a temporal detection problem rather than a token classification problem. In joint ASR-prosody modeling, pitch accents are converted into binary frame-level labels, where frames aligned to segments annotated as accented are labeled positive and all others negative; evaluation then uses precision, recall, and F1 under several tolerance windows around annotated event boundaries (Sasu et al., 6 Aug 2025). This frame-level view is closely related to prosodic event detection.

Outside English, the label space is often richer and the unit of analysis shifts. Japanese work uses accent nucleus position or accent type at the mora or accent-phrase level, while Seoul Korean work models Accentual Phrases as members of discrete tonal categories such as LH, LHLH, HHLH, and HHLL (Luong et al., 2018, Joo et al., 21 Apr 2026). Rigvedic Sanskrit work addresses automatic restoration of accent marks in text rather than spoken detection, but it remains relevant as a sequence-labeling or sequence-to-sequence view of accent placement (P et al., 28 Nov 2025).

Setting	Unit	Label formulation
English binary detection	Word	accented vs none (Stehwien et al., 2018)
English contextual sequence labeling	Word sequence	pitch accent per token (Nielsen et al., 2020)
Joint ASR-prosody detection	Frame/time step	binary accented-region labels (Sasu et al., 6 Aug 2025)
Japanese accent assessment	Mora / accent phrase	nucleus correctness or error severity (Kawamura et al., 18 Jun 2026)
Seoul Korean contour classification	Accentual Phrase	16 tonal categories (Joo et al., 21 Apr 2026)
Rigvedic accent restoration	Unicode sequence	udātta, anudātta, svarita marks (P et al., 28 Nov 2025)

This diversity of task formulations is significant because reported metrics and model architectures are rarely directly comparable across them. A plausible implication is that “pitch accent detection” functions less as a single benchmark than as a family of related inference problems centered on prominence, accentual alignment, and tonal categorization.

2. Acoustic and contextual neural models for English

A central result in English pitch accent detection is that broader context improves performance. One model extends earlier CNN-based work by taking full utterances as input, applying CNN layers to frame-level acoustic-prosodic features, pooling those frames into token representations using word boundaries, and then passing token embeddings through a bidirectional LSTM followed by a feed-forward output layer (Nielsen et al., 2020). The input frames contain six acoustic-prosodic features: smoothed $F_0$, RMS energy, loudness, zero-crossing rate, voicing probability, and harmonics-to-noise ratio. Development-set comparisons show a consistent ordering: full utterance + CNN+LSTM reaches 89.1%, full utterance + CNN only 87.9%, three tokens + CNN+LSTM 88.6%, three tokens + CNN only 87.3%, and one token + CNN only 85.5% (Nielsen et al., 2020). On the test set, the speech-only model reaches 88.4%, the text-only model 82.2%, and the speech+text model 89.1%; the abstract frames the gain as an improvement from 87.5% to 88.7% accuracy (Nielsen et al., 2020).

The same study argues that a simple content-word heuristic is the appropriate baseline for this task. Labeling every content word as accented yields 82.2% accuracy, equal to the text-only model and far above the majority baseline of 54.4%; a duration-only baseline reaches 81.2% (Nielsen et al., 2020). This is important because English pitch accents usually appear on content words, though they can also appear on function words when those words are unusually informative or contrastive. The ablation analysis further reports that pitch is the most important acoustic feature, with removing pitch causing the largest drop in performance, while voicing is the weakest feature group (Nielsen et al., 2020).

A second English line of work augments a strong acoustic CNN with lexical embeddings. The acoustic baseline is a 2-layer CNN over frame-based features extracted with OpenSMILE from the current word plus immediate left and right context, using root mean square energy, loudness, smoothed $F_0$, voicing probability of the $F_0$ candidate, harmonics-to-noise ratio, and zero-crossing rate (Stehwien et al., 2018). A lexical branch then feeds GloVe or word2vec embeddings through a bottleneck layer before concatenation with the acoustic representation. Within corpus, embeddings improve accuracy: BURNC rises from 87.1 to 87.5, BDC from 78.0 to 81.2, and LeaP from 80.5 to 83.5 in representative settings (Stehwien et al., 2018). Cross corpus, however, generalization can degrade, particularly when training on LeaP and testing elsewhere, which the authors interpret as lexical overfitting to corpus-specific vocabulary and genre (Stehwien et al., 2018). The broader lesson is that lexical cues are useful but unstable under distribution shift.

3. Self-supervised representations and multitask learning

Recent work shifts pitch accent detection from hand-crafted acoustic front ends toward self-supervised speech representations. A joint model built on a pretrained wav2vec 2.0 encoder shares the encoder between an ASR head and a pitch accent head, with the additive objective

$$\mathcal{L}_j = \mathcal{L}_{asr} + \mathcal{L}_{pad}.$$

The pitch accent branch classifies whether each time step is accented, using only wav2vec 2.0 hidden states and explicitly dropping external fundamental frequency features used by wav2TOBI; it adds an extra linear layer and layer normalization before the final classifier (Sasu et al., 6 Aug 2025).

The supervised prosody data come from BURNC, described as about 11 hours of broadcast-news-style read speech from 7 speakers, with transcriptions, alignments, POS tags, and prosodic labels. The corpus is split 75% train, 15% dev, 10% test, avoiding news-story overlap between train and test, and audio is preprocessed into overlapping 20-second clips with 10-second hops following wav2TOBI (Sasu et al., 6 Aug 2025). Because LibriSpeech and Libri-light lack pitch accent labels, the study uses semi-supervised self-training with model voting: BURNC training is partitioned into three folds, three models are trained, majority vote pseudo-labels LS1, and the process is iterated until performance stops improving after four iterations (Sasu et al., 6 Aug 2025).

The reported gains on pitch accent detection are large. At 0 ms tolerance, wav2TOBI obtains precision 0.13, recall 0.11, and F1 0.12; the simpler wav2vec-based prosody model reaches precision 0.37, recall 0.36, and F1 0.36; the semi-supervised version reaches precision 0.49, recall 0.48, and F1 0.48 (Sasu et al., 6 Aug 2025). At 40 ms tolerance, F1 rises from 0.65 for wav2TOBI to 0.81 and 0.83 for the supervised and semi-supervised systems; at 80 ms, from 0.79 to 0.87 and 0.90; at 100 ms, from 0.81 to 0.88 and 0.90 (Sasu et al., 6 Aug 2025). The paper interprets this as evidence that wav2vec 2.0 representations retain prosodic information and that explicit prosody supervision can reorganize those representations in a more useful way.

The same multitask setup improves ASR in limited-resource fine-tuning. Without a LLM, LibriSpeech WER drops from 6.0 to 4.3, a 28.3% relative reduction, while BURNC WER drops from 23.0 to 20.0. With a 4-gram LLM, LibriSpeech improves from 5.0 to 3.0 and BURNC from 17.59 to 13.25; with a Transformer LLM, LibriSpeech changes from 5.0 to 4.0 and BURNC from 18.0 to 14.9 (Sasu et al., 6 Aug 2025). CER changes relatively little, while WER improves more substantially, which the paper argues is consistent with prosody helping word-level recognition rather than only local character correction.

4. Japanese pitch accent as annotation, conditioning, and quality assessment

Japanese work emphasizes that pitch accent is a linguistically essential but operationally fragile feature. In a neural TTS pipeline, pitch accent type (Acc_Type) is defined as the location of the accented mora in a Japanese accent phrase, or 0 for a no-accent phrase, and interrogative phrase mark (Question_Flag) is treated as a second crucial prosodic feature (Luong et al., 2018). A controlled corruption study perturbs Acc_Type by a discrete offset in [-2, +2] with 50% probability and flips Question_Flag with 30% probability. When training uses Oracle labels and testing uses corrupted labels, performance drops sharply; objective results show RMSE rising from 23.31 to 31.09 and CORR falling from 0.94 to 0.89, while mixed clean+corrupted training partly recovers robustness (Luong et al., 2018). The authors interpret this as a train-test mismatch problem and conclude that test-set annotation errors are especially harmful, while some training noise can act as regularization.

A distinct TTS line uses learned pitch-accent-conditioned synthesis rather than standalone detection. One English Tacotron-2 variant trains a classifier on phoneme embedding outputs with two Bi-LSTM layers, a fully connected network, and sigmoid activation to predict a “two dimensional binary vector” corresponding to stress syllable and pitch accent; the predicted vector is then injected at pre-encoder, pre-decoder, or intra-decoder points (Elyasi et al., 2021). The best configuration is the combination of pre-encoder and intra-decoder conditioning. In objective evaluation, baseline PTCorr is 0.60 and partial $F_0$ variation is 41.23, while the combo model reaches PTCorr 0.76 and partial $F_0$ variation 46.62; MCD improves from 5.36 to 5.09, and MOS rises from $3.918 \pm 0.223$ to $4.148 \pm 0.192$, compared with ground truth at $4.288 \pm 0.101$ (Elyasi et al., 2021). The paper explicitly notes that this is better described as learned pitch-accent-conditioned synthesis than as standalone pitch accent detection.

Japanese self-supervised text representations have also been studied as implicit accent predictors. A PnG-BERT-based TTS system fine-tunes only the last 0, 2, 4, or 6 Transformer layers and adds tone prediction as an auxiliary downstream task using X-JToBI tone labels aligned at the phoneme level (Yasuda et al., 2022). The pretrained representation already carries phrase-level information better than accent-nucleus information: for example, PGB0 obtains TA 72.8, PA 74.1, but AA only 20.6. Auxiliary tone prediction markedly improves these to TA 81.6, PA 80.5, and AA 56.7 in PGB2T (Yasuda et al., 2022). In listening tests, plain Tacotron2 with phoneme input achieves accent-correctness MOS $1.89 \pm 0.03$, while PGB2 reaches $F_0$0 and PGB2T $F_0$1, even though Tacotron remains better on naturalness (Yasuda et al., 2022). The paper interprets this as evidence that grapheme+phoneme pretraining helps infer pitch accent without tone labels at test time.

PASQA reframes Japanese pitch accent as accent-quality assessment rather than naturalness MOS prediction. It is trained on a controlled synthetic dataset generated with NANSY-TTS by modifying accent nucleus positions in sampled accent phrases, creating error-free, low-severity, and high-severity conditions (Kawamura et al., 18 Jun 2026). The model uses wav2vec 2.0 features, mora-conditioned fusion by cross-attention, a Bradley–Terry ranking loss, an auxiliary frame-level accent-error localization task, and speaker-invariant training with a gradient reversal layer (Kawamura et al., 18 Jun 2026). Its main metric is order accuracy over triplets satisfying error-free > low-severity > high-severity. On seen speakers, PASQA reaches 0.754 order accuracy with LCC 0.829, SRCC 0.711, and KTAU 0.524; on unseen speakers, 0.785, 0.879, 0.751, and 0.559, respectively (Kawamura et al., 18 Jun 2026). Human listeners reach order accuracy 0.925, and PASQA shows stronger agreement with human accent-correctness judgments than ACC-SSL-MOS (Kawamura et al., 18 Jun 2026). This suggests a shift from categorical detection toward severity-sensitive assessment of localized accent nucleus errors.

5. Contour-level classification, probing, and typology

In Seoul Korean, the task is formulated as classification of whole Accentual Phrase contours into discrete tonal categories. Dual-Glob encodes complete $F_0$2 contours, generates clean and augmented views, and applies supervised contrastive learning so that same-class contours cluster while different classes separate in a shared latent space (Joo et al., 21 Apr 2026). The benchmark consists of 10,093 manually annotated Accentual Phrases from 18 professional broadcasters and 16 tonal labels: H, HH, HHL, HHLH, HHLL, HL, HLH, HLL, L, LH, LHH, LHL, LHLH, LHLL, LL, and LLH (Joo et al., 21 Apr 2026). With Logistic Regression on frozen embeddings, Dual-Glob reaches accuracy $F_0$3 and macro-F1 $F_0$4, improving on the strongest baseline BiLSTM at $F_0$5 and $F_0$6 (Joo et al., 21 Apr 2026). A syllable-aware extension that concatenates clipped syllable count raises accuracy to 0.894 with Logistic Regression (Joo et al., 21 Apr 2026). The authors argue that global contour shape is a better inductive bias than local predictive modeling for this task.

Neural audio codecs provide a different perspective: not direct detection, but probing whether pitch-accent information is recoverable from learned representations. A probing study on Mimi codec latents evaluates several tasks on phrase-final nuclear tunes, including hxx-vs-lxx, defined as a distinction only in pitch accent and identified as the key task for pitch accent detection (Vigneaux et al., 14 Mar 2026). Linear probes on unquantized latents or some codebooks reach top average test accuracy of about 0.31 for 8-class tune discrimination and 0.45 for 5-class clustering; binary rising-versus-falling tasks attain 0.74–0.89, but hxx-vs-lxx is weaker than edge-tone discrimination (Vigneaux et al., 14 Mar 2026). The study further reports that unquantized embeddings always perform as well as or better than codewords, Codebook 1 is consistently better than Codebook 0, and tune information is distributed across several codebooks rather than isolated in a “semantic” one (Vigneaux et al., 14 Mar 2026). The conclusion is that codec representations contain pitch-accent information, but less cleanly than edge-tone information and still far below human performance on the 5-class problem.

A typological perspective replaces categorical labels with mutual information between lexical identity and pitch. Using Common Voice data from ten languages, word-aligned pitch contours are resampled to 100 points and compressed into the first four DCT coefficients, and mutual information is estimated as

$F_0$7

The main empirical result is that tonal languages show the highest word–pitch dependency, pitch-accent languages are intermediate, and stress-accent languages the weakest, though the distributions overlap and the overall pattern is gradient rather than categorical (Wilcox et al., 12 May 2025). This does not define a detector in the usual sense, but it provides a quantitative criterion for how strongly lexical identity predicts pitch contour structure.

6. Evaluation regimes, recurrent limitations, and evidentiary boundaries

Pitch accent detection research uses heterogeneous evaluation protocols. English binary classification studies often report accuracy because the task is treated as balanced binary classification; representative values include 88.4% for speech-only and 89.1% for speech+text on BURNC, with a content-word baseline of 82.2% (Nielsen et al., 2020). Frame-level prosodic event detection instead uses precision, recall, and F1 under timing tolerances from 0 ms to 100 ms (Sasu et al., 6 Aug 2025). TTS-oriented work introduces local $F_0$8 metrics such as PTCorr and partial $F_0$9 variation, while PASQA adds order accuracy, LCC, SRCC, and KTAU for severity-aware accent correctness (Elyasi et al., 2021, Kawamura et al., 18 Jun 2026). Adjacent text-based accent restoration tasks use WER, CER, and a task-specific Diacritic Error Rate that isolates accent edits; in Rigvedic Sanskrit restoration, full fine-tuning of ByT5 achieves WER 0.1023, CER 0.0246, and DER 0.0685, outperforming BiLSTM-CRF and LoRA on all three metrics (P et al., 28 Nov 2025).

Several limitations recur across the literature. English detection results are often based on read speech, especially BURNC broadcast news, rather than spontaneous conversation (Nielsen et al., 2020). Self-supervised multitask work is evaluated in a limited-resource fine-tuning regime because prosodic labels are scarce, uses only pitch accent as binary hard labels, and notes that preprocessing mismatches and clipped audio segments can affect BURNC results (Sasu et al., 6 Aug 2025). Lexical augmentation improves within-corpus accuracy but can harm cross-corpus generalization through lexical overfitting (Stehwien et al., 2018). Korean contour classification is constrained by heavy label imbalance and by reliance mainly on $F_0$0, ignoring other prosodic cues such as duration and intensity (Joo et al., 21 Apr 2026). Japanese accent-quality assessment depends on synthetic data and pseudo labels, and assumes access to input text and mora sequence (Kawamura et al., 18 Jun 2026).

A persistent source of confusion is that not every suprasegmental or accent-related paper actually provides evidence on pitch accent detection. The provided record for "Automatic recognition of suprasegmentals in speech" states that it is an ASRU 2019 manuscript template containing no experiments, no speech dataset, no pitch accent annotations, no model architecture, and no results relevant to pitch accent detection (Yuan et al., 2021). This is a useful boundary condition for the field: bibliographic proximity to prosody is not itself evidence of a pitch accent detection method.

Taken together, these works indicate that pitch accent detection now spans at least four technical regimes: binary event detection in speech, multitask representation learning with ASR, contour-level category classification, and accent-focused quality assessment or restoration. This suggests that future comparisons will remain difficult unless task units, label inventories, and evaluation criteria are aligned more systematically across languages and application settings.