Prominence-Aware ASR

Updated 10 July 2026

Prominence-aware ASR is a technique that incorporates prosodic cues, such as word-level emphasis and pitch accents, to enrich transcription output.
It employs diverse annotation methods—categorical labels, scalar values, and binary detection—to capture multifaceted prosodic salience.
Empirical studies reveal that integrating prominence cues can maintain recognition accuracy while providing additional prosodic information for downstream tasks.

Prominence-aware automatic speech recognition (ASR) denotes ASR systems that model word- or frame-level prosodic salience jointly with transcription, or that augment transcripts with prominence estimates derived from the speech signal. In the recent literature, “prominence” is treated as a perceptual or prosodic property linked to emphasis and information structure, but its operationalization varies: as categorical word-level labels in conversational Austrian German, as scalar word-level values in English speech prominence estimation, or as binary frame-level pitch accent events in English broadcast and read speech (Linke et al., 12 Sep 2025). The common premise is that explicit prosodic supervision may either enrich ASR output with salient-word information or bias pretrained acoustic representations toward supra-segmental structure. Empirical results to date are mixed: one study reports unchanged ASR word recognition with reliable prominence labeling conditional on correct word segmentation, whereas another reports substantial word error rate reductions under limited-resource fine-tuning through joint pitch accent detection (Linke et al., 12 Sep 2025).

1. Conceptual scope and relation to prosody

Prominence-aware ASR is motivated by the observation that prominence is a multi-dimensional suprasegmental phenomenon linked to emphasis and information structure, and that downstream systems may benefit from knowing which words are salient. In conversational Austrian German, prominence was modeled at the word level under the Kiel Intonation Model (KIM), with annotators assigning one of four levels—0 (no prominence, PL0), 1 (weak prominence, PL1), 2 (strong prominence), and 3 (emphatic prominence)—with levels 2 and 3 collapsed to PL2 for modeling. This yielded either a binary formulation, PL0 versus PL2, or a three-level formulation, PL0 versus PL1 versus PL2 (Linke et al., 12 Sep 2025).

A complementary English-language formulation defines prominence as “the degree to which an average native listener perceives the word as salient or emphasized relative to its context,” and distinguishes it from binary emphasis labeling by treating it as a scalar in $[0,1]$ . In that formulation, the emphasis label $e_i$ for word $i$ is a Bernoulli random variable with parameter $m_i$ , so that $p(e_i = 1) = m_i$ (Morrison et al., 2023). This probabilistic view is compatible with prominence estimation pipelines that average multiple binary annotations into a scalar target.

A narrower but operationally important specialization is pitch accent detection. One recent study treats pitch accent as a binary frame-level label derived from ToBI-style annotations and uses it as the auxiliary prosodic target in a joint ASR model. This suggests that prominence-aware ASR is not tied to a single annotation ontology; rather, it encompasses several related strategies for injecting prosodic salience into ASR, ranging from explicit word-level prominence tags to frame-level accent supervision (Sasu et al., 6 Aug 2025).

2. Annotation regimes and target representations

Existing prominence-aware ASR research differs substantially in how prosodic targets are obtained and represented. The following summary captures the principal regimes described in the recent literature.

Study	Prosodic target	Representation
(Linke et al., 12 Sep 2025)	Word-level prominence	PL0/PL2 or PL0/PL1/PL2 with “\
(Morrison et al., 2023)	Word-level prominence	Scalar $p_i \in [0,1]$ from averaged binary annotations
(Sasu et al., 6 Aug 2025)	Pitch accent	Binary frame-level labels

In conversational Austrian German, the prominence-annotated subset of the Graz corpus of Read and Spontaneous Speech (GRASS), specifically its conversational speech component (GRCS), contains 4.4 hours of speech, 4,944 utterances with 15,664 word tokens from 34 speakers. Inter-annotator agreements were reported as 0.72 for PL0/PL1, 0.92 for PL0/PL2, and 0.57 for PL1/PL2, indicating that PL1 is noisier and harder to detect reliably than PL0 and PL2. The same study used a small human-annotated subset to train detectors and then automatically labeled the larger conversational corpus, with alignment to forced-aligned word boundaries succeeding for 52.06% $\pm$ 8.57% of utterances per speaker for the three-level detector and 42.3% $\pm$ 8.4% for the binary detector in the large-scale automatic labeling phase (Linke et al., 12 Sep 2025).

In the crowdsourced English prominence-estimation framework, one eighth of LibriTTS train-clean-100 was annotated, yielding 3,626 utterances totaling 6.42 hours and 69,809 words from 18 adult speakers. Emphasis was binary per word at annotation time, but scalar prominence was obtained by averaging multiple annotations:

$p_i = \frac{1}{N} \sum_{j=1}^{N} r_{ij}.$

Quality control included listening-environment screening, minimum listening requirements, recruitment filters, and a bot filter that excluded annotators who marked more than $2/3$ of words as emphasized in eight or more utterances within a batch. Average Cohen’s $e_i$ 0 across overlapping annotator pairs was 0.226, and a one-parameter logistic model predicted held-out annotations from overlapping annotator scores with 77.7% accuracy (Morrison et al., 2023).

In the pitch-accent setting, the prosody corpus is the Boston University Radio News Corpus (BURNC), which includes audio, transcriptions, phonetic alignments, part-of-speech tags, and ToBI-style prosodic labels. Accent labels are mapped to frame indices, and overlapping 20 s clips with 10 s stride are used for training. No explicit forced alignment beyond the provided alignments is performed, and no handcrafted prosodic features are added in the proposed model (Sasu et al., 6 Aug 2025).

3. Architectural patterns for prominence-aware ASR

Three architectural patterns recur in the current literature: standalone prominence estimation followed by downstream use, joint single-objective transcription with embedded prominence symbols, and multi-task learning with separate ASR and prosody heads.

The conversational Austrian German system adopts a single-task CTC formulation built on wav2vec2 XLSR. A prominence detector is first trained by fine-tuning wav2vec2 XLSR to map raw audio directly to a sequence of prominence labels and word-boundary markers “|”, without phone or syllable segmentation and without handcrafted prosodic features. The detector output alphabet is $e_i$ 1 interleaved with “|”. Two detectors are used: PDET02 for binary PL0 versus PL2, trained on 1,770 utterances with an average of 2.09 $e_i$ 2 1.39 tokens per utterance, and PDET012 for PL0 versus PL1 versus PL2, trained on 4,944 utterances with an average of 3.17 $e_i$ 3 2.13 tokens per utterance. Detection uses standard CTC loss, $e_i$ 4, where $e_i$ 5 denotes frame-level label distributions and $e_i$ 6 the target sequence (Linke et al., 12 Sep 2025).

The same study then integrates prominence into ASR by augmenting the output vocabulary so that the CTC model predicts characters decorated with prominence digits. Word boundaries are explicitly represented by “|”, and no additional prominence loss term or separate head is introduced:

$e_i$ 7

By contrast, the English pitch-accent study uses a multi-task architecture on a pretrained wav2vec 2.0 base-960h backbone. The shared encoder is followed by a CTC ASR head and a lightweight accent classifier attached at frame level. The accent head applies a linear layer and layer normalization, then a final classification layer to produce $e_i$ 8. The joint loss is

$e_i$ 9

with $i$ 0 in the reported experiments. The pretrained model is frozen for the first 15k steps, then the Transformer is unfrozen for the remaining 15k steps, while the feature encoder remains frozen throughout (Sasu et al., 6 Aug 2025).

A third pattern, described as a practical integration route rather than an ASR experiment proper, uses a separate word-level prominence estimator trained on 80-channel log-Mel spectrograms and forced-aligned word spans. That estimator comprises a six-layer 1D convolutional framewise encoder and a wordwise decoder, with the best-performing configuration using intermediate downsampling and sum over frames per channel. The prominence loss is

$i$ 1

where $i$ 2 is BCE, though MSE with output bounding to $i$ 3 is reported as comparable. The same work explicitly formulates a joint ASR–prominence objective,

$i$ 4

as a plausible integration mechanism (Morrison et al., 2023).

4. Corpora, decoding conditions, and evaluation methodology

Prominence-aware ASR research is currently shaped by corpus characteristics and by the difficulty of aligning prosodic targets to recognized words. The conversational Austrian German study focuses on GRCS, a spontaneous face-to-face corpus containing 19 conversations between closely acquainted native speakers. The paper emphasizes frequent overlapping speech, approximately 42% of GRASS utterances, highly variable voice quality including breathy and creaky speech, disfluencies, and pronunciation variation. These factors are presented as reasons to avoid classical prosodic feature extraction pipelines based on $i$ 5, RMS, and phone- or syllable-level segmentation (Linke et al., 12 Sep 2025).

For ASR training in that study, GRCS was post-filtered to approximately 14.4 hours and 33,734 utterances after excluding laughter, singing, imitations or onomatopoeia, unintelligible tokens, and artifacts. Text was normalized through lowercasing, punctuation removal, and standardization of backchannels. The baseline ASR system is a wav2vec2 XLSR encoder fine-tuned with a CTC head and character-level tokenizer of approximately 37 tokens. Decoding conditions are lexicon-free greedy decoding, beam search with a lexicon, and beam search with a lexicon plus a 3-gram character-level LLM in KenLM with modified Kneser-Ney smoothing (Linke et al., 12 Sep 2025).

The prominence detector and prominence-aware ASR studies both rely on word-boundary information, but in different ways. The Austrian German pipeline uses Kaldi forced alignment to align detector outputs to word boundaries during automatic labeling, then uses majority voting over character-level tags within “| … |” spans to recover word-level prominence at inference. If all characters in a word agree, the word is assigned that label; if multiple tags appear, majority voting is used; ambiguous cases are discarded. If the ASR produces an incorrect number of words through merges or splits, the mapping from character tags to words may fail, leaving an empty prominence assignment (Linke et al., 12 Sep 2025).

The English prominence-estimation work also relies on forced alignment, here via P2FA through the pyfoal library, but for a different target representation: it maps acoustic frames to reference word spans so that the model can predict one scalar prominence per word. Its principal metrics are Pearson correlation and BCE between predicted and human prominence values (Morrison et al., 2023). The pitch-accent study evaluates prosody as an event-detection task with precision, recall, and $i$ 6 under tolerance windows of $i$ 7, 40, 80, and 100 ms, alongside standard ASR WER and CER (Sasu et al., 6 Aug 2025).

The metric definitions used in this literature are standard but task-specific. In the Austrian German work, word error rate is

$i$ 8

character error rate is defined analogously, and prominence error rate is

$i$ 9

with substitutions, deletions, and insertions counted over prominence labels rather than words (Linke et al., 12 Sep 2025).

5. Empirical findings

The main empirical result for conversational prominence-aware ASR is that explicit prominence augmentation did not change word recognition performance relative to a baseline ASR system, while still yielding usable prominence labels. On held-out conversation 003M023F, baseline WERs were 26.04% for lexicon-free decoding, 21.78% for lexicon decoding, and 18.57% for lexicon plus 3-gram LM. On 004M024F, the corresponding baseline WERs were 31.25%, 27.52%, and 23.71%. Prominence-aware variants based on PDET02 remained close to these values: for example, on 003M023F, ASR0(PDET02) achieved 26.54%, 22.31%, and 18.58%, while ASR2(PDET02) achieved 26.27%, 22.24%, and 18.50%; on 004M024F, ASR0(PDET02) obtained 32.32%, 28.64%, and 24.50%, and ASR2(PDET02) 32.34%, 28.31%, and 24.32%. Slight deteriorations of approximately 1.6–2.3% were most likely when both PL0 and PL2 tags were included in ASR02, consistent with the larger output space (Linke et al., 12 Sep 2025).

The same study reports substantially better prominence performance for binary than for three-level detection. PDET02 achieved a PER of 24.83% $m_i$ 0 1.79% in cross-validation and 29.58% on held-out conversation 004M024F, with aligned-word accuracy of 89.72% $m_i$ 1 3.26% in cross-validation and 87.40% on the held-out conversation. PDET012 was weaker: PER 36.54% $m_i$ 2 0.92% in cross-validation and 41.02% on the held-out conversation, with aligned-word accuracy 69.45% $m_i$ 3 2.11% and 64.97%, respectively. For PDET012 on the held-out conversation, PL1 recall was 49%, with confusions to PL0 of 30% and to PL2 of 21%, reinforcing the interpretation that intermediate prominence is the hardest category (Linke et al., 12 Sep 2025).