---
title: CommonVoice-SpeechRE Benchmark
url: https://www.emergentmind.com/topics/commonvoice-speechre
type: topic
---

# CommonVoice-SpeechRE Benchmark

CommonVoice-SpeechRE is a benchmark for Speech Relation Extraction (SpeechRE) built from the English subset of Common Voice 17.0 and paired with an end-to-end model family, RPG-MoGe, for extracting relation triplets directly from speech without an intermediate transcription stage. In this setting, the target output is a set of triples of the form $(h,r,t)$, where $h$ and $t$ are entities and $r$ is a predefined relation type. The benchmark was introduced to address two persistent constraints in SpeechRE research: the heavy reliance of earlier benchmarks on synthetic speech, and the limited quantity and speaker diversity of real-human speech in prior datasets. Its source corpus, Common Voice, is a crowd-sourced, CC0-licensed speech collection designed for open speech technology research and development [2509.08438][1912.06670].

## 1. Provenance within the Common Voice ecosystem

CommonVoice-SpeechRE is derived from the English subset of Common Voice 17.0. In the benchmark description, Common Voice is characterized as a large-scale, volunteer-contributed, multilingual speech corpus spanning 124 languages and 20,408 validated hours, with CC-0 licensing that enables unrestricted use and redistribution. The decision to focus on English is motivated by the need for consistent relation schemas and annotation quality, even though the underlying source corpus is multilingual [2509.08438].

Common Voice itself was originally designed primarily for Automatic Speech Recognition, while also being useful for other speech tasks such as language identification. Its construction is crowd-sourced at both the recording and validation stages, and it distributes metadata such as `client_id`, votes, and optional self-reported demographic attributes in per-language TSV files. This design makes it unusually accessible for secondary annotation, including semantic annotation layers that were not part of the original release [1912.06670].

The relevance of Common Voice to SpeechRE is methodological as well as practical. Prior SpeechRE benchmarks relied heavily on text-to-speech data, which attenuates natural variability in pronunciation, prosody, disfluencies, devices, and background conditions. CommonVoice-SpeechRE was explicitly introduced to close that gap by supplying a real-human-speech benchmark rather than a synthetic proxy [2509.08438].

## 2. Curation pipeline and annotation protocol

The curation pipeline begins from Common Voice transcriptions rather than from raw audio alone. Because most Common Voice samples are negative for entities and relations, the authors first used a pre-trained BERT-based NER tagger, `flair/ner-english-ontonotes`, to filter candidate sentences likely to contain entities and relations. They then adopted entity and relation type definitions from ACE04/ACE05 and created a tailored annotation guideline. Annotation was carried out in Label Studio by a team of 10 graduate annotators described as CET-6 certified. Data were partitioned into batches of at most 1,000 sentences; for each batch, 10% were randomly sampled for verification, and if sentence-level accuracy was below 95%, the batch was re-annotated by experienced annotators [2509.08438].

The resulting benchmark statistics are as follows.

| Component | Train / Dev / Test | Total |
|---|---:|---:|
| Speech instances | 14,557 / 2,495 / 2,494 | 19,546 |
| Triplets | 15,948 / 2,696 / 2,728 | 21,372 |
| Relation schema size | — | 45 |
| Average audio length | — | 11.6 seconds |
| Speakers | — | approximately 20,000 unique volunteers |

Labels consist of relation triplets per instance. The paper does not report timestamps or alignment between entities and audio spans, and counts per relation type are not enumerated. Annotation was performed on the transcriptions associated with the speech rather than via forced alignment, which means the dataset is semantically labeled at the utterance level rather than at the span-timestamp level [2509.08438].

This curation design has two immediate consequences. First, it preserves the acoustic diversity of real Common Voice recordings. Second, it keeps the semantic ontology close to established text relation extraction practice by importing ACE-style definitions, thereby making cross-modal comparison more straightforward.

## 3. Formal task definition and data representation

SpeechRE in CommonVoice-SpeechRE is defined as recovering a set of relational triples directly from a speech signal $\boldsymbol{S}$. Formally, the target is

$$
\varGamma = \{ (h_i, r_i, t_i) \mid h_i, t_i \in \text{E}, r_i \in \text{R} \},
$$

where $\text{E}$ is the entity set and $\text{R}$ is the relation set [2509.08438].

The triplet formalism inherits the semantics of relation extraction but shifts the input modality from text to audio. In CommonVoice-SpeechRE, this means that the model must jointly handle acoustic recognition, entity boundary recovery, and relation decoding. The paper evaluates entities, relations, and full triplets with Micro-F1 under exact-match criteria; no partial matching is used. A triplet is therefore counted as correct only when the predicted head entity, relation, and tail entity all exactly match the ground truth [2509.08438].

A notable representational limitation is that the benchmark does not provide timestamp-level grounding for entities or relations. The labels are utterance-level semantic targets attached to audio clips. This makes the task well suited to end-to-end semantic decoding, but less suitable for studies that require temporal localization of entity mentions or segment-level interpretability.

The benchmark is also explicitly positioned against pipeline approaches of the form ASR $\rightarrow$ TextRE. In that contrast, CommonVoice-SpeechRE functions not merely as a dataset but as a testbed for whether a model can learn speech-native semantic extraction rather than inheriting all errors from a transcription front end.

## 4. RPG-MoGe: relation-prompt-guided multi-order generation

RPG-MoGe, the model introduced alongside CommonVoice-SpeechRE, is an end-to-end framework with three principal components: a speech encoder, a CNN-based latent relation prediction head, and a text decoder. The architecture uses Whisper as both encoder and decoder backbone so as to improve cross-modal alignment relative to designs that combine mismatched speech and text models. Raw speech is converted to log-mel spectrogram features $\boldsymbol{X}$, and Whisper’s encoder produces high-level features

$$
\boldsymbol{H} = \text{WhisperEncoder}(\boldsymbol{X}) \in \mathbb{R}^{L_H \times d_h}.
$$

The Latent Relation Prediction Head (LRPH) applies four CNN layers with ReLU to capture local patterns in $\boldsymbol{H}$, then flattens and linearly projects the result to multi-label relation scores with a sigmoid. The associated loss is multi-label BCE, and LRPH includes dropout with $p=0.5$. During training, 0–50% of positive relations predicted by LRPH are randomly masked to mitigate train–test discrepancies; this masking is disabled at inference [2509.08438].

The model’s second key idea is multi-order triplet generation. Because a triplet can be linearized in six permutations of $(h,r,t)$, RPG-MoGe constructs a multi-view relation tree and generates textual sequences under all six order views. For a set of triplets $\mathcal{T}$ and an order view $\psi_i$ in the six-order set $\varPsi$,

$$
\mathcal{G}_{\psi_i} = \text{Treeify}(\mathcal{T}, \psi_i), \quad
T_{\text{lin}^{\psi_i}} = \text{SeqLin}(\mathcal{G}_{\psi_i}).
$$

The decoder input concatenates three token sequences: relation prompt tokens $\boldsymbol{T}_{rel}$ derived from LRPH predictions, order-control tokens $\boldsymbol{T}_{\psi_i}^{ctrl}$, and linearized relation tree tokens $\boldsymbol{T}_{\psi_i}^{lin}$:

$$
\boldsymbol{T}_{dec} = [\boldsymbol{T}_{rel}, \boldsymbol{T}_{\psi_i}^{ctrl}, \boldsymbol{T}_{\psi_i}^{lin}].
$$

Whisper’s decoder then performs autoregressive generation, and the overall training objective is

$$
\mathcal{L}_{total} = \mathcal{L}_{lrp} + \mathcal{L}_{dec}.
$$

At inference, the model decodes under all order views and accepts a triplet if it appears in more than a voting threshold $\lambda_{vote}$ views, with $\lambda_{vote}=2$ in the reported experiments. The paper does not specify beam versus greedy decoding. For CommonVoice-SpeechRE, Whisper-small.en ($\sim244$M) and Whisper-medium.en ($\sim769$M) are the backbone variants, with Adam, learning rate $1\times10^{-5}$, batch size 12, and 10 training epochs [2509.08438].

## 5. Evaluation results and comparative standing

Evaluation on CommonVoice-SpeechRE uses Micro-F1 for entities, relations, and triplets, all under exact-match criteria. On this benchmark, RPG-MoGe(770M) with Whisper backbones achieves Entity F1 47.20, Relation F1 53.48, and Triplet F1 18.29. Under the same backbone scale, MCAM(770M) attains Entity F1 45.34, Relation F1 50.34, and Triplet F1 15.71. The reported gains are therefore +1.86 for Entity F1, +3.14 for Relation F1, and +2.58 for Triplet F1 [2509.08438].

The paper attributes these gains to two mechanisms: multi-order generation with ensemble voting, and prompt-guided decoding using LRPH relation predictions. It also reports that RPG-MoGe outperforms end-to-end baselines such as LNA-ED and MCAM across datasets, while pipeline approaches lag because of error accumulation from ASR and mismatch with TextRE assumptions. Large language models in zero-shot or in-context modes, including GPT-3.5/4, GPT-4o-audio, and Qwen2-audio, are also reported to underperform fine-tuned task-specific SpeechRE models on these benchmarks [2509.08438].

These results are technically significant for two reasons. First, they indicate that end-to-end semantic extraction from speech remains a low-F1 regime on real-human audio even with large pretrained backbones. Second, they show that output linearization strategy remains a substantial variable in generative SpeechRE, rather than a superficial formatting choice.

## 6. Corpus-specific confounds, limitations, and broader research context

Because CommonVoice-SpeechRE inherits its audio from Common Voice, it also inherits several corpus-level characteristics documented elsewhere in the Common Voice literature. Common Voice recordings are crowd-sourced, and the source corpus exhibits broad variability in device, environment, accent, and speaker participation [1912.06670]. A related study on Common Voice speaker identity quality shows that the released `client ID` is only an anonymized account-level identifier and that multiple distinct speakers can contribute under the same client ID. Using ResNet-based voice embeddings and human-audited thresholding, that study derives a cleaning threshold of $\tau=0.354$ for cosine similarity and reports that this removes a median of 1.6% utterances per language, with mean 3.5% [2506.00733].

This suggests that speaker purity can matter even in tasks not framed as speaker verification. For SpeechRE, a plausible implication is that real-human acoustic diversity is beneficial, but unresolved speaker heterogeneity within source metadata may complicate analyses of speaker-conditioned robustness or per-speaker error structure. CommonVoice-SpeechRE itself reports approximately 20,000 unique volunteers from the source corpus, but detailed demographic and accent distributions are not reported [2509.08438].

A separate Common Voice fairness study on French ASR shows that speaker imbalance can be severe in Common Voice train splits: the top single speaker accounts for 7% of all training utterances, the top 10 speakers account for 25%, and half of the training utterances come from only 1% of speakers. That work also finds that maximizing the number of unique speakers is a stronger lever for robustness and subgroup performance than subgroup-targeted fine-tuning at fixed training size [2306.03773]. Although CommonVoice-SpeechRE is not an ASR benchmark, the finding is relevant context for any future expansion of SpeechRE training protocols on Common Voice-derived audio.

The benchmark’s limitations are explicit. It is English-only; detailed demographic and accent distributions are not reported; performance depends heavily on Whisper pretraining; timestamp-level alignment is absent; and multi-order decoding increases inference cost. The paper identifies multilingual SpeechRE, timestamp-level entity and relation alignment, more efficient ensemble strategies, stronger noise robustness, and domain adaptation as future directions [2509.08438].

Taken together, CommonVoice-SpeechRE occupies a distinct position in the speech-semantic pipeline. It uses a genuinely open, real-human, crowd-sourced speech source; it imports a text-derived relation ontology into audio semantics; and it frames speech-native triplet extraction as a benchmarkable task rather than a by-product of ASR. Its main importance lies less in absolute F1 values than in establishing a public benchmark where robustness to real acoustic variability is unavoidable rather than simulated.

Source: https://www.emergentmind.com/topics/commonvoice-speechre