Spoken NER: Audio to Entity Mapping

• Spoken NER is the task of identifying and classifying named entities from audio inputs, addressing challenges like ASR errors, disfluencies, and absent punctuation.
• It encompasses transcript-based approaches using text NER on ASR outputs and end-to-end methods that directly integrate speech-to-entity extraction.
• Recent advances leverage subword features, pretrained models, and knowledge distillation to enhance performance across multilingual and domain-specific scenarios.

Spoken Named Entity Recognition (spoken NER) is the task of identifying named-entity mentions and assigning entity types when the input originates in speech rather than clean written text. In the literature, the term covers two closely related settings: transcript-based NER, where a text NER model operates on manual or automatic transcripts of speech, and end-to-end speech NER, where a model maps audio directly to entity-annotated output without a separate downstream tagger. The distinction matters because spoken inputs introduce ASR error propagation, pronunciation variation, disfluencies, missing punctuation and casing, and domain-specific acoustic ambiguity; at the same time, direct access to speech can expose phonological or acoustic cues unavailable to text-only systems (Abujabal et al., 2018, Ghannay et al., 2018, Yadav et al., 2020, Chen et al., 2022).

1. Scope, task variants, and relation to spoken language understanding

Spoken NER sits within spoken language understanding and speech-to-semantics. In voice-assistant and telephony pipelines, the classic formulation is sequential: speech is first transcribed by ASR and the resulting text is then passed to a text NER system. This transcript-based formulation appears in large-scale spoken SLU settings for voice-controlled devices, in noisy business telephone conversations, and in voice-assistant named-entity understanding systems that integrate downstream entity linking (Abujabal et al., 2018, Fu et al., 2022, Muralidharan et al., 2020).

A more integrated formulation treats spoken NER as direct speech-to-entity extraction. Early end-to-end work on French and English speech replaced the pipeline with a single neural model trained to emit a transcript enriched with named-entity markers, so that transcription and tagging are optimized jointly (Ghannay et al., 2018, Yadav et al., 2020). Chinese work later described this as “entity-aware ASR,” while Arabic work serialized BIO tags inline in the generated output and compared that design against classical ASR+text-NER cascades (Chen et al., 2022, Saidi et al., 2 Apr 2026).

The literature also includes adjacent formulations that widen the meaning of spoken NER. Audio de-identification reframes the task around redacting the corresponding time spans in audio rather than merely tagging transcript tokens, thereby turning spoken NER into an audio-grounded localization problem (Cohn et al., 2019). In voice assistants, named-entity understanding couples NER with entity linking so that KB evidence can rerank alternative entity analyses under spoken-query noise (Muralidharan et al., 2020). This suggests that spoken NER is best understood not as a single model family but as a layer in a broader speech understanding stack.

2. Model formulations and output representations

Transcript-based spoken NER often reuses high-performing text sequence models, but modifies the token representation or training regime to handle speech-derived noise. A representative formulation is the hierarchical BiLSTM-CRF in which each token is composed from characters, phonemes, and bytes, with optional word embeddings. For a token $w_i$, the subword-only embedding is explicitly written as

$x_i = [V_c(w_i); V_{ph}(w_i); V_{by}(w_i)],$

and the hybrid version as

$$x_i = [V_c(w_i); V_{ph}(w_i); V_{by}(w_i); E_w(w_i)].$$

The resulting contextual sequence is decoded with a linear-chain CRF. In spoken SLU settings this is notable because phoneme sequences are derived from ASR/TTS lexica, giving the NER model access to pronunciation-level information while retaining a much smaller vocabulary than word-level models (Abujabal et al., 2018).

End-to-end spoken NER usually abandons BIO-over-tokens and instead treats entity extraction as sequence transduction. In the French and English Deep Speech 2–style systems, named entities are encoded by inserting special symbols directly into the output character sequence, so that the model predicts characters plus entity boundary/type markers under a CTC objective (Ghannay et al., 2018, Yadav et al., 2020). Chinese entity-aware ASR follows the same principle with paired markers—[ ] for PER, ( ) for LOC, and < > for ORG—so that a standard ASR architecture can be trained on entity-annotated transcripts rather than plain text (Chen et al., 2022). Arabic end-to-end systems instead serialize BIO tags inline within the target transcription and add those tags to the tokenizer vocabulary as single atomic units (Saidi et al., 2 Apr 2026).

Hybrid systems combine speech-aware representations with strong text NER components. In Chinese, the best reported system uses Conformer entity-aware ASR followed by pretrained BERT NER, rather than choosing between pipeline and end-to-end extremes (Chen et al., 2022). In low-resource English spoken NER, external-data methods distill pseudo-labels from stronger teachers into an end-to-end speech model or into pipeline components, showing that the decisive factor is often not the architectural label “pipeline” or “E2E” in isolation, but where semantic supervision enters the system (Pasad et al., 2021).

Production-oriented transcript NER has also emphasized model compression and deployment constraints. In noisy business telephone transcripts, LUKE is fine-tuned as a teacher on human-annotated spoken data, used to pseudo-label hundreds of thousands of in-domain utterances, and then distilled into a smaller DistilBERT student in a “distill-then-fine-tune” pipeline (Fu et al., 2022). Voice-assistant named-entity understanding adds a different axis: top-$l$ NER hypotheses are reranked jointly with entity-linking candidates, so KB popularity, aliases, relations, and character-level mention–entity similarity can repair NER decisions made under speech noise (Muralidharan et al., 2020).

3. Benchmarks, corpora, and domains

The empirical landscape of spoken NER is shaped by a heterogeneous benchmark ecology that spans transcript-based SLU, end-to-end speech benchmarks, multilingual corpora, medical speech, and synthetic-data testbeds.

Resource	Languages / domain	Key reported scale
Voice-assistant transcript NER (Abujabal et al., 2018)	EN, DE, FR, ES; voice-controlled devices	Up to 5.5M utterances per language
SLUE VoxPopuli spoken NER (Pasad et al., 2021)	English parliamentary speech	15 h train, 5 h dev, 5 h test
AISHELL-NER (Chen et al., 2022)	Chinese speech	141,600 utterances, 68,604 entities
MSNER (Meeus et al., 2024)	NL, FR, DE, ES; VoxPopuli	590 h silver train, 15 h silver dev, 17 h gold test
VietMed-NER (Le-Duc et al., 2024)	Vietnamese medical speech	9,000 sentences, 18 entity types
CV-18 NER (Saidi et al., 2 Apr 2026)	Arabic Common Voice 18	8h15m train, 2h54m dev, 2h59m test after filtering
ST-CMDS-NER / HeardU (Yu et al., 2024)	Chinese voice chat / voice control	3,000 sentences; NED with 8,853 entities

AISHELL-NER is presented as the first publicly available dataset for NER from Chinese speech and preserves the default AISHELL-1 train/dev/test split, with 120,098 training utterances, 14,326 development utterances, 7,176 test utterances, and 68,604 total entities over PER, LOC, and ORG (Chen et al., 2022). CV-18 NER is introduced as the first publicly available Arabic dataset for NER from speech, built by augmenting Arabic Common Voice 18 with manual annotations under the 21-type Wojood schema; after removing utterances with no named entity, the released benchmark retains 8h15m of train audio, 2h54m of dev audio, and 2h59m of test audio (Saidi et al., 2 Apr 2026).

MSNER extends VoxPopuli with multilingual spoken NER annotations in Dutch, French, German, and Spanish. Its training and development partitions are silver-labeled, while the test set is gold, manually refined, and totals 6,254 sentences over 17 hours (Meeus et al., 2024). VietMed-NER is described as the first publicly available spoken NER dataset in the medical domain and, with 18 distinct entity types, is presented as the largest spoken NER dataset in the world regarding number of entity types; it contains 9,000 sentences split into 8 hours train, 2 hours dev, and 6 hours test (Le-Duc et al., 2024).

Other resources stress different research questions. The English work on end-to-end spoken NER constructs a 150-hour candidate corpus by filtering nearly 1,000 hours of LibriSpeech, Common Voice, TED-LIUM, and VoxForge with a Flair NER model, then manually annotates PER, LOC, and ORG (Yadav et al., 2020). Cross-lingual spoken NER uses pseudo-annotated Common Voice corpora in English, German, and Dutch, with Dutch representing the sharply low-resource case at 54 training hours (Benaicha et al., 2023). HeardU adds a benchmark for unseen entities via ST-CMDS-NER and a corresponding named entity dictionary of 8,853 Chinese entities, explicitly targeting the emergence of previously unseen names in spoken NER deployment (Yu et al., 2024).

4. Evaluation protocols and what they measure

Spoken NER evaluation is notably less standardized than text NER evaluation because transcript fidelity, entity typing, and speech localization are separable sources of error. Several lines of work retain exact-match entity F1. Low-resource spoken NER on SLUE VoxPopuli evaluates micro-averaged F1 over unordered lists of $(\text{entity phrase}, \text{tag})$ tuples, so a prediction is correct only if both text span and entity type match exactly (Pasad et al., 2021). English end-to-end spoken NER similarly reports strict entity-level precision, recall, and micro-F1, and discards malformed “half-labeled” outputs lacking a closing entity marker (Yadav et al., 2020).

Other benchmarks introduce spoken-specific variants. MSNER reports exact entity F1 and a looser label-F1 that ignores spelling and segmentation errors and checks only entity type, which is particularly informative under ASR corruption (Meeus et al., 2024). Arabic CV-18 NER evaluates speech systems with ASR-style error rates: $\text{Error Rate} = \frac{S + D + I}{N},$ instantiated as WER, CoER, and CVER. CoER evaluates entity labels only, while CVER is stricter and requires both entity type and lexical value/span to be correct (Saidi et al., 2 Apr 2026). Cross-lingual spoken NER introduces Entity Error Rate,

$$EER = \frac{N_{\text{Incorrectly Transcribed Entities}}}{N_{\text{Total Entities}}},$$

and computes micro-$F_1$ as

$F_1 = \frac{TP}{TP+\frac{1}{2}(FP+FN)},$

over triples consisting of entity transcription, entity type, and entity occurrence index (Benaicha et al., 2023).

Medical spoken NER on Vietnamese explicitly argues that exact phrase matching on ASR output is too harsh. It therefore supplements seqeval with a modified SLUE evaluation in which the entity phrase is replaced by a dummy token, so that type recovery is rewarded even when ASR slightly alters the surface form (Le-Duc et al., 2024). Audio de-identification pushes evaluation farther toward the signal level by defining a redaction function $R(t)$ over time and measuring whether at least a proportion $x_i = [V_c(w_i); V_{ph}(w_i); V_{by}(w_i)],$0 of a sensitive word’s audio interval is covered. The resulting $x_i = [V_c(w_i); V_{ph}(w_i); V_{by}(w_i)],$1 and $x_i = [V_c(w_i); V_{ph}(w_i); V_{by}(w_i)],$2 distinguish mislocalization from outright classification failure (Cohn et al., 2019). A plausible implication is that spoken NER evaluation cannot be reduced to a single universal metric; task design determines whether exact value recovery, concept labeling, or audio-span coverage is the primary objective.

5. Empirical regularities across the literature

A first regularity is that transcript-based baselines remain strong when text models are powerful and ASR quality is adequate, but speech-aware modeling materially changes the error profile. In the large transcript-based voice-assistant study, subword-only models approach word-level models in high-resource settings while drastically reducing vocabulary size; for English, subword-only reaches 91.35 F1 versus 93.92 F1 for word embeddings, with vocabulary size reduced from 74K to 332, and combining word and subword features is best in all four languages (Abujabal et al., 2018). In noisy business telephone transcripts, a distilled DistilBERT student reaches 85.29 F1 at 40 ms CPU inference, while the LUKE teacher reaches 86.07 F1 at 2980 ms, showing that spoken-transcript NER has a substantial deployment-oriented literature distinct from direct speech modeling (Fu et al., 2022).

A second regularity is that end-to-end spoken NER often improves entity robustness relative to simple cascades, but not uniformly across all benchmarks or output criteria. On French broadcast speech, an end-to-end system reaches F-measure $x_i = [V_c(w_i); V_{ph}(w_i); V_{by}(w_i)],$3 for named-entity category detection on test data, compared with $x_i = [V_c(w_i); V_{ph}(w_i); V_{by}(w_i)],$4 for the classical pipeline, while still trailing the pipeline for exact category+value extraction (Ghannay et al., 2018). On English speech, DeepSpeech2 with entity markers and a 4-gram LM reaches about 0.90 micro-F1, versus 0.80 for the classical two-step ASR→Flair pipeline (Yadav et al., 2020). On Arabic CV-18 NER, end-to-end systems substantially outperform the best pipeline configuration, reducing CoER from 51.3 to 37.0 and CVER from 50.2 to 38.0 (Saidi et al., 2 Apr 2026).

A third regularity is that the relative merits of pipeline and end-to-end systems depend strongly on pretraining and supervision. In Chinese, pure end-to-end Conformer entity-aware ASR reaches 73.37 F1, Conformer ASR + pretrained BERT reaches 74.04 F1, and the best result comes from combining entity-aware ASR with pretrained BERT, at 74.90 F1 (Chen et al., 2022). In low-resource English spoken NER, the best baseline is pipeline at 72.0 F1, but the best externally augmented system is end-to-end, reaching 82.2 F1 when distilling text NER labels from transcribed speech (Pasad et al., 2021). On MSNER, an end-to-end cross-lingual model has far stronger label-F1 than a pipeline in every language, yet exact F1 still shows a Dutch exception where the pipeline exceeds the end-to-end system (Meeus et al., 2024).

A fourth regularity is that entity distribution and entity type matter. Organization is consistently harder than person and location in the English end-to-end dataset because the class is much rarer (Yadav et al., 2020). In Chinese spoken NER, person names degrade most sharply under speech because homophones and polyphones create acoustically correct but character-incorrect transcriptions; the paper’s illustrative confusion between 郭京 and 郭晶 exemplifies this failure mode (Chen et al., 2022). Arabic CV-18 NER reports severe long-tail degradation for rare classes such as LAW, PERCENT, PRODUCT, QUANTITY, and CURR, while common classes like PERS and GPE are much stronger (Saidi et al., 2 Apr 2026). Medical Vietnamese spoken NER likewise shows large per-type disparities, with OCCUPATION, GENDER, and LOCATION much easier than PREVENTIVEMED or MEDDEVICETECHNIQUE (Le-Duc et al., 2024).

6. Low-resource adaptation, unseen entities, and application-specific extensions

External data and transfer learning are central because spoken NER labels are expensive. In the low-resource SLUE setting with only 15 hours of labeled training speech, self-supervised pretraining already yields large gains over systems trained from scratch, and further improvements come from self-training, knowledge distillation, and transfer learning on unlabeled speech, transcribed speech, and text. The strongest result, 82.2 F1, comes from distilling text NER labels into an end-to-end model using 500 hours of transcribed speech (Pasad et al., 2021). Cross-lingual transfer pushes the same logic across languages: German-to-Dutch transfer improves spoken NER by about 7 absolute F1 over the Dutch end-to-end baseline and about 4 over the Dutch pipeline baseline, whereas German-to-English transfer is much weaker (Benaicha et al., 2023).

Unseen entities have become an explicit research target. HeardU argues that existing spoken NER systems perform poorly on previously unseen named entities and proposes generating spoken NER data from a named entity dictionary using an LLM plus TTS, with a WER-based round-trip noise filter. On English zero-shot domain adaptation, the end-to-end model improves from 11.86 F1 to 31.62 F1 when synthetic target-domain data are added; on Chinese zero-shot adaptation, the end-to-end model improves from 27.86 F1 to 60.84 F1 (Yu et al., 2024). The same work reports that named entity dictionary construction is much cheaper than full spoken NER annotation, at 3660 entities/hour versus 148 for spoken NER annotation (Yu et al., 2024). This suggests a data-centric route to spoken NER scalability in environments where lexicons exist but audio annotations do not.

Application-specific extensions modify the task definition itself. Voice-assistant named-entity understanding couples NER with entity linking, showing that reranking NER hypotheses with KB evidence improves NER by up to 3.13 F1 and EL by up to 3.6 F1 (Muralidharan et al., 2020). Audio de-identification treats spoken NER as privacy-preserving redaction and shows that transcript NER on manual transcripts can reach 0.90 F1 while end-to-end audio de-ID peaks at 0.51 F1, largely because ASR remains the dominant bottleneck (Cohn et al., 2019). Medical spoken NER introduces highly specialized label inventories and evaluation needs driven by clinical semantics rather than news-style entity categories (Le-Duc et al., 2024).

7. Persistent challenges and research directions

The literature converges on several unresolved problems. ASR remains the principal bottleneck in many realistic settings: Arabic work attributes strong end-to-end gains to reduced error propagation (Saidi et al., 2 Apr 2026), audio de-identification identifies ASR as the main obstacle to final audio redaction quality (Cohn et al., 2019), and multilingual benchmarks repeatedly show large drops when text NER models are exposed to ASR transcripts instead of gold text (Meeus et al., 2024). This suggests that spoken NER is not merely text NER with noisier input; it is a coupled recognition-and-semantics problem.

Data quality and supervision regime are equally constraining. MSNER relies on silver train/dev annotations and reports no formal inter-annotator agreement statistic (Meeus et al., 2024). CV-18 NER uses a single annotator and only about 8h15m of entity-containing training speech after filtering (Saidi et al., 2 Apr 2026). Cross-lingual spoken NER explicitly calls for human-annotated multilingual corpora with consistent entity annotation across languages (Benaicha et al., 2023). HeardU identifies TTS quality as a major bottleneck in synthetic-data generation for unseen entities, especially in Chinese fully zero-shot settings (Yu et al., 2024).

A final persistent issue is mismatch between evaluation criterion and intended use. Exact text-span matching is too strict for some ASR-noisy scenarios, but relaxed label-only metrics can understate the importance of value recovery. Audio-grounded tasks such as de-identification require temporal coverage metrics rather than text-only span metrics (Cohn et al., 2019). Medical Vietnamese spoken NER therefore introduces modified SLUE evaluation for ASR outputs, while Arabic work distinguishes CoER from the stricter CVER (Le-Duc et al., 2024, Saidi et al., 2 Apr 2026). A plausible implication is that future progress in spoken NER will continue to depend on benchmark-specific alignment between model design, annotation scheme, and evaluation target, rather than on a single dominant architecture or metric.