CV-18 NER: Augmented Common Voice for Named Entity Recognition from Arabic Speech

Abstract: End-to-end speech Named Entity Recognition (NER) aims to directly extract entities from speech. Prior work has shown that end-to-end (E2E) approaches can outperform cascaded pipelines for English, French, and Chinese, but Arabic remains under-explored due to its morphological complexity, the absence of short vowels, and limited annotated resources. We introduce CV-18 NER, the first publicly available dataset for NER from Arabic speech, created by augmenting the Arabic Common Voice 18 corpus with manual NER annotations following the fine-grained Wojood schema (21 entity types). We benchmark both pipeline systems (ASR + text NER) and E2E models based on Whisper and AraBEST-RQ. E2E systems substantially outperform the best pipeline configuration on the test set, reaching 37.0% CoER (AraBEST-RQ 300M) and 38.0% CVER (Whisper-medium). Further analysis shows that Arabic-specific self-supervised pretraining yields strong ASR performance, while multilingual weak supervision transfers more effectively to joint speech-to-entity learning, and that larger models may be harder to adapt in this low-resource setting. Our dataset and models are publicly released, providing the first open benchmark for end-to-end named entity recognition from Arabic speech https://huggingface.co/datasets/Elyadata/CV18-NER.

• The paper introduces CV-18 NER, the first public dataset for end-to-end Arabic speech NER leveraging fine-grained Wojood annotations.
• The methodology benchmarks both pipeline (ASR + BERT-based NER) and end-to-end systems using models like Whisper and AraBEST-RQ with metrics such as WER, CoER, and CVER.
• The results demonstrate that end-to-end systems outperform cascaded approaches, emphasizing the need for tailored model selection and data augmentation in low-resource scenarios.

Summary of "CV-18 NER: Augmented Common Voice for Named Entity Recognition from Arabic Speech" (2604.02209)

Introduction and Motivation

The paper addresses the problem of end-to-end (E2E) named entity recognition (NER) directly from Arabic speech. While E2E architectures have shown promising performance in English, French, and Chinese, Arabic remains less explored due to morphological complexity, the absence of short vowels in script, and scarcity of annotated corpora. This work introduces CV-18 NER, the first public dataset for speech NER in Arabic, which comprises manual NER annotations over the Modern Standard Arabic subset of Common Voice 18 using the fine-grained Wojood schema (21 entity types). Both pipeline (ASR + text-based NER) and E2E systems are benchmarked—specifically, large-scale multilingual (Whisper) and Arabic-specific SSL (AraBEST-RQ) models.

Dataset Construction

Source and Annotation Schema

CV-18 NER is constructed from the Arabic portion of Common Voice 18, which is openly licensed, moderately sized (∼32h train / 12h dev / 12h test), and characterized by diverse speakers and acoustic conditions. The authors filter out speech segments without any named entities to focus learning on informative samples, reducing the total size to 8h15m (train), 2h54m (dev), and 2h59m (test). Annotation leverages Wojood for its high tagset granularity (21 entity types), unlike more typical Arabic NER datasets (e.g., ANERCorp, AQMAR).

Annotation Pipeline

A two-stage annotation is used: (1) automatic pre-annotation with supervised BERT-based models (AraBERT v2) fine-tuned on Wojood, and (2) manual post-correction by a trained annotator. The BIO tagging scheme is used for entity marking. Notably, only utterances with at least one entity are retained, resulting in a realistic, long-tailed distribution of entity classes dominated by PERS and GPE while categories like LAW, PRODUCT, and PERCENT are rare.

System Architectures and Training Protocols

Pipeline Baselines

The pipeline approach decouples ASR and NER: fine-tuned Whisper (medium, large-v3) or AraBEST-RQ (300M, 600M) models transcribe speech; then, BERT derivatives (AraBERT v2, v0.2, CAMeLBERT-MSA/MIX) yield entity predictions. All models are fine-tuned on the filtered CV-18 NER splits.

End-to-End NER from Speech

In E2E systems, models generate enriched transcriptions with inline BIO entity tags directly from audio input, modifying tokenizer vocabularies to treat NER markers as atomic units. Whisper (medium, large-v3) and AraBEST-RQ (300M, 600M) are trained for joint transcription and entity prediction.

Training and Evaluation

Pipeline NER is evaluated with micro-F1, ASR with WER, and speech NER with Concept Error Rate (CoER) and Concept-Value Error Rate (CVER). These metrics measure entity detection accuracy and joint entity-span (type + value) correctness, accounting for utterance-level tokenization/segmentation mismatches.

Experimental Results

ASR Performance

• Whisper in zero-shot yields prohibitive WERs (e.g., 107.4% for Whisper-medium); fine-tuning is critical.
• Fine-tuned Whisper-medium achieves 22.0% WER (test); Whisper-large-v3 underperforms at 32.3%.
• AraBEST-RQ 300M yields the lowest WER (15.1% test), validating the utility of Arabic-specific SSL in low-resource regimes; larger AraBEST models (600M) do not consistently improve upon the 300M baseline.

Text-based NER

• BERT-based models fine-tuned on CV-18 NER reach up to 81.1% (validation) and 77.3% (test) micro-F1.
• AraBERT v0.2 performs best overall.

Pipeline Speech NER

• The Whisper-medium + AraBERT v0.2 configuration is best: 51.3 CoER and 50.2 CVER on test.
• AraBEST-RQ shows better ASR (lower WER) but underperforms in entity extraction due to NER module interaction.
• Entity recognition degrades sharply for rare classes, as expected in long-tailed scenarios.

E2E Speech NER

• AraBEST-RQ 300M is optimal for WER (16.0) and CoER (37.0) in E2E, but Whisper-medium achieves superior CVER (38.0).
• Larger models (Whisper-large-v3, AraBEST-RQ 600M) underperform, likely due to over-parameterization and low-resource adaptation limits.
• E2E systems outperform pipelines by a large margin: Whisper-medium E2E yields 13.3 points lower CVER compared to the best pipeline.

Entity-Level Analysis

• Both pipelines and E2E systems perform well on high-frequency classes (PERS, GPE).
• E2E models are less sensitive to entity span disruptions and show improved robustness on mid-frequency categories (ORDINAL, TIME).
• Recognition for low-frequency classes remains poor due to data scarcity.

Implications and Future Directions

This work empirically substantiates that E2E speech NER is not only feasible for Modern Standard Arabic but also delivers superior entity-level accuracy compared to classical cascaded approaches, even under constrained data settings. Several key implications arise:

• E2E training reduces information loss from transcription errors and enables learning of entity-aware acoustic patterns, yielding improved semantic robustness.
• Arabic-specific SSL representations (AraBEST-RQ) can outperform large multilingual systems in ASR and match them in E2E NER efficiency, particularly with moderate parameter counts.
• Model size increases do not guarantee performance gains in low-resource adaptation; careful architecture/model selection and regularization are crucial.
• Dataset class imbalance remains a central challenge; accruing more annotated samples for rare entity types or leveraging data balancing strategies is necessary for comprehensive entity coverage.

The public release of CV-18 NER establishes a new benchmark for Arabic speech NER and is expected to accelerate research in multi-dialectal and code-switched NER, model robustness, and data-efficient sequence tagging in spoken Arabic.

Conclusion

The introduction of CV-18 NER and the accompanying empirical analysis demonstrate clear performance advantages for E2E architectures over pipelines in Arabic speech NER. The dataset and baseline systems constitute a substantive foundation for future research in Arabic speech understanding, with necessary developments focused on data augmentation for rare entities, better adaptation strategies for large-scale models, and robust evaluation frameworks for morphologically rich, low-resource languages.