---
title: Qur'anic Mispronunciation Benchmark (QuranMB.v1)
url: https://www.emergentmind.com/topics/qur-anic-mispronunciation-benchmark-quranmb-v1
type: topic
---

# Qur'anic Mispronunciation Benchmark (QuranMB.v1)

The Qur’anic Mispronunciation Benchmark (QuranMB.v1) is a standardized, publicly available test set and evaluation framework designed to rigorously assess automatic mispronunciation detection in Quranic recitation. QuranMB.v1 provides a unified protocol for data, annotation, phonetic representation, and evaluation, grounding the measurement of automated pronunciation assessment systems within the highly constrained and phonetically detailed context of Quranic Arabic, including its unique rules of Tajweed. The benchmark integrates recent advances from data curation, machine learning, computational orthography, and specializes in the precise identification of segmental and suprasegmental errors relevant to both native and non-native reciters.

## 1. Data Collection, Annotation, and Processing Pipeline

QuranMB.v1 construction follows a comprehensive multi-stage pipeline that addresses intrinsic linguistic and data scarcity challenges in Arabic mispronunciation detection [2506.07722, 2509.00094]. The process involves:

- **Dataset Curation:** Primary data are drawn from controlled recordings where native speakers recite 98 selected Quranic verses in Modern Standard Arabic (MSA) style, deliberately injecting specified pronunciation errors. This ensures coverage of a wide range of authentic mispronunciation phenomena. In the large-scale variant, over 850 hours of expert reciter audio with automatic and verified segmentation (at pause points or waqf) is included, with each recording annotated for Tajweed correctness and segmented at linguistically meaningful boundaries [2509.00094].
- **Synthetic Mispronunciation Generation:** Where naturally occurring error data are sparse, synthetic corpora are generated by Text-to-Speech (TTS) systems that produce both canonical and mispronounced renderings. Mispronunciation is controlled through automated modification of either the transcription (by substituting, deleting, or replacing specific phonemes/diacritics based on a confusion matrix) or via TTS systems fitted with custom pronunciation lexicons [2506.07722].
- **Segmentation and Transcription:** Segmenters based on fine-tuned Wav2Vec2-BERT models classify and segment the audio at precise waqf points, followed by automatic transcription (using Whisper or Tarteel ASR adaptations for Quranic speech). Automated verification (the Tasmeea algorithm) achieves a 98% automation rate in aligning text and audio, relying on sliding window matching and edit-distance scoring to minimize manual correction [2509.00094].
- **Phonetic Conversion:** Phoneme sequences are derived by applying advanced phonetizers, which map fully vowelized text to a standardized inventory, with coverage for diphones, gemination, and MSA prosody [2506.07722].

This pipeline ensures high-fidelity alignment between recited audio, canonical text, and ground-truth phonetic targets, supporting fine-grained error localization and assessment.

## 2. Phonetic Representation and Tajweed Encoding

Central to the benchmark is the adoption of a computational phonetic script specifically tailored for Quranic and MSA Arabic recitation [2509.00094, 2505.11379]. Notable characteristics include:

- **Quran Phonetic Script (QPS):** A multi-level system, comprising:
  - **Phoneme Level:** Encodes all Arabic consonants, short and long vowels, as well as gemination and diacritic features specific to Quranic Arabic.
  - **Sifat Level:** Ten distinct parallel tracks, each encoding one articulation characteristic (e.g., hams/jahr, shidda/rakhawa, tafkheem/taqeeq, etc.), crucial for evaluating Tajweed application.
- **Contemporary Quranic Orthography (CQO) and Tajweed Layer:** The benchmark interfaces directly with digital CQO, which combines the consonantal skeleton (QCT) with an overlay of diacritic and Tajweed marks. Computational modules strip, reapply, or transform this layer via sets of regular expressions and linguistic rules, facilitating dynamic generation or transformation of recitation targets for alignment and comparison [2505.11379].
- **Phonetic Set Size and Specifics:** The canonical MSA benchmark employs a 68-phoneme inventory, which includes explicit geminate representation and, where appropriate, phonetic distinctions such as vague versus emphatic vowels (though these may be collapsed for certain tasks) [2506.07722].

This approach enables rigorous mapping between the orthographic, phonetic, and auditory realities of recitation, essential for detecting both surface and latent pronunciation errors.

## 3. Baseline Architectures and Modeling Approaches

The evaluation of QuranMB.v1 relies on state-of-the-art modeling paradigms for sequence-to-sequence mispronunciation detection. Leading approaches include:

- **Self-Supervised Learning (SSL) Frontends:** Models such as Wav2vec2, HuBERT, WavLM, and the multilingual mHuBERT provide acoustic feature extraction pipelines with robust representation learning from limited labels [2506.07722]. The features from these encoders feed into bidirectional LSTM heads for sequence modeling.
- **Multi-Level CTC Modeling:** The QPS-based systems employ multi-task CTC (Connectionist Temporal Classification) heads—one for each phonetic or sifat attribute. The total loss is computed as a weighted sum:
  $$
  L_\text{total} = \frac{0.4 \cdot L_1 + L_2 + \cdots + L_{11}}{11}
  $$
  where $L_1$ is the phoneme-level CTC loss and $L_2$–$L_{11}$ are for the sifat tracks (weighted to address the vocabulary and learning difficulty differentials) [2509.00094].
- **Monolingual vs. Multilingual Models:** mHuBERT, trained on 147 languages, consistently outperforms monolingual SSL architectures, delivering true acceptance rates up to 87% and F1-scores approaching 30% when trained on combined real and synthetic data [2506.07722].
- **Auxiliary Systems:** The pipeline includes specialized ASR models for segmentation and transcription, as well as custom alignment (Tasmeea) and verification algorithms [2509.00094].

This modeling ensemble supports simultaneous detection and localization of mispronunciation across segmental and suprasegmental layers.

## 4. Performance, Metrics, and Baseline Results

QuranMB.v1 employs the following metrics for system evaluation:

| Metric           | Use                                    | Benchmark Value (Best Reported)       |
|------------------|----------------------------------------|---------------------------------------|
| Phoneme Error Rate (PER) | Segmental mispronunciation assessment         | 0.16% (multi-level CTC on QPS, [2509.00094])    |
| F1-Score         | Precision/recall tradeoff for error detection | ~29–30% (mHuBERT, [2506.07722])        |
| True Acceptance  | Correct error detection rate           | Up to ~87% ([2506.07722])             |

Phoneme Error Rate is computed as the proportion of substitutions, deletions, and insertions necessary to align predicted and reference phoneme sequences. Additional layers of evaluation consider per-attribute error rates (for each sifat), and the system is capable of diagnosing error types such as omitted diacritics, incorrect gemination, or Tajweed misapplication.

Reported results show that phoneme-level performance can reach PERs as low as 0.16% in expert reciter test sets when leveraging the multi-level CTC architecture with extensive ground-truth annotation [2509.00094]. However, for the more challenging case of controlled mispronunciation detection in diverse speakers, F1-scores remain below 30%, pointing to the intrinsic complexity of the task and motivating further research [2506.07722].

## 5. Challenges, Limitations, and Future Directions

Several technical and scientific challenges are highlighted by the QuranMB.v1 literature:

- **Data Scarcity and Diversity:** There remains a lack of large-scale, naturally occurring error datasets, especially from non-native or L2 reciters. Synthetic augmentation mitigates this but may under-represent subtle, real-world mispronunciations [2506.07722].
- **Annotation Complexity:** Fine-grained, multi-level (phoneme, sifat, suprasegmental) annotation requires robust, partially automated pipelines and significant verification effort, as in the Tasmeea post-processing algorithm [2509.00094].
- **Modeling Limitations:** Despite high accuracy on expert reciter benchmarks, error detection for "in-the-wild" recitation with variable mispronunciation presents challenges, including low precision in F1 metrics and the difficulty of capturing prosodic and context-sensitive errors.
- **Script and Orthography Variations:** Applying the benchmark to diverse recitation traditions or new manuscript encodings requires rigorous alignment strategies, given the highly codified CQO/Tajweed overlay [2505.11379].

Planned improvements include extending authentic error collection (particularly from L2 learners), refining multi-tier annotation (possibly incorporating dynamic context-aware phoneme sets), and integrating advanced augmentation or transfer learning techniques to boost generalizability [2506.07722].

## 6. Applications and Impact

The QuranMB.v1 benchmark is positioned as a foundational resource for Arabic speech assessment research and practical applications:

- **Automated Pronunciation Training:** It supports the development of intelligent tutoring systems and mobile applications that deliver precise, Tajweed-aware feedback for learners lacking access to expert instructors [2509.00094, 2503.23470].
- **Linguistic and Manuscript Studies:** The computational methods for Tajweed layer manipulation and CQO alignment enable detailed scholarship into prosodic and phonetic variation across Qur’anic manuscripts and recitation traditions [2505.11379].
- **Model Evaluation and Comparison:** QuranMB.v1 establishes public baselines and protocols that allow direct, reproducible comparison across future mispronunciation detection systems, promoting progress and standardization in the field [2506.07722].
- **Open-Source Ecosystem:** All data, code, and models are released under open licenses, stimulating community-driven improvement, benchmarking, and technology transfer for both academic and educational sectors [2509.00094].

## 7. Historical and Research Context

QuranMB.v1 synthesizes decades of progress in Arabic ASR, pronunciation modeling, and digital orthography research:

- **Early Methods:** Previous systems relied on HMMs, explicit multi-variant phonetic dictionaries, and Bayesian decoding to accommodate pronunciation variability in Arabic speech recognition [1204.1566].
- **Machine Learning Advancements:** The evolution to SVMs, ensemble methods, and tailored deep learning models (LSTM, CNN-GRU, EfficientNet-B0) provided incremental improvements in Tajweed rule recognition and segmental error detection [2101.04200, 2305.06429, 2305.07034, 2503.23470].
- **Scriptural Encoding and Digital Philology:** CQO and the associated computational tools for Tajweed layer manipulation represent a bridge between historical orthographic practices and high-precision computational analysis [2505.11379].
- **Benchmark Standardization:** QuranMB.v1 is the first large-scale, rigorously annotated, and publicly available framework supporting phoneme-level and attribute-level mispronunciation evaluation in Quranic Arabic, setting new standards for research in computational pronunciation assessment [2506.07722, 2509.00094].

This benchmark represents a pivotal advance, operationalizing both the academic rigor and practical frameworks required for future research and technology in Quranic speech processing.

Source: https://www.emergentmind.com/topics/qur-anic-mispronunciation-benchmark-quranmb-v1