Misraj-DocOCR: Arabic Doc-to-Markdown Benchmark

• The paper introduces Misraj-DocOCR, a benchmark designed for end-to-end conversion of Arabic printed documents into Markdown, addressing challenges like cursive scripts and ligatures.
• It details a normalization pipeline that converts Markdown tables to HTML and standardizes headers and formatting to preserve document structure.
• The benchmark evaluates various systems, highlighting trade-offs between text accuracy and structure fidelity, with models like Baseer setting a strong performance baseline.

Misraj-DocOCR is an Arabic document-to-Markdown OCR benchmark introduced to support rigorous evaluation of end-to-end OCR systems on Arabic pages, with emphasis on both transcription fidelity and structural conversion to Markdown, using HTML for tables. It was proposed alongside Baseer, a vision-LLM fine-tuned for Arabic document OCR, in response to persistent limitations of Arabic OCR arising from cursive script, context-sensitive letter forms and ligatures, optional but meaning-bearing diacritics, right-to-left orientation, and wide font and layout diversity. The benchmark is described as expert-verified, highly diverse, and intended for broad document understanding across synthetic and real-world printed pages rather than handwritten material (Hennara et al., 17 Sep 2025).

1. Definition, task, and motivation

Misraj-DocOCR is defined as a focused benchmark for Arabic document-to-Markdown OCR. Its task is not limited to plain text extraction: systems are expected to perform end-to-end document conversion, preserving textual content and document structure in a Markdown representation, with tables encoded in HTML. This design makes the benchmark structurally aware by construction, rather than reducing evaluation to isolated text recognition (Hennara et al., 17 Sep 2025).

The benchmark is motivated by recurring failure modes in Arabic OCR. The underlying paper identifies a fully cursive script with context-sensitive letter forms and ligatures, optional but semantically consequential diacritics, right-to-left text orientation, and broad variation in fonts and layouts as persistent obstacles for general-purpose systems. The same paper reports that existing Arabic benchmarks, especially the KITAB pdf-to-markdown subset, contained unreliable labels, including hallucinatory text, missing page numbers, and omissions of small-font content, while also lacking sufficient diversity. Misraj-DocOCR is presented as a response to these shortcomings through careful curation and expert review (Hennara et al., 17 Sep 2025).

A common misconception is to treat Arabic OCR benchmarks as interchangeable with generic OCR test sets. Misraj-DocOCR is narrower and more specific: it targets Arabic document-to-Markdown conversion, not generic scene text OCR, not handwritten transcription, and not only plain-text extraction. This suggests that its evaluation signal is especially relevant for systems intended to preserve reading structure, headings, lists, and tables, rather than only maximizing character-level recovery.

2. Dataset composition and representational format

The benchmark consists of 400 high-quality images. The paper explicitly states that it comprises both synthetic and real-world pages, and that it was curated to include a wide variation of document types, layouts, and fonts. It emphasizes printed content and diverse publishing layouts; handwritten content is not discussed. The focus is broad document understanding, including books, magazines, and academic-like pages, rather than a single application domain (Hennara et al., 17 Sep 2025).

Several dataset statistics are intentionally not specified in the source. The exact synthetic-versus-real breakdown is not given. Per-category counts, token counts, font inventories, ligature coverage, language-mixing patterns, numeral variants, and layout-type counts are also not reported. Likewise, there is no formal enumeration of document categories such as invoices or receipts. This absence is itself part of the benchmark’s current profile: it is a curated evaluation set whose reliability is emphasized more strongly than exhaustive descriptive metadata.

Ground truth is represented in Markdown for text, with tables represented in HTML. The paper indicates that this representation is used consistently in evaluation, and that predictions are normalized to consistent Markdown/HTML conventions before scoring. Standard Markdown semantics such as headings, lists, and emphasis are implied, while encoding specifics for items such as links, code blocks, and mathematics are not detailed. Structural fidelity is enforced through expert verification, but the paper does not define a formal schema for nested structures, equations, or links (Hennara et al., 17 Sep 2025).

This representational choice aligns Misraj-DocOCR with contemporary document-understanding benchmarks that evaluate structural conversion rather than plain transcription alone. In the broader Arabic benchmarking landscape, KITAB-Bench also treats PDF-to-Markdown as a first-class task and evaluates structural fidelity using MARS, reinforcing the importance of markup-aware evaluation for Arabic document systems (Heakl et al., 20 Feb 2025).

3. Annotation quality, access, and benchmark protocol

The paper characterizes Misraj-DocOCR as having Expert-Verified Ground Truth. Every image was “meticulously reviewed by human experts” to ensure accurate text transcription and structural fidelity. This quality-control emphasis is central to the benchmark’s identity, especially because it is introduced in direct contrast to label errors observed in earlier Arabic resources (Hennara et al., 17 Sep 2025).

At the same time, several annotation-process details remain unspecified. The paper does not report detailed annotation guidelines for right-to-left reading order, hyphenation, diacritics, ligatures, punctuation, or line breaks. It does not provide inter-annotator agreement, double-check sampling procedures, or error-auditing statistics. Nor does it describe a validation toolchain beyond human review. These omissions do not negate the expert-review claim, but they limit formal analysis of annotation consistency.

Misraj-DocOCR is used as a single evaluation set of 400 images. No train/validation/test splits are reported, and no category-wise splits are described. This is an important practical point: the benchmark is framed as an evaluation resource rather than a training corpus. Public access is provided through Hugging Face at https://huggingface.co/datasets/Misraj/Misraj-DocOCR, although licensing terms are not stated in the paper (Hennara et al., 17 Sep 2025).

Another common misunderstanding is to assume that release of a benchmark necessarily entails a complete evaluation toolkit. The paper describes a standardized post-processing pipeline, but does not provide links to official evaluation scripts or a toolkit. This suggests that exact replication depends on reimplementing the documented normalization protocol with care.

4. Evaluation methodology and metrics

Misraj-DocOCR evaluates both text accuracy and structure preservation. Before scoring, outputs are normalized so that equivalent but syntactically different Markdown/HTML variants are not penalized. The documented normalization steps are:

1. Remove HTML tags outside table structures.
2. Convert Markdown tables to HTML.
3. Normalize horizontal line representations such as --- and *** to ---.
4. Standardize header formatting.
5. Unify formatting tags within HTML tables, for example <strong> and <b> to <b>.
6. Remove model-specific tags such as <page_number> and <watermark> that appear only in some systems’ outputs (Hennara et al., 17 Sep 2025).

The benchmark uses six reported metrics. The two explicit error-rate formulas are:

$\mathrm{WER} = \frac{S + D + I}{N}$

where $S$, $D$, and $I$ are the number of substitutions, deletions, and insertions, and $N$ is the number of words in the reference, and

$\mathrm{CER} = \frac{S + D + I}{N}$

at the character level (Hennara et al., 17 Sep 2025).

BLEU and ChrF are also reported to capture n-gram and character-level agreement, although the paper does not provide formal equations. For structure-aware evaluation, it uses TEDS, described conceptually as measuring similarity between hierarchical document structures such as Markdown or HTML, and MARS, described as evaluating layout-aware alignment between predicted and reference renderings. The paper does not provide formal definitions for TEDS or MARS within this benchmark description (Hennara et al., 17 Sep 2025).

Notably, the source does not describe special metric handling for right-to-left text, Unicode normalization, diacritic stripping, or punctuation weighting beyond the formatting normalization listed above. This is methodologically consequential: benchmark scores incorporate Arabic-specific recognition difficulty, but without an explicitly Arabic-specific scoring normalization layer for diacritics or RTL order.

5. Baselines and reported results

The paper evaluates both open-source and commercial systems on Misraj-DocOCR and reports that Baseer establishes a new state of the art overall. The main findings are that Baseer achieves the lowest WER and the strongest structure-aware scores, Gemini-2.5-pro attains the highest BLEU and ChrF, and Azure AI Document Intelligence achieves the lowest CER, but trails Baseer on structure (Hennara et al., 17 Sep 2025).

System	Text metrics	Structure metrics
Baseer	WER 0.25; CER 0.53; BLEU 76.18; ChrF 87.77	TEDS 66; MARS 76.885
Gemini-2.5-pro	WER 0.37; CER 0.31; BLEU 77.92; ChrF 89.55	TEDS 52; MARS 70.775
Azure AI Document Intelligence	WER 0.44; CER 0.27; BLEU 62.04; ChrF 82.49	TEDS 42; MARS 62.245
dots.ocr	WER 0.50; CER 0.40; BLEU 58.16; ChrF 78.41	TEDS 40; MARS 59.205
Nanonets	WER 0.71; CER 0.55; BLEU 42.22; ChrF 67.89	TEDS 37; MARS 52.445
QARI	WER 0.76; CER 0.64; BLEU 38.59; ChrF 64.50	TEDS 21; MARS 42.750
Qwen2.5-VL-32B	WER 0.76; CER 0.59; BLEU 37.62; ChrF 62.64	TEDS 41; MARS 51.820
GPT-5	WER 0.86; CER 0.62; BLEU 40.67; ChrF 61.6	TEDS 48; MARS 54.8
Qwen2.5-VL-3B-Instruct	WER 0.87; CER 0.71; BLEU 25.39; ChrF 53.42	TEDS 27; MARS 40.210
Qwen2.5-VL-7B	WER 0.92; CER 0.77; BLEU 31.57; ChrF 54.70	TEDS 27; MARS 40.850
Gemma3-12B	WER 0.96; CER 0.80; BLEU 19.75; ChrF 44.53	TEDS 33; MARS 38.765
Gemma3-4B	WER 1.01; CER 0.85; BLEU 9.57; ChrF 31.39	TEDS 28; MARS 29.695
AIN	WER 1.23; CER 1.11; BLEU 1.25; ChrF 2.24	TEDS 21; MARS 11.620
GPT-4o-mini	WER 1.36; CER 1.10; BLEU 22.63; ChrF 47.04	TEDS 26; MARS 36.52
Aya-vision	WER 1.41; CER 1.07; BLEU 2.91; ChrF 9.81	TEDS 26; MARS 17.905

These results are notable for the mismatch between different metric leaders. Baseer has the best overall profile according to the paper’s interpretation, but not the lowest CER and not the highest BLEU or ChrF. This suggests that Misraj-DocOCR rewards structural fidelity and end-to-end document conversion quality in a way that is not reducible to character-level error alone. It also underscores that systems can be strong on local text similarity while remaining weaker on document structure.

The paper does not provide per-category breakdowns such as tables versus paragraphs, nor robustness analyses for fonts, diacritics, or other Arabic-specific subconditions. As a result, the leaderboard is informative at the aggregate level but not diagnostic about which failure modes dominate specific systems.

6. Relation to prior Arabic benchmarks, limitations, and significance

Misraj-DocOCR is positioned partly through critique of prior Arabic benchmark resources. The paper specifically identifies problems in the KITAB pdf-to-markdown subset: hallucinatory ground-truth text, missing page numbers, systematic omissions of small-font text such as footers, and insufficient diversity. The authors therefore release a corrected KITAB subset and present Misraj-DocOCR as a more reliable, expert-verified, and structurally faithful alternative for evaluation (Hennara et al., 17 Sep 2025).

This positioning matters because Arabic OCR benchmarking has recently expanded in scope. KITAB-Bench covers 8,809 samples across 9 major domains and 36 sub-domains, including handwritten text, layout detection, table recognition, charts, diagrams, and VQA, whereas Misraj-DocOCR is a smaller, more focused benchmark centered on document-to-Markdown OCR. The relationship is therefore complementary rather than redundant: KITAB-Bench is broader in task coverage, while Misraj-DocOCR is narrower and more targeted toward high-quality Arabic document conversion evaluation (Heakl et al., 20 Feb 2025).

The benchmark’s current limitations are mostly those of specification granularity rather than stated design flaws. Only the total size of 400 images is reported. Detailed coverage statistics, official splits, language-mixture information, diacritic distributions, and richer annotation schemas are not described. The paper also does not provide a dedicated error analysis for Misraj-DocOCR, such as diacritic omission rates, ligature confusions, RTL reading-order mistakes, or table-structure error taxonomies (Hennara et al., 17 Sep 2025).

A further plausible implication is that Misraj-DocOCR’s principal contribution is epistemic reliability: it functions as a controlled, expert-reviewed evaluation instrument for Arabic document OCR at a time when the field has suffered from unreliable labels and structurally weak evaluation. In that sense, it resembles earlier calls in low-resource script OCR research for better public benchmarks and standardized protocols, including work on Arabographic OCR and Indic OCR that repeatedly identified benchmark quality as a major bottleneck (Romanov et al., 2017, Mathew et al., 2022).

7. Practical use in Arabic document OCR research

Misraj-DocOCR is intended to evaluate end-to-end document-to-Markdown OCR systems for Arabic across text and structure. Systems should output Markdown, with tables represented in HTML for best comparability. The paper recommends prompts tailored for document understanding when evaluating multimodal LLMs, and it specifies that normalization should be applied before scoring according to the documented post-processing steps (Hennara et al., 17 Sep 2025).

For researchers, the benchmark is best understood as a final-stage evaluation resource rather than a development dataset. Because there are no official train/validation/test splits, using it as a model-selection set would blur its role as an external benchmark. Because the paper does not release an official evaluation harness, reproducible use requires faithful reimplementation of the normalization protocol and careful handling of Markdown/HTML equivalences.

The benchmark is also structurally revealing in how it frames Arabic OCR itself. Misraj-DocOCR does not treat OCR as isolated line transcription; it treats it as a document-conversion problem in which headings, lists, tables, and layout fidelity matter. That framing is consistent with recent document-understanding research in Arabic, where the difficulty of PDF-to-Markdown conversion remains substantial even for strong multimodal systems, and where structure-sensitive metrics are necessary to expose errors that text-only metrics conceal (Heakl et al., 20 Feb 2025).

In summary, Misraj-DocOCR occupies a specific place in the Arabic OCR ecosystem: a 400-image, expert-verified, document-to-Markdown benchmark for printed Arabic pages, designed to measure both textual correctness and structural fidelity, and introduced in direct response to annotation unreliability and limited diversity in earlier resources. Its main significance lies less in scale than in curation, evaluation rigor, and its insistence that Arabic OCR benchmarking must account for document structure as well as text (Hennara et al., 17 Sep 2025).