--- title: 'DharmaOCR Benchmark: Structured OCR Evaluation' url: https://www.emergentmind.com/topics/dharmaocr-benchmark type: topic --- # DharmaOCR Benchmark: Structured OCR Evaluation Searching arXiv for the specified paper and the cited DPO method. Searching for 2604.14314. DharmaOCR-Benchmark is a held-out benchmark for structured OCR introduced together with DharmaOCR Full and DharmaOCR Lite in "DharmaOCR: Specialized Small Language Models for Structured OCR that outperform Open-Source and Commercial Baselines" [2604.14314]. It covers printed, handwritten, and legal/administrative documents, and is designed around a unified evaluation protocol that jointly measures fidelity, structure accuracy, degeneration rate, and unit cost. Within that framework, text degeneration is treated as a first-class benchmark metric rather than a secondary failure mode, and the associated training methodology combines Supervised Fine-Tuning (SFT) for a strict JSON schema with Direct Preference Optimization (DPO) that explicitly penalizes degenerate generations. ## 1. Corpus design and benchmark composition The underlying fine-tuning corpus contains **39 680 pages** in **Brazilian Portuguese**, drawn from **public (e.g. arXiv) + proprietary** sources, with a **training split (≈ 85 %)** and a **held-out evaluation split (≈ 15 %)** [2604.14314]. Annotation was produced through **prompted extraction via three LLMs (Claude Sonnet 4, llama-4-maverick-17b, Gemini 2.5 Pro)** followed by **human verification**. The output format is a strict JSON schema with four fields: ```json { "header": "", "margin": "", "footer": "", "text": "" } ``` DharmaOCR-Benchmark itself is a distinct final evaluation set of **\(N = 496\) pages total**, explicitly described as **never seen during any training, DPO pair construction, or quantization**. It has **no further train/dev/test splits—the entire 496 instances serve as the benchmark**. Its three subsets are as follows: | Subset | Description | Samples | |---|---|---:| | ESTER-Pt | printed Portuguese pages | 363 | | Legal/administrative | internal, 2-stage LLM+manual audit | 83 | | BRESSAY | handwritten | 50 | The labels were **produced by multiple LLMs + full manual audit for Legal subset**. This organization places printed, handwritten, and legal/administrative material within a single evaluation target, rather than treating them as separate tasks. ## 2. Unified evaluation protocol Each model is evaluated on **four complementary dimensions under a single protocol**: **fidelity (text-transcription quality)**, **structure accuracy (correct JSON schema-adherence)**, **degeneration rate (stability)**, and **unit cost (inference economic efficiency)** [2604.14314]. The fidelity metric is the **DharmaOCR-Benchmark Score**, which aggregates character-level and n-gram sequence fidelity: $$ \text{Score} = \frac{\text{Levenshtein\_Ratio} + \text{BLEU}}{2}. $$ Here, **Levenshtein\_Ratio \(\in [0,1]\)** is defined as $$ 1 - \frac{\text{EditDistance}(\text{predicted}, \text{reference})}{\text{max\_len}}, $$ and **BLEU \(\in [0,1]\)** is the **corpus-level BLEU score on tokenized sequences**. Structure accuracy is defined as the fraction of outputs that both **parse as valid JSON** and **assign each of the four fields ("header," "margin," "footer," "text") either a string or null exactly as prescribed by the schema**: $$ \text{StructureAccuracy} = \frac{1}{N}\sum_{i=1}^{N} I_{\text{schema}(i)}. $$ Degeneration rate is defined operationally. A request is flagged as **degenerate** if and only if both conditions hold: **(1) it hits the configured maximum-token limit (8 192) without emitting an EOS token, and (2) it exhibits repeated spans, detected via an n-gram repetition detector**. If \(D\) is the number of degenerate requests and \(N = 496\), then $$ \text{DegenerationRate} = \frac{D}{N}\times 100\%. $$ Unit cost is reported as **unit cost per million pages**. For locally hosted models on a GPU with hourly price \(P_{\rm hour}\) and average time per page \(T_{\rm avg}\) in seconds, $$ C_{1M} = 10^6 \times T_{\rm avg} \times \frac{P_{\rm hour}}{3600}. $$ For commercial APIs priced per 1 000 pages at \(P_k\), $$ C_{1M} = 10^6 \times \frac{P_k}{1000}. $$ For token-priced APIs, with average input/output tokens per page \(n_{\rm in}, n_{\rm out}\) and prices \(p_{\rm in}, p_{\rm out}\) in dollars per million tokens, $$ C_{1M} = 10^6 \left(\frac{n_{\rm in}\,p_{\rm in}}{10^6} + \frac{n_{\rm out}\,p_{\rm out}}{10^6}\right). $$ This protocol makes stability and economic efficiency coequal with extraction quality, rather than downstream or auxiliary reporting dimensions. ## 3. Degeneration as an explicit benchmark target A central methodological feature is the treatment of **text degeneration as a first-class metric** [2604.14314]. The manuscript states that degeneration is **not just a rare quality glitch—it materially impacts latency, throughput, and cost**, with Appendix A.1 cited for **queuing and VRAM effects**. The reported examples include **infinite loops of table rows** and **repeated numeric lines**. The benchmark therefore does not limit itself to transcription fidelity and schema correctness. Degeneration is explicitly tracked using the formal criterion above, and production relevance is emphasized: abnormal generations increase response time, reduce throughput, and inflate computational cost because they continue until the maximum-token limit is reached. This framing is consequential for OCR evaluation. A plausible implication is that a model with acceptable transcription quality but unstable generation dynamics may still be unsuitable for deployment, because the benchmark interprets repeated-span failures as operationally significant rather than as isolated decoding anomalies. ## 4. DPO-based mitigation and schema-constrained training The training methodology combines **Supervised Fine-Tuning (SFT)** and **Direct Preference Optimization (DPO)** [2604.14314]. SFT is used **for enforcing a strict JSON schema (header, margin, footer, and text)**. DPO is then applied to reduce degeneration, with the manuscript stating: **"To the best of the author's knowledge, as a methodological contribution, this is the first application of Direct Preference Optimization (DPO) for OCR, explicitly using degenerate generations as rejected examples to penalize looping behavior."** During **DPO pair construction (§3.4)**, **any generation containing degeneration is actively selected as the "rejected" example**. The DPO objective is given as $$ \mathcal{L}_{\rm DPO}(\pi_\theta;\,\pi_{\rm ref}) = -\mathbb{E}_{(x,y_w,y_\ell)}\biggl[ \log \sigma\!\Bigl( \beta\bigl( \log \frac{\pi_\theta(y_w|x)}{\pi_{\rm ref}(y_w|x)} -\log \frac{\pi_\theta(y_\ell|x)}{\pi_{\rm ref}(y_\ell|x)} \bigr) \Bigr) \biggr], $$ where \(y_w\) is the preferred high-quality response and \(y_\ell\) is the rejected response, often degenerate. The manuscript states that by including degenerate \(y_\ell\), **DPO explicitly pushes down its probability, "unlearning" looping attractors**. The reported empirical result is that **combined with SFT**, DPO **consistently reduces degeneration rate across model families (up to 87.6% relative) while preserving or improving extraction quality**. This establishes a direct connection between preference optimization and structured OCR robustness, not merely stylistic alignment. ## 5. Comparative performance on DharmaOCR-Benchmark The benchmark compares **fine-tuned models (vanilla → SFT → SFT+DPO → Quant)**, **open-source vanilla models**, and **commercial OCR services** [2604.14314]. Among the open-source baselines, the reported DharmaOCR-Benchmark scores and degeneration rates are: | Model | Score | Degeneration rate | |---|---:|---:| | Qwen3-VL-8B-Instruct | 0.829 | 5.65 % | | Dots OCR | 0.738 | 6.85 % | | GLM-OCR | 0.710 | 11.69 % | | Qwen3-VL-2B-Instruct | 0.623 | 11.69 % | | DeepSeek-OCR | 0.196 | 21.98 % | For commercial models, the reported scores are **Claude-Opus-4.6: 0.833**, **Gemini-3.1-Pro: 0.820**, **GPT-5.4: 0.750**, **Google Vision: 0.686**, **Google Document AI: 0.640**, **GPT-4o: 0.635**, **Amazon Textract: 0.618**, and **Mistral OCR 3: 0.574**. The manuscript notes that **degeneration rates are not exposed for commercial APIs**. The two principal DharmaOCR models are reported as follows. **DharmaOCR Full = olmOCR-2-7B (SFT + DPO + AWQ-Quant)** reaches **Score 0.925**, **DegRate 0.40 %**, and **Time 2.132 s/page**, corresponding to **49.50 % relative cost vs. olmOCR-2-7B vanilla**. The manuscript describes this as **new SOTA**. It also states that DharmaOCR Full **outperforms the top open-source baseline by ≈ 0.003 in score and reduces degeneration from ~1 % to 0.40 %**. **DharmaOCR Lite = Nanonets-OCR2-3B (SFT + DPO + AWQ-Quant)** reaches **Score 0.911**, **DegRate 0.20 %**, and **Time 1.464 s/page**, corresponding to **33.99 % relative cost vs. Nanonets-vanilla**. The reported characterization is that, **despite having ~3 B parameters, it outperforms all commercial services and open-source OCR models**. Its score is **only 0.010 below Full**, while its degeneration rate is **the lowest of all models**. ## 6. AWQ quantization and quality-cost trade-offs The manuscript applies **Activation-aware Weight Quantization (AWQ)** to the two best models in order to explore **quality–efficiency trade-offs** [2604.14314]. The reported appendix results distinguish several quantization modes. For **olmOCR-2-7B (SFT+DPO)**, the quantized variants are: - **AWQ W8A8 8/8 bits**: **Score 0.926**, **DegRate 0.60 %**, **2.122 s/page** - **AWQ FP8 8-bit float**: **0.925**, **1.21 %**, **2.187 s** - **AWQ W4A16 4-bit weights, 16-bit acts**: **0.903**, **1.61 %**, **2.004 s** The manuscript summarizes this configuration by stating that **W8A8 yields a 22 % cost reduction with only –0.002 score loss**. For **Nanonets-OCR2-3B (SFT+DPO)**, the quantized variants are: - **AWQ FP8**: **0.912**, **0.20 %**, **1.463 s** - **AWQ W8A8**: **0.903**, **1.81 %**, **1.500 s** - **AWQ W4A16**: **0.831**, **11.09 %**, **1.821 s** The manuscript’s overall summary is that **AWQ enables ~20–25 % unit-cost savings with negligible quality degradation (–0.002 in score) and without increasing the degeneration rate of the best configurations**. More specifically, the abstract states that **AWQ quantization reduced up to 22% per-page cost with negligible quality loss, enabling a strong quality-cost trade-off in comparison to proprietary OCR APIs and open-source alternatives**. ## 7. Position within structured OCR evaluation DharmaOCR-Benchmark is defined not simply by its document mixture but by the way it formalizes structured OCR as a joint problem of **transcription fidelity**, **schema conformity**, **generation stability**, and **inference cost** [2604.14314]. The benchmark’s schema—**header, margin, footer, and text**—anchors evaluation in structured extraction rather than plain text recovery, and the use of a single held-out set of **496 instances** provides one common basis for comparing fine-tuned, open-source, and commercial systems. The benchmark also re-specifies model failure. In this setting, a malformed or looping output is not merely counted as low-quality text; it is measured through **StructureAccuracy** and **DegenerationRate**, and it is tied to explicit latency and cost consequences. This suggests a broader evaluation stance in which OCR systems are judged as generative systems with operational constraints, not only as text recognizers. Within that framing, the reported results position DharmaOCR Full and Lite as the strongest systems evaluated on the benchmark, with the smaller model retaining near-top fidelity while attaining the lowest degeneration rate. The benchmark therefore functions simultaneously as an evaluation suite and as a methodological vehicle for the claim that structured OCR quality, stability, and economic efficiency can be optimized together. Source: https://www.emergentmind.com/topics/dharmaocr-benchmark