DharmaOCR Benchmark: Structured OCR Evaluation

• DharmaOCR-Benchmark is a specialized evaluation suite for OCR that unifies metrics for fidelity, structure accuracy, degeneration rate, and unit cost.
• It covers printed, handwritten, and legal documents using a strict JSON schema and a held-out set of 496 pages for unbiased assessment.
• It integrates an innovative DPO training method to penalize generation degeneration, achieving robust quality-cost trade-offs compared to other OCR systems.

Searching arXiv for the specified paper and the cited DPO method. Searching for (Cardoso et al., 15 Apr 2026). DharmaOCR-Benchmark is a held-out benchmark for structured OCR introduced together with DharmaOCR Full and DharmaOCR Lite in "DharmaOCR: Specialized Small LLMs for Structured OCR that outperform Open-Source and Commercial Baselines" (Cardoso et al., 15 Apr 2026). It covers printed, handwritten, and legal/administrative documents, and is designed around a unified evaluation protocol that jointly measures fidelity, structure accuracy, degeneration rate, and unit cost. Within that framework, text degeneration is treated as a first-class benchmark metric rather than a secondary failure mode, and the associated training methodology combines Supervised Fine-Tuning (SFT) for a strict JSON schema with Direct Preference Optimization (DPO) that explicitly penalizes degenerate generations.

1. Corpus design and benchmark composition

The underlying fine-tuning corpus contains 39 680 pages in Brazilian Portuguese, drawn from public (e.g. arXiv) + proprietary sources, with a training split (≈ 85 %) and a held-out evaluation split (≈ 15 %) (Cardoso et al., 15 Apr 2026). Annotation was produced through prompted extraction via three LLMs (Claude Sonnet 4, llama-4-maverick-17b, Gemini 2.5 Pro) followed by human verification. The output format is a strict JSON schema with four fields:

$\in [0,1]$1

DharmaOCR-Benchmark itself is a distinct final evaluation set of $N = 496$ pages total, explicitly described as never seen during any training, DPO pair construction, or quantization. It has no further train/dev/test splits—the entire 496 instances serve as the benchmark. Its three subsets are as follows:

Subset	Description	Samples
ESTER-Pt	printed Portuguese pages	363
Legal/administrative	internal, 2-stage LLM+manual audit	83
BRESSAY	handwritten	50

The labels were produced by multiple LLMs + full manual audit for Legal subset. This organization places printed, handwritten, and legal/administrative material within a single evaluation target, rather than treating them as separate tasks.

2. Unified evaluation protocol

Each model is evaluated on four complementary dimensions under a single protocol: fidelity (text-transcription quality), structure accuracy (correct JSON schema-adherence), degeneration rate (stability), and unit cost (inference economic efficiency) (Cardoso et al., 15 Apr 2026).

The fidelity metric is the DharmaOCR-Benchmark Score, which aggregates character-level and n-gram sequence fidelity:

$$\text{Score} = \frac{\text{Levenshtein\_Ratio} + \text{BLEU}}{2}.$$

Here, Levenshtein_Ratio $\in [0,1]$ is defined as

$$1 - \frac{\text{EditDistance}(\text{predicted}, \text{reference})}{\text{max\_len}},$$

and BLEU $\in [0,1]$ is the corpus-level BLEU score on tokenized sequences.

Structure accuracy is defined as the fraction of outputs that both parse as valid JSON and assign each of the four fields ("header," "margin," "footer," "text") either a string or null exactly as prescribed by the schema:

$$\text{StructureAccuracy} = \frac{1}{N}\sum_{i=1}^{N} I_{\text{schema}(i)}.$$

Degeneration rate is defined operationally. A request is flagged as degenerate if and only if both conditions hold: (1) it hits the configured maximum-token limit (8 192) without emitting an EOS token, and (2) it exhibits repeated spans, detected via an n-gram repetition detector. If $D$ is the number of degenerate requests and $N = 496$, then

$\text{DegenerationRate} = \frac{D}{N}\times 100\%.$

Unit cost is reported as unit cost per million pages. For locally hosted models on a GPU with hourly price $P_{\rm hour}$ and average time per page $$\text{Score} = \frac{\text{Levenshtein\_Ratio} + \text{BLEU}}{2}.$$0 in seconds,

$$\text{Score} = \frac{\text{Levenshtein\_Ratio} + \text{BLEU}}{2}.$$1

For commercial APIs priced per 1 000 pages at $$\text{Score} = \frac{\text{Levenshtein\_Ratio} + \text{BLEU}}{2}.$$2,

$$\text{Score} = \frac{\text{Levenshtein\_Ratio} + \text{BLEU}}{2}.$$3

For token-priced APIs, with average input/output tokens per page $$\text{Score} = \frac{\text{Levenshtein\_Ratio} + \text{BLEU}}{2}.$$4 and prices $$\text{Score} = \frac{\text{Levenshtein\_Ratio} + \text{BLEU}}{2}.$$5 in dollars per million tokens,

$$\text{Score} = \frac{\text{Levenshtein\_Ratio} + \text{BLEU}}{2}.$$6

This protocol makes stability and economic efficiency coequal with extraction quality, rather than downstream or auxiliary reporting dimensions.

3. Degeneration as an explicit benchmark target

A central methodological feature is the treatment of text degeneration as a first-class metric (Cardoso et al., 15 Apr 2026). The manuscript states that degeneration is not just a rare quality glitch—it materially impacts latency, throughput, and cost, with Appendix A.1 cited for queuing and VRAM effects. The reported examples include infinite loops of table rows and repeated numeric lines.

The benchmark therefore does not limit itself to transcription fidelity and schema correctness. Degeneration is explicitly tracked using the formal criterion above, and production relevance is emphasized: abnormal generations increase response time, reduce throughput, and inflate computational cost because they continue until the maximum-token limit is reached.

This framing is consequential for OCR evaluation. A plausible implication is that a model with acceptable transcription quality but unstable generation dynamics may still be unsuitable for deployment, because the benchmark interprets repeated-span failures as operationally significant rather than as isolated decoding anomalies.

4. DPO-based mitigation and schema-constrained training

The training methodology combines Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) (Cardoso et al., 15 Apr 2026). SFT is used for enforcing a strict JSON schema (header, margin, footer, and text). DPO is then applied to reduce degeneration, with the manuscript stating: "To the best of the author's knowledge, as a methodological contribution, this is the first application of Direct Preference Optimization (DPO) for OCR, explicitly using degenerate generations as rejected examples to penalize looping behavior."

During DPO pair construction (§3.4), any generation containing degeneration is actively selected as the "rejected" example. The DPO objective is given as

$$\text{Score} = \frac{\text{Levenshtein\_Ratio} + \text{BLEU}}{2}.$$7

where $$\text{Score} = \frac{\text{Levenshtein\_Ratio} + \text{BLEU}}{2}.$$8 is the preferred high-quality response and $$\text{Score} = \frac{\text{Levenshtein\_Ratio} + \text{BLEU}}{2}.$$9 is the rejected response, often degenerate. The manuscript states that by including degenerate $\in [0,1]$0, DPO explicitly pushes down its probability, "unlearning" looping attractors.

The reported empirical result is that combined with SFT, DPO consistently reduces degeneration rate across model families (up to 87.6% relative) while preserving or improving extraction quality. This establishes a direct connection between preference optimization and structured OCR robustness, not merely stylistic alignment.

5. Comparative performance on DharmaOCR-Benchmark

The benchmark compares fine-tuned models (vanilla → SFT → SFT+DPO → Quant), open-source vanilla models, and commercial OCR services (Cardoso et al., 15 Apr 2026). Among the open-source baselines, the reported DharmaOCR-Benchmark scores and degeneration rates are:

Model	Score	Degeneration rate
Qwen3-VL-8B-Instruct	0.829	5.65 %
Dots OCR	0.738	6.85 %
GLM-OCR	0.710	11.69 %
Qwen3-VL-2B-Instruct	0.623	11.69 %
DeepSeek-OCR	0.196	21.98 %

For commercial models, the reported scores are Claude-Opus-4.6: 0.833, Gemini-3.1-Pro: 0.820, GPT-5.4: 0.750, Google Vision: 0.686, Google Document AI: 0.640, GPT-4o: 0.635, Amazon Textract: 0.618, and Mistral OCR 3: 0.574. The manuscript notes that degeneration rates are not exposed for commercial APIs.

The two principal DharmaOCR models are reported as follows. DharmaOCR Full = olmOCR-2-7B (SFT + DPO + AWQ-Quant) reaches Score 0.925, DegRate 0.40 %, and Time 2.132 s/page, corresponding to 49.50 % relative cost vs. olmOCR-2-7B vanilla. The manuscript describes this as new SOTA. It also states that DharmaOCR Full outperforms the top open-source baseline by ≈ 0.003 in score and reduces degeneration from ~1 % to 0.40 %.

DharmaOCR Lite = Nanonets-OCR2-3B (SFT + DPO + AWQ-Quant) reaches Score 0.911, DegRate 0.20 %, and Time 1.464 s/page, corresponding to 33.99 % relative cost vs. Nanonets-vanilla. The reported characterization is that, despite having ~3 B parameters, it outperforms all commercial services and open-source OCR models. Its score is only 0.010 below Full, while its degeneration rate is the lowest of all models.

6. AWQ quantization and quality-cost trade-offs

The manuscript applies Activation-aware Weight Quantization (AWQ) to the two best models in order to explore quality–efficiency trade-offs (Cardoso et al., 15 Apr 2026). The reported appendix results distinguish several quantization modes.

For olmOCR-2-7B (SFT+DPO), the quantized variants are:

• AWQ W8A8 8/8 bits: Score 0.926, DegRate 0.60 %, 2.122 s/page
• AWQ FP8 8-bit float: 0.925, 1.21 %, 2.187 s
• AWQ W4A16 4-bit weights, 16-bit acts: 0.903, 1.61 %, 2.004 s

The manuscript summarizes this configuration by stating that W8A8 yields a 22 % cost reduction with only –0.002 score loss.

For Nanonets-OCR2-3B (SFT+DPO), the quantized variants are:

• AWQ FP8: 0.912, 0.20 %, 1.463 s
• AWQ W8A8: 0.903, 1.81 %, 1.500 s
• AWQ W4A16: 0.831, 11.09 %, 1.821 s

The manuscript’s overall summary is that AWQ enables ~20–25 % unit-cost savings with negligible quality degradation (–0.002 in score) and without increasing the degeneration rate of the best configurations. More specifically, the abstract states that AWQ quantization reduced up to 22% per-page cost with negligible quality loss, enabling a strong quality-cost trade-off in comparison to proprietary OCR APIs and open-source alternatives.

7. Position within structured OCR evaluation

DharmaOCR-Benchmark is defined not simply by its document mixture but by the way it formalizes structured OCR as a joint problem of transcription fidelity, schema conformity, generation stability, and inference cost (Cardoso et al., 15 Apr 2026). The benchmark’s schema—header, margin, footer, and text—anchors evaluation in structured extraction rather than plain text recovery, and the use of a single held-out set of 496 instances provides one common basis for comparing fine-tuned, open-source, and commercial systems.

The benchmark also re-specifies model failure. In this setting, a malformed or looping output is not merely counted as low-quality text; it is measured through StructureAccuracy and DegenerationRate, and it is tied to explicit latency and cost consequences. This suggests a broader evaluation stance in which OCR systems are judged as generative systems with operational constraints, not only as text recognizers.

Within that framing, the reported results position DharmaOCR Full and Lite as the strongest systems evaluated on the benchmark, with the smaller model retaining near-top fidelity while attaining the lowest degeneration rate. The benchmark therefore functions simultaneously as an evaluation suite and as a methodological vehicle for the claim that structured OCR quality, stability, and economic efficiency can be optimized together.