olmOCR2-synthmix-1025: Synthetic OCR Benchmark

• olmOCR2-synthmix-1025 is a synthetic corpus designed for evaluating document OCR systems in complex, layout-rich environments using VLM-generated HTML.
• It utilizes a three-stage PDF-to-HTML conversion pipeline that produces verifiable binary unit tests for text, tables, mathematics, and layout structures.
• The dataset is optimized for reinforcement learning protocols, enabling precise benchmarking and improvement of OCR models through automated, test-driven evaluation.

The olmOCR2-synthmix-1025 dataset is a synthetic corpus specifically constructed to support the development and evaluation of document optical character recognition (OCR) systems in complex, layout-rich settings. All supervision within this dataset is derived from vision LLM (VLM)–generated HTML, providing a controlled environment for reinforcement learning with verifiable, binary unit tests. olmOCR2-synthmix-1025 is released under the Apache 2.0 license and is intended for RL-based training, benchmarking, and evaluation of document OCR models that require granular layout and semantic fidelity (Poznanski et al., 22 Oct 2025).

1. Dataset Generation Pipeline

Construction of the olmOCR2-synthmix-1025 dataset begins with PDF sourcing targeted at maximizing layout complexity. Documents are chosen specifically to stress-test OCR capabilities in parsing mathematics, tables, and multi-column layouts. For instance, arXiv papers are sampled for mathematical content, while documents with complex tables or multi-column text are targeted for evaluation of those features.

The PDF-to-HTML conversion pipeline is orchestrated via claude-sonnet-4-20250514 and operates in three stages:

1. Layout analysis: The VLM receives a rasterized page image and is prompted to identify macro-structural elements, including the number of columns, presence and location of <header>/<footer> blocks, figures, tables, and math regions.
2. Content rendering: With the original page image and detected layout hints, the VLM emits a semantic HTML representation aligned to the page’s canonical canvas.
3. Output refinement: The generated HTML is rendered back to an image, which, alongside the original rasterized PDF image, is re-fed to the VLM for error correction or refinement of the HTML.

Following this pipeline, unit tests compatible with olmOCR-Bench are automatically extracted from the final HTML. Test types include:

• Text-presence/absence: Asserts the non-transference of header content into the main body.
• Natural reading order: Checks for correct sequencing of specified phrases.
• Table accuracy: Compares cell values and their positional relationships.
• Math-formula accuracy: Renders each KaTeX block and checks for bounding-box layout correspondence.
• Baseline robustness: Detects spurious repeated n-grams or extraneous scripts.

All unit tests are derived from synthetic HTML ground truth, ensuring that VLM hallucinations do not contaminate the evaluation process (Poznanski et al., 22 Oct 2025).

2. Composition, Layouts, and Statistics

olmOCR2-synthmix-1025 consists of 2,186 rasterized PDF pages, furnishing a total of 30,381 binary unit tests, with an average of approximately 13.9 unit tests per page. The composition of layouts demonstrates substantial diversity, reflecting:

• Single-column and multi-column article formats
• Pages containing zero to over five mathematical formulas
• Tables ranging from 2×2 to 10×8 in structure
• Inclusion of headers, footers, captions, and figures

The dataset is exclusively synthetic, derived wholly from VLM-generated HTML without any admixture of human-annotated pages. For comparison, a related but discrete resource ("olmOCR-mix-1025") encompasses 267,962 real, human-verified pages for supervised fine-tuning (SFT). This suggests the intent is to provide both a highly controllable synthetic cohort and an expansive real-world corpus for complementary training paradigms (Poznanski et al., 22 Oct 2025).

3. Dataset Format and Structure

olmOCR2-synthmix-1025 is distributed on Hugging Face as "allenai/olmOCR-synthmix-1025," with a consistent, page-centric organizational schema. Each page is packaged in a directory containing:

File	Content	Format/Role
image.png	Rendered synthetic page image	Model input/reference
page.html	Ground-truth semantic HTML	Supervision & unit tests
page.md	Reference Markdown/LaTeX conversion	Auxiliary evaluation
unit_tests.json	Array of binary unit test specifications	RL/reward computation
metadata.yaml	(Optional) PDF source, layout statistics	Provenance/statistics

HTML annotations are fully semantic, using tags such as <header>, <table><tr><td>, and <span class="math">. An illustrative HTML snippet combining math and table content is:

<section> <p>… some explanatory text …</p> <span class="math">%%%%3%%%%</span> <table> <tr> <th>Variable</th><th>Value</th> </tr> <tr> <td>m</td><td>1.0</td> </tr> <tr> <td>c</td><td>3e8</td> </tr> </table> </section>

A corresponding unit test in unit_tests.json is:

{ "type": "math_formula", "latex": "E = mc^2", "test_id": "page1234_formula1" }

4. Licensing, Distribution, and Recommended Use

olmOCR2-synthmix-1025 is made available under the Apache 2.0 license, providing permissive access for research and commercial utilization. The dataset, documentation, and associated scripts are accessible for download and inspection via Hugging Face at https://huggingface.co/datasets/allenai/olmOCR-synthmix-1025. Citation should reference Poznanski, Soldaini & Lo (2025), “Unit Test Rewards for Document OCR,” (Poznanski et al., 22 Oct 2025).

Recommended applications include:

• Reinforcement learning with verifiable rewards (RLVR) for OCR-focused VLMs
• Binary unit test–driven benchmarking of new and existing OCR systems
• Design and deployment of novel unit-test-based evaluation protocols (Poznanski et al., 22 Oct 2025)

5. Integration into Reinforcement Learning Workflows

The dataset is optimized for RL environments where output verifiability and coverage are critical. The reward computation protocol executes all $T$ unit tests for a model's Markdown output and records binary pass/fail outcomes. The page-level RL reward is defined as:

$$r_{\rm page} = \frac{1}{T}\sum_{i=1}^{T} \mathbf{1}\bigl[\text{test}_i\text{ passes}\bigr] \in [0,1]$$

Additional reward terms include a bonus if the model output concludes with an EOS token, as well as credit for correctly emitting required metadata fields.

The reinforcement learning algorithm employed is Group Relative Policy Optimization (GRPO) with KL-penalty $\beta=0.01$. The training regime includes a single epoch over synthmix-1025, with 28 sampled completions per page, and six random seeds. Final model performance is stabilized by “soupling” checkpoints via weight averaging (Poznanski et al., 22 Oct 2025).

6. Significance and Context

olmOCR2-synthmix-1025 constitutes a tightly controlled and reproducible synthetic benchmark for OCR research, enabling exhaustive and granular unit test–based reward computation. Its exclusive reliance on VLM-generated HTML as ground truth ensures that benchmarking is immune to human annotation biases and hallucinated artifacts. A plausible implication is that such a dataset enables fairer, more scalable evaluation and training for cutting-edge vision–language OCR models—especially in document domains where layout complexity and semantic integrity are critical. The methodology aligns with broader trends in leveraging synthetic corpora and automated supervision for state-of-the-art RL training in structured document understanding (Poznanski et al., 22 Oct 2025).