GutenOCR-7B: Grounded Vision-Language OCR

• GutenOCR-7B is a grounded vision–language OCR model that unifies reading, detection, and text grounding via a natural-language prompt interface.
• It leverages the Qwen2.5-VL-7B backbone and multi-scenario training data to deliver significant improvements in detection (F1 from 0.111 to 0.787) and reading accuracy.
• The model outputs spatially precise JSON objects, enabling end-to-end document parsing and bridging traditional OCR with modern document understanding.

GutenOCR-7B is a grounded vision–LLM for document OCR derived by fully fine-tuning the Qwen2.5-VL-7B backbone on a broad, multi-scenario OCR corpus, yielding a single checkpoint that supports unified prompt-driven reading, detection, and text grounding for complex business, scientific, and synthetic documents. It demonstrates substantial gains in grounded OCR accuracy over its base VLM (Vision-LLM) and establishes a new empirical baseline for spatially aware OCR on challenging real-world layouts (Heidenreich et al., 20 Jan 2026).

1. Model Architecture and Functional Primitives

GutenOCR-7B leverages the Qwen2.5-VL-7B vision–language backbone (7B parameters) without architectural modifications, adapters, or LoRA modules during fine-tuning. The model comprises:

• Vision Encoder: NaViT-style Vision Transformer operating on a full 72 DPI document page, removing the need for tiling or cropping.
• Language Decoder: Instruction-tuned Qwen2.5 multimodal transformer supporting generation of long-sequence transcripts (up to 16,000 tokens) and JSON-structured outputs.
• Spatial Outputs: Direct support for predicting bounding box coordinates in absolute pixel units, enabling grounding of recognized text.

A central feature is the prompt-based interface: four OCR task primitives (full-page reading, detection, conditional detection, localized reading) are invoked by specifying task, schema, and output type via a natural-language template. All grounded outputs are emitted as JSON objects of the form {“text”:string, “bbox”:[x₁, y₁, x₂, y₂]}, making the system amenable to downstream structured document parsing (Heidenreich et al., 20 Jan 2026).

2. Training Data Composition and Augmentation Strategy

The model is trained on a multi-source corpus spanning:

• Real-World Documents:
  • OCR-IDL: 4.6M documents (26M pages) of business forms and IDs.
  • TabMe++: 44.8k documents (122.5k pages) akin to business paperwork.
  • PubMed-OCR: 209.5k documents (1.5M pages) of multi-column scientific literature (including math and references).
• Synthetic Grounding Data:
  • SynthDoG Grounding: 1.2M pages featuring word-level layout perturbations and varied backgrounds.
  • Grounded LaTeX: 3.0M pages rendering isolated equations (Wikimedia source) at randomized positions, scales, and orientations.

Preprocessing routines include Unicode NFKC normalization, geometry standardization (clipping, discarding degenerate boxes), synthetic noise injection (blur, JPEG artifacts, geometric jitter, textures), and variation of prompt templates to enhance schema generalization (Heidenreich et al., 20 Jan 2026).

3. Fine-Tuning Curriculum and Optimization

GutenOCR-7B adopts a curriculum-based, staged fine-tuning protocol:

• Optimizer and Schedule: AdamW (learning rate 1e-6, weight decay 0.01), bfloat16 precision, ZeRO-3 memory optimization, gradient norm clipping (max 1.0), effective batch size of 128 pages, and greedy decoding (temperature=0) at inference.
• Progressive Staging:
  • Stage 1 (<2k tokens): Predominantly synthetic (SDG 6%, GL 48%), and OCR-IDL (45%).
  • Stage 2 (2k–8k tokens): OCR-IDL and TabMe++ only.
  • Stage 3a (2k–8k tokens): Blend of OCR-IDL, TabMe++, and PubMed-OCR.
  • Stage 3b (8k–16k tokens): Pure PubMed-OCR, emphasizing long-form scientific layouts.
• Early Stopping and Held-Out Evaluation: Early stopping is applied using 2,048 held-out samples per stage, with broader in-domain testing on an additional 10,500 pages (Heidenreich et al., 20 Jan 2026).

4. Grounded OCR Evaluation Protocol and Metrics

Performance is quantified using a task-family-specific metric suite:

Task Type	Primary Metric(s)	Aggregation/Notes
Full-page/structured reading	CER, WER, [email protected], CER_e2e	CER: raw char error; [email protected]: for localized spans (IoU≥0.5); CER_e2e: penalizes sequence/order errors
Detection	[email protected], [email protected]	Evaluated on predicted vs GT boxes, IoU≥0.5
Conditional detection	F1 (by query)	Query-image→BOX task performance, weighted equally

A composite grounded OCR score is then computed per sample: $$\mathrm{CompositeScore} = \frac{1}{3} (1-\mathrm{CER}_\mathrm{text}) + \frac{1}{3}\mathrm{F1}_\mathrm{detection} + \frac{1}{3}\mathrm{F1}_\mathrm{conditional}$$ which produces a summary measure over [0,1] that balances reading accuracy and spatial/conditional grounding performance (Heidenreich et al., 20 Jan 2026).

5. Performance Benchmarks and Comparative Analysis

In-domain (Business/Scientific):

Relative to its Qwen2.5-VL-7B backbone, GutenOCR-7B more than doubles composite OCR performance on 10.5k held-out real-world pages (0.396→0.819). Character error rates for structured lines drop from 0.633 to 0.147 ([email protected]), and detection F1 jumps from 0.111 to 0.787. Conditional detection F1 reaches 0.882 (Heidenreich et al., 20 Jan 2026).

Fox Benchmark (English subtasks):

• Region-level CER improves from 0.163 to 0.067; line CER from 0.701 to 0.211.
• Page F1 remains above 0.97, but page CER rises (0.025→0.164) due to nontrivial linearization mismatches with Fox’s markdown reference.
• Color-guided OCR fails (0.109→0.963 CER), attributed to zero color-pointer examples in training.

OmniDocBench v1.5 Stress Tests:

• Large recall increase on line-level text detection ([email protected] improves from ≈0.02 to ≈0.55).
• Formula recognition (CDM, CER) exhibits mild negative transfer: formula-heavy supervision is not retained across all training stages, leading to slightly degraded performance on math-intensive pages (Heidenreich et al., 20 Jan 2026).

6. Failure Modes and Domain Trade-Offs

GutenOCR-7B reveals key trade-offs:

• Page Linearization vs Layout Sensitivity:

Optimized for TEXT2D/structured LINES; accurate spatial grouping yields high F1 but misaligns with flat, reference Markdown orderings, increasing page-level CER.

• Catastrophic Forgetting (Color-Guided OCR):

Fine-tuning erases the base VLM’s colored pointer-reading behavior; mitigated by adding explicit color-pointer examples in future recipes.

• Formula-Heavy Degradation:

Dedicated formula grounding in early training is not maintained; math accuracy drops after shift to generic OCR objectives.

• Domain Mismatch on Diverse Visuals:

Multi-color backgrounds and visually challenging layouts (magazines, slides) produce higher CER due to limited training diversity.

• Lack of Table/Discourse Structure:

Outputs are limited to lines and paragraphs, not supporting deep structural parsing (tables, row headers, cross-page anchors) (Heidenreich et al., 20 Jan 2026).

This suggests that targeted data augmentation, prompt expansion, and staged task-specific recurrency during fine-tuning are necessary to retain and extend multimodal capabilities.

7. Historical and Technical Context: Scaling OCR to New Document Domains

GutenOCR-7B occupies a space between traditional line-level OCR and modern VLM-based document understanding systems. Earlier neural OCR systems such as OCRopus, as characterized in the RIDGES herbal corpus work, employ BiLSTM-CTC stacks trained on ground-truth line images with substantial data augmentation and postcorrection. These classical models optimize for character error and require document segmentation, but struggle with layout complexity and do not natively unify detection and recognition (Springmann et al., 2016).

By contrast, GutenOCR-7B is conceived as a prompt-driven, fully grounded front-end, bridging detection and recognition in a single VLM context. Its JSON-output interface and spatial comprehension sidestep the need for rigid segmentation, while its composite evaluation metric rewards end-to-end extraction performance—aligning with contemporary efforts to build “document holograms” that support evidence-grounded, multi-level querying (Heidenreich et al., 20 Jan 2026).

A plausible implication is that such models, when extended with broader structural supervision (tables, semantically typed fields), could operationalize end-to-end pipelines for large-scale, heterogeneous archival and scientific document corpora, building on the modular design and iterative postcorrection strategies pioneered by OCRopus-based historical OCR systems (Springmann et al., 2016).