Papers
Topics
Authors
Recent
Search
2000 character limit reached

ParseBench Benchmark Suite

Updated 3 July 2026
  • ParseBench is a set of open benchmarks designed to evaluate semantic extraction from diverse documents including scholarly articles, logs, PDFs, and enterprise records.
  • It covers tasks such as citation parsing, log template extraction, and PDF table/figure recovery using both traditional algorithms and advanced AI models.
  • The benchmarks employ heterogeneous datasets, adversarial sampling, and semantic evaluation metrics to rigorously assess structured-data extraction performance.

ParseBench is a collective term denoting a set of open benchmarks and evaluation protocols for document and data parsing systems, developed to address the increasing demand for reliable, semantically rich extraction in real-world, enterprise, and scientific workflows. Across its different instantiations and domains, ParseBench seeks to rigorously measure the structured-data extraction capabilities of both traditional algorithms and state-of-the-art AI models in scenarios spanning scholarly reference parsing, log template extraction, PDF table/figure/math extraction, and document understanding for autonomous agents (Zhang et al., 9 Apr 2026, Sarin et al., 26 Mar 2026, Zhu et al., 2018, Horn et al., 10 Dec 2025, Horn et al., 19 Mar 2026).

1. Evolution and Motivation

The emergence of enterprise automation, AI-driven agents, and large-scale scholarly infrastructure has fundamentally shifted the requirements for document parsing. The primary goal has evolved from mere text recovery to semantic correctness and structured fidelity, as domains such as insurance, finance, and metascience demand reliable mapping between unstructured document content and machine-interpretable structured representations. Traditional benchmarks often relied on synthetic data, narrow text similarity metrics, and under-represented the complexity of real-world documents, resulting in limited stress-testing of modern parsing systems (Zhang et al., 9 Apr 2026, Sarin et al., 26 Mar 2026).

ParseBench benchmarks were thus conceived to provide reproducible, heterogeneous, and robust evaluation standards, spanning citation parsing (RenoBench), log template mining, mathematical formula and table extraction from academic PDFs, and complex document parsing for AI agents. These benchmarks implement protocol-level advances, such as semantics-driven and LLM-mediated metrics, multilingual and multi-domain data, adversarial sampling, and human-in-the-loop corrections (Zhang et al., 9 Apr 2026, Sarin et al., 26 Mar 2026, Horn et al., 19 Mar 2026, Zhu et al., 2018, Horn et al., 10 Dec 2025).

2. Dataset Construction and Annotation

ParseBench benchmarks are built around the principle of domain and modality diversity, combining curated and programmatically generated datasets.

  • Citation Parsing (RenoBench): 10,000 citations sampled from 161,625 extracted pairs across SciELO, Redalyc, PKP, and Open Research Europe, filtered and balanced over language (eight languages, led by English, Portuguese, Spanish), publication type (journals, books), and persistent identifier presence. Annotation follows the JATS schema with tight field-level validation (Sarin et al., 26 Mar 2026).
  • Log Parsing: 16 real logs from LogHub, covering distributed systems, mobile, supercomputers, and server apps, each with 2,000 manually labeled templates (Zhu et al., 2018).
  • Math Formula Extraction: Synthetic PDFs generated from Wikipedia LaTeX, with precise formula location and structure annotations for 2,000+ formulas per 100-document benchmark, enabling absolute attribution of parsing errors (Horn et al., 10 Dec 2025).
  • Table and Figure Extraction: Synthetic datasets using scientific arXiv LaTeX tables, with controlled layout and complexity, produce ground-truth-aligned PDF sets for table extraction, matched and scored using LLMs (Horn et al., 19 Mar 2026).
  • Enterprise Documents: For agentic parsing, ∼2,000 human-verified pages from insurance, finance, and government, stratified by table/chart/formatting/grounding task and manually corrected by annotators (Zhang et al., 9 Apr 2026).

All benchmarks apply multi-stage data validation, with automated feature balancing and, in select cases, adversarial sampling (e.g., pages or books with high parser disagreement) to ensure that evaluated systems are stress-tested beyond uniform/easy cases (Yang et al., 31 May 2026, Zhang et al., 9 Apr 2026).

3. Task Definitions and Target Outputs

ParseBench encompasses a broad spectrum of parsing tasks, each grounded in well-defined, real-world outputs:

  • Citation Parsing: Convert plain-text reference strings to JATS-structured XML (fields: author names, article title, source, volume/issue, pub-id, etc.).
  • Log Message Parsing: Group unstructured log lines into templates, accurately identifying variable fields for downstream analytics (Zhu et al., 2018).
  • Formula/Table Extraction: Recover LaTeX or HTML-structured mathematical objects from rendered or scanned PDF content (Horn et al., 10 Dec 2025, Horn et al., 19 Mar 2026).
  • Enterprise Document Parsing: Extract full tables (including hierarchical headers and merged cells), chart data points (with series/axis matching and numeric tolerances), faithful page text (markdown with order and formatting), and produce localization bounding boxes for all elements, traceable to their source (Zhang et al., 9 Apr 2026).
  • Structured Data Extraction from NL Input: Parse emails or requests into structured JSON adhering to application schemas (LLMStructBench) (Tenckhoff et al., 16 Feb 2026).

Distinct ParseBench tracks formalize subtasks such as table record matching, chart data extraction, code block and LaTeX formatting retrieval, and precise visual grounding, reflecting the multi-dimensional requirements of enterprise- and research-grade parsing.

4. Evaluation Protocols and Metrics

ParseBench methodologies emphasize metrics that reflect semantic equivalence, structure preservation, and practical downstream utility.

  • Field/Element-Level Precision, Recall, F1: For structured outputs (e.g., citation fields, log template assignment), classic P/R/F1 are applied, but with context-specific interpretations and “lenient/strict” variants (e.g., XML tag presence for JATS-grounded fields) (Sarin et al., 26 Mar 2026, Zhu et al., 2018).
  • Semantically-Driven Table and Math Metrics: Adoption of LLM-judge paradigms, where models (e.g., GPT-5-mini, Gemini-3-Flash) rate semantic preservation on scale 0–10, outclassing conventional Tree Edit Distance Similarity (TEDS, r=0.68 vs r=0.93 correlation with human ratings) (Horn et al., 19 Mar 2026, Horn et al., 10 Dec 2025).
  • Task-Specific Metrics: TableRecordMatch, ChartDataPointMatch, and content faithfulness scores capture the fidelity of mapped records, correct data extraction from charts, and comprehensive content reproduction, often leveraging bipartite matching and rule-based scoring over element alignment (Zhang et al., 9 Apr 2026).
  • Visual Grounding: Element Pass Rate (EPR) combines IoA-based localization, class attribution, and content-level F1 for robust spatial provenance evaluation.
  • Composite Metrics: Aggregate dimension scores (tables, charts, content faithfulness, semantic formatting, grounding) are averaged to produce overall scores in multipurpose benchmarks (cf. “Overall” in (Zhang et al., 9 Apr 2026)), or trackwise means in robustness benchmarks (cf. Avg₃ in (Li et al., 8 May 2026)).

Notably, precision in ground truth is actively scrutinized; manual audits often reveal ground truth omissions leading to underestimated performance, especially in recall-favored pipelines (Sarin et al., 26 Mar 2026).

5. Benchmarking Results and Model Comparisons

Empirical findings across ParseBench variants consistently reveal fragmented strengths among parsing systems.

  • Citation Parsing: Fine-tuned small LLMs (Qwen3-0.6B + LoRA) outperform traditional systems (GROBID) on author and source fields, while large generic LLMs dominate numeric fields but struggle with noisy name splitting (Sarin et al., 26 Mar 2026).
  • Log Parsing: Drain-based templates achieve the highest mean accuracy and robustness; however, no system is optimal for all log types and scales. Efficiency and accuracy degrade for parsers under large-template or high-volume logs (Zhu et al., 2018).
  • Tables and Math: Top vision-LLMs (Gemini 3 Pro, Qwen3-VL, PaddleOCR-VL) achieve mean LLM-judge scores >9.6/10, while classic rule-based parsers exhibit lower semantic fidelity (<8.0). Table complexity increases error rates for most models, but multi-modal models maintain higher robustness (Horn et al., 19 Mar 2026, Horn et al., 10 Dec 2025).
  • Enterprise Document Parsing: No method demonstrates uniform excellence across table, chart, text, formatting, and grounding axes. LlamaParse Agentic achieves the best aggregate performance (all-dimension lead except one), with pipeline and open-VLM methods splitting individual tasks (Zhang et al., 9 Apr 2026).
  • Domain and Noise Robustness: General-purpose VLMs lose less accuracy under digital and real-world degradation compared to specialist pipelines; clean-only evaluations are shown to overstate deploy-time performance by up to 14 points on the scale (Li et al., 8 May 2026).

6. Limitations and Forward-Looking Directions

ParseBench benchmarks expose both the boundaries and promising directions of current parsing technology:

  • Ground Truth Completeness: Incompleteness of ground truth—especially for optional/nested fields or layout-sensitive attributes—limits the reliability of precision-based scores. Manual audits and LLM-assisted validation are required to counteract false error inflation (Sarin et al., 26 Mar 2026).
  • Language and Domain Coverage: While benchmarks cover multiple languages and document classes, low-resource languages, non-Latin scripts, and rare enterprise templates remain underrepresented (Sarin et al., 26 Mar 2026, Yang et al., 31 May 2026).
  • Multimodal and Adversarial Complexity: Significant gaps persist in music/MusicXML, chemistry, cross-page table and reading order, rotated layouts, and visually challenging objects (small seals, handwriting) (Yang et al., 31 May 2026, Zhou et al., 21 May 2026).
  • LLM Evaluation Risk: LLM-semantic metrics demonstrably align better with human ratings but remain subject to prompt dependency, model drift, and local hallucination risks, requiring periodic recalibration (Horn et al., 10 Dec 2025, Horn et al., 19 Mar 2026).
  • Active and Human-in-the-Loop Annotation: Future ParseBench protocols recommend integration of active learning sampling and domain expert review for challenging and edge cases (Sarin et al., 26 Mar 2026).

Ongoing efforts include expanding benchmarks to new domains (forms, institutional repositories), improving ground-truth annotation depth, broadening language and script coverage, establishing downstream utility metrics, and designing parsing+reasoning composite tasks (Sarin et al., 26 Mar 2026, Yang et al., 31 May 2026, Zhang et al., 9 Apr 2026).

7. Reproducibility, Adoption, and Community Impact

ParseBench datasets, code, annotator instructions, and evaluation scripts are openly available on platforms such as HuggingFace and GitHub, permitting researchers to evaluate both commercial and open-source parsers, add new systems via unified Python interfaces, and extend protocols to new data types (Zhang et al., 9 Apr 2026, Horn et al., 10 Dec 2025, Horn et al., 19 Mar 2026, Zhu et al., 2018). The benchmarks have established themselves as references in their respective communities for rigorous, reproducible, and semantically meaningful evaluation of structured data extraction in academic and enterprise contexts.

References:

Topic to Video (Beta)

No one has generated a video about this topic yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Sign Up to Generate

Follow Topic

Get notified by email when new papers are published related to ParseBench.

Sign Up to Follow Topic by Email