ExtractBench: PDF-to-JSON Extraction Benchmark
- ExtractBench is an open-source benchmark designed for evaluating enterprise PDF-to-JSON extraction with schema-aware, nested output evaluation.
- It employs field-specific metrics like exact matching, tolerance-based numeric evaluation, and semantic alignment to assess complex, deeply nested JSON outputs.
- The benchmark reveals that output validity alone is insufficient, emphasizing the challenges of schema breadth and nuanced evaluation in realistic document automation.
ExtractBench is an open-source benchmark and evaluation framework for PDF-to-JSON structured extraction that is intended to reflect the conditions of enterprise document automation rather than simplified document-understanding tasks. In its 2026 formulation, it evaluates document-grounded extraction from PDFs under enterprise-scale JSON Schemas and scores predictions with a schema-aware methodology in which the schema serves as an executable specification of both structure and field-level correctness (Ferguson et al., 12 Feb 2026). The name has also been used in earlier extraction benchmarking work, most notably for a multi-task, multi-domain evaluation framework for academic PDF information extraction built on DocBank (Meuschke et al., 2023). In current usage, however, the term most directly denotes the benchmark for realistic PDF-to-JSON extraction with nested outputs, heterogeneous field semantics, and explicit treatment of omission versus hallucination (Ferguson et al., 12 Feb 2026).
1. Scope, problem formulation, and benchmark intent
ExtractBench was introduced to address two gaps in structured extraction evaluation. First, there was no end-to-end benchmark for PDF-to-JSON extraction under enterprise-scale schema breadth. Second, there was no principled methodology for scoring nested extraction, where different fields require different notions of correctness, arrays require alignment, and missingness must be distinguished from hallucination (Ferguson et al., 12 Feb 2026).
The benchmark is explicitly designed around production-style conditions. The input is a long, noisy PDF; the output is a large, deeply nested JSON object; and correctness is not uniform across fields. The paper states that an identifier should be checked by exact match, a monetary value by tolerance-based numeric matching, and a person or company name by semantic equivalence. Arrays of objects must be aligned before they can be scored. This departs from earlier benchmarks that evaluated only fragments of the overall problem, such as span extraction, small fixed field sets, or schema conformance without document grounding (Ferguson et al., 12 Feb 2026).
A central design claim is that JSON Schema is not merely a serialization target. In ExtractBench, it is the formal specification of what should be extracted and how each field should be evaluated. The schema is parsed into an AST and traversed jointly with the gold and predicted JSON objects. This makes evaluation recursive, modular, and field-sensitive. A plausible implication is that ExtractBench treats structural extraction as a typed semantic matching problem rather than as uniform string comparison.
2. Dataset composition and domain structure
The dataset is intentionally small in sample count but high in structural complexity. It contains 35 PDF documents across 5 domains, spanning 2,076 pages and 12,867 evaluatable field values (Ferguson et al., 12 Feb 2026). The five domains are:
| Domain | Distinctive property |
|---|---|
| SEC 10-K/Q filings | 369 evaluatable fields |
| Credit agreements | 100–250 pages, 13 fields |
| Research papers | 16 fields, very large citation arrays |
| Professional resumes | 31 fields |
| Sports results | Schema depth 6 |
The benchmark is designed to expose failure modes caused by breadth, nesting, and array expansion rather than by sample scarcity alone. The five domains were chosen because they stress different extraction regimes. Sports results have the deepest schema but relatively regular structure. Credit agreements are long documents with only 13 fields, making them “needle-in-a-haystack” extraction problems. Research papers have only 16 fields but very large citation arrays that expand to roughly 25k output tokens. Professional resumes are semi-structured with 31 fields. SEC 10-K/Q filings are the hardest in schema breadth, with 369 evaluatable fields, and the paper describes them as an order of magnitude larger than prior enterprise benchmarks (Ferguson et al., 12 Feb 2026).
Gold-label creation was correspondingly expensive. The paper reports 67.9 hours of expert human annotation, of which 56.1 hours were spent on SEC filings alone (Ferguson et al., 12 Feb 2026). This is presented as evidence that high-quality ground truth becomes costly when schemas are broad and nested. This suggests that ExtractBench is intended less as a large-scale training corpus than as a diagnostic evaluation set for realistic structured extraction.
3. Schema-driven evaluation and field-specific metrics
ExtractBench’s defining methodological feature is schema-driven evaluation. Each node in the schema can specify its own metric via evaluation_config, and evaluation recurses over the schema AST jointly with the predicted and gold JSON objects (Ferguson et al., 12 Feb 2026). The paper gives examples such as:
1 |
\texttt{"borrower\_name": \{"type": "string", "evaluation\_config": "string\_semantic"\} |
and
1 |
\texttt{"principal": \{"type": "number", "evaluation\_config": \{"metric\_id": "number\_tolerance", "params": \{"tolerance": 0.001\}\}\} |
The metric library includes:
- exact and case-insensitive string matching
- Levenshtein-based fuzzy matching
- LLM-based semantic equivalence
- exact and tolerance-based numeric matching
- exact boolean matching
- semantic array alignment
This evaluation framework is motivated by the claim that a single global comparison rule is inadequate for nested extraction. An identifier, a free-form name, and a quantity do not admit the same notion of correctness. ExtractBench therefore formalizes correctness locally, at the field level, and composes those judgments recursively over the whole JSON object (Ferguson et al., 12 Feb 2026).
The paper reports two headline aggregate metrics. Valid JSON is the fraction of outputs that are parseable and schema-conforming. Pass Rate is the field-level correctness rate across all evaluated fields, with invalid outputs contributing zero-passing fields in the denominator (Ferguson et al., 12 Feb 2026). This design prevents structurally invalid generations from being excluded from aggregate scoring. In the structured-output experiments, the authors also use a Gemini 2.5 Flash judge with a 0.7 pass threshold, and a Levenshtein similarity threshold of 0.8 for fuzzy string matching (Ferguson et al., 12 Feb 2026).
4. Missingness, arrays, and nested-object semantics
ExtractBench explicitly distinguishes three states for each field: present, null, and MISSING (Ferguson et al., 12 Feb 2026). The distinction is operationally important. A field that exists in the document but is not extracted is an omission; a field that should be empty but is filled is a hallucination. The paper emphasizes that these are different failure modes with different downstream risks. Evaluation is therefore policy-aware: before values are compared, the framework first resolves the gold/predicted state combination.
Arrays receive separate treatment because position-based comparison is brittle under reordering, omissions, and spurious items. ExtractBench uses LLM-based semantic alignment to match predicted array items to gold items, using the item schema and per-field criteria. The matcher returns matched pairs, missed gold items, and extra predicted items. From these sets, the framework computes precision, recall, and F1 at the array level, after which aligned items are scored recursively using the same field-level metrics as elsewhere in the schema (Ferguson et al., 12 Feb 2026).
This array treatment is one of the benchmark’s main responses to enterprise-style nested extraction, where the complexity often lies less in scalar fields than in repeated substructures. Research papers, for example, have only 16 fields but very large citation arrays, and the paper reports that this causes output volume to explode to roughly 25k output tokens (Ferguson et al., 12 Feb 2026). A plausible implication is that array handling, rather than raw field typing, is a primary determinant of end-to-end robustness on realistic schemas.
5. Baseline evaluation and empirical results
The baseline study evaluates six frontier models in zero-shot mode: GPT-5, GPT-5.2, Gemini 3 Flash, Gemini 3 Pro, Claude Sonnet 4.5, and Claude Opus 4.5 (Ferguson et al., 12 Feb 2026). Across 210 extraction attempts—that is, 35 documents × 6 models—only 107 outputs were valid JSON, and the overall field-level Pass Rate was 4.6% (Ferguson et al., 12 Feb 2026).
The best overall model was Gemini 3 Flash, with 25/35 valid outputs and a 6.9% pass rate. GPT-5.2 achieved 21/35 valid and 5.2% pass rate, while Gemini 3 Pro also reached 21/35 valid with 5.5% pass rate. GPT-5 had lower validity, at 13/35, but somewhat better valid-only accuracy on the subset it produced. Claude Sonnet 4.5 and Claude Opus 4.5 achieved 15/35 and 12/35 valid outputs respectively, though the paper notes that both were blocked from some domains by provider constraints, notably PDF ingestion limits (Ferguson et al., 12 Feb 2026).
The most striking domain result is that all six models produced 0% valid output on the 369-field SEC 10-K/Q schema (Ferguson et al., 12 Feb 2026). Not a single one of the seven SEC documents yielded parseable, schema-conforming JSON from any model. Because the SEC domain contributes 15,498 of the 18,516 field evaluations in the main table, it dominates overall benchmark difficulty. The paper uses this to argue that schema breadth, rather than document length alone, is the dominant bottleneck.
Performance differs sharply by domain. Credit agreements are described as the strongest domain despite their length; Gemini 3 Flash, GPT-5, GPT-5.2, and Gemini 3 Pro all achieved around 80–87% field pass rate there, whereas Claude models scored 0% because their PDF support was capped at 100 pages (Ferguson et al., 12 Feb 2026). Research papers show a different profile: validity ranges from 0% valid for GPT-5 to 83% valid for some Claude and Gemini runs, but pass rates remain low. Sports results reached validity as high as 90%, yet only 12.5% pass rate overall, showing that syntactically valid JSON can remain semantically incorrect across many nested fields (Ferguson et al., 12 Feb 2026).
6. Failure modes, structured-output APIs, and relation to earlier ExtractBench usage
The paper provides a failure analysis of invalid outputs. Common failure modes include empty responses, trailing commas, PDF page-limit rejections for Claude on long documents, truncated JSON, and occasional context-length errors (Ferguson et al., 12 Feb 2026). A substantial fraction of failures are therefore formatting failures rather than purely extraction failures. However, the paper argues that this does not mean constrained decoding solves the problem.
To test that proposition, the authors repeated the evaluation using provider-specific structured output mode: OpenAI Structured Outputs, Google response schema, and Anthropic JSON outputs (Ferguson et al., 12 Feb 2026). Contrary to expectation, structured output did not improve performance overall. Instead, validity dropped from 51% to 37%, and the best overall Pass Rate dropped from 6.9% to 5.5%. Some schemas were rejected outright because they were too large or too complex, and the resume schema was rejected in structured-output mode across providers. Even when a schema was accepted, accuracy could decrease; for example, GPT-5’s credit-agreement pass rate fell from 86.9% to 70.0% (Ferguson et al., 12 Feb 2026). The paper interprets this as evidence that constrained decoding scales poorly with grammar complexity, schema breadth, and nesting depth.
ExtractBench also sits within a broader benchmark lineage. An earlier benchmark using the same name in practice evaluated ten freely available tools for academic PDF information extraction over DocBank, covering 500K pages, 1.5M annotated content elements, and tasks such as metadata, references, tables, and general layout elements (Meuschke et al., 2023). In that earlier framework, GROBID was strongest overall, while Adobe Extract was best for tables, and lists, footers, and equations remained hard extraction targets (Meuschke et al., 2023). The coexistence of these usages indicates that “ExtractBench” has referred to more than one extraction benchmark family. In the later PDF-to-JSON benchmark, the term is specialized to enterprise-scale structured extraction, with schema-driven evaluation as its defining methodological contribution (Ferguson et al., 12 Feb 2026).
Overall, ExtractBench is significant because it shows that enterprise PDF-to-JSON extraction is not solved by simply requesting valid JSON from frontier models. The benchmark’s results imply that output validity is necessary but insufficient, that schema breadth and output volume are major bottlenecks, and that evaluation must move beyond generic exact match toward field-aware, schema-driven scoring for nested structured outputs (Ferguson et al., 12 Feb 2026).
