---
title: 'ExtractBench: PDF-to-JSON Extraction Benchmark'
url: https://www.emergentmind.com/topics/extractbench
type: topic
---

# ExtractBench: PDF-to-JSON Extraction Benchmark

ExtractBench is an open-source benchmark and evaluation framework for **PDF-to-JSON structured extraction** that is intended to reflect the conditions of enterprise document automation rather than simplified document-understanding tasks. In its 2026 formulation, it evaluates **document-grounded extraction from PDFs** under **enterprise-scale JSON Schemas** and scores predictions with a **schema-aware** methodology in which the schema serves as an **executable specification** of both structure and field-level correctness [2602.12247]. The name has also been used in earlier extraction benchmarking work, most notably for a **multi-task, multi-domain evaluation framework** for academic PDF information extraction built on DocBank [2303.09957]. In current usage, however, the term most directly denotes the benchmark for realistic PDF-to-JSON extraction with nested outputs, heterogeneous field semantics, and explicit treatment of omission versus hallucination [2602.12247].

## 1. Scope, problem formulation, and benchmark intent

ExtractBench was introduced to address two gaps in structured extraction evaluation. First, there was **no end-to-end benchmark** for **PDF-to-JSON extraction** under **enterprise-scale schema breadth**. Second, there was **no principled methodology** for scoring **nested extraction**, where different fields require different notions of correctness, arrays require alignment, and missingness must be distinguished from hallucination [2602.12247].

The benchmark is explicitly designed around production-style conditions. The input is a **long, noisy PDF**; the output is a **large, deeply nested JSON object**; and correctness is not uniform across fields. The paper states that an identifier should be checked by **exact match**, a monetary value by **tolerance-based numeric matching**, and a person or company name by **semantic equivalence**. Arrays of objects must be aligned before they can be scored. This departs from earlier benchmarks that evaluated only fragments of the overall problem, such as span extraction, small fixed field sets, or schema conformance without document grounding [2602.12247].

A central design claim is that **JSON Schema** is not merely a serialization target. In ExtractBench, it is the formal specification of what should be extracted and how each field should be evaluated. The schema is parsed into an AST and traversed jointly with the gold and predicted JSON objects. This makes evaluation recursive, modular, and field-sensitive. A plausible implication is that ExtractBench treats structural extraction as a typed semantic matching problem rather than as uniform string comparison.

## 2. Dataset composition and domain structure

The dataset is intentionally small in sample count but high in structural complexity. It contains **35 PDF documents** across **5 domains**, spanning **2,076 pages** and **12,867 evaluatable field values** [2602.12247]. The five domains are:

| Domain | Distinctive property |
|---|---|
| SEC 10-K/Q filings | **369 evaluatable fields** |
| Credit agreements | **100–250 pages**, **13 fields** |
| Research papers | **16 fields**, very large citation arrays |
| Professional resumes | **31 fields** |
| Sports results | Schema **depth 6** |

The benchmark is designed to expose failure modes caused by **breadth, nesting, and array expansion** rather than by sample scarcity alone. The five domains were chosen because they stress different extraction regimes. **Sports results** have the deepest schema but relatively regular structure. **Credit agreements** are long documents with only **13 fields**, making them “needle-in-a-haystack” extraction problems. **Research papers** have only **16 fields** but very large citation arrays that expand to roughly **25k output tokens**. **Professional resumes** are semi-structured with **31 fields**. **SEC 10-K/Q filings** are the hardest in schema breadth, with **369 evaluatable fields**, and the paper describes them as an order of magnitude larger than prior enterprise benchmarks [2602.12247].

Gold-label creation was correspondingly expensive. The paper reports **67.9 hours of expert human annotation**, of which **56.1 hours** were spent on SEC filings alone [2602.12247]. This is presented as evidence that high-quality ground truth becomes costly when schemas are broad and nested. This suggests that ExtractBench is intended less as a large-scale training corpus than as a diagnostic evaluation set for realistic structured extraction.

## 3. Schema-driven evaluation and field-specific metrics

ExtractBench’s defining methodological feature is **schema-driven evaluation**. Each node in the schema can specify its own metric via `evaluation_config`, and evaluation recurses over the schema AST jointly with the predicted and gold JSON objects [2602.12247]. The paper gives examples such as:

```latex
\texttt{"borrower\_name": \{"type": "string", "evaluation\_config": "string\_semantic"\}
```

and

```latex
\texttt{"principal": \{"type": "number", "evaluation\_config": \{"metric\_id": "number\_tolerance", "params": \{"tolerance": 0.001\}\}\}
```

The metric library includes:

- **exact and case-insensitive string matching**
- **Levenshtein-based fuzzy matching**
- **LLM-based semantic equivalence**
- **exact and tolerance-based numeric matching**
- **exact boolean matching**
- **semantic array alignment**

This evaluation framework is motivated by the claim that a single global comparison rule is inadequate for nested extraction. An identifier, a free-form name, and a quantity do not admit the same notion of correctness. ExtractBench therefore formalizes correctness locally, at the field level, and composes those judgments recursively over the whole JSON object [2602.12247].

The paper reports two headline aggregate metrics. **Valid JSON** is the fraction of outputs that are parseable and schema-conforming. **Pass Rate** is the field-level correctness rate across all evaluated fields, with invalid outputs contributing zero-passing fields in the denominator [2602.12247]. This design prevents structurally invalid generations from being excluded from aggregate scoring. In the structured-output experiments, the authors also use a **Gemini 2.5 Flash judge** with a **0.7 pass threshold**, and a **Levenshtein similarity threshold of 0.8** for fuzzy string matching [2602.12247].

## 4. Missingness, arrays, and nested-object semantics

ExtractBench explicitly distinguishes three states for each field: **present**, **null**, and **MISSING** [2602.12247]. The distinction is operationally important. A field that exists in the document but is not extracted is an **omission**; a field that should be empty but is filled is a **hallucination**. The paper emphasizes that these are different failure modes with different downstream risks. Evaluation is therefore **policy-aware**: before values are compared, the framework first resolves the gold/predicted state combination.

Arrays receive separate treatment because position-based comparison is brittle under reordering, omissions, and spurious items. ExtractBench uses **LLM-based semantic alignment** to match predicted array items to gold items, using the item schema and per-field criteria. The matcher returns matched pairs, missed gold items, and extra predicted items. From these sets, the framework computes **precision, recall, and F1** at the array level, after which aligned items are scored recursively using the same field-level metrics as elsewhere in the schema [2602.12247].

This array treatment is one of the benchmark’s main responses to enterprise-style nested extraction, where the complexity often lies less in scalar fields than in repeated substructures. Research papers, for example, have only **16 fields** but very large citation arrays, and the paper reports that this causes output volume to explode to roughly **25k output tokens** [2602.12247]. A plausible implication is that array handling, rather than raw field typing, is a primary determinant of end-to-end robustness on realistic schemas.

## 5. Baseline evaluation and empirical results

The baseline study evaluates six frontier models in zero-shot mode: **GPT-5, GPT-5.2, Gemini 3 Flash, Gemini 3 Pro, Claude Sonnet 4.5, and Claude Opus 4.5** [2602.12247]. Across **210 extraction attempts**—that is, **35 documents × 6 models**—only **107 outputs** were valid JSON, and the overall field-level **Pass Rate** was **4.6%** [2602.12247].

The best overall model was **Gemini 3 Flash**, with **25/35 valid outputs** and a **6.9% pass rate**. **GPT-5.2** achieved **21/35 valid** and **5.2% pass rate**, while **Gemini 3 Pro** also reached **21/35 valid** with **5.5% pass rate**. **GPT-5** had lower validity, at **13/35**, but somewhat better valid-only accuracy on the subset it produced. **Claude Sonnet 4.5** and **Claude Opus 4.5** achieved **15/35** and **12/35** valid outputs respectively, though the paper notes that both were blocked from some domains by provider constraints, notably PDF ingestion limits [2602.12247].

The most striking domain result is that **all six models produced 0% valid output on the 369-field SEC 10-K/Q schema** [2602.12247]. Not a single one of the seven SEC documents yielded parseable, schema-conforming JSON from any model. Because the SEC domain contributes **15,498 of the 18,516 field evaluations** in the main table, it dominates overall benchmark difficulty. The paper uses this to argue that **schema breadth**, rather than document length alone, is the dominant bottleneck.

Performance differs sharply by domain. **Credit agreements** are described as the strongest domain despite their length; **Gemini 3 Flash, GPT-5, GPT-5.2, and Gemini 3 Pro** all achieved around **80–87% field pass rate** there, whereas Claude models scored **0%** because their PDF support was capped at **100 pages** [2602.12247]. **Research papers** show a different profile: validity ranges from **0% valid** for GPT-5 to **83% valid** for some Claude and Gemini runs, but pass rates remain low. **Sports results** reached validity as high as **90%**, yet only **12.5% pass rate** overall, showing that syntactically valid JSON can remain semantically incorrect across many nested fields [2602.12247].

## 6. Failure modes, structured-output APIs, and relation to earlier ExtractBench usage

The paper provides a failure analysis of invalid outputs. Common failure modes include **empty responses**, **trailing commas**, **PDF page-limit rejections** for Claude on long documents, **truncated JSON**, and occasional **context-length errors** [2602.12247]. A substantial fraction of failures are therefore formatting failures rather than purely extraction failures. However, the paper argues that this does not mean constrained decoding solves the problem.

To test that proposition, the authors repeated the evaluation using provider-specific **structured output mode**: **OpenAI Structured Outputs**, **Google response schema**, and **Anthropic JSON outputs** [2602.12247]. Contrary to expectation, structured output did **not** improve performance overall. Instead, **validity dropped from 51% to 37%**, and the best overall **Pass Rate** dropped from **6.9% to 5.5%**. Some schemas were rejected outright because they were too large or too complex, and the **resume schema** was rejected in structured-output mode across providers. Even when a schema was accepted, accuracy could decrease; for example, **GPT-5’s credit-agreement pass rate fell from 86.9% to 70.0%** [2602.12247]. The paper interprets this as evidence that constrained decoding scales poorly with grammar complexity, schema breadth, and nesting depth.

ExtractBench also sits within a broader benchmark lineage. An earlier benchmark using the same name in practice evaluated **ten freely available tools** for **academic PDF information extraction** over **DocBank**, covering **500K pages**, **1.5M annotated content elements**, and tasks such as metadata, references, tables, and general layout elements [2303.09957]. In that earlier framework, **GROBID** was strongest overall, while **Adobe Extract** was best for tables, and **lists, footers, and equations** remained hard extraction targets [2303.09957]. The coexistence of these usages indicates that “ExtractBench” has referred to more than one extraction benchmark family. In the later PDF-to-JSON benchmark, the term is specialized to **enterprise-scale structured extraction**, with schema-driven evaluation as its defining methodological contribution [2602.12247].

Overall, ExtractBench is significant because it shows that enterprise PDF-to-JSON extraction is not solved by simply requesting valid JSON from frontier models. The benchmark’s results imply that **output validity is necessary but insufficient**, that **schema breadth and output volume** are major bottlenecks, and that evaluation must move beyond generic exact match toward **field-aware, schema-driven scoring** for nested structured outputs [2602.12247].

Source: https://www.emergentmind.com/topics/extractbench