OfficeQA Pro: Grounded Reasoning Benchmark
- OfficeQA Pro is a benchmark that evaluates AI agents on end-to-end document parsing, retrieval, and numerical analysis over nearly 100 years of U.S. Treasury Bulletins.
- It integrates heterogeneous data from unstructured text and complex tables, requiring multi-step reasoning, revision tracking, and precise output verification.
- Empirical results highlight that even state-of-the-art systems reach around 67% accuracy in ideal settings, underscoring the challenges of enterprise-grade document reasoning.
Searching arXiv for the benchmark paper and closely related QA benchmarks for accurate citations. OfficeQA Pro is a benchmark for evaluating AI agents on end-to-end grounded, multi-document reasoning over a large and heterogeneous document corpus derived from U.S. Treasury Bulletins spanning nearly 100 years. It comprises 133 hard questions that require precise document parsing, retrieval, and analytical reasoning across both unstructured text and tabular data, with single, numerically precise answers that can be automatically checked. The benchmark is intended as a proxy for enterprise workflows in finance, compliance, and operations, where systems must reason over messy archives, revisions, footnotes, and heterogeneous layouts rather than answer short, closed-world prompts (Opsahl-Ong et al., 9 Mar 2026).
1. Benchmark objective and scope
OfficeQA Pro is organized around what the paper calls Grounded Reasoning: document parsing, retrieval, grounded analysis, and verifiable answer production over a fixed corpus rather than reliance on parametric memory. The target setting is not abstract puzzle solving, single-table QA, or prompt-contained reasoning. Instead, it stresses an entire pipeline that resembles enterprise analytical work: turning thousands of heterogeneous PDFs into machine-readable form, finding the correct bulletin, page, and table, reconciling revisions, and carrying out precise numerical or statistical operations before returning a single answer (Opsahl-Ong et al., 9 Mar 2026).
This design addresses a recurrent misconception in benchmark interpretation: high-capacity LLM performance on general reasoning tasks does not imply reliability on archival, revision-sensitive, numerically exact workflows. In OfficeQA Pro, prompt-only models remain below 3% accuracy at 0.0% allowable error, and even strong web-enabled systems remain below roughly 12%, indicating that the benchmark is not primarily testing memorized Treasury statistics. A second misconception is that the task reduces to retrieval alone. Even with oracle pages and high-quality parsed representations, substantial error remains because the benchmark couples retrieval with table interpretation, revision tracking, unit handling, and analytical correctness (Opsahl-Ong et al., 9 Mar 2026).
The benchmark is explicitly framed as a test of enterprise-grade grounded reasoning. The motivating use cases are tasks such as computing trends in net interest outlays, comparing nominal and inflation-adjusted defense spending, using the most recent revised value of a liability series, or identifying which investor class held the largest share of long-term debt in a specific year. These tasks are structurally similar to work performed by FP&A teams, risk officers, compliance analysts, and BI users, but OfficeQA Pro requires that the entire reasoning chain be grounded in the supplied corpus rather than reconstructed from vague prior knowledge (Opsahl-Ong et al., 9 Mar 2026).
2. Corpus composition, document representations, and question design
The corpus consists of U.S. Treasury Bulletins published monthly from 1939 to 1982 and quarterly afterwards, continuing into the 2010s–2020s. It contains approximately 89,000 pages and 26+ million numerical values, with individual bulletins typically spanning 100–200 pages. The content is heterogeneous: narrative analyses, dense multi-page statistical tables, figures and charts, appendices, footnotes, and methodological notes. The benchmark deliberately preserves realistic archival heterogeneity, including layout drift from scanned physical reports to digital-native PDFs, nested table hierarchies, changing units, and repeated statistics that are later revised (Opsahl-Ong et al., 9 Mar 2026).
To make parsing both difficult and realistic, embedded text layers are stripped from PDFs where present and the corpus is treated as images for parsing. The paper evaluates multiple representation strategies. In the raw-PDF setting, agents use libraries such as pytesseract, pdfplumber, pymupdf, and tesseract, often with OCR pipelines based on pdftoppm and Pillow. In the structured setting, Databricks’ ai_parse_document produces page-level layout, text blocks, paragraphs, table structures, header hierarchies, and bounding boxes. These rich outputs are then simplified into page text in reading order and optionally serialized tables in HTML or hierarchical Markdown. Additional parser ablations use Docling with RapidOCR + TableFormer and unstructured.io, with outputs normalized to plain text plus serialized tables (Opsahl-Ong et al., 9 Mar 2026).
OfficeQA Pro contains 133 “Pro” questions, while OfficeQA-Full augments these with 113 “Easy” questions. The Pro questions are constructed to satisfy three constraints: enterprise data complexity, multi-step reasoning, and verifiable answers. Some questions require a single page, others more than 20 pages; 11% require 3+ bulletins; 22% require web search for exogenous inputs such as CPI series or exchange rates; 3% require explicit visual reasoning over figures; and 62% require operations beyond basic arithmetic, including OLS, forecasts, forecast errors, growth rates, deflators, multi-step aggregation, and cross-table joins (Opsahl-Ong et al., 9 Mar 2026).
The question construction pipeline is correspondingly controlled. Initial seed questions were written by Databricks researchers, then scaled through annotation partners, reviewed by USAFacts, and subjected to a verification pipeline involving independent answering and adjudication. Questions answerable from memory by early frontier models without corpus or web access were removed, and questions solved by the strongest agent systems with parsed documents were tagged Easy and excluded from Pro. This filtering process is central to the benchmark’s claim that it measures grounded reasoning rather than mere recall (Opsahl-Ong et al., 9 Mar 2026).
Two exemplar questions illustrate the technical character of the dataset. One asks for an OLS fit over fiscal-year federal individual income tax receipts from 1929–1942 and requires output in the format [slope,intercept], rounded to the nearest thousandth. Another asks for the inflation-corrected absolute difference between 1940 and 1953 monthly defense expenditures using the annual average BLS CPI-U from the Federal Reserve Bank of Minneapolis. In both cases, the difficulty is distributed across retrieval, interpretation of fiscal versus calendar year, unit verification, computation, and strict output formatting rather than any single step alone (Opsahl-Ong et al., 9 Mar 2026).
3. Evaluation protocol, metrics, and experimental settings
Evaluation is designed for fully automatic, high-precision scoring. The core metric is Exact Match accuracy,
supplemented by Absolute Relative Error (ARE) for the predominantly numerical answers,
A prediction is counted as correct when the ARE does not exceed a threshold , with headline results reported at . The public reward.py additionally normalizes punctuation and numeric formatting, handles commas versus periods as thousand separators, and applies stricter matching for multipart or non-numeric answers to avoid false positives (Opsahl-Ong et al., 9 Mar 2026).
The benchmark supports several evaluation modes. In Prompt Only, the model receives only the question and must rely on parametric knowledge. In Web Search Enabled, native browsing tools are allowed but there is no direct corpus access beyond what is publicly discoverable. In Direct Corpus Access, models or agents receive either oracle pages or the entire bulletin archive. Numeric evaluation is identical across modes; what changes is the available evidence and the extent of the parsing and retrieval burden (Opsahl-Ong et al., 9 Mar 2026).
The paper evaluates both direct LLM calls and tool-using agents. Frontier models include GPT-5.4, GPT-5.1, GPT-5 Mini, Claude Opus 4.6, Claude Opus 4.5, Sonnet 4.6, Sonnet 4.5, Haiku 4.5, Gemini 3.1 Pro Preview, Gemini 3 Pro Preview, and Gemini 3 Flash. Agent settings use provider toolchains such as OpenAI Codex CLI Agent, Claude Agent SDK, and Gemini CLI, as well as a custom agent framework. In the full-corpus agent setup, all systems receive file search, shell access, code execution or notebooks, and web search, together with a common system prompt that emphasizes use of ../officeqa_corpus/, instructs agents to prefer the most recent revision when metrics recur across documents, and requires a <FINAL_ANSWER> tag (Opsahl-Ong et al., 9 Mar 2026).
4. Empirical performance profile
A central empirical result is that OfficeQA Pro remains difficult even under strong assistance. The paper reports that frontier agents provided directly with the document corpus score 34.1% on average, and that ai_parse_document yields a 16.1% average relative performance gain across agents. Yet the strongest configurations still leave substantial headroom: even with oracle pages and Databricks-parsed representations, approximately one third of questions remain unsolved (Opsahl-Ong et al., 9 Mar 2026).
| Configuration | Best reported 0.0% accuracy | Observation |
|---|---|---|
| Prompt Only | <3% | Parametric knowledge is insufficient |
| Web Search Enabled | 11.3% | GPT-5.4 is the strongest reported web-only result |
| Full corpus, raw PDFs | 48.12% | Claude Opus 4.6 is best in this setting |
| Full corpus, Databricks parsed | 56.39% | GPT-5.4 benefits strongly from parsed text |
| Oracle pages, Databricks parsed | 66.92% | Claude Opus 4.6 is the strongest overall baseline |
Prompt-only performance is extremely low: all models are below 3% accuracy at 0.0% allowable error, although at 5% tolerance they reach 17–24%, indicating occasional Fermi-style approximation rather than exact recovery. With web access, GPT-5.4 reaches 11.3%, while Claude and Gemini remain around 3.01% in the reported table. The web-enabled setting is limited by series confusion, preliminary-versus-revised data errors, and single-turn token exhaustion; for example, Opus fails to finish in approximately 80% of cases, Gemini in approximately 15%, and GPT-5.4 in approximately 2% (Opsahl-Ong et al., 9 Mar 2026).
In the LLM oracle-page setting, retrieval is removed and the remaining challenge is parsing plus reasoning. With raw PDF oracle pages and web search, GPT-5.4 scores 57.1%, Claude Opus 4.6 36.1%, and Gemini 3.1 Pro Preview 52.6%. Replacing raw pages with Databricks-parsed oracle pages raises these to 65.4%, 57.1%, and 56.4%, respectively. The paper summarizes this oracle-page transition as a +50.2% relative gain on average at strict 0.0% ARE, showing that document representation materially alters the ceiling even before retrieval is considered (Opsahl-Ong et al., 9 Mar 2026).
The full-corpus agent setting further exposes the interaction between retrieval and parsing. Over raw PDFs, Claude Opus 4.6 reaches 48.12%, GPT-5.4 36.09%, and Gemini 3.1 Pro Preview 18.05%. Over Databricks-parsed documents, these increase to 54.14%, 56.39%, and 29.32%, while latency drops by 4–9× relative to PDF. The paper gives the example of GPT-5.4 improving from 13.1 to 3.6 minutes per question and cost decreasing from \$1.79** to **\$1.26 per sample. When both retrieval and parsing are made easy through oracle parsed pages, Claude reaches 66.92%, GPT-5.4 65.41%, and Gemini 46.62%, but even then approximately one third of questions still fail (Opsahl-Ong et al., 9 Mar 2026).
Model-selection ablations using a common custom agent show that the frontier is not dominated by a single provider under all resource budgets. Claude Opus 4.6 attains 57.1% at 5.3 minutes, GPT-5.4 High 51.1% at 10.9 minutes, Claude Sonnet 4.6 51.9% at 5.4 minutes, and Gemini 3.1 Pro Preview 42.9% at 2.6 minutes. Smaller models degrade gradually rather than catastrophically: Gemini 3 Flash, Claude Haiku 4.5, and GPT-5 Mini High still solve roughly 30–34% of questions with the full tool stack, albeit with differing cost–latency tradeoffs (Opsahl-Ong et al., 9 Mar 2026).
5. Ablations, failure modes, and technical bottlenecks
The parsing stack is one of the dominant determinants of performance. In a parser-comparison ablation with the same custom agent and full-corpus setting, average accuracy is 50.4% for ai_parse_document, 38.4% for Docling, and 31.1% for unstructured.io. Average cost per sample is \$5.29** for `ai_parse_document` and **\$8.18 for Docling, while full-corpus parsing itself costs \$178** with Databricks versus **\$2,670 for unstructured.io. The benchmark therefore treats parsing not as incidental preprocessing but as a first-order component of end-to-end system quality (Opsahl-Ong et al., 9 Mar 2026).
Table serialization matters, but less than parser quality. Comparing HTML with hierarchical Markdown on the same agent and parsed corpus, HTML wins in 7/11 cases, though differences are often modest. GPT-5.4 scores 51.1% with HTML versus 50.4% with Markdown; Claude Opus 4.6 scores 57.1% versus 55.6%; Gemini 3.1 Pro scores 42.9% versus 39.1%. Claude Opus 4.5 is the notable exception, improving from 46.6% with HTML to 55.6% with Markdown. This suggests that representation interacts with model-specific inductive biases rather than having a uniform effect (Opsahl-Ong et al., 9 Mar 2026).
Retrieval ablations likewise show that dense retrieval is not automatically superior in table-heavy corpora. Vanilla vector search (VS) underperforms simple file search (FS), plausibly because rows become detached from headers and document-level context. Contextual vector search (C-VS) augments each chunk with document name, bulletin date, page number, page header, and table or section titles, partially repairing this loss. For GPT-5.4, accuracy moves from 39.1% with VS to 49.6% with C-VS and 51.9% with FS + C-VS; for Claude Opus 4.6, from 45.9% to 49.6% to 53.4%; and for Gemini 3.1 Pro, from 26.3% to 33.8% to 45.1%. The paper notes that contextual embeddings are both more accurate and more efficient than vanilla vector search, and that weaker models are especially sensitive to retrieval quality (Opsahl-Ong et al., 9 Mar 2026).
Four major failure modes recur throughout the study. Temporal revision verification is difficult because Treasury Bulletins repeatedly revise earlier values; agents often seize on the first plausible number and ignore later issues, even when explicitly instructed to use the most recent revision. Parsing faithfulness remains fragile: in raw-PDF agent runs, up to 40–50% of failures are attributed to OCR and table-extraction errors, and even ai_parse_document can shift rows or columns, lose nested headers, or detach footnotes from the data they qualify. Visual understanding remains weak for the approximately 3% of questions requiring chart interpretation; image-based systems struggle with fine-grained value reading, while parsed-text systems may omit the figure entirely. Analytical reasoning itself is still unstable: agents choose nearby but incorrect columns, apply the wrong formula, reverse the sign of a forecast error, mishandle units, or round prematurely (Opsahl-Ong et al., 9 Mar 2026).
The benchmark also studies test-time scaling through multiple rollouts with plurality voting. All three examined models—Gemini 3.1 Pro, GPT-5.4, and Claude Opus 4.6—benefit from repeated runs, but gains saturate quickly for stronger models. This suggests that ensemble-style inference can improve reliability, yet the residual error is still far too high for high-stakes enterprise deployment (Opsahl-Ong et al., 9 Mar 2026).
6. Position within QA research, distinction from ProQA, and practical significance
OfficeQA Pro sits at the intersection of tabular reasoning, multi-hop QA, long-document QA, and financial-domain evaluation, but it differs materially from each. Unlike WikiTableQuestions, WikiSQL, Spider, TabFact, or ToTTo, it does not assume clean, isolated tables; tables are embedded in long PDFs with hierarchical headers, footnotes, and revisions. Unlike HotpotQA or 2WikiMultihopQA, the hops are not primarily entity-chain inference across Wikipedia pages but temporal and numerical operations across Treasury Bulletins. Unlike QuALITY, NarrativeQA, or LongBench v2, the outputs are usually exact numbers rather than narrative judgments. Relative to FinanceBench, the benchmark emphasizes a single deep, noisy corpus with nearly a century of definitional drift and explicit end-to-end evaluation of parsing, retrieval, computation, and exact numeric answer matching (Opsahl-Ong et al., 9 Mar 2026).
Despite the similarity in naming, OfficeQA Pro is distinct from “ProQA: Structural Prompt-based Pre-training for Unified Question Answering” (Zhong et al., 2022). ProQA is a unified QA paradigm built around a T5-based text-to-text model, a key–value structural prompt, task/format/domain soft prompts, and structural prompt-based pre-training over a synthetic QA corpus. It is designed to unify extractive, abstractive, multiple-choice, and yes/no QA within a single model and to improve few-shot, zero-shot, transfer, and continual learning performance. OfficeQA Pro, by contrast, is not a prompting or pre-training method. It is a benchmark for enterprise-style grounded reasoning in which parsing quality, retrieval strategy, revision handling, and analytical precision jointly determine success (Zhong et al., 2022).
The practical implication is that OfficeQA Pro measures whether an agent can operate as an “AI analyst” over real archives rather than whether it can answer benchmark questions in a short context window. The strongest reported performance is approximately 67% accuracy in the oracle parsed-page setting and approximately 54–57% in the full parsed-corpus setting, whereas enterprise deployment might demand >95% accuracy or strong calibrated abstention. The benchmark therefore exposes a substantial gap between current frontier systems and reliable enterprise-grade grounded reasoning (Opsahl-Ong et al., 9 Mar 2026).
The benchmark is also designed for reproducibility. Code and data are released through https://github.com/databricks/officeqa, including the OfficeQA Pro and OfficeQA-Full question sets, reward.py, scripts and prompts for non-agent and agent baselines, and instructions for corpus parsing, search-index construction, and custom-agent execution. The Treasury Bulletins are public documents, with the paper referencing FRASER and the processed corpus used in experiments. This release structure makes OfficeQA Pro not merely a leaderboard artifact but a practical evaluation harness for diagnosing failures in parser choice, retrieval design, agent tooling, latency, and cost under realistic enterprise constraints (Opsahl-Ong et al., 9 Mar 2026).