---
title: 'OfficeQA Pro: Grounded Reasoning Benchmark'
url: https://www.emergentmind.com/topics/officeqa-pro
type: topic
---

# OfficeQA Pro: Grounded Reasoning Benchmark

Searching arXiv for the benchmark paper and closely related QA benchmarks for accurate citations.
OfficeQA Pro is a benchmark for evaluating AI agents on end-to-end grounded, multi-document reasoning over a large and heterogeneous document corpus derived from U.S. Treasury Bulletins spanning nearly 100 years. It comprises 133 hard questions that require precise document parsing, retrieval, and analytical reasoning across both unstructured text and tabular data, with single, numerically precise answers that can be automatically checked. The benchmark is intended as a proxy for enterprise workflows in finance, compliance, and operations, where systems must reason over messy archives, revisions, footnotes, and heterogeneous layouts rather than answer short, closed-world prompts [2603.08655].

## 1. Benchmark objective and scope

OfficeQA Pro is organized around what the paper calls **Grounded Reasoning**: document parsing, retrieval, grounded analysis, and verifiable answer production over a fixed corpus rather than reliance on parametric memory. The target setting is not abstract puzzle solving, single-table QA, or prompt-contained reasoning. Instead, it stresses an entire pipeline that resembles enterprise analytical work: turning thousands of heterogeneous PDFs into machine-readable form, finding the correct bulletin, page, and table, reconciling revisions, and carrying out precise numerical or statistical operations before returning a single answer [2603.08655].

This design addresses a recurrent misconception in benchmark interpretation: high-capacity LLM performance on general reasoning tasks does not imply reliability on archival, revision-sensitive, numerically exact workflows. In OfficeQA Pro, prompt-only models remain below 3% accuracy at 0.0% allowable error, and even strong web-enabled systems remain below roughly 12%, indicating that the benchmark is not primarily testing memorized Treasury statistics. A second misconception is that the task reduces to retrieval alone. Even with oracle pages and high-quality parsed representations, substantial error remains because the benchmark couples retrieval with table interpretation, revision tracking, unit handling, and analytical correctness [2603.08655].

The benchmark is explicitly framed as a test of enterprise-grade grounded reasoning. The motivating use cases are tasks such as computing trends in net interest outlays, comparing nominal and inflation-adjusted defense spending, using the most recent revised value of a liability series, or identifying which investor class held the largest share of long-term debt in a specific year. These tasks are structurally similar to work performed by FP&A teams, risk officers, compliance analysts, and BI users, but OfficeQA Pro requires that the entire reasoning chain be grounded in the supplied corpus rather than reconstructed from vague prior knowledge [2603.08655].

## 2. Corpus composition, document representations, and question design

The corpus consists of **U.S. Treasury Bulletins** published monthly from 1939 to 1982 and quarterly afterwards, continuing into the 2010s–2020s. It contains approximately **89,000 pages** and **26+ million numerical values**, with individual bulletins typically spanning **100–200 pages**. The content is heterogeneous: narrative analyses, dense multi-page statistical tables, figures and charts, appendices, footnotes, and methodological notes. The benchmark deliberately preserves realistic archival heterogeneity, including layout drift from scanned physical reports to digital-native PDFs, nested table hierarchies, changing units, and repeated statistics that are later revised [2603.08655].

To make parsing both difficult and realistic, embedded text layers are stripped from PDFs where present and the corpus is treated as images for parsing. The paper evaluates multiple representation strategies. In the raw-PDF setting, agents use libraries such as `pytesseract`, `pdfplumber`, `pymupdf`, and `tesseract`, often with OCR pipelines based on `pdftoppm` and `Pillow`. In the structured setting, Databricks’ `ai_parse_document` produces page-level layout, text blocks, paragraphs, table structures, header hierarchies, and bounding boxes. These rich outputs are then simplified into page text in reading order and optionally serialized tables in HTML or hierarchical Markdown. Additional parser ablations use Docling with RapidOCR + TableFormer and unstructured.io, with outputs normalized to plain text plus serialized tables [2603.08655].

OfficeQA Pro contains **133 “Pro” questions**, while **OfficeQA-Full** augments these with **113 “Easy” questions**. The Pro questions are constructed to satisfy three constraints: enterprise data complexity, multi-step reasoning, and verifiable answers. Some questions require a single page, others more than 20 pages; **11%** require **3+ bulletins**; **22%** require **web search** for exogenous inputs such as CPI series or exchange rates; **3%** require explicit visual reasoning over figures; and **62%** require operations beyond basic arithmetic, including OLS, forecasts, forecast errors, growth rates, deflators, multi-step aggregation, and cross-table joins [2603.08655].

The question construction pipeline is correspondingly controlled. Initial seed questions were written by Databricks researchers, then scaled through annotation partners, reviewed by USAFacts, and subjected to a verification pipeline involving independent answering and adjudication. Questions answerable from memory by early frontier models without corpus or web access were removed, and questions solved by the strongest agent systems with parsed documents were tagged Easy and excluded from Pro. This filtering process is central to the benchmark’s claim that it measures grounded reasoning rather than mere recall [2603.08655].

Two exemplar questions illustrate the technical character of the dataset. One asks for an OLS fit over fiscal-year federal individual income tax receipts from 1929–1942 and requires output in the format `[slope,intercept]`, rounded to the nearest thousandth. Another asks for the inflation-corrected absolute difference between 1940 and 1953 monthly defense expenditures using the annual average BLS CPI-U from the Federal Reserve Bank of Minneapolis. In both cases, the difficulty is distributed across retrieval, interpretation of fiscal versus calendar year, unit verification, computation, and strict output formatting rather than any single step alone [2603.08655].

## 3. Evaluation protocol, metrics, and experimental settings

Evaluation is designed for fully automatic, high-precision scoring. The core metric is **Exact Match accuracy**,
$$
\text{EM} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}[\hat{y}_i = y_i],
$$
supplemented by **Absolute Relative Error (ARE)** for the predominantly numerical answers,
$$
\text{Absolute Relative Error (\%)} = \left| \frac{\hat{y} - y}{y} \right| \times 100.
$$
A prediction is counted as correct when the ARE does not exceed a threshold $\tau$, with headline results reported at **$\tau = 0.0\%$**. The public `reward.py` additionally normalizes punctuation and numeric formatting, handles commas versus periods as thousand separators, and applies stricter matching for multipart or non-numeric answers to avoid false positives [2603.08655].

The benchmark supports several evaluation modes. In **Prompt Only**, the model receives only the question and must rely on parametric knowledge. In **Web Search Enabled**, native browsing tools are allowed but there is no direct corpus access beyond what is publicly discoverable. In **Direct Corpus Access**, models or agents receive either oracle pages or the entire bulletin archive. Numeric evaluation is identical across modes; what changes is the available evidence and the extent of the parsing and retrieval burden [2603.08655].

The paper evaluates both direct LLM calls and tool-using agents. Frontier models include GPT-5.4, GPT-5.1, GPT-5 Mini, Claude Opus 4.6, Claude Opus 4.5, Sonnet 4.6, Sonnet 4.5, Haiku 4.5, Gemini 3.1 Pro Preview, Gemini 3 Pro Preview, and Gemini 3 Flash. Agent settings use provider toolchains such as **OpenAI Codex CLI Agent**, **Claude Agent SDK**, and **Gemini CLI**, as well as a custom agent framework. In the full-corpus agent setup, all systems receive file search, shell access, code execution or notebooks, and web search, together with a common system prompt that emphasizes use of `../officeqa_corpus/`, instructs agents to prefer the most recent revision when metrics recur across documents, and requires a `<FINAL_ANSWER>` tag [2603.08655].

## 4. Empirical performance profile

A central empirical result is that OfficeQA Pro remains difficult even under strong assistance. The paper reports that frontier agents provided directly with the document corpus score **34.1% on average**, and that `ai_parse_document` yields a **16.1% average relative performance gain across agents**. Yet the strongest configurations still leave substantial headroom: even with oracle pages and Databricks-parsed representations, approximately one third of questions remain unsolved [2603.08655].

| Configuration | Best reported 0.0% accuracy | Observation |
|---|---:|---|
| Prompt Only | \<3% | Parametric knowledge is insufficient |
| Web Search Enabled | 11.3% | GPT-5.4 is the strongest reported web-only result |
| Full corpus, raw PDFs | 48.12% | Claude Opus 4.6 is best in this setting |
| Full corpus, Databricks parsed | 56.39% | GPT-5.4 benefits strongly from parsed text |
| Oracle pages, Databricks parsed | 66.92% | Claude Opus 4.6 is the strongest overall baseline |

Prompt-only performance is extremely low: all models are below 3% accuracy at 0.0% allowable error, although at 5% tolerance they reach 17–24%, indicating occasional Fermi-style approximation rather than exact recovery. With web access, GPT-5.4 reaches **11.3%**, while Claude and Gemini remain around **3.01%** in the reported table. The web-enabled setting is limited by series confusion, preliminary-versus-revised data errors, and single-turn token exhaustion; for example, Opus fails to finish in approximately 80% of cases, Gemini in approximately 15%, and GPT-5.4 in approximately 2% [2603.08655].

In the LLM oracle-page setting, retrieval is removed and the remaining challenge is parsing plus reasoning. With **raw PDF oracle pages** and web search, GPT-5.4 scores **57.1%**, Claude Opus 4.6 **36.1%**, and Gemini 3.1 Pro Preview **52.6%**. Replacing raw pages with **Databricks-parsed oracle pages** raises these to **65.4%**, **57.1%**, and **56.4%**, respectively. The paper summarizes this oracle-page transition as a **+50.2% relative gain on average** at strict 0.0% ARE, showing that document representation materially alters the ceiling even before retrieval is considered [2603.08655].

The full-corpus agent setting further exposes the interaction between retrieval and parsing. Over raw PDFs, Claude Opus 4.6 reaches **48.12%**, GPT-5.4 **36.09%**, and Gemini 3.1 Pro Preview **18.05%**. Over Databricks-parsed documents, these increase to **54.14%**, **56.39%**, and **29.32%**, while latency drops by **4–9×** relative to PDF. The paper gives the example of GPT-5.4 improving from **13.1** to **3.6 minutes** per question and cost decreasing from **\$1.79** to **\$1.26** per sample. When both retrieval and parsing are made easy through oracle parsed pages, Claude reaches **66.92%**, GPT-5.4 **65.41%**, and Gemini **46.62%**, but even then approximately one third of questions still fail [2603.08655].

Model-selection ablations using a common custom agent show that the frontier is not dominated by a single provider under all resource budgets. Claude Opus 4.6 attains **57.1%** at **5.3 minutes**, GPT-5.4 High **51.1%** at **10.9 minutes**, Claude Sonnet 4.6 **51.9%** at **5.4 minutes**, and Gemini 3.1 Pro Preview **42.9%** at **2.6 minutes**. Smaller models degrade gradually rather than catastrophically: Gemini 3 Flash, Claude Haiku 4.5, and GPT-5 Mini High still solve roughly **30–34%** of questions with the full tool stack, albeit with differing cost–latency tradeoffs [2603.08655].

## 5. Ablations, failure modes, and technical bottlenecks

The parsing stack is one of the dominant determinants of performance. In a parser-comparison ablation with the same custom agent and full-corpus setting, average accuracy is **50.4%** for `ai_parse_document`, **38.4%** for Docling, and **31.1%** for unstructured.io. Average cost per sample is **\$5.29** for `ai_parse_document` and **\$8.18** for Docling, while full-corpus parsing itself costs **\$178** with Databricks versus **\$2,670** for unstructured.io. The benchmark therefore treats parsing not as incidental preprocessing but as a first-order component of end-to-end system quality [2603.08655].

Table serialization matters, but less than parser quality. Comparing **HTML** with **hierarchical Markdown** on the same agent and parsed corpus, HTML wins in **7/11** cases, though differences are often modest. GPT-5.4 scores **51.1%** with HTML versus **50.4%** with Markdown; Claude Opus 4.6 scores **57.1%** versus **55.6%**; Gemini 3.1 Pro scores **42.9%** versus **39.1%**. Claude Opus 4.5 is the notable exception, improving from **46.6%** with HTML to **55.6%** with Markdown. This suggests that representation interacts with model-specific inductive biases rather than having a uniform effect [2603.08655].

Retrieval ablations likewise show that dense retrieval is not automatically superior in table-heavy corpora. Vanilla **vector search (VS)** underperforms simple **file search (FS)**, plausibly because rows become detached from headers and document-level context. **Contextual vector search (C-VS)** augments each chunk with document name, bulletin date, page number, page header, and table or section titles, partially repairing this loss. For GPT-5.4, accuracy moves from **39.1%** with VS to **49.6%** with C-VS and **51.9%** with **FS + C-VS**; for Claude Opus 4.6, from **45.9%** to **49.6%** to **53.4%**; and for Gemini 3.1 Pro, from **26.3%** to **33.8%** to **45.1%**. The paper notes that contextual embeddings are both more accurate and more efficient than vanilla vector search, and that weaker models are especially sensitive to retrieval quality [2603.08655].

Four major failure modes recur throughout the study. **Temporal revision verification** is difficult because Treasury Bulletins repeatedly revise earlier values; agents often seize on the first plausible number and ignore later issues, even when explicitly instructed to use the most recent revision. **Parsing faithfulness** remains fragile: in raw-PDF agent runs, up to **40–50% of failures** are attributed to OCR and table-extraction errors, and even `ai_parse_document` can shift rows or columns, lose nested headers, or detach footnotes from the data they qualify. **Visual understanding** remains weak for the approximately **3%** of questions requiring chart interpretation; image-based systems struggle with fine-grained value reading, while parsed-text systems may omit the figure entirely. **Analytical reasoning** itself is still unstable: agents choose nearby but incorrect columns, apply the wrong formula, reverse the sign of a forecast error, mishandle units, or round prematurely [2603.08655].

The benchmark also studies **test-time scaling** through multiple rollouts with plurality voting. All three examined models—Gemini 3.1 Pro, GPT-5.4, and Claude Opus 4.6—benefit from repeated runs, but gains saturate quickly for stronger models. This suggests that ensemble-style inference can improve reliability, yet the residual error is still far too high for high-stakes enterprise deployment [2603.08655].

## 6. Position within QA research, distinction from ProQA, and practical significance

OfficeQA Pro sits at the intersection of tabular reasoning, multi-hop QA, long-document QA, and financial-domain evaluation, but it differs materially from each. Unlike WikiTableQuestions, WikiSQL, Spider, TabFact, or ToTTo, it does not assume clean, isolated tables; tables are embedded in long PDFs with hierarchical headers, footnotes, and revisions. Unlike HotpotQA or 2WikiMultihopQA, the hops are not primarily entity-chain inference across Wikipedia pages but temporal and numerical operations across Treasury Bulletins. Unlike QuALITY, NarrativeQA, or LongBench v2, the outputs are usually exact numbers rather than narrative judgments. Relative to FinanceBench, the benchmark emphasizes a single deep, noisy corpus with nearly a century of definitional drift and explicit end-to-end evaluation of parsing, retrieval, computation, and exact numeric answer matching [2603.08655].

Despite the similarity in naming, OfficeQA Pro is distinct from **“ProQA: Structural Prompt-based Pre-training for Unified Question Answering”** [2205.04040]. ProQA is a unified QA paradigm built around a T5-based text-to-text model, a key–value **structural prompt**, task/format/domain soft prompts, and structural prompt-based pre-training over a synthetic QA corpus. It is designed to unify extractive, abstractive, multiple-choice, and yes/no QA within a single model and to improve few-shot, zero-shot, transfer, and continual learning performance. OfficeQA Pro, by contrast, is not a prompting or pre-training method. It is a benchmark for enterprise-style grounded reasoning in which parsing quality, retrieval strategy, revision handling, and analytical precision jointly determine success [2205.04040].

The practical implication is that OfficeQA Pro measures whether an agent can operate as an “AI analyst” over real archives rather than whether it can answer benchmark questions in a short context window. The strongest reported performance is approximately **67% accuracy** in the oracle parsed-page setting and approximately **54–57%** in the full parsed-corpus setting, whereas enterprise deployment might demand **>95% accuracy** or strong calibrated abstention. The benchmark therefore exposes a substantial gap between current frontier systems and reliable enterprise-grade grounded reasoning [2603.08655].

The benchmark is also designed for reproducibility. Code and data are released through `https://github.com/databricks/officeqa`, including the OfficeQA Pro and OfficeQA-Full question sets, `reward.py`, scripts and prompts for non-agent and agent baselines, and instructions for corpus parsing, search-index construction, and custom-agent execution. The Treasury Bulletins are public documents, with the paper referencing FRASER and the processed corpus used in experiments. This release structure makes OfficeQA Pro not merely a leaderboard artifact but a practical evaluation harness for diagnosing failures in parser choice, retrieval design, agent tooling, latency, and cost under realistic enterprise constraints [2603.08655].

Source: https://www.emergentmind.com/topics/officeqa-pro