---
title: 'FBBench: Auditable Financial Research Benchmark'
url: https://www.emergentmind.com/topics/fbbench
type: topic
---

# FBBench: Auditable Financial Research Benchmark

FBBench, introduced as **BigFinanceBench**, is a workflow-grounded benchmark for evaluating AI agents on the **full derivation** of financial-research answers rather than only on isolated subskills or final numbers. It consists of **928 expert-authored, open-ended financial-research tasks**, each paired with a **ground-truth reference answer** and a **point-weighted rubric** that decomposes the derivation into independently checkable steps. Across **15,656** atomic rubric criteria and **36,241** total rubric points, FBBench is designed to support partial-credit evaluation, auditability, and failure localization across the analyst workflow. Initial evaluation on ten frontier and open-weight agents reports substantial headroom: the best systems achieve only **58.8% rubric score**, while the highest **final-answer accuracy** is **44.3%** [2606.03829].

## 1. Motivation and problem setting

FBBench is motivated by the observation that financial-research answers are decision-relevant only when another analyst can audit how they were produced: **which source was chosen, which period and accounting definition were used, which assumptions were made, and how the calculation was performed**. In financial research, analysts must identify and disambiguate the company or ticker, locate and retrieve multiple filings or press releases, normalize accounting definitions such as **GAAP vs non-GAAP**, period boundaries, and amortization treatment, choose and document assumptions such as **software capitalization vs expense**, perform stepwise calculations, and present a final figure with explicit units and rounding rules [2606.03829].

The benchmark is therefore designed to replicate both the **open-ended, multi-source nature** of analyst queries and the **fine-grained, rubrics-based review process** used to audit derivations. Existing finance benchmarks are described as largely evaluating isolated subskills or final answers, leaving the **auditable derivation** under-measured. FBBench addresses that gap by treating the analyst workflow itself as the evaluation target.

This emphasis on derivation has an immediate methodological consequence: a correct number is not treated as sufficient evidence of a correct workflow. Conversely, an incorrect final number does not imply total failure if the earlier stages of the derivation were correct. This suggests a benchmark philosophy aligned with auditability rather than outcome-only scoring.

## 2. Benchmark composition and task structure

FBBench contains **928 expert-authored, open-ended tasks**. Each question was written by a **current or former investment banker, private-equity professional, or equity-research analyst** and then audited by a **separate reviewer**. The tasks cover core public-equity workflows, including **earnings quality, M&A, valuation, and operating KPIs**, and require combinations of **cross-filing synthesis, accounting adjustment, scenario projection, and market-data lookup** [2606.03829].

A defining feature is that the queries are both **multi-source** and **assumption-driven**. Roughly **30 percent** of items require more than one filing or press release. Every question is **time-anchored**, for example “Fiscal Year Ended December 31 2024,” and any non-obvious assumptions are stated explicitly.

| Component | Value | Description |
|---|---:|---|
| Tasks | 928 | Expert-authored, open-ended financial-research questions |
| Rubric criteria | 15,656 | Atomic yes/no checklist items |
| Total rubric points | 36,241 | Sum of all point-weighted criteria |

For each item, authors provide a single reference answer and a rubric decomposed into atomic steps. The paper gives examples such as **“Overstated by \$90.1 m”** and **“1.74 % ARR growth”** for reference answers, and rubric criteria such as **“Identifies DAY as ticker,” “Calculates Q3 ’26 retained ARR as \$511.38 m,”** and **“Notes that NDRR = 97 %.”** The mean number of criteria is **16.9 per question**. Integer weights range from **+1 through +10**, and the final conclusion is always assigned a significant share of total points [2606.03829].

The structure of the tasks makes FBBench closer to analyst-grade review than to fixed-context QA. A plausible implication is that it can separate failures caused by retrieval, accounting interpretation, or numerical execution even when all three appear in a single problem.

## 3. Rubric formalism and scoring methodology

FBBench scores each **agent trajectory**—defined as the chain of tool calls, intermediate outputs, and final answer—against every rubric criterion. Criterion $i$ receives a binary judgment $s_i \in \{0,1\}$ and has weight $w_i$. The per-question rubric score is defined as

$$
S = \frac{\sum_{i=1}^n w_i s_i}{\sum_{i=1}^n w_i} \times 100\%.
$$

The benchmark reports model performance by **macro-averaging** $S$ over all **928 questions**. In parallel, the final answer is judged **correct or incorrect** to compute **final-answer accuracy** [2606.03829].

This design yields two distinct evaluation signals. The rubric score measures derivational completeness and correctness at the level of individual workflow steps; final-answer accuracy collapses the same task into an outcome-only metric. The benchmark therefore formalizes a distinction that is often implicit in financial analysis: an answer can be partially correct in its method even when its terminal value is wrong.

The weighting scheme also matters. Because each criterion has an integer weight and the final conclusion always carries meaningful weight, the scoring procedure avoids reducing the benchmark to either trivial step counting or purely end-state correctness. This suggests a compromise between process supervision and answer verification.

## 4. Workflow-grounded evaluation and failure localization

In FBBench, **workflow-grounded** means that the benchmark explicitly evaluates the entire audit trail: **entity identification, retrieval, definition, calculation, and synthesis**. The large number of rubric points enables both **partial-credit evaluation** and **failure localization** [2606.03829].

Partial credit is central to the benchmark’s design. A model that correctly retrieves the **10-K**, identifies the correct **EBIT** line item, but miscalculates the adjustment still receives credit for the earlier steps even if the final figure is wrong. This prevents “flat failure” scoring from obscuring partially solved derivations.

Failure localization is obtained by aggregating criterion pass rates by rubric stage. The paper reports that benchmark points are distributed across **Retrieval (29.5 % of points)**, **Definition (5.8 %)**, and **Calculation (63.2 %)** stages. Because these stages are scored separately, FBBench can identify where an agent breaks down rather than only whether it succeeded overall.

A common misconception in benchmark interpretation is to treat final-answer accuracy as equivalent to derivation quality. FBBench directly contests that reduction. Its results show that final-answer accuracy is a **useful but lossy proxy** for derivation quality, and its rubric decomposition exposes what the paper describes as **“half-solved” derivations** that conventional answer-only grading would classify as failures.

## 5. Experimental setup and empirical results

The initial evaluation covers **ten current frontier and open-weight agents** under a **ReAct-style harness** with up to **50 steps** and four public tools: **web_search, edgar_search, fetch_url, python_exec**. Two **LLM-based graders** independently score each trajectory [2606.03829].

The evaluated models are divided into two groups. The **closed-weight** systems are **Claude Opus 4.7, GPT-5.5, Claude Sonnet 4.6, and GLM 5.1**. The **open-weight** systems are **Qwen 3.6 27B, Kimi K2.6, Gemini 3 Flash, Gemini 3.1 Pro, Gemma 4 31B, and GPT-5.4 Mini**.

The headline results are as follows:

| Metric | Best result | Model(s) |
|---|---:|---|
| Average rubric score | 58.8 % | Claude Opus 4.7 and GPT-5.5 |
| Final-answer accuracy | 44.3 % | GPT-5.5 |
| Rubric criterion pass rate | ~67 % | Top model |

Several empirical patterns are emphasized. First, every model’s rubric score exceeds its final-answer accuracy by approximately **16 percentage points on average**, with **Pearson $r = 0.94$ across models**. This indicates that partial progress is widespread. Second, top closed models cluster around **58–59 % rubric score overall**, but each leads on different analyst workflows such as **Earnings Quality**, **M&A**, or **Valuation**. Third, many failures occur **before arithmetic**. When **Retrieval + Definition** steps are perfect—described as a **“clean setup”**—models still score approximately **85 % on Calculation**, indicating that retrieval and setup are the larger barrier [2606.03829].

These findings materially refine the interpretation of model capability. Arithmetic is not presented as the dominant bottleneck once the relevant sources, entities, and definitions have been established. The residual difficulty lies upstream in workflow construction.

## 6. Significance, future directions, and release

FBBench frames current financial-research agents as substantially below **human audit quality**. No evaluated system exceeds **60 % derivation correctness**, and final-answer accuracy remains below **45 %**. The benchmark therefore positions auditable derivation as an unresolved systems problem rather than a nearly solved QA task [2606.03829].

The authors identify several directions for improvement. They state that targeted advances in **multi-source retrieval, entity disambiguation, and accounting-definition parsing** would likely produce large gains, because retrieval and setup dominate residual errors once calculation is instrumented. They also report that the jagged frontier across workflows implies utility in **learned routing or ensembling**: a **simple router** based on **question workflow and source type** improves rubric score by approximately **4.5 percentage points** over the best single model, with approximately **9 percentage points** of additional headroom to the oracle.

Additional proposed directions include **fine-tuning on rubric-scored derivations**, improving **tool-use policies for financial filings**, building **clarifying-question capabilities**, and developing **specialized prompting or retrieval chains** for underperforming workflows such as **8-K event analysis**. These proposals remain forward-looking, but they are tightly coupled to the failure patterns revealed by the benchmark.

FBBench’s release strategy also reflects its methodological emphasis. The authors state that they are operationalizing and publicly releasing the benchmark’s **questions, rubrics, and grading harness**, including a **50-question subset under CC BY 4.0 with a leaderboard**. This suggests an effort to standardize evaluation around **fully auditable derivations**, a requirement the paper presents as essential for **real-world decision support and regulatory compliance** [2606.03829].

Source: https://www.emergentmind.com/topics/fbbench