FBBench: Auditable Financial Research Benchmark
- FBBench is a benchmark for evaluating AI agents' financial research derivations, featuring 928 expert tasks and a comprehensive, audit-friendly rubric.
- It measures both rubric scores and final-answer accuracy to capture methodological rigor and highlight areas of partial progress.
- The design promotes partial-credit evaluation and failure localization across retrieval, definition, and calculation stages for actionable insights.
FBBench, introduced as BigFinanceBench, is a workflow-grounded benchmark for evaluating AI agents on the full derivation of financial-research answers rather than only on isolated subskills or final numbers. It consists of 928 expert-authored, open-ended financial-research tasks, each paired with a ground-truth reference answer and a point-weighted rubric that decomposes the derivation into independently checkable steps. Across 15,656 atomic rubric criteria and 36,241 total rubric points, FBBench is designed to support partial-credit evaluation, auditability, and failure localization across the analyst workflow. Initial evaluation on ten frontier and open-weight agents reports substantial headroom: the best systems achieve only 58.8% rubric score, while the highest final-answer accuracy is 44.3% (Wang et al., 2 Jun 2026).
1. Motivation and problem setting
FBBench is motivated by the observation that financial-research answers are decision-relevant only when another analyst can audit how they were produced: which source was chosen, which period and accounting definition were used, which assumptions were made, and how the calculation was performed. In financial research, analysts must identify and disambiguate the company or ticker, locate and retrieve multiple filings or press releases, normalize accounting definitions such as GAAP vs non-GAAP, period boundaries, and amortization treatment, choose and document assumptions such as software capitalization vs expense, perform stepwise calculations, and present a final figure with explicit units and rounding rules (Wang et al., 2 Jun 2026).
The benchmark is therefore designed to replicate both the open-ended, multi-source nature of analyst queries and the fine-grained, rubrics-based review process used to audit derivations. Existing finance benchmarks are described as largely evaluating isolated subskills or final answers, leaving the auditable derivation under-measured. FBBench addresses that gap by treating the analyst workflow itself as the evaluation target.
This emphasis on derivation has an immediate methodological consequence: a correct number is not treated as sufficient evidence of a correct workflow. Conversely, an incorrect final number does not imply total failure if the earlier stages of the derivation were correct. This suggests a benchmark philosophy aligned with auditability rather than outcome-only scoring.
2. Benchmark composition and task structure
FBBench contains 928 expert-authored, open-ended tasks. Each question was written by a current or former investment banker, private-equity professional, or equity-research analyst and then audited by a separate reviewer. The tasks cover core public-equity workflows, including earnings quality, M&A, valuation, and operating KPIs, and require combinations of cross-filing synthesis, accounting adjustment, scenario projection, and market-data lookup (Wang et al., 2 Jun 2026).
A defining feature is that the queries are both multi-source and assumption-driven. Roughly 30 percent of items require more than one filing or press release. Every question is time-anchored, for example “Fiscal Year Ended December 31 2024,” and any non-obvious assumptions are stated explicitly.
| Component | Value | Description |
|---|---|---|
| Tasks | 928 | Expert-authored, open-ended financial-research questions |
| Rubric criteria | 15,656 | Atomic yes/no checklist items |
| Total rubric points | 36,241 | Sum of all point-weighted criteria |
For each item, authors provide a single reference answer and a rubric decomposed into atomic steps. The paper gives examples such as “Overstated by \$90.1 m”** and **“1.74 % ARR growth”** for reference answers, and rubric criteria such as **“Identifies DAY as ticker,” “Calculates Q3 ’26 retained ARR as \$511.38 m,” and “Notes that NDRR = 97 %.” The mean number of criteria is 16.9 per question. Integer weights range from +1 through +10, and the final conclusion is always assigned a significant share of total points (Wang et al., 2 Jun 2026).
The structure of the tasks makes FBBench closer to analyst-grade review than to fixed-context QA. A plausible implication is that it can separate failures caused by retrieval, accounting interpretation, or numerical execution even when all three appear in a single problem.
3. Rubric formalism and scoring methodology
FBBench scores each agent trajectory—defined as the chain of tool calls, intermediate outputs, and final answer—against every rubric criterion. Criterion receives a binary judgment and has weight . The per-question rubric score is defined as
The benchmark reports model performance by macro-averaging over all 928 questions. In parallel, the final answer is judged correct or incorrect to compute final-answer accuracy (Wang et al., 2 Jun 2026).
This design yields two distinct evaluation signals. The rubric score measures derivational completeness and correctness at the level of individual workflow steps; final-answer accuracy collapses the same task into an outcome-only metric. The benchmark therefore formalizes a distinction that is often implicit in financial analysis: an answer can be partially correct in its method even when its terminal value is wrong.
The weighting scheme also matters. Because each criterion has an integer weight and the final conclusion always carries meaningful weight, the scoring procedure avoids reducing the benchmark to either trivial step counting or purely end-state correctness. This suggests a compromise between process supervision and answer verification.
4. Workflow-grounded evaluation and failure localization
In FBBench, workflow-grounded means that the benchmark explicitly evaluates the entire audit trail: entity identification, retrieval, definition, calculation, and synthesis. The large number of rubric points enables both partial-credit evaluation and failure localization (Wang et al., 2 Jun 2026).
Partial credit is central to the benchmark’s design. A model that correctly retrieves the 10-K, identifies the correct EBIT line item, but miscalculates the adjustment still receives credit for the earlier steps even if the final figure is wrong. This prevents “flat failure” scoring from obscuring partially solved derivations.
Failure localization is obtained by aggregating criterion pass rates by rubric stage. The paper reports that benchmark points are distributed across Retrieval (29.5 % of points), Definition (5.8 %), and Calculation (63.2 %) stages. Because these stages are scored separately, FBBench can identify where an agent breaks down rather than only whether it succeeded overall.
A common misconception in benchmark interpretation is to treat final-answer accuracy as equivalent to derivation quality. FBBench directly contests that reduction. Its results show that final-answer accuracy is a useful but lossy proxy for derivation quality, and its rubric decomposition exposes what the paper describes as “half-solved” derivations that conventional answer-only grading would classify as failures.
5. Experimental setup and empirical results
The initial evaluation covers ten current frontier and open-weight agents under a ReAct-style harness with up to 50 steps and four public tools: web_search, edgar_search, fetch_url, python_exec. Two LLM-based graders independently score each trajectory (Wang et al., 2 Jun 2026).
The evaluated models are divided into two groups. The closed-weight systems are Claude Opus 4.7, GPT-5.5, Claude Sonnet 4.6, and GLM 5.1. The open-weight systems are Qwen 3.6 27B, Kimi K2.6, Gemini 3 Flash, Gemini 3.1 Pro, Gemma 4 31B, and GPT-5.4 Mini.
The headline results are as follows:
| Metric | Best result | Model(s) |
|---|---|---|
| Average rubric score | 58.8 % | Claude Opus 4.7 and GPT-5.5 |
| Final-answer accuracy | 44.3 % | GPT-5.5 |
| Rubric criterion pass rate | ~67 % | Top model |
Several empirical patterns are emphasized. First, every model’s rubric score exceeds its final-answer accuracy by approximately 16 percentage points on average, with Pearson across models. This indicates that partial progress is widespread. Second, top closed models cluster around 58–59 % rubric score overall, but each leads on different analyst workflows such as Earnings Quality, M&A, or Valuation. Third, many failures occur before arithmetic. When Retrieval + Definition steps are perfect—described as a “clean setup”—models still score approximately 85 % on Calculation, indicating that retrieval and setup are the larger barrier (Wang et al., 2 Jun 2026).
These findings materially refine the interpretation of model capability. Arithmetic is not presented as the dominant bottleneck once the relevant sources, entities, and definitions have been established. The residual difficulty lies upstream in workflow construction.
6. Significance, future directions, and release
FBBench frames current financial-research agents as substantially below human audit quality. No evaluated system exceeds 60 % derivation correctness, and final-answer accuracy remains below 45 %. The benchmark therefore positions auditable derivation as an unresolved systems problem rather than a nearly solved QA task (Wang et al., 2 Jun 2026).
The authors identify several directions for improvement. They state that targeted advances in multi-source retrieval, entity disambiguation, and accounting-definition parsing would likely produce large gains, because retrieval and setup dominate residual errors once calculation is instrumented. They also report that the jagged frontier across workflows implies utility in learned routing or ensembling: a simple router based on question workflow and source type improves rubric score by approximately 4.5 percentage points over the best single model, with approximately 9 percentage points of additional headroom to the oracle.
Additional proposed directions include fine-tuning on rubric-scored derivations, improving tool-use policies for financial filings, building clarifying-question capabilities, and developing specialized prompting or retrieval chains for underperforming workflows such as 8-K event analysis. These proposals remain forward-looking, but they are tightly coupled to the failure patterns revealed by the benchmark.
FBBench’s release strategy also reflects its methodological emphasis. The authors state that they are operationalizing and publicly releasing the benchmark’s questions, rubrics, and grading harness, including a 50-question subset under CC BY 4.0 with a leaderboard. This suggests an effort to standardize evaluation around fully auditable derivations, a requirement the paper presents as essential for real-world decision support and regulatory compliance (Wang et al., 2 Jun 2026).