FIRM-Bench: A Firm-Level Evaluation Benchmark

• FIRM-Bench is a benchmark archetype defined by realistic, firm-centered evaluations across corporate document QA, financial reasoning, and firmware fuzzing.
• It emphasizes full-workflow realism by integrating retrieval, provenance tracking, and auditable evaluation rather than relying on synthetic, snippet-level tasks.
• Key insights include performance metrics in document QA retrieval (e.g., up to 62.1% accuracy improvements), nuanced audit scoring in financial tasks, and robust bug detection in firmware fuzzing.

FIRM-Bench is not a single uniformly defined benchmark name across recent arXiv literature. The available literature instead suggests three closely related usages: a reference point for firm-level document question answering over long, heterogeneous corporate filings; a broader benchmark archetype for firm-centric financial reasoning, auditing, graph analysis, and prediction; and, in a different technical domain, the concrete firmware-fuzzing target suite FirmBench released with FirmReBugger (Duong et al., 22 Jan 2026). Across these usages, the common denominator is evaluation on realistic firm artifacts and workflows rather than short-context, snippet-level, or purely synthetic tasks.

1. Terminological scope and neighboring benchmarks

Several papers position themselves relative to something like FIRM-Bench without claiming identity with a benchmark of that exact name. Climate Finance Bench states that it is “best understood as a climate-disclosure analogue of firm-level document QA benchmarks such as FinanceBench and, by extension, any FIRM-Bench-style benchmark built around long, heterogeneous corporate filings” (Mankour et al., 28 May 2025). LDBC FinBench explicitly states that it does not mention the term “FIRM-Bench” and should be treated as a closely related benchmark in the same problem space rather than the same benchmark by name (Qi et al., 2023). FinRule-Bench is described similarly: not “FIRM-Bench” by name, but a neighboring benchmark for financial reasoning evaluation (Malarkkan et al., 11 Mar 2026). BigFinanceBench also does not mention FIRM-Bench and is relevant by benchmark characteristics rather than direct naming overlap (Wang et al., 2 Jun 2026).

Benchmark or reference	Relation to FIRM-Bench	Core task
Climate Finance Bench (Mankour et al., 28 May 2025)	“FIRM-Bench-style” analogue	QA over full sustainability reports
LDBC FinBench (Qi et al., 2023)	Not the same benchmark by name	Financial graph transaction workload
FinRule-Bench (Malarkkan et al., 11 Mar 2026)	Neighboring benchmark	Rule-based auditing over financial tables
BigFinanceBench (Wang et al., 2 Jun 2026)	Methodological comparator	Workflow-grounded financial-research agents
FirmBench / FirmReBugger (Duong et al., 22 Jan 2026)	Near-identical name, concrete suite	Bug-based benchmark for firmware fuzzers

This suggests that “FIRM-Bench” currently functions less as a stable proper noun than as a benchmark archetype for firm-centered, workflow-relevant evaluation. In one branch of the literature it refers to corporate-document QA and financial reasoning; in another, it names the specific firmware benchmark FirmBench. The distinction is essential because the surrounding methodologies, metrics, and target artifacts differ substantially.

2. FIRM-Bench as a firm-document QA pattern

The clearest explicit description of a FIRM-Bench-like setting appears in Climate Finance Bench, which treats realistic firm-level QA as an end-to-end retrieval-augmented question answering problem over full reports, not as reading comprehension over pre-extracted snippets (Mankour et al., 28 May 2025). Its corpus contains 33 recent sustainability reports in English drawn from companies across all 11 GICS sectors, with 330 expert-validated question-answer pairs. The benchmark operates effectively as a single 330-question evaluation set, with no train/dev/test split reported.

Its annotation design is particularly informative for firm-document benchmarks. Each row contains 13 fields, including company, fiscal year, question ID, question text, question type, gold answer, source document, page numbers, extracted evidence, and evidence modality. Evidence is typed as text, table, figure, or mixed combinations such as text+table or text+figure. The paper treats this provenance structure as central because many failures in corporate QA arise specifically from table and figure handling.

The task taxonomy is three-part: Pure Extraction, Numerical Reasoning, and Logical Reasoning. The evaluation uses a strict semantic grading scale

$$2\;(\text{correct}),\quad 1\;(\text{incomplete}),\quad 0\;(\text{incorrect}),$$

with agreement analysis between human grading and LLM-as-a-Judge. The reported conclusion is that the benchmark’s chief bottleneck is retrieval: “the retriever’s ability to locate passages that actually contain the answer is the chief performance bottleneck” (Mankour et al., 28 May 2025). In the reported ablations, Minimal RAG achieved 54.8% correct, +BM25 lexical reached 59.1%, and +reranking hybrid reached 62.1%. The benchmark further reports that in the best configuration, numerical reasoning slightly outperforms pure extraction: 69.7% vs. 65.7% correct, because many nominally numerical items collapse into retrieval-plus-normalization.

From a FIRM-Bench perspective, this document-QA branch establishes several durable design commitments: use full documents rather than snippets, keep retrieval in scope, preserve page-level evidence provenance, separate retrieval-only extraction from compositional tasks, and evaluate with partial credit rather than exact match only. Climate Finance Bench also adds an unusual systems dimension by tracking carbon cost and quantization efficiency, with reported per-query emissions for, among others, LLaMA 3.1-8B (full precision) at 2.79 g CO$_2$eq, LLaMA 3.1-8B (4-bit) at 0.70, GPT-4o at 7.18 $\pm 4.00$, and Claude 3.5 Sonnet at 8.15 $\pm 4.52$ (Mankour et al., 28 May 2025).

3. Workflow-grounded auditing and verification

A second major meaning of FIRM-Bench in the literature is as a benchmark family for financial reasoning under explicit audit structure rather than answer-only QA. BigFinanceBench is the clearest workflow-grounded formulation. It defines a 928-item benchmark of open-ended financial-research tasks, authored by 52 financial-research subject-matter experts and audited by 12 separate reviewers, with 15,656 rubric criteria carrying 36,241 rubric points (Wang et al., 2 Jun 2026). Its central claim is that finance-agent evaluation should grade the auditable derivation, not only the final answer. Reported headline results show substantial headroom: the best systems achieve 58.8% rubric score, while final-answer accuracy is materially lower.

The benchmark decomposes analyst work into checkable steps across document retrieval, period selection, accounting definition choice, adjustment logic, and calculation. A fixed-effects analysis is used to argue that much of the residual model separation lies upstream of arithmetic itself. In the notation of the paper,

$C_{mqt} = \alpha_q + \beta_m + \epsilon_{mqt},$

where $C_{mqt}$ is conditional Calculation score for model $m$, question $q$, trial $t$. The reported interpretation is that once models reach a clean Retrieval + Definition setup, remaining differences in Calculation compress sharply (Wang et al., 2 Jun 2026).

FinRule-Bench and FinVerBench refine the same auditing orientation in more formal verification settings. FinRule-Bench evaluates rule verification, rule identification, and joint rule diagnosis over real financial statements paired with explicit accounting principles and deterministic validators (Malarkkan et al., 11 Mar 2026). Its core validator formalism is

$v_r : T \rightarrow \{True, False\},$

and joint diagnosis requires record-level localization through

$_2$0

Across four statement types and 16 rules, the benchmark reports that models do moderately well on isolated verification but degrade sharply on rule discrimination and multi-violation localization. For GPT-4o, overall Rule verification accuracy is 0.640 / 0.736 / 0.738 under zero-shot / few-shot / few-shot+CR, while Diagnosis Step-2 Exact Match is only 0.234 / 0.297 / 0.342 (Malarkkan et al., 11 Mar 2026).

FinVerBench narrows the problem further to financial statement verification: determining whether a set of balance sheet, income statement, and cash flow statement numbers is internally consistent from the information shown to the model (Panda, 28 May 2026). The task is binary at the top level,

$_2$1

but the paper emphasizes that observability is part of the construct. Generated positives whose perturbed field is not rendered are relabeled not enough information and excluded from binary scoring. Its main evaluated subset is a 105-instance observable diagnostic subset consisting of 43 clean and 62 error-injected instances. The central empirical finding is calibration failure: on the original guided-checklist prompt and the unrounded diagnostic subset, nine of fourteen complete LLM runs produce 95–100% false positives on clean statements, whereas one run achieves 0% observed false positives (Panda, 28 May 2026). The paper argues that verification should therefore be treated as calibrated judgment under incomplete observability rather than as generic arithmetic anomaly detection.

Taken together, these benchmarks define a FIRM-Bench-like subfield centered on firm-level auditability: explicit rules, provenance, localization, partial credit, and sensitivity to prompt design and rendering choices.

4. Structured firm data, graphs, and forecasting

A broader interpretation of FIRM-Bench extends beyond documents and auditing to structured firm data, graphs, and predictive tasks. LDBC FinBench is the clearest graph-system benchmark in this neighborhood. It defines a financial graph workload for risk control, anti-money laundering, KYC, and related scenarios over a schema with vertices such as Person, Company, Account, Loan, and Medium, and edges such as transfer, withdraw, deposit, repay, invest, and guarantee (Qi et al., 2023). Its fully specified Transaction Workload contains 12 complex read queries, 6 simple read queries, 19 write queries, and 3 read-write queries, with benchmark execution rules requiring full ACID support, a 95% on-time requirement, and an audited 2-hour measurement window.

V4FinBench represents a different branch: large-scale company-year distress prediction. It contains 1,106,879 company-year observations, 203,900 unique companies, and 131 features from the Visegrád Group economies for 2006–2021, with six prediction horizons $_2$2 (Kostrzewa et al., 11 May 2026). Positive rates range from 0.19% to 0.36%, making it a rare-event benchmark rather than an artificially balanced task. Its main modeling result is that imbalance-aware TabPFN can match or exceed gradient boosting at longer horizons, whereas QLoRA-finetuned Llama-3-8B trails XGBoost on ROC-AUC at every horizon and is generally weaker on $_2$3-score.

A third structured-data variant appears in Industry Aware Firm Level Network Reconstruction, which treats firm-to-firm production network inference as reconstruction from firm marginals and sector-level input–output totals (Devetak et al., 23 Mar 2026). The paper reconstructs a directed weighted network $_2$4 from firm out-strengths, firm in-strengths, sector labels, sector-pair total flows, and a target mean degree / density. Its industry-aware topology model uses

$_2$5

and reports that adding input-output constraints can reduce IO table absolute error from 110% in a strength-only CReM pipeline to 4.10% in the dcIAGM-IPF pipeline. The paper is equally clear, however, that all methods remain weak at recovering microstructural properties and ESRI rankings.

These benchmarks show that FIRM-Bench-like evaluation is not limited to question answering. It also encompasses transaction workloads, rare-event firm prediction, and network reconstruction under partial information, provided the benchmark remains anchored in firm-level objects and operationally meaningful evaluation.

5. FirmBench as a concrete benchmark suite in firmware fuzzing

The only benchmark in the supplied literature whose name nearly coincides with FIRM-Bench is FirmBench, the target suite released with FirmReBugger (Duong et al., 22 Jan 2026). Here the domain is not finance but monolithic firmware fuzzing, and the nomenclature is explicit: FirmReBugger is the framework and service, while FirmBench is the benchmark corpus.

The benchmark addresses a different problem—reliable bug-based evaluation of firmware fuzzers—but many of its design choices are benchmark-theoretically relevant. The paper argues that code coverage and unique crashes are unreliable evaluation proxies in firmware. Its motivating example shows that, for the Thermostat target, patching out an exploitable bug removed more than 35% of the original coverage, implying that measured coverage had been inflated by bug exploitation rather than legitimate exploration (Duong et al., 22 Jan 2026).

FirmBench comprises 61 binary targets and 313 bug oracles across three subsets: FirmBench with 31 binaries and 166 bugs, FirmBenchDMA with 8 binaries and 31 bugs, and FirmBenchX with 22 binaries and 109 bugs. The underlying benchmark is drawn from 33 base binaries containing 187 unique software bugs across 34 distinct CWEs. The paper defines four bug states: Not Reached, Reached, Triggered, and Detected, and implements evaluation through replay of saved fuzzing seeds against external bug oracles called Ravens rather than by modifying the target binary.

This “no target modification” principle is central. A Raven contains reflection hooks and introspection logic in C-like syntax, and the framework replays seeds after the fuzzing campaign to determine bug state. The benchmark thereby attempts to avoid the leaky oracle problem. Its authors evaluate 9 state-of-the-art monolithic firmware fuzzers in 10 repeated 24-hour fuzzing runs per configuration, using a total effort of 10 CPU-years. Across the benchmark sets, fuzzers trigger 181 of 295 benchmark bugs considered in evaluation; the strongest performers are Hoedur with 150 bugs triggered and MultiFuzz with 144. The benchmark also reports a marked difficulty gap between simplified and challenge subsets: 73% of FirmBench bugs were triggered by at least one fuzzer in 24 hours, compared with only 43% of FirmBenchX bugs (Duong et al., 22 Jan 2026).

Although this is a distinct technical field, FirmBench is relevant to FIRM-Bench as an example of bug-based, provenance-aware, replay-based benchmarking that rejects weak indirect metrics in favor of semantically grounded outcomes.

6. Common design principles and open issues

Across these literatures, several common design principles recur. First, realistic benchmarks keep the full workflow in scope. In document QA this means keeping retrieval in the loop rather than assuming the relevant passage is preselected (Mankour et al., 28 May 2025). In financial research it means grading the derivation, not only the delivered answer (Wang et al., 2 Jun 2026). In financial verification it means respecting observability constraints and not scoring hidden perturbations as ordinary binary misses (Panda, 28 May 2026). In firmware fuzzing it means replaying seeds against bug oracles rather than trusting proxy metrics such as coverage or crash counts (Duong et al., 22 Jan 2026).

Second, strong benchmarks emphasize provenance and localization. Climate Finance Bench records page numbers, extracted evidence, and evidence modality (Mankour et al., 28 May 2025). FinRule-Bench localizes violated rules at the record level through $_2$6 (Malarkkan et al., 11 Mar 2026). BigFinanceBench decomposes analyst work into weighted rubric lines and reports failure localization across retrieval, definition, and calculation (Wang et al., 2 Jun 2026). FirmBench distinguishes whether a bug was merely reached, actually triggered, or detected through a crash (Duong et al., 22 Jan 2026).

Third, benchmark validity depends heavily on what is being measured. FinVerBench is explicit that its contribution is a construct-validity conclusion rather than a final leaderboard (Panda, 28 May 2026). The same caution applies more broadly. Climate Finance Bench does not report classical retrieval metrics such as Recall@$_2$7, MRR, or nDCG, so its retrieval claims are downstream-performance based (Mankour et al., 28 May 2025). V4FinBench reports $_2$8-score and ROC-AUC but not PR-AUC despite extreme imbalance (Kostrzewa et al., 11 May 2026). Industry-aware network reconstruction can achieve an almost perfect fit to input-output totals while remaining poor at recovering microstructure (Devetak et al., 23 Mar 2026). These cases show that benchmark results are inseparable from the chosen observables, metrics, and abstractions.

Finally, the literature suggests a persistent misconception: FIRM-Bench should not be assumed to denote a single settled benchmark artifact. In current usage it is better understood as a family resemblance term linking firm-centered evaluation problems—corporate-document QA, workflow-grounded financial analysis, rule-based statement auditing, graph operational workloads, distress prediction, network reconstruction, and, under the specific name FirmBench, firmware-fuzzing evaluation. A plausible implication is that future work may either consolidate these strands under a more stable naming convention or continue to treat “FIRM-Bench” as a benchmark style defined by firm-level realism, auditable provenance, and task-faithful evaluation rather than by one canonical dataset.