FinanceBench SEC Filings Dataset

• FinanceBench SEC Financial Filings Dataset is a comprehensive benchmark that evaluates LLM agents' multi-step reasoning using expert-crafted QA pairs on authentic SEC filings.
• It employs structured JSONL and CSV files with granular metadata to facilitate precise retrieval and quantitative as well as qualitative financial analysis.
• The agentic harness integrates tool-assisted, iterative workflows, mirroring real-world SEC filing analysis and enabling systematic evaluation of financial research tasks.

The FinanceBench SEC Financial Filings Dataset is a rigorously validated benchmark for evaluating LLM agents on real-world financial research and analysis tasks, with a particular emphasis on complex reasoning using recent U.S. Securities and Exchange Commission (SEC) filings. It underpins the Finance Agent Benchmark for assessing agentic LLMs with access to retrieval tools and mimics authentic workflows in the finance industry (Bigeard et al., 20 May 2025). The dataset features granular, expert-authored question–answer (QA) pairs, structured data files, and an agentic harness supporting interaction with live EDGAR filings, enabling systematic measurement of progress and limitations in LLM-driven financial analysis.

1. Dataset Composition and Content

FinanceBench draws on public company filings submitted to the SEC—including 10-K, 10-Q, and 8-K forms—retrieved programmatically from the EDGAR system via its REST API. Each filing is split into semantically meaningful sections, such as MD&A (“Item 2. Management’s Discussion and Analysis”), with boilerplate and HTML tags removed and section text indexed for retrieval. The corpus includes 537 QA pairs, each crafted by domain experts drawn from banking, hedge funds, and private equity. Each QA instance includes:

• Metadata: company ticker, SEC Central Index Key (CIK), filing type, filing date, section ID and title, and provenance URL.
• Questions file (CSV): fields for question ID, category, difficulty (Easy/Medium/Hard), prompt text, answer text, step‐wise expert reasoning trace, and rubric key points.
• Filings file (JSONL): one JSON per document section, with all metadata and up to 2,000 tokens of source text.

A schematic representation is provided in the following table, summarizing the key elements of each file:

File	Format	Main Fields
filings.jsonl	JSON Lines	company_ticker, cik, filing_type, filing_date, section_id, section_title, section_text, url
questions.csv	CSV	question_id, category, difficulty, company_ticker, filing_type, filing_date, prompt_text, answer_text, reasoning_steps, rubric

Expert annotation strictly pairs each question with a reference answer and a list of requisite steps/rubric checkpoints, imposing structure for both human and agent evaluation (Bigeard et al., 20 May 2025).

2. Task Taxonomy and Benchmark Objectives

FinanceBench is organized around nine expert-defined task categories that reflect common analytical flows in real-world financial research. Each category is associated with a precisely defined set of expected skills:

Category	Definition	Count
Quantitative Retrieval	Fetch a single numeric value	102
Qualitative Retrieval	Summarize descriptive sections	97
Numerical Reasoning	Compute percentages or CAGRs	83
Complex Retrieval	Multi-document event synthesis	29
Adjustments	Non-GAAP/addback calculations	43
Beat or Miss	Compare actuals vs. management guidance	69
Trends	Detect and analyze time-series shifts	33
Financial Modeling	Aggregation/projection of balance sheet data	47
Market Analysis	Cross-company comparison	34

Questions span the spectrum from simple retrieval (e.g., “What was the revenue of CRM for Q4 2024?”) to multi-hop synthesis and modeling (e.g., “Project net cash flows for the next fiscal year using reported segment breakdowns”) (Bigeard et al., 20 May 2025).

The dataset includes a public validation set (50 samples; CC BY 4.0), a private validation set (150 samples; research use), and a test set (337 samples; leaderboard), with the schema supporting both closed/private and open evaluation settings.

3. Agentic Benchmarking Infrastructure

The Finance Agent Benchmark incorporates a ReAct-style agentic harness that empowers LLMs to perform tool-augmented analysis workflows. The agent can invoke the following tools in discrete decision steps:

• GoogleSearch(query): Returns relevant top-K web results.
• EdgarSearch(ticker, type, date): Retrieves URLs of SEC filings matching specified parameters.
• ParseHTML(url): Parses a filing into its section-wise structure.
• RetrieveInformation(section_id): Returns the text of a desired filing section.

Agents reason iteratively using “Thought:” and “Action:” prompts, reflecting the Observe-Orient-Decide-Act paradigm and enabling dynamic, evidence-retrieval-driven answering. This explicit modeling of multi-step financial research procedures differentiates FinanceBench from static retrieval datasets (Bigeard et al., 20 May 2025, Nguyen et al., 2024).

A worked example illustrates: To answer, “What was the quarterly revenue of Salesforce (NYSE:CRM) for the quarter ended December 31, 2024?”, the agent calls EdgarSearch, parses the appropriate 10-Q, retrieves “Item 2,” and extracts the correct figure, matching the expert reference answer.

4. Dataset Schema, Splits, and Access Modalities

FinanceBench enforces schema regularity to facilitate reproducibility:

• Question schema fields: unique ID, category, difficulty, company/filer metadata, filing type/date, section excerpt (token-limited), natural language prompt, expert answer, rubric, reasoning trace.
• Output types: single values (for numeric queries), JSON dictionaries (complex reasoning/multi-part answers), or rubric-validated free-text (qualitative tasks).

The public validation split is available under CC BY 4.0 at github.com/vals-ai/finance-agent, with the full dataset and filings distributed via Zenodo. File structure choices (JSONL for filings, CSV for queries) support rapid query and section-level retrieval, enabling scalable benchmarking (Bigeard et al., 20 May 2025, Nguyen et al., 2024).

5. Evaluation Protocols and Performance Metrics

FinanceBench defines multiple performance metrics and reporting protocols, tailored to the nuances of agentic financial tasks:

• Naïve Accuracy:

$$\text{Acc}_\text{naïve} = \frac{\# \text{correct responses}}{\# \text{total queries}} \times 100\%,$$

requiring exact or near-exact matches to truth values or rubrics.

• Class-balanced Accuracy: Category-wise mean accuracy to mitigate class imbalance.
• Contradiction checks: Any contradiction of a rubric point results in zero credit for the response.

Operational costs are explicitly measured:

• Cost per query: Sum of token-based API fees across all tool and answer calls, e.g., OpenAI o3 averaged \$3.79 per question.
• Time per query: Aggregates wall-clock time per query, including tool calls and inference (humans averaged 16.8 minutes; top models 2–4 minutes).

These metrics provide multidimensional feedback on retrieval, reasoning, efficiency, and economic feasibility of LLM-powered financial analysis (Bigeard et al., 20 May 2025).

6. Impact, Comparative Benchmarks, and Practical Insights

The FinanceBench SEC Financial Filings Dataset has enabled empirical comparison of various technical advancements in QA architectures. For instance, domain-specific fine-tuning of retrieval models has consistently yielded higher answer correctness than generator fine-tuning. The adoption of iterative reasoning loops (e.g., OODA) atop the agentic QA pipeline increases accuracy, sometimes approaching chartered financial analyst (CFA) quality levels (Nguyen et al., 2024). The agentic design and section indexing facilitate compositional tasks that are otherwise infeasible for generic QA benchmarks.

FinanceBench is directly compared with contemporaries such as SECQUE (Yoash et al., 6 Apr 2025), which focuses on four high-level categories and leverages a multi-judge LLM evaluation approach rather than an agentic harness. SECQUE includes 565 open-ended questions from 45 filings and features its own structured annotation and automated evaluation, but does not provide the agentic tool ecosystem or detailed section-level indexing found in FinanceBench.

7. Distribution, Licensing, and Limitations

FinanceBench is accessible under dual licensing: the 50-question public validation set is CC BY 4.0, while the full 537-question set is governed by a research-only agreement. The data and code for agent infrastructure, formatting, and evaluation are provided via GitHub and Zenodo, supporting robust reproducibility (Bigeard et al., 20 May 2025). Key limitations include the predominance of recent filings (time span not fully specified) and restrictions on the broader release beyond the validation subset. The dataset’s focus is on SEC filings; it excludes other document types such as proxy statements or ESG disclosures, which are deferred for future expansion, a feature that contrasts with the coverage decisions in SECQUE (Yoash et al., 6 Apr 2025).

A plausible implication is that future benchmarks will extend schema, tool integration, and QA object diversity to address evolving research and regulatory needs in financial NLP and agentic AI.