FinResearchBench: Evaluating Financial Research

• FinResearchBench is a domain-specific benchmark that evaluates financial research reports using a logic-tree-based Agent-as-a-Judge pipeline.
• It extracts the argumentative structure of reports to measure analysis depth, width, and information density through rule-based and LLM-evaluated metrics.
• The framework aids deep research agents in producing professionally structured outputs while highlighting challenges like reliable multimodal synthesis in financial contexts.

Searching arXiv for FinResearchBench and closely related financial benchmark papers to ground the article in current literature. FinResearchBench is a domain-specific benchmark and evaluation framework for financial research agents that evaluates open-ended research outputs through a logic-tree-based Agent-as-a-Judge pipeline rather than direct assignment of a single overall score to a final report. It is designed for deep research agents operating in finance, where quality depends on domain terminology, factual support, analytical structure, source attribution, and professional presentation. The framework is presented as the first logic-tree-based Agent-as-a-Judge system for financial research agents and covers 70 typical financial research questions across 7 frequently encountered task types in the financial research domain (Sun et al., 22 Jul 2025).

1. Conceptual motivation and evaluation target

FinResearchBench is motivated by two claims. First, deep research agents are increasingly important because they can browse the web, call tools, collect information, and generate long, source-attributed reports. Second, financial research is unusually difficult to evaluate automatically because tasks are open-ended, there is usually no single ground-truth answer, and report quality depends on structure, evidence, reasoning, and professional presentation. The paper argues that directly asking an LLM to assign a single overall quality score is unreliable, especially for long, open-ended financial reports (Sun et al., 22 Jul 2025).

The framework treats professional financial writing as a process with three stages: pre-writing, writing, and re-writing. In this formulation, the “underlying structure of ideas, arguments, evidences, etc.” is the logic tree. FinResearchBench therefore attempts to reverse the writing process: instead of judging only surface prose, it extracts the report’s argumentative structure and then evaluates that structure. This suggests that the benchmark is aimed not merely at answer correctness, but at the analyst-style organization of claims and supporting evidence (Sun et al., 22 Jul 2025).

2. Benchmark construction and task coverage

The benchmark contains 70 questions, with 10 typical research problems per category, across 7 key task types. These 7 categories are reported to cover more than 95% of online traffic in the product data. Queries are sampled from the online product AI Cashcat under strict privacy protection, including anonymization, removal of personally identifiable information, and disposal of session information. A general-purpose LLM is used to classify queries into categories; the authors consulted 3 domain experts, created a union of categories, and removed semantically duplicate answers. Domain experts from finance news agencies, security company research departments, and professional investors were then interviewed to generate the 70 benchmark questions (Sun et al., 22 Jul 2025).

Task type	Representative focus
Stock-Specific Analysis	Deep research on a single company
Event Analysis	Market or supply-chain effects of a major event
Stock Selection from Given Plate	Selecting securities within a specified theme
Sector Analysis	Industry development trends
Investment Morning Brief	Daily pre-market briefing
Post-Market Recap	Fund-flow or market-close synthesis
Expert Opinion	Summarization of sell-side analyst views

The appendix gives one example for each category, including Geely Automobile for stock-specific analysis, Trump tariffs and Apple supply chain for event analysis, eVTOL sector selection, cosmetics industry development trends, a daily morning brief, fund-flow analysis for post-market recap, and summarization of sell-side analyst opinions. The paper does not enumerate all 70 benchmark questions in the provided text, but it does specify the seven task families and their representative examples (Sun et al., 22 Jul 2025).

3. Logic tree extraction and the Agent-as-a-Judge pipeline

The core pipeline has three stages: extract the logic tree from the research output, evaluate the tree with another LLM-based judge and rule-based metrics, and then produce scores across multiple dimensions of financial research quality. The extracted logic tree serves as intermediate information for evaluation rather than as a user-facing output (Sun et al., 22 Jul 2025).

Logic-tree extraction is a two-step process derived by “reversing the writing method.” In the first step, a well-prompted LLM extracts the report into a hierarchical tree whose nodes are of two types: argument type and evidence type. Argument nodes contain claims, hypotheses, and analytical insights derived from reasoning; evidence nodes contain concrete facts, data points, statistical evidence, and verifiable statements. The extraction must preserve the original report and must not add or interpret any new information. In the second step, a second well-prompted LLM evaluates whether each argument node is supported by its subordinate argument nodes or evidence nodes. The resulting tree is therefore არა a mere topic outline; it is intended to encode argumentative support relations (Sun et al., 22 Jul 2025).

This design is meant to make evaluation more robust than direct end-to-end scoring. The paper’s explicit claim is that the logic tree captures intermediate evidence about the report’s reasoning process, allowing the judge to assess whether claims are supported, whether the analysis is sufficiently deep, and whether the report is structurally coherent. A plausible implication is that FinResearchBench evaluates financial research reports as structured analytical artifacts rather than as long-form free text alone (Sun et al., 22 Jul 2025).

4. Scoring dimensions and formal metrics

FinResearchBench combines rule-derived structural metrics with LLM-evaluated qualitative metrics. The rule-derived metrics are computed from the extracted logic tree. “Analysis Width” is based on the average number of child nodes and is defined as

$$S_{\text{width}} = \min\left(100, \max\left(0, 33.33 \cdot (\bar{N}_{\text{child}} - 1)\right)\right).$$

“Analysis Depth” is defined as a weighted combination of maximum depth and average leaf depth:

$$S_{D_{\max}} = \min\left(100, \max\left(0, 25 \cdot (D_{\max} - 2)\right)\right),$$

$$S_{\bar{D}_{\text{leaf}}} = \min\left(100, \max\left(0, 40 \cdot (\bar{D}_{\text{leaf}} - 1.5)\right)\right),$$

$$S_{\text{depth}} = 0.4 \cdot S_{D_{\max}} + 0.6 \cdot S_{\bar{D}_{\text{leaf}}}.$$

“Information Density” combines total node count and evidence-node ratio:

$$S_{\text{nodes}} = \min\left(100, \max\left(0, 2 \cdot (N_{\text{total}} - 5)\right)\right),$$

$$R_{\text{evidence}} = \begin{cases} \frac{N_{\text{evidence}}}{N_{\text{total}}} & \text{if } N_{\text{total}} > 0 \ 0 & \text{otherwise} \end{cases},$$

$$S_{\text{density}} = \min\left(100, 100 \cdot R_{\text{evidence}}\right),$$

$$S_{\text{info}} = 0.7 \cdot S_{\text{nodes}} + 0.3 \cdot S_{\text{density}}.$$

“Paragraph Richness” is a piecewise function of the average number of words per subtitle. For subjective evaluation, the framework uses an LLM with rubrics for Articulation Professionalism, Logic Consistency, and Viewpoint Clarity; the main result table also includes Visual-Linguistic Synergy, Articulation Coherence, and Analysis Framework Quality. The reported final score is a weighted aggregate in which each dimension has weight 10% (Sun et al., 22 Jul 2025).

The paper separately evaluates logic-tree extraction quality using three similarity metrics against ground-truth trees:

$$S_{\text{nodes}} = 1 - \frac{|N_a - N_b|}{\max(N_a, N_b)},$$

$$S_{\text{depth}} = 1 - \frac{|d_a - d_b|}{\max(d_a, d_b)},$$

$$S_{D_{\max}} = \min\left(100, \max\left(0, 25 \cdot (D_{\max} - 2)\right)\right),$$0

where $$S_{D_{\max}} = \min\left(100, \max\left(0, 25 \cdot (D_{\max} - 2)\right)\right),$$1 are total node counts, $$S_{D_{\max}} = \min\left(100, \max\left(0, 25 \cdot (D_{\max} - 2)\right)\right),$$2 are average depths, and $$S_{D_{\max}} = \min\left(100, \max\left(0, 25 \cdot (D_{\max} - 2)\right)\right),$$3 are average children counts (Sun et al., 22 Jul 2025).

5. Experimental setup, benchmark results, and extraction reliability

The experiments use Claude 3.7 Sonnet as the foundational LLM for the agentic evaluation system. In the first stage, FinResearchBench is run on several leading international and domestic AI agents, including the finance-focused deep research agent Cashcat Pro. In the second stage, the top 3 agents are compared against 20 financial research reports from reputable institutions, focusing on stock-specific analysis and sector analysis; the report topic is extracted from each human report, and the agents generate corresponding reports under the same evaluation framework (Sun et al., 22 Jul 2025).

The main table reports Gemini with a Final Score of 76.60, OpenAI with 74.15, and Cashcat Pro with 70.73. The paper highlights Gemini as strong in Articulation Professionalism, Articulation Coherence, Analysis Framework Quality, and Analysis Depth, while OpenAI is reported as strong in Logic Consistency, Viewpoint Clarity, and Paragraph Richness. Cashcat Pro is described as showing strong Visual-Linguistic Synergy and strong structure and methodology. Against these systems, the Expert Written Report scores 81.43 overall, with especially high scores in Visual-Linguistic Synergy at 100.00 and Paragraph Richness at 99.96. The stated conclusion is that top agents are competitive but human experts still outperform them overall, especially in holistic synthesis and integrated presentation (Sun et al., 22 Jul 2025).

Because the framework depends on reliable tree extraction, the paper separately evaluates extraction on 100 report–ground-truth logic tree pairs with different tree widths and depths. Reported similarity scores are 0.54 for Total Nodes Similarity, 0.85 for Depth Similarity, 0.76 for Width Similarity, and 0.72 for Average Similarity. A bin analysis by ground-truth tree size reports average similarity of 0.61 for $$S_{D_{\max}} = \min\left(100, \max\left(0, 25 \cdot (D_{\max} - 2)\right)\right),$$4, 0.83 for $$S_{D_{\max}} = \min\left(100, \max\left(0, 25 \cdot (D_{\max} - 2)\right)\right),$$5, and 0.71 for $$S_{D_{\max}} = \min\left(100, \max\left(0, 25 \cdot (D_{\max} - 2)\right)\right),$$6. The paper also reports a Spearman correlation between extracted tree size and ground-truth tree size of $$S_{D_{\max}} = \min\left(100, \max\left(0, 25 \cdot (D_{\max} - 2)\right)\right),$$7 with $$S_{D_{\max}} = \min\left(100, \max\left(0, 25 \cdot (D_{\max} - 2)\right)\right),$$8, and states that the method is effective especially up to about 16,000 words but begins to plateau for extremely long reports. The interpretation given is that extraction is most robust for moderately complex trees and weaker for very small or very large trees (Sun et al., 22 Jul 2025).

6. Position within financial benchmark research and reported limitations

FinResearchBench belongs to a broader wave of finance-specific evaluation frameworks, but its emphasis is specifically on long-form research-report evaluation through extracted argumentative structure. Related work has targeted adjacent capabilities: Finance Agent Benchmark evaluates 537 expert-authored real-world finance research problems in an agentic harness with Google Search, EdgarSearch, ParseHTML, and RetrieveInformation (Bigeard et al., 20 May 2025); BigFinanceBench evaluates 928 open-ended financial-research tasks with 15,656 rubric criteria and 36,241 rubric points, emphasizing auditable derivations and partial-credit rubric scoring (Wang et al., 2 Jun 2026); FinSearchComp evaluates realistic, expert-level open-domain financial search and reasoning over 635 questions (Hu et al., 16 Sep 2025); Fin-RATE benchmarks detail reasoning, cross-entity comparison, and longitudinal tracking on 7,500 SEC-filing QA instances (Jiang et al., 7 Feb 2026); FinTrust evaluates trustworthiness across seven dimensions, including safety, fairness, privacy, transparency, and knowledge discovery (Hu et al., 17 Oct 2025); and FinRule-Bench evaluates rule verification, rule identification, and joint rule diagnosis over real-world financial statements and explicit accounting principles (Malarkkan et al., 11 Mar 2026).

This suggests that FinResearchBench occupies the report-evaluation layer of the finance-agent benchmark landscape. Its distinctive contribution is not a retrieval corpus, a question-answer dataset, or a point-weighted derivation rubric, but a logic-tree-based Agent-as-a-Judge mechanism for assessing research outputs as structured arguments (Sun et al., 22 Jul 2025).

The paper reports two main limitations. First, there remains a substantial gap between AI agents and human experts in creating meaningful visuals such as charts, graphs, and diagrams; future work is proposed on multimodal generation so that visuals become informative and tightly aligned with textual analysis. Second, logic-tree extraction is less robust for very short reports and very long reports, making extraction sensitivity to document length a central reliability issue. The benchmark is therefore both an evaluation instrument and a statement about what remains difficult in financial research automation: not only factual correctness and fluency, but also breadth of analysis, depth of argumentation, evidence density, logical consistency, and multimodal presentation quality (Sun et al., 22 Jul 2025).