FinTradeBench: A Financial Reasoning Benchmark for LLMs

Abstract: Real-world financial decision-making is a challenging problem that requires reasoning over heterogeneous signals, including company fundamentals derived from regulatory filings and trading signals computed from price dynamics. Recently, with the advancement of LLMs, financial analysts have begun to use them for financial decision-making tasks. However, existing financial question answering benchmarks for testing these models primarily focus on company balance sheet data and rarely evaluate reasoning over how company stocks trade in the market or their interactions with fundamentals. To take advantage of the strengths of both approaches, we introduce FinTradeBench, a benchmark for evaluating financial reasoning that integrates company fundamentals and trading signals. FinTradeBench contains 1,400 questions grounded in NASDAQ-100 companies over a ten-year historical window. The benchmark is organized into three reasoning categories: fundamentals-focused, trading-signal-focused, and hybrid questions requiring cross-signal reasoning. To ensure reliability at scale, we adopt a calibration-then-scaling framework that combines expert seed questions, multi-model response generation, intra-model self-filtering, numerical auditing, and human-LLM judge alignment. We evaluate 14 LLMs under zero-shot prompting and retrieval-augmented settings and witness a clear performance gap. Retrieval substantially improves reasoning over textual fundamentals, but provides limited benefit for trading-signal reasoning. These findings highlight fundamental challenges in the numerical and time-series reasoning for current LLMs and motivate future research in financial intelligence.

• The paper introduces FinTradeBench, a novel benchmark that evaluates LLMs’ ability to integrate financial fundamentals with market trading signals.
• The paper employs a dual-track RAG architecture to retrieve and combine SEC filing text and time-series market data, ensuring context fidelity.
• The paper reveals that while RAG enhances fundamentals reasoning, it degrades pure trading signal analysis, underscoring challenges in hybrid financial reasoning.

Comprehensive Analysis of "FinTradeBench: A Financial Reasoning Benchmark for LLMs" (2603.19225)

Motivation and Problem Statement

"FinTradeBench: A Financial Reasoning Benchmark for LLMs" (2603.19225) addresses a pivotal deficiency in current financial question answering (QA) benchmarks: the omission of reasoning over trading signals and the integration of heterogeneous financial data modalities. While existing benchmarks predominantly evaluate LLMs’ ability to extract and reason over company fundamentals from SEC filings, actual financial analysis demands the synthesis of information from both fundamentals and dynamic market signals (e.g., price trends, momentum, volatility). The paper empirically demonstrates that LLMs frequently fail when trading-based reasoning or fundamentals-market divergence is required, exemplified by tasks such as detecting the absence of a pullback in Nvidia’s stock price (Figure 1).

Figure 1: Performance comparison of proprietary LLMs on a trading signal-focused question, showing that only Claude identified the absence of a pullback, with all LLMs failing to capture the correct trading reasoning.

FinTradeBench Construction and Design

The proposed FinTradeBench is constructed as a large-scale, multicomponent benchmark designed to evaluate LLM reasoning over both company fundamentals and market-based trading signals, with explicit probing of cross-modal (hybrid) reasoning. The dataset comprises 1,400 questions grounded in a decade of NASDAQ-100 data (2015–2025), methodically partitioned into fundamentals-focused (F), trading-signal-focused (T), and hybrid (FT) categories.

The benchmark’s construction relies on a calibration-then-scaling paradigm, securing initial domain-expert-authored seed questions (50 per category) and scaling them using automated LLM response generation, self-filtration, rigorous numerical auditing, and a human–LLM judge alignment step to ensure quality control and reproducibility. The pipeline’s architecture (Figure 2) guarantees interpretability, empirical relevance, and the capture of diverse market phenomena including fundamentals–market divergences.

Figure 2: FinTradeBench design pipeline, outlining the sequential flow from data selection, question taxonomy design, human and LLM calibration, to the scalable generation of historical question instances.

Retrieval-Augmented Generation System

The benchmark is deployed within a domain-specific dual-track RAG architecture, separating the ingestion and retrieval of unstructured textual (SEC filing) and structured time-series (OHLCV market data) evidence (Figure 3). For text, hierarchical chunking and metadata injection ensure context fidelity and mitigate temporal hallucination. For numerical time-series, dedicated retrieval (bypassing semantic re-rankers) and temporal filtering enhance the supply of actionable market evidence. Response generation leverages the TELeR prompting taxonomy for prompt diversity and robust model probing.

Figure 3: Overview of the RAG architecture, exhibiting dual-track retrieval for text and time-series data, integrated candidate response generation, and quality-controlled self-selection.

Model Evaluation and Key Findings

A total of 14 LLMs, spanning proprietary and open-source, large to compact architectures (from $\sim 1$B to 100B+ parameters), are benchmarked under both zero-shot and realistic RAG scenarios. Results are quantitatively robust, with rigorous paired-statistics for retrieval delta evaluation and signal integration quality metrics.

Key findings include:

• RAG provides substantial accuracy gains for fundamentals reasoning and hybrid tasks (e.g., up to +55.1% for R1-Distill-Qwen 32B on FT questions), but generally degrades performance on pure trading-signal questions (up to -19.7%, LLaMA 3.3 Instruct 70B) (see Table below and Figure 4).
• Hybrid reasoning remains a significant challenge, with only models incorporating latent chain-of-thought or advanced reasoning achieving nontrivial gains. For example, DeepSeek-R1 attains +39.8% improvement for hybrid queries, implying that inductive capability (beyond instruction tuning) is critical in these cross-modal scenarios.
• Model family and pretraining corpus composition are highly determinative. LLaMA variants, despite parameter count, are susceptible to information overload and context distraction from dense or unstructured financial data, whereas Qwen and distilled DeepSeek architectures are more robust. This points to architecture and data mixture as central rather than parameter scale per se.
• RAG introduces an information overload effect, where factual anchoring improves, but precision of expert indicator extraction and reasoning depth drop; Golden Indicator F1 falls by over 56.5% under RAG (global metrics, Figure 4).
• Qualitative analysis reveals that performance ceilings under RAG are a function of preprocessing, not LLM ability: when provided with precomputed, structured context ("ideal RAG"), even mid-scale LLMs execute robust hybrid and trading-signal reasoning.

Figure 4: Global quality metrics (Precision, Recall, F1, Integration, Reasoning Depth) for No-RAG vs RAG, illustrating that RAG boosts factual grounding but reduces analytical precision and reasoning depth.

Implications, Limitations, and Future Directions

FinTradeBench sets a new minimum evaluation standard for LLM-based financial reasoning by incorporating temporal, heterogeneous, and conflicting signals, crucial for tasks such as risk assessment, trading strategy evaluation, and real-world financial decision support. The modal performance split (fundamental vs trading signal) documented here challenges claims of general reasoning proficiency by today's LLMs and delineates concrete architectural and corpus gaps. Architectures relying heavily on context retrieval are shown to be brittle without adequate pretraining on structured and time-series financial data; simply adding evidence is detrimental without proper pre-parsing or abstraction.

Practically, the results indicate current LLMs are not ready for autonomous financial reasoning on tasks requiring trading signal analysis, especially where integration across market and fundamental data is required. Agentic systems or RAG architectures targeting the financial domain must address the information overload and numerical distraction effect, potentially incorporating intermediate computation or explicit signal extraction/code execution modules. There is a strong indication that future development should move toward more sophisticated agent pipelines that allow for dynamic context compression, integration of external analytic engines for numerical/time-series analytics, or architectural priors for financial data.

On the theoretical front, the systematic benchmarking and evaluation protocol outlined here sets a precedent for other structured-reasoning tasks (e.g., scientific discovery, engineering diagnostics) where mere factual retrieval is insufficient. The fine-grained metrics, including human–LLM judge calibration and golden indicator F1, reveal failure modes that aggregate accuracy scores obscure, promoting a more nuanced and actionable understanding of model limitations versus domain complexity.

Conclusion

FinTradeBench (2603.19225) fills a crucial methodological gap in financial NLP by systematically evaluating reasoning across company fundamentals, trading signals, and hybrid information states. The results challenge the efficacy of retrieval augmentation for time-series-based trading tasks and highlight the importance of model architecture, pretraining data, and context selection in achieving robust financial reasoning. While fundamental signal reasoning can be enhanced via RAG, genuine cross-modal financial intelligence remains elusive for current LLMs, providing a blueprint for future research and architectural intervention. The benchmark’s public release and transparent, extensible pipeline will facilitate rigorous comparison of future financial LLMs and agentic systems, informing both academic development and industrial application.