FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol
Abstract: This paper introduces \textbf{FinMCP-Bench}, a novel benchmark for evaluating LLMs in solving real-world financial problems through tool invocation of financial model context protocols. FinMCP-Bench contains 613 samples spanning 10 main scenarios and 33 sub-scenarios, featuring both real and synthetic user queries to ensure diversity and authenticity. It incorporates 65 real financial MCPs and three types of samples, single tool, multi-tool, and multi-turn, allowing evaluation of models across different levels of task complexity. Using this benchmark, we systematically assess a range of mainstream LLMs and propose metrics that explicitly measure tool invocation accuracy and reasoning capabilities. FinMCP-Bench provides a standardized, practical, and challenging testbed for advancing research on financial LLM agents.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What this paper is about (in simple terms)
The paper introduces FinMCP-Bench, a big “test set” to check how good AI chatbots (LLMs, or LLMs) are at using finance tools in real situations. These tools can fetch real stock data, look up fund holdings, analyze markets, and more. The benchmark focuses on whether AIs can choose the right tools, use them in the right order, and explain results clearly—just like a smart finance assistant would.
The main questions the authors ask
- Can an AI understand a finance question and pick the right tool (or several tools) to solve it?
- Can it use multiple tools in the correct sequence, where each step depends on the previous one (like following a recipe)?
- Can it handle back-and-forth conversations with a user across several messages?
- How do we fairly measure all this, beyond just checking the final text answer?
How the benchmark was built (with everyday examples)
The authors collected real conversation logs (10,000 of them) from a live finance assistant app and then carefully cleaned and reviewed them. They also created tougher, synthetic cases to push AIs to their limits. In the end, FinMCP-Bench has 613 test cases, covering 10 main finance topics and 33 sub-topics, using 65 real tools.
To make the benchmark realistic and varied, each test case fits one of three types:
- Single-tool: one question solved with one tool in one message.
- Multi-tool: one message but multiple tools chained together (like following steps in a recipe, sometimes doing two steps in parallel).
- Multi-turn: a full conversation over several messages, possibly using different tools at different times.
Here’s how they crafted the complex cases:
What is MCP (Model Context Protocol)?
Think of MCP as a universal plug that lets an AI talk to many different tools in a standard way. Instead of each tool using its own strange shape of plug, MCP gives them a shared plug so the AI can connect smoothly.
Making multi-tool chains (the “recipe” idea)
- Tool map (dependency graph): They built a “map” showing which tool’s output can feed into another tool. For example, “get fund holdings” might feed into “risk analysis,” which might feed into “portfolio advice.”
- Query generation: They then created user questions that would naturally require following a path on this map (like a recipe: Step 1 → Step 2 → Step 3).
- Trajectory creation: An AI connected to the real tool server ran through these questions. They kept only those runs that followed the intended tool chain properly.
Making multi-turn dialogs (the “role-play” idea)
They gave the AI a user “persona” (e.g., age, income, goals) and a realistic finance goal. Then the AI played both the user and the assistant to create a natural back-and-forth conversation. These were filtered automatically and then checked by finance experts.
Quality checks (so it’s trustworthy)
First, they used automatic checks to make sure tools ran without errors. Then six finance experts reviewed each sample for:
- Relevance of the question
- Whether the tool chain is complete and logically sound
- If the answer is reliable and traceable (you can see how the data was used)
- Whether the data is up to date
Only high-scoring samples were kept.
How models were scored (with simple analogies)
To judge AIs, the paper doesn’t just read the final answer—it looks at the tools used.
- Tool Recall (TR): Of all the tools you should have used, how many did you actually use? Like packing for a trip: did you remember all the essentials?
- Tool Precision (TP): Of all the tools you used, how many were actually needed? Like not overpacking junk you won’t use.
- Tool F1 (TF1): A balance of the two above—doing well at both remembering what’s needed and not overdoing it.
- Exact Match Rate (EMR): Did your full tool plan match the correct one exactly (including which tools go together in the same step), allowing for parallel steps? This is the strictest score.
What they found and why it matters
They tested six well-known AI models (including three Qwen3 models, DeepSeek-R1, GPT-OSS-20B, and Seed-OSS-36B):
- The Qwen3 family generally did the best overall. But “bigger” wasn’t always “better”—a smaller Qwen3 sometimes beat a larger one on certain scores.
- Single-tool tasks were easier to recall (models often remembered the needed tool), but many models still overused tools, which hurt precision.
- Multi-turn conversations were the hardest: exact matches were rare, showing that staying consistent and logical across several messages is challenging.
- Across different finance scenarios (like market research or investment planning), the top models were strong and balanced, while weaker models dipped more in complex areas.
- As tasks got harder, the best models sometimes improved their balanced scoring because complex tasks rewarded smart planning and correct chaining of tools.
Why is this important? Because in finance, you often need the right data at the right time, and a convincing-sounding answer isn’t enough. You need to show your work, use the right tools, and follow a clear plan.
What this could change in the real world
- Better finance assistants: This benchmark gives researchers a realistic way to spot weaknesses in AI tool use and fix them, leading to more trustworthy assistants for investors and customers.
- Safer, more explainable AI: By focusing on tool steps and traceability, it encourages AIs that can show how they got an answer—not just what the answer is.
- Faster progress: A shared, tough test set helps the whole field compare models fairly and improve faster, especially on complex, multi-step finance problems.
In short, FinMCP-Bench is like a demanding “driving test” for finance AIs—one that checks not only if they reach the destination, but if they followed the right route, used the right tools, and drove safely along the way.
Practical Applications
Overview
FinMCP-Bench introduces a standardized, tool-centric benchmark for testing how LLM agents invoke real financial tools via the Model Context Protocol (MCP). It provides 613 scenarios (single-tool, multi-tool, multi-turn), a tool dependency graph across 65 real financial MCP tools, and explicit evaluation metrics for tool selection and orchestration (Tool Precision/Recall/F1, Exact Match Rate). Below are concrete applications derived from the benchmark’s findings, methods, and innovations.
Immediate Applications
The following items are deployable with current tools/models and modest integration effort.
- Model and vendor selection for financial assistants (Sector: finance/software). Use FinMCP-Bench to A/B test LLMs on scenario- and difficulty-specific TF1/EMR to choose the best model for retail investment chatbots, research assistants, and advisor tools; define SLAs per scenario (e.g., Market Analysis and Research vs. Investment Planning). Assumptions/dependencies: access to benchmark, MCP integration, representative alignment between your production tools and the 65 benchmarked MCP tools.
- Release gating and regression testing in LLMOps (Sector: software/finance). Add FinMCP-Bench as a CI/CD test suite to prevent tool-orchestration regressions (especially for multi-tool and multi-turn flows with dependencies/parallel calls); set pass/fail thresholds using TF1/EMR. Assumptions/dependencies: reproducible test harnesses, stable MCP endpoints, change management acceptance by product teams.
- Prompt and policy tuning for tool use (Sector: finance/software). Use single-tool vs. multi-tool findings (e.g., over-calling reduces precision on easy tasks) to tune prompts, tool-calling policies, and guardrails that reduce unnecessary tool calls; deploy per-scenario prompt templates and planner parameters. Assumptions/dependencies: prompt management infrastructure, permissioning for tool-restricted environments.
- Fine-tuning and reward modeling for tool orchestration (Sector: AI/finance). Leverage multi-tool/multi-turn traces to instruction-tune agents on tool schemas or train reward models for ToolRL-style methods to improve EMR in complex tasks. Assumptions/dependencies: training pipeline and compute, licensing/usage rights for training, data privacy controls.
- Heuristic toolchain planning using the dependency graph (Sector: software/finance). Implement simple planners that select minimal tool chains, schedule parallel calls where safe, and prefetch intermediate results guided by the benchmark’s dependency graph; deploy to reduce latency and failure rates. Assumptions/dependencies: accurate mapping between your tools and benchmark graph; reliable tool semantics.
- Agent QA dashboards for compliance and reliability (Sector: finance/compliance). Monitor TF1/EMR by scenario and conversation length; add “data freshness” and “traceability” checks (mirroring expert review dimensions) to surface compliance risks (e.g., stale market data) and improve auditability. Assumptions/dependencies: logging of tool calls and outputs, data provenance tagging, agreed KPIs with risk teams.
- Retail investor support assistants calibrated to common needs (Sector: finance—wealth/retail). Deploy assistants tuned to high-frequency benchmark scenarios (e.g., Market Analysis, Investment Planning) with improved sequencing of fund/stock lookups, holdings analysis, and cross-source synthesis. Assumptions/dependencies: MCP access to internal/brokerage data, jurisdiction-specific compliance.
- Training and coursework in fintech engineering (Sector: education). Incorporate FinMCP-Bench into practical labs on tool-augmented LLMs, teaching students tool schemas, dependency-aware planning, and multi-turn orchestration. Assumptions/dependencies: academic access to benchmark and an MCP-compatible sandbox.
- Tool inventory rationalization and API design (Sector: software/platform). Use the benchmark’s tool dependency insights to prune redundant tools, define clearer API boundaries, and expose parallelizable endpoints; improve tool ergonomics (argument schemas, error messages) to reduce agent confusion. Assumptions/dependencies: ownership of API/MCP server, change control in platform teams.
- Productization: “MCP Agent QA Harness” (Sector: software). Build a lightweight product or internal service that runs FinMCP-Bench on candidate agents, reports scenario-wise TF1/EMR, flags over/under-calling patterns, and recommends prompt/policy tweaks. Assumptions/dependencies: stable benchmark packaging, internal buy-in.
- Domain-adapted synthetic case generation (Sector: software/finance). Reuse the role-playing multi-turn synthesis method to extend your own domain/test suite (e.g., specialty portfolios, tax-aware advice) and to simulate user personas/goals for targeted QA. Assumptions/dependencies: access to strong LLMs for generation, expert review bandwidth.
Long-Term Applications
These require further research, scaling, standardization, or regulatory/organizational changes before widespread deployment.
- Industry-wide MCP standardization across financial institutions (Sector: finance/standards). Extend and harmonize MCP tool schemas across brokers, banks, and data providers to create interoperable “financial tool ecosystems” that agents can orchestrate reliably. Assumptions/dependencies: industry consortiums, governance, security standards, vendor incentives.
- Certification and audit frameworks for AI agents (Sector: policy/finance). Regulators and model risk teams can formalize certification using tool-centric metrics (TF1/EMR) and traceability/freshness checks as part of model validation and ongoing monitoring. Assumptions/dependencies: regulatory acceptance, mapping metrics to investor-protection outcomes, audit infrastructure.
- Autonomous (or semi-autonomous) multi-tool co-pilots for advisors and operations (Sector: finance). Deploy agents that prepare orders, rebalance proposals, and compliance-ready reports by chaining KYC, holdings, market data, and risk tools, leaving final approvals to humans. Assumptions/dependencies: robust tool accuracy beyond invocation metrics, strict human-in-the-loop controls, SOC/IT approvals, latency SLAs.
- RL-driven tool planners that maximize EMR under constraints (Sector: AI/finance). Train planners with reward functions derived from FinMCP-Bench to handle long dependency chains and reduce error propagation in multi-turn dialogues; integrate safe exploration/off-policy data. Assumptions/dependencies: high-quality reward design, safety layers, offline logs.
- Cross-domain MCPBench suites (healthcare, energy, logistics) (Sector: healthcare/energy/industry). Port the methodology—tool dependency graphs, role-playing synthesis, and EMR/TF1 metrics—to other domains requiring multi-tool reasoning (e.g., patient triage with EHR/imaging tools, grid diagnostics). Assumptions/dependencies: domain-specific MCP servers, privacy compliance (HIPAA/GDPR), expert annotation.
- Explainability and provenance tooling for regulated AI (Sector: finance/compliance/software). Build systems that automatically bind each model statement to the specific tool call outputs (evidence chaining) to satisfy audit and client-disclosure needs; surface gaps when freshness/traceability are weak. Assumptions/dependencies: consistent tool output schemas, provenance metadata, UI for auditors.
- Adversarial and stress testing of tool orchestration (Sector: finance/safety). Develop red-team suites that test agents for hallucinated tools, mis-ordered dependencies, and parallelization bugs; integrate into model risk frameworks. Assumptions/dependencies: curated adversarial cases, safe sandboxes, incident reporting.
- Globalization and multilingual adaptation (Sector: finance/global). Expand benchmarks and agents to US/EU/APAC markets with localized tools, instruments, and regulations; validate across languages and data providers. Assumptions/dependencies: new MCP adapters, localized data rights, multilingual LLM quality.
- Real-time trading and portfolio execution co-pilots (Sector: finance—trading). Extend tool chains from analysis to execution with risk/limit controls, pre-trade checks, and post-trade TCA; start with simulation/sandbox and move to limited production. Assumptions/dependencies: broker connectivity, stringent compliance, sub-second reliability, liability frameworks.
- Model-agnostic agent evaluation platforms in LLMOps (Sector: software/platform). Create horizontal platforms that continuously benchmark any agent against evolving tool suites, track drift, and auto-suggest prompt/policy/model upgrades per scenario. Assumptions/dependencies: integration with model registries/feature stores, scalable orchestration.
- Data/tool marketplaces for MCP adapters (Sector: software/finance). Foster marketplaces where vetted MCP tools (e.g., risk analyzers, ESG scorers) are discoverable and benchmarked with standardized metrics, accelerating safe agent composition. Assumptions/dependencies: commercial agreements, security vetting, consistent schemas.
- Human-agent workflow redesign (Sector: finance/operations). Redesign back-office and advisory workflows to allocate multi-tool chains to agents (data gathering, preliminary analysis) and reserve judgment/approval to experts; measure productivity gains against TF1/EMR improvements. Assumptions/dependencies: change management, training, quality baselines.
- Policy guidance tying tool-use metrics to consumer outcomes (Sector: policy). Develop methodologies that link TF1/EMR to tangible client outcomes (error rates, mis-selling risk), informing future rules on AI disclosures and suitability. Assumptions/dependencies: longitudinal studies, shared metrics between firms and regulators.
Notes across all applications:
- FinMCP-Bench evaluates tool invocation quality (precision/recall, EMR), not financial correctness of final answers; production deployment must add task-level accuracy, risk, and compliance checks.
- Feasibility depends on MCP adoption, stability of underlying tools, data freshness/latency, and jurisdiction-specific regulation.
- The benchmark is rooted in real logs from a Chinese wealth-management context; cross-market generalization requires adaptation of tools, data sources, personas, and regulations.
Collections
Sign up for free to add this paper to one or more collections.