---
title: 'API-Bank: LLM Benchmark & Financial APIs'
url: https://www.emergentmind.com/topics/api-bank
type: topic
---

# API-Bank: LLM Benchmark & Financial APIs

API-Bank is a term with multiple meanings across open banking ecosystems, large language model (LLM) tool-use evaluation, and secure financial API design. In the context of LLMs, API-Bank is a comprehensive benchmark and dataset for assessing and training tool-augmented language models. In financial systems, it denotes the architectural transformation of traditional banks into platform-centric organizations, built around open and standardized APIs for account, payment, and data services. Across both domains, API-Bank captures the intersection of robust technical infrastructure, rigorous security modeling, and practical evaluation at scale.

## 1. Benchmark for Tool-Augmented Language Models

API-Bank is the first end-to-end benchmark crafted specifically to rigorously quantify, train, and analyze LLMs’ abilities to plan for, retrieve, and invoke external APIs [2304.08244]. Its design addresses three pivotal research questions: current LLM proficiency in tool-use, strategies for enhancement, and persistent obstacles.

The benchmark decomposes tool-use ability into three axes:
- **Call:** Given an API pool and user query, can the model select and parameterize the correct call?
- **Retrieve+Call:** For a large candidate pool, can the model identify and then invoke the appropriate API?
- **Plan+Retrieve+Call:** Can the model autonomously orchestrate multi-step sequences of search and function invocations in response to complex user needs?

### Benchmark Dataset and Evaluation System

API-Bank comprises:
- **Evaluation Suite:** 73 real-world, executable APIs (including weather, search, CRUD, and image generation) with a “frozen” backend, enabling reproducible, deterministic responses.
- **Annotated Dialogues:** 314 human-curated dialogues containing 753 API calls, systematically labeled for granularity and logical consistency.
- **Synthetic Training Corpus:** 1,888 dialogues with 2,138 unique APIs, spanning 1,000 domains, generated via a multi-agent LLM pipeline and passing strict quality controls.
- **Evaluation Metrics:** Exact-match accuracy for API calls and ROUGE-L for response quality. Retrieval precision/recall and end-to-end success rate are reported for retrieval and planning subtasks.

The benchmark supports zero-shot, few-shot, and instruction-tuned evaluations. LLMs interact with input contexts (dialogue histories and API docs/specs) and produce API calls executable by an API-Bank backend.

## 2. Model Architectures and the FANTASE Approach

FANTASE (“Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and Reranking”) represents a methodological advance in LLM tool-use, introducing state-tracked constrained decoding (SCD) and reranking [2407.13945].

### State-Tracked Constrained Decoding (SCD)

SCD strictly enforces constraints derived from API documentation during generation:
- **Extraction and Trie Construction:** For each API, packages, functions, argument names (required/optional), and allowed values are parsed and mapped into a token-level constrained trie indexed by subwords.
- **Finite-State Machine (FSM) State Tracking:** The decoder tracks structural position—package, function, arguments, etc.—and updates the allowed next-token set accordingly, ensuring only permissible continuations at every step.
- **Masked Decoding:** Given LLM output logits, a masked distribution is formed over allowed vocabulary elements, with sampling or beam search restricted to valid continuations. The algorithm ensures exact adherence to API signatures without LLM fine-tuning.

### Reranking Module

After SCD beam search generates B candidates, a lightweight RoBERTa-base discriminator reranks them:
- **Input:** [CLS]〈dialogue + API-docs〉[SEP]〈candidate API call〉[SEP]
- **Scoring:** Candidates are scored against the gold API call by means of a hybrid loss (mean squared error and Spearman rank).
- **Selection:** The candidate with the highest reranker score is output, which empirically yields higher exact-match accuracy than SCD beam search alone.

### Quantitative Performance on API-Bank

| Model/Setting                | Exact-Match Accuracy (%) |
|------------------------------|-------------------------|
| GPT-4                        | 63.66                   |
| GPT-3.5-turbo                | 59.40                   |
| Alpaca-13B (greedy)          | 24.06                   |
| Alpaca-13B + SCD (greedy)    | 56.64                   |
| Alpaca-13B + SCD (beam)      | 62.66                   |
| + SCD beam + reranker        | 64.41                   |
| Lynx-7B (fine-tuned)         | 50.53                   |

SCD delivers near-GPT-4 accuracy at nearly 1/4th the inference time of regular beam search, with the reranker yielding a further increase of approximately 2 percentage points [2407.13945].

## 3. API-Bank as Financial Platform: Open Banking Transformation

“API-Bank” also refers to the strategic, architectural, and organizational reconfiguration of incumbent banks around open APIs [2210.01109]. Post-PSD2, European financial institutions are increasingly adopting API-first platforms, exposing products and services to external parties (FinTechs, consumers) via RESTful interfaces.

### Strategic Framework

Banks’ maturity and innovation are modeled on two axes:
- **Internal digitalization (API portfolio):** Number and variety of internally developed/published APIs (“inside-out”).
- **External partnerships:** Number of equity investments or commercial agreements with FinTechs (“outside-in”).

This produces a typology:
- **Invisible banks:** Large, diversified API stacks and many external partnerships.
- **Technological hubs:** Rich APIs, moderate partnerships.
- **Innovative banks:** High partnerships, modest API portfolios.
- **Follower banks:** Minimum regulatory compliance on both axes.

API portfolios range from a PSD2-mandated floor of 3 to approximately 20, with value-added APIs (e.g., credit assessment, personalized PFM) marking technological leaders.

### Organizational and Regulatory Drivers

- **Branch transformation** (e.g., increased ATM/POS focus)
- **Component-based architectures** for micro-service integration
- **Chief Digital/Chief API Officer appointments**
- **Sandbox environments** for third-party code/testing
- **ECB/EBA** guidelines on system resilience, cross-border operability, and digital inclusion

A major consequence is the shift of customer interaction from physical to digital channels, with APIs as the principal front door to the institution.

## 4. Formal Security Modeling and Verification of API-Bank Protocols

Security is paramount in banking APIs, and formal analyses are used to verify both protocol correctness and resilience against various attacker models [2003.12776, 1901.11520].

### Formal Methods for Open Banking Protocols

- **Modeling Languages:** Extended Alice & Bob notation (AnBx), Web Infrastructure Model (WIM) in Dolev-Yao style.
- **Authentication, Authorization, Integrity Goals:** Encoded as secrets, agent bindings, freshness (nonces), and strongly typed messages.
- **Verification Tools:** OFMC (Open-Source Fixed-Point Model-Checker) and ProVerif.
- **Protocol Flows:** Typically, OAuth2 + mTLS for token, consent, and data exchange among PSU (end-user), AISP (third party), ASPSP (bank), and resource servers.

### Key Properties and Mitigations

Eight core security goals cover secrecy, authentication, and integrity of client/user secrets, consent, and resource access (see Section 3, [2003.12776]). Verification proceeds through:
- Automated proof of these goals under unlimited sessions and strong typing.
- Discovery and rectification of modeling flaws (replay attacks mitigated via fresh nonces).
- Practical testing on real (e.g., NatWest) open banking sandboxes, confirming mTLS, OAuth2 grant security, and replay protection.

### FAPI (Financial-grade API) Analysis

- **High-level Flows:** OAuth2 Authorization Code + PKCE, JWS assertion, mTLS/Token Binding.
- **Attack Classes:** Cuckoo’s Token (token substitution), endpoint misconfiguration (token replay), PKCE challenge attacks, CSRF-style session swapping.
- **Fixes:** Inclusion of token issuer, at_hash binding, request JWTs, and token binding for web flows. Proofs show that these strengthen authorization, authentication, and session integrity [1901.11520].

## 5. Empirical Results and Error Analysis

The API-Bank benchmark yields deep insights into LLM tool-use error modes and highlights practical challenges in financial API design.

### LLM Tool-Use Failures

Frequent error categories on API-Bank benchmark [2304.08244, 2407.13945]:
- Omitted API call
- Hallucinated API
- Missing/wrong input parameters
- Malformed syntax
- Unhandled exceptions
- Missing argument values

SCD in FANTASE eliminates almost all missing/invalid parameter errors. Reranking ensures that the correct sequence, even if not top-ranked by sequence probability, can be recovered from the beam. Residual errors are concentrated in argument-value pairings dependent on fine-grained world knowledge or in cases of ambiguous documentation.

### Financial API Practical Security

Penetration tests and practical sandboxing (e.g., replay without nonce, malformed/JWT signature errors) validate that formal models have operational significance. For real deployments:
- All requests must be bound to client secrets/tokens
- Freshness via nonces is essential for replay resistance
- Whitelisting of redirect URIs and strict JSON schema adherence are non-negotiable

## 6. Implications, Extensions, and Open Challenges

### LLM-Centric Tool Use

FANTASE demonstrates that constrained decoding coupled with lightweight supervised reranking enables API-call generation with near SOTA accuracy, requiring no LLM fine-tuning and offering significant inference speedups [2407.13945]. The approach generalizes to other structured prediction domains (e.g., Text-to-SQL, JSON), provided structural FSM and trie constraints are well-defined.

A challenge remains in accommodating APIs with large or unbounded parameter/value spaces, hybrid or runtime-derived constraints, and APIs at web scale (>10,000 endpoints), where existing Trie/FSM approaches may need hierarchical or vector search augmentation.

### API-Bank in Finance

Most European banks are converging toward the “invisible bank” model, wherein dense internal API portfolios and extensive external partnerships become the basis for digital value creation [2210.01109]. Risks include over-outsourcing (loss of core competence/data governance), digital exclusion (asymmetric access), and cyber-physical resilience (dependency on FinTech providers).

A plausible implication is that API-Bank architectures will increasingly require rigorous, ongoing formal verification not only at the protocol level but across the CI/CD workflow, especially as banks migrate to component-based and composable financial services.

## References

- "API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs" [2304.08244]
- "FANTAstic SEquences and Where to Find Them: Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and Reranking" [2407.13945]
- "Security Analysis of the Open Banking Account and Transaction API Protocol" [2003.12776]
- "An Extensive Formal Security Analysis of the OpenID Financial-grade API" [1901.11520]
- "Digital financial services and open banking innovation: are banks becoming invisible?" [2210.01109]

Source: https://www.emergentmind.com/topics/api-bank