---
title: Berkeley Function Calling Leaderboard v4
url: https://www.emergentmind.com/topics/berkeley-function-calling-leaderboard-v4-bfclv4
type: topic
---

# Berkeley Function Calling Leaderboard v4

The Berkeley Function Calling Leaderboard v4 (BFCLv4) is an academic benchmark for evaluating the function-calling capabilities of large language models (LLMs). It systematically quantifies models’ proficiency in selecting, formatting, and executing calls to external APIs or functions based on natural language user requests, across a diverse range of domains and levels of task complexity. BFCLv4 is recognized as a de facto standard for empirical evaluation in function-calling agent research, offering rigorous, multi-faceted metrics and a high degree of transparency and reproducibility.

## 1. Benchmark Design and Evaluation Protocol

BFCLv4 is designed as a multi-domain, multi-language testbed encompassing both synthetic and real-world function-calling scenarios. Each benchmark instance presents the model with a natural language query, a schema of available functions/APIs (including argument names, types, and return types), and, in some cases, multi-turn dialogue histories [2406.18518][2407.00121][2412.17754].

The task for the model is to produce either:
- A valid function call in canonical JSON or code format, or
- "no_call" if no API in the provided schema is suitable.

The required output JSON format is:
```json
{
  "name": "<function_name>",
  "args": { "param1": value1, ... }
}
```
with strict type-matching (e.g., quoted strings, unquoted numerics, literal booleans).

BFCLv4 aggregates multiple task categories:
- Simple: single-API, single-call queries
- Multiple: multiple candidate APIs, single call
- Parallel: repeated calls to a single API
- Parallel Multiple: multiple APIs and multiple calls in a single response
- Relevance Detection: queries for which none of the presented APIs are relevant [2406.18518][2407.00121]

Zero-shot evaluation is the standard regime: no in-context learning or prompt exemplars.

## 2. Core Metrics and Scoring Formulation

BFCLv4 employs orthogonal, automated evaluation metrics with category-specific reporting [2406.18518][2407.00121][2412.17754]:

- **AST Accuracy (ASTAcc)**: Fraction of predictions whose parsed abstract syntax tree (function name + argument names/types) exactly matches the gold reference.
- **Execution Accuracy (ExecAcc)**: Fraction of calls that, when invoked in a sandbox, yield the ground-truth output or the correct HTTP response.
- **Irrelevance Detection (IrrelAcc)**: Fraction of “no_call” cases where the model correctly abstains from outputting a function call.
- **Relevance Detection (RelAcc)**: Fraction of queries requiring calls for which the model emits at least one correct function call.
- **Overall Score**: Default in v4 is the (unweighted or category-weighted) average of the four principal metrics [2412.17754]:
  $$
  \mathrm{Overall} = \frac{1}{4}(\mathrm{ASTAcc} + \mathrm{ExecAcc} + \mathrm{IrrelAcc} + \mathrm{RelAcc}) \times 100
  $$
- **Relevance F1** (Relevance Detection split): 
  $$
  \text{F1} = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}
  $$
  
Per-category (Simple, Multiple, Parallel, Parallel Multiple) and global overall metrics are published, ensuring fine-grained diagnostics and aggregate comparability.

## 3. Dataset Construction and Verification Methodologies

Dataset curation is a multi-stage, semi-automated process, combining data diversity and rigorous verification:

- **Synthetic Data Synthesis**: Leading pipelines such as ToolACE and APIGen utilize hierarchical API evolution, multi-agent dialog simulation, and combinatorial sampling to generate diverse function schemas and user queries across 21–30 high-level categories and 3,000+ endpoints [2409.00920][2406.18518].
- **Verification Stages**:
  - *Format Checking*: Structural adherence to JSON schema, argument validity, and parseability.
  - *Execution Checking*: Candidate calls are executed in sandboxed environments; runtime errors lead to rejection.
  - *Semantic Verification*: LLM-judges validate call correctness and query alignment, often via majority-vote or decomposed expert prompting.
- Quality bottlenecks are quantitatively monitored (e.g., pass rates of ~60% after all filters in ToolACE, final precision ~95% post-APIGen pipeline) [2409.00920][2406.18518].

Notably, adversarial and edge-case data generation is actively used to prevent models from overfitting to "easy" schema patterns, with adversarial refinement loops embedded in data pipelines [2412.17754].

## 4. Model Training Paradigms and Leaderboard Performance

The leaderboard catalogs both closed-source and open-source models, evaluated strictly on held-out test sets. Key methodologies represented include:

- **Supervised Fine-Tuning (SFT)**: Standard approach using high-quality, verified datasets such as those from APIGen and ToolACE.
- **Granular Task Decomposition**: Multi-task architectures (e.g., Granite-20B-FunctionCalling) train on tasks including function chaining, parallel calls, parameter-value mapping, and abstention detection [2407.00121].
- **Process Supervision and Feedback**: Line-level code execution feedback (as in ADC) provides stepwise guidance, boosting format adherence and parameter inference [2412.17754].
- **Reinforcement Learning (RL) with Policy Optimization**:
  - *Group Relative Policy Optimization (GRPO)* and reward-conditioning, as in RC-GRPO, aim to address reward sparsity and improve exploration in multi-turn tool calling scenarios [2602.03025].
  - *Entropy-regularized RL* and strategic CoT exploration enhance robustness on complex, multi-call tasks [2508.05118].

### Representative Results

A selection of models and their metrics are summarized below (all results taken directly from source):

| Model                        | ASTAcc | ExecAcc | IrrelAcc | RelAcc | Overall |
|------------------------------|--------|---------|----------|--------|---------|
| GPT-4-0125                   | 85.50  | 89.25   | 61.35    | 97.56  | 83.42   |
| Meta-Llama-3-70B             | 80.15  | 88.04   | 50.47    | 92.68  | 77.84   |
| xLAM-7B-fc-r                 | 72.77  | 85.68   | 79.76    | 80.49  | 79.17   |
| **ADC (LLM+Adv Datasets)**   | 70.46  | 87.50   | 75.67    | 82.89  | 79.13   |
| **ToolACE-8B**               | ≈92    | ≈90     | ≈89      | --     | ≈91.5   |
| **Qwen2.5-Coder-7B-Instruct (FunRL)** | 90.40   | 81.64    | --      | --      | 86.02   |
| **Granite-20B-FunctionCalling** | 84.11 | 86.50  | 87.08    | --     | 84.71   |
| **RC-GRPO (Qwen-2.5-7B-Instruct)** | -- | --   | --       | --     | 85.0    |

BFCLv4 rankings consistently feature ToolACE’s 8B model as surpassing GPT-4 and Claude-3.5 in overall accuracy, and Granite-20B as leading all openly licensed models. Recent reinforcement learning approaches (FunRL, RC-GRPO) on code-pretrained or reward-conditioned models now match or exceed proprietary systems [2409.00920][2508.05118][2602.03025].

## 5. Robustness, Generalization, and Failure Modes

Recent investigations highlight the sensitivity of function-calling agents to perturbations:
- *Naturalistic Paraphrases*: Rephrasings of the input query often cause exact-match AST accuracy to drop by 13–19 percentage points.
- *Toolkit Expansion*: Introducing semantically related distractor functions yields an average drop of 1–8 points; most errors are due to wrong function selection or incorrect parameter mapping [2504.00914].

Robustness metrics include:
$$
R_{\mathrm{nat}} = \frac{\text{AST}_{\mathrm{nat}}}{\text{AST}_0},\quad R_{\mathrm{stab}} = \frac{\text{AST}_{\mathrm{exp}}}{\text{AST}_0}
$$
These analyses underscore the need for model architectures and training regimes that explicitly target semantic slot-value matching and adversarial query coverage.

## 6. Advances in Data Synthesis and Multi-turn Function Calling

Emerging methodologies for overcoming data and supervision bottlenecks include:
- **Structured Data Synthesis**: Frameworks such as FunReason-MT leverage environment–API interaction graphs, advanced abstraction, and guided iterative reasoning to generate high-quality, logically dependent multi-turn trajectories [2510.24645].
- **Multi-agent and Reward-conditioned RL**: RC-GRPO combines reward-conditioned trajectory policy pretraining and group-relative RL, yielding state-of-the-art multi-turn tool calling results and demonstrating the necessity of explicit reward diversity to prevent reward collapse [2602.03025].

In multi-turn (3–10 step) environments, trajectory-level binary rewards aggregate state accuracy and action correctness, with overall accuracy on realistic agentic tool-use tasks now improved significantly over early SFT and standard RL baselines.

## 7. Implications, Best Practices, and Limitations

BFCLv4 provides an empirically robust platform for evaluating both closed and open models. Rigorous data validation, task diversity, and zero-shot protocols foster results with substantial generalizability. Leading papers recommend:
- Adopting AST-parsing, execution validation, and LLM-judge based semantic checking for both data curation and evaluation
- Integrating adversarial perturbation and tool expansion both in training and robustness testing [2412.17754][2504.00914]
- Designing training pipelines with process supervision, reward diversity, and staged format/semantic objectives

Current limitations include the benchmark’s primary focus on JSON-style single/multi-turn calls, with active research aimed at incorporating richer toolchains, more diverse API paradigms, and improved real-world deployment relevance [2406.18518][2504.00914].

Source: https://www.emergentmind.com/topics/berkeley-function-calling-leaderboard-v4-bfclv4