Function Calling in LLMs

Updated 12 July 2026

Function Calling is a mechanism where LLMs use structured API schemas to determine when and how to invoke external tools for task completion.
Research in this area focuses on schema-constrained decision-making, robust evaluation regimes, and optimizing runtime efficiency through methods like reinforcement learning.
Recent advances leverage data synthesis, dynamic tool selection, and execution optimizations to improve accuracy, reduce latency, and enhance safety in LLM-integrated tool use.

In contemporary large-language-model research, function calling (FC) denotes the mechanism by which a model is given a set of function or API specifications, decides whether external tools are relevant, emits a structured invocation with arguments, and may use returned outputs to continue solving a task. The paradigm is treated as a core capability for tool-using agents, copilots, database interfaces, and online question-answering systems, and in several recent model families it is exposed as a native structured interface rather than only as a prompt-level heuristic (Singh et al., 2024, Chen et al., 2024, Ji et al., 18 Sep 2025, Shorten et al., 23 Jan 2025).

1. Task semantics and interface design

FC is usually formulated as a structured decision problem over a tool inventory. Given a user query and a set of available functions, the model must determine whether to answer directly or invoke one or more tools, select the appropriate function name, and populate arguments in a schema-constrained format. Recent work distinguishes at least two subproblems: relevance detection, namely whether any tool should be used at all, and structured argument generation, namely whether the produced call obeys the required schema and parameterization (Chen et al., 2024).

In mechanistic analyses, FC is treated as a built-in capability of modern LLMs such as Llama-3.1, Mistral, and Hermes: instead of emitting free-form text, the model is instructed to produce a structured function invocation, typically JSON-like output. This framing matters because it separates FC from prompt patterns such as ReAct and makes it possible to study FC as a native control interface with its own internal computational effects (Ji et al., 18 Sep 2025).

The interface itself is typically schema-centric. A representative example is database querying through a single unified tool, query_database, whose required collection_name argument is combined with optional search, filtering, aggregation, and groupby_property fields. This design is intended to support mixed queries that combine search, filters, aggregations, and grouping within one structured call, rather than forcing the model to generate database-specific syntax such as SQL (Shorten et al., 23 Jan 2025).

Conversation adds another layer of task structure. In interactive settings, FC includes clarification, confirmation, follow-ups, goal correction, goal switching, chained tool calls, and the possibility that no tool should be called. CONFETTI formalizes this by evaluating multi-turn trajectories in which the model receives prior user turns, prior agent turns, prior function calls, prior function results, and the available API schemas before predicting the next action (Alkhouli et al., 2 Jun 2025).

2. Evaluation regimes and benchmark design

The dominant FC evaluation tradition has emphasized whether the model picked the correct function and arguments. Benchmarks such as the Berkeley Function Calling Leaderboard (BFCL), tau $^2$ -Bench, and ACEBench are described as measuring function selection, argument correctness, tool-use success, or conversational behavior, but not whether a model obeys verifiable formatting instructions embedded inside parameter descriptions (Skripko, 22 Sep 2025).

This omission motivated "Instruction-Following Evaluation in Function Calling for LLMs" (Skripko, 22 Sep 2025), which introduced IFEval-FC. IFEval-FC encodes verifiable constraints directly in JSON schema descriptions, such as lowercase requirements, quotation requirements, punctuation restrictions, spacing rules, comma counts, ISO-format dates, JSON format, or Python list format. The final benchmark contains 750 test cases and 19 instruction types grouped into seven categories. Scoring is fully algorithmic and binary: for a response $r$ and instruction $i$ , $\text{score}(r,i)=1$ if the instruction is followed and $0$ otherwise. The reported result is that no evaluated model exceeded 80% overall accuracy, and even proprietary models including GPT-5 and Claude 4.1 Opus frequently failed basic formatting constraints (Skripko, 22 Sep 2025).

Turn-level conversational evaluation exposes a different failure mode. CONFETTI contains 109 human-simulated conversations, 313 user turns, 86 APIs, and 506 total turn-level FC examples, with an average conversation length of 8.8 turns. Its off-policy evaluation design isolates the next action at a given turn while conditioning on the gold history. The benchmark shows that long conversations and large API inventories remain difficult, and that performance on chained function-calls is severely limited across models; the top reported AST Soft Accuracy is 40.91% for Nova Pro, followed by 35.46% for Claude Sonnet v3.5 and 33.19% for Llama 3.1 405B (Alkhouli et al., 2 Jun 2025).

Robustness evaluation introduces yet another dimension. "On the Robustness of Agentic Function Calling" (Rabinovich et al., 1 Apr 2025) expands a challenging BFCL subset along two axes: meaning-preserving query rephrasings and toolkit expansion with semantically related tools. The original test cases had 2.7 tools on average, while the expanded setting had 5.6. Under rephrasing, top models dropped by about 8–19%, but the study argues that much of this decline reflects brittle exact-match slot filling rather than true semantic failure; 70–90% of the errors in the rephrasing setting come from wrong parameter assignment, often due to exact string matching such as “Miami, Florida” versus “Miami, FL” (Rabinovich et al., 1 Apr 2025).

Uncertainty evaluation adds an execution-oriented perspective. The first systematic UQ study for FC uses BFCL task families such as Simple, Multiple, Parallel, Parallel-Multiple, and Irrelevance, with correctness determined by AST matching. Its main finding is that simple single-sample logit-based scores often outperform multi-sample semantic methods, and that FC-aware modifications such as AST-based clustering and semantically meaningful token selection improve uncertainty estimation in the structured-output setting (Ye et al., 24 Apr 2026).

3. Data synthesis, supervision, and reinforcement learning

A central research theme in FC is that data quality, coverage, and verification may matter as much as model scale. ToolACE addresses this directly through an automatic agentic synthesis pipeline with three components: Tool Self-evolution Synthesis for API generation, Multi-Agent Interactive dialog generation, and a Dual-Layer Validation Process combining rule-based and model-based checks. The pipeline constructs an API pool of 26,507 distinct APIs across 390 domains, including nested parameter types, and produces training data covering single, parallel, dependent, and non-tool-use cases. ToolACE-8B reaches 91.41 overall accuracy on BFCL-v1 and 85.77 on BFCL-v2, surpassing several GPT-4-class systems in the reported tables (Liu et al., 2024).

Multi-turn data synthesis has required different machinery. FunReason-MT is designed to overcome what it calls the complexity barrier in multi-turn FC data by combining Environment-API Graph Interactions, Advanced Tool-Query Synthesis, and a Guided Iterative Chain that validates generated chains of thought against ground-truth call sequences. Training with 10,000 multi-turn samples on Qwen3-4B-Instruct-2507 raises BFCLv3 multi-turn performance from 15.75 in the base model to 46.90 with supervised fine-tuning and 56.50 with RL, while also improving single-turn performance and transferring to BFCLv4 Web Search and Memory tasks (Xu et al., 28 Oct 2025).

Instruction-following data also contributes materially. "Enhancing Function-Calling Capabilities in LLMs" (Chen et al., 2024) reports that mixing IF-110k instruction-following data with FC-110k function-calling data improves both AST Summary and Relevance Detection relative to FC-only training. The same study introduces a Decision Token mechanism using <|answer|> and <|use_tool|>, and shows that pairing this mechanism with synthetic non-function-call data improves relevance detection to 57.50% or 65.42%, depending on prompt format, albeit with a small drop in AST Summary (Chen et al., 2024).

Reinforcement learning has been adapted to FC in multiple forms. FunRL augments Group Relative Policy Optimization with entropy-based exploration on chain-of-thought tokens and uses a two-stage data cleaning pipeline based on LLM evaluation and AST validation. Starting from 60,000 xLAM samples and retaining 58,759 cleaned samples, the method yields 86.02% overall accuracy on BFCL v2 for Qwen2.5-Coder-7B-Instruct, with particularly strong gains on complex multi-function scenarios (Hao et al., 7 Aug 2025). A related adversarial augmentation line trains a query model with RL to rewrite seed queries into semantically valid but failure-inducing variants, then fine-tunes the FC model on those bad cases. The reported gains are strongest for smaller models; for Qwen3-0.6B, the round-2 RL-trained FC model reaches 44.21% overall accuracy, compared with 42.60% for the baseline LLM+RL setup (Guo et al., 27 Jan 2026).

4. Runtime optimization and inference scaling

As FC moved into real agent deployments, the research emphasis broadened from pure accuracy to latency, token cost, power, prompt length, and concurrency. The resulting methods operate at distinct layers: tool-space construction, inference-time search, retrieval, execution scheduling, and decoding.

Method	Mechanism	Reported result
LLM-Tool Compiler (Singh et al., 2024)	Runtime fusion of similar tools into unified fused tools	Up to 4× more parallel calls; token costs and latency reduced by up to 40% and 12%
Less-is-More (Paramanayakam et al., 2024)	Fine-tuning-free dynamic tool selection for edge devices	Execution time reduced by up to 70%; power consumption by up to 40%
DTDR (Patel et al., 18 Dec 2025)	Dynamic retrieval conditioned on query and execution history	Success-rate gains of 23%–104% over static retrievers
ToolPRM (Lin et al., 16 Oct 2025)	Fine-grained beam search with a process reward model	ToolPRM reaches loss 0.0286, step acc 99.11%, trajectory acc 99.38%
AsyncFC (Feng et al., 14 May 2026)	Future-based asynchronous execution without model changes	BFCL v4 Web Search speedups of 1.26× for AsyncFC(S) and 1.12× for AsyncFC(P) with preserved accuracy
SimpleTool (Shi et al., 4 Feb 2026)	Special-token compression plus parallel decoding heads	3–6× end-to-end speedup, up to 9.6×; 61.2 ms P50 on quantized Qwen3-4B

LLM-Tool Compiler changes the tool inventory seen by the agent rather than the reasoning procedure itself. A GPT-based fuser groups tools at runtime and exposes fused functions, while an executor de-fuses them into underlying primitive operations. This reduces model round-trips and increases the chance that related operations are bundled into one decision (Singh et al., 2024).

Retrieval-centric methods instead shrink the candidate set. Less-is-More first asks the LLM, without any tools in the prompt, to describe the ideal tools needed for the query, then retrieves a subset of actual tools from an embedding space and falls back to the full tool list when confidence is low. On edge hardware, this yields substantial success, time, and power gains (Paramanayakam et al., 2024). DTDR extends this idea to multi-step settings by conditioning retrieval on both the initial query and the executed history, $\omega(q,f_{0:t-1})$ , thereby adapting the retrieved tool subset as the plan unfolds (Patel et al., 18 Dec 2025).

Inference scaling for FC must contend with the irrecoverability of structured errors. ToolPRM formalizes FC as a sequence of intra-call states such as selecting the function name, selecting a parameter name, filling a parameter value, and terminating the call, and trains a fine-grained process reward model on 192,061 annotated samples. Its main principle is “explore more but retain less”: broaden local search while pruning aggressively, because wrong function or argument choices are often unrecoverable downstream (Lin et al., 16 Oct 2025).

Execution-layer and decoding-layer optimizations attack latency from different directions. AsyncFC decouples decoding from execution by returning symbolic futures immediately and allowing the model to continue reasoning over unresolved results. This introduces decode–execution overlap and inter-function parallelism without fine-tuning or protocol changes (Feng et al., 14 May 2026). SimpleTool instead restructures generation itself: 17 special tokens compress low-entropy structural tokens by about 4–6× and act as mode selectors for parallel heads that independently decode the function name and arguments. On quantized Qwen3-4B, the method reaches 61.2 ms P50 latency on RTX 4090, corresponding to 16 Hz real-time control (Shi et al., 4 Feb 2026).

5. Robustness, safety, and uncertainty

FC improves the actionability of LLMs, but it also creates brittle interfaces and new attack surfaces. Robustness studies show that clean benchmark accuracy can obscure substantial sensitivity to paraphrase, toolset growth, and evaluation conventions. Under query rephrasing, the strongest BFCL models degrade materially, but much of the drop is attributable to strict exact-match evaluation of parameter values rather than genuine semantic confusion. Under toolkit expansion, by contrast, failures are more substantive: wrong function choice, wrong number of functions, wrong parameter assignment, and parameter hallucination become more common (Rabinovich et al., 1 Apr 2025).

Mechanistic safety work adds a more nuanced picture. "Digging Into the Internal: Causality-Based Analysis of LLM Function Calling" (Ji et al., 18 Sep 2025) finds that FC-based instructions do more than format outputs: they alter internal computation. Layer-level and clause-level interventions indicate that FC makes causal effects more concentrated, increases the influence of middle layers, and aligns clause importance more strongly with the core malicious intent of jailbreak prompts. In a safety-robustness evaluation on Wildjailbreak, FC-based instructions improve malicious-input detection by over 135% on average relative to semantically equivalent conventional prompting, while keeping helpfulness drops small on MMLU-Pro (Ji et al., 18 Sep 2025). This suggests that FC can function as a behavior-shaping interface as well as a tool-use interface.

At the same time, blue-team evaluations show that FC models are not safe by default. A study of four open-source FC models on BFCL-derived data evaluates three attacks—Direct Prompt Injection, Simple Tool Poisoning, and Renaming Tool Poisoning—and eight defenses. Baseline accuracy ranges from 66% to 92% depending on the model, while attack success rates can be very high, especially for prompt injection and description poisoning. The strongest practical defenses in the reported experiments are Description Rewriting for tool poisoning and cryptographic Watermarking for tool authenticity, but the paper concludes that current LLM-based guardian defenses have either narrow coverage or false-positive rates that are too high for real-world deployment (Dolcetti et al., 14 Jan 2026).

Uncertainty quantification is therefore increasingly treated as a pre-execution guardrail. In FC, multi-sample methods such as Semantic Entropy show no clear advantage over simple single-sample methods such as greedy negative log-likelihood, contrary to some free-form QA settings. However, FC-specific adaptations matter: clustering alternative calls by AST structure improves multi-sample UQ, and restricting token-level uncertainty to semantically meaningful positions improves both discrimination and calibration for single-sample scores. On mixed BFCL-style settings including irrelevance, $\text{G-NLL}_{\text{SMT}}$ reaches mean AUROC 0.80, and abstaining on the most uncertain cases substantially raises executed-call accuracy (Ye et al., 24 Apr 2026).

6. Application domains and unresolved problems

The application range of FC is broad, but the most mature studies are those in which structured calls interface with external systems that are operationally meaningful. Database querying is one such case. By modeling database access as a unified tool rather than SQL generation, "Querying Databases with Function Calling" (Shorten et al., 23 Jan 2025) reports Exact Match scores of 74.3% for Claude 3.5 Sonnet, 73.7% for GPT-4o mini, and 71.8% for GPT-4o. The study finds that models are highly effective at boolean-property filters and routing to the correct collection, but they struggle with text-property filters and sometimes choose not to call the tool at all (Shorten et al., 23 Jan 2025).

Industrial deployment has begun to close the loop between offline benchmarks and online traffic. A data-driven FC pipeline for online financial QA constructs a periodically updated dataset from real queries, augments blind spots in parameter-value coverage via AugFC, and trains with supervised fine-tuning followed by GRPO-based reinforcement learning. The system is reported as deployed in YuanBao, where online evaluation shows F1 score +39.1%, tool execution rate +40.7%, and final answer accuracy +20.3% (Tang et al., 7 Apr 2026).

Despite these advances, the literature converges on several open problems. Precise instruction following in FC remains unsolved: schema-embedded formatting constraints are still violated even by frontier proprietary models (Skripko, 22 Sep 2025). Multi-turn orchestration remains substantially harder than single-turn selection, as shown by the large gap between simple action turns and long chained function-calls in CONFETTI (Alkhouli et al., 2 Jun 2025). Structured-output inference scaling appears to obey a different regime from free-form reasoning, because many FC errors are unrecoverable once committed (Lin et al., 16 Oct 2025). Multilingual FC also remains data-sensitive; careful translation pipelines that preserve function names and JSON structure can improve Traditional Chinese performance, but naive translation is unsafe for tool metadata and argument structure (Chen et al., 2024).

Taken together, these results portray FC not as a narrow formatting feature but as a full research area at the intersection of structured generation, tool retrieval, systems optimization, evaluation methodology, safety, and domain adaptation. The most plausible implication is that future progress will depend less on any single trick than on coordinated advances in schemas, data synthesis, search, execution runtimes, and guardrails around when and how tool calls are allowed to execute.