---
title: Multi-Turn Tool-Calling LLMs
url: https://www.emergentmind.com/topics/multi-turn-tool-calling-large-language-models-llms
type: topic
---

# Multi-Turn Tool-Calling LLMs

Multi-turn tool-calling large language models (LLMs) are agent architectures that interact with external functions or APIs over multi-turn dialogues, leveraging both sequential tool invocations and their intermediate results to solve compositional, context-dependent tasks. This paradigm arises from the growing need to move beyond single-step retrieval augmented generation (RAG) and single-turn function calling, allowing LLMs to handle scenarios such as iterative search, data transformation, symbolic computation, and real-world API orchestration. Architectures employ explicit reasoning/planning steps, action selection for tool invocation, structured state tracking, reward-driven learning (via RL), and memory management to support long-horizon interaction and dynamic information flows.

## 1. Formal Problem Definition and Agent Frameworks

The multi-turn tool-calling setting formalizes the agent’s state at turn $t$ as $s_t = (q, h_t, z_0,\ldots,z_{t-1})$, with $q$ the user query, $h_t$ the hidden embedding capturing history, and $z_i$ the outputs of prior tool calls [2510.24126]. The action space $A$ comprises tool invocations—each with arguments—and the final answer. At every turn, the model policy $\pi^{A}_\theta$ can (a) select a tool and generate its arguments, or (b) terminate with the answer. State transitions include appending tool feedback to the context and updating the internal representation.

A canonical multi-turn framework can be described as:
- Observation: full history of queries, tool calls (with arguments), and tool outputs.
- Planning: select which tool to call next or whether to answer.
- Tool Execution: call external API or function, receive and record output.
- Termination: output the answer, possibly with supporting citations or structured responses.

In advanced settings, e.g., "From Tool Calling to Symbolic Thinking: LLMs in a Persistent Lisp Metaprogramming Loop" [2506.10021], agent-tool creation, versioning, and persistent stateful memory (via REPL) enable the model to invent new symbolic procedures and refine them across turns.

## 2. Training Algorithms: Supervised, Preference, and Reinforcement Learning

Training multi-turn tool-calling LLMs involves (i) supervised fine-tuning (SFT) on high-fidelity multi-turn trajectories, (ii) preference optimization, and (iii) outcome-driven RL.

**Supervised approaches** synthesize or collect multi-turn trajectories, where each step involves tool selection, argument filling, tool output parsing, and propagation into subsequent reasoning [2503.07826, 2410.12952]. The loss is typically next-token cross-entropy, possibly masked for assistant turns only.

**Preference learning** (e.g., Magnet [2503.07826]) contrasts positive trajectories (ground-truth tool usage hints) against negative/adversarial trajectories (error patterns injected as context), optimizing an mDPO-like objective that drives the model to prefer the correct sequence over misleading alternatives.

**RL algorithms:** Recent advances employ actor-critic style methods, especially Group Relative Policy Optimization (GRPO) [2510.24126, 2505.01441]. Here, batches of multi-turn trajectories are grouped; the agent’s parameters are updated to increase the likelihood of higher-reward rollouts relative to their groupmates, with KL regularization for stability. The reward function is outcome-centric, incorporating solution correctness, tool citation match, efficiency (fewer calls), and formatting penalties. This enables the agent to discover dynamic strategies, such as iterative refinement, multi-hop search, or error recovery in symbolic tool interaction.

## 3. Data Synthesis for Multi-Turn Function Calling

Curating high-quality multi-turn training data is essential [2510.24645, 2410.12952, 2503.07826]. Synthesis frameworks like FunReason-MT [2510.24645], Magnet [2503.07826], and BUTTON [2410.12952] construct data with compositional trajectories, varied logical dependencies, and realistic tool usage.

- **Graph-based sampling:** Build API relation graphs modeling inter-tool dependencies and simulate environment traversals constrained by prerequisite satisfaction.
- **Advanced query synthesis:** Abstract chains of tool calls into composite "advanced tools," then generate reverse-engineered queries that force the use of such chains.
- **Multi-agent simulation:** BUTTON creates complex scenarios and decomposes tasks bottom-up, synthesizes atomic and compositional subtasks, and then simulates multi-agent dialogue with rigorous turn-by-turn state and function definitions.
- **Guided iterative chain-of-thought:** In FunReason-MT, agents engage in self-critique correction loops, validating function call traces against the API graph and revising their trajectories to enforce full logical coherence.

These frameworks demonstrate that targeted graph exploration and composition-centric tuning yield state-of-the-art multi-turn tool use accuracy (e.g., +40.75 pp multi-turn gain in FunReason-MT RL over base models [2510.24645]).

## 4. Memory Management and Scalability

Tool-calling agents require dynamic memory management—especially in environments with thousands of APIs and finite context windows [2507.21428, 2505.10570]. The MemTool framework [2507.21428] addresses efficient short-term memory in large-scale, multi-turn sessions:

- **Instantaneous Removal Ratio:** $ \mathrm{RemovalRatio} = \frac{\sum_{t=1}^T R_t}{\sum_{t=1}^T A_t} $; measures tool-pruning efficiency.
- **Sliding Window Metrics:** Capture recent memory churn and average residual tool count.
- **Design modes:** Autonomous Agent Mode delegates tool-add/remove decisions to the LLM; Workflow Mode deterministically prunes/searches up front; Hybrid Mode combines upfront pruning with dynamic recovery.

Top-performing LLMs in Autonomous/Hybrid modes maintain 90–94% removal ratio and average 5–10 tools in context, supporting 0.80–0.90 task completion accuracy over 100-turn sessions [2507.21428]. Efficient memory management is crucial for scaling multi-turn tool-calling without exhausting context or incurring excessive inference latency.

## 5. Empirical Results, Benchmarks, and Performance Degradation

Agent performance on multi-turn tool-calling is quantitatively evaluated on benchmarks such as BFCL v3/v4, ToolQuery, MINT, ScaleMCP, and in synthetic test environments [2510.24126, 2510.24645, 2505.10570, 2309.10691]. Key findings include:

- **RL-trained agents (Qwen3-14B+GRPO) achieve 85% accuracy on legal document search (vs. 78% for Gemini 2.5 Pro, 33% for naive RAG), with monotonic improvement as turn count increases past 6, revealing deep search strategies [2510.24126].**
- **Magnified multi-turn gains:** Magnet-14B-mDPO yields 68.01% BFCL-v3 success (vs. teacher Gemini-1.5-Pro-002’s 62.19%) and 73.3% ToolQuery success [2503.07826].
- **Disambiguation-centric finetuning (DiaFORGE) improves dynamic tool-call accuracy by 27–49 pp compared to leading closed-source models [2507.03336].**
- **Long context is still a bottleneck:** As catalog size and tool response length increase (to 128K tokens / 80K tokens), accuracy degrades by up to 85–91% [2505.10570]. Multi-turn dialog length increases induce up to 68–95% drop in AST-match accuracy for all but the largest models.
- **RL, supervised instruction, and feedback:** MINT [2309.10691] demonstrates 1–8% improvement per additional tool call and 2–17% absolute gain from natural-language feedback, but observes that SIFT/RLHF can decrease multi-turn capabilities unless multi-turn examples are explicitly included.

## 6. System Architectures and Symbolic Tool Integration

Tool invocation can be tightly coupled via explicit tagging, generative tokenization, or symbolic code emission. ToolGen [2410.03439] virtualizes each external tool as a vocabulary token, supporting unified retrieval and calling via a standard next-token prediction, achieving 92.7 NDCG@5 retrieval accuracy in a 47,000 tool space with no external index.

In symbolic and persistent settings [2506.10021], LLMs generate code snippets (e.g., Lisp expressions); a middleware intercepts, evaluates, and injects results into an interactive loop, with versioning and garbage collection for long-lived tool registries. This enables agentic introspection and dynamic tool evolution within multi-turn workflows.

Memory augmentation, pointer calls, chain-of-thought priors, and persistent REPLs represent current strategies for increasing agent robustness to long-horizon tasks, deep compositionality, and evolving toolsets.

## 7. Outstanding Challenges and Future Directions

Immediate research targets include:
- Scaling multi-turn RL training to $T\gg 10$ (while alleviating reward sparsity) [2510.24126, 2507.21428].
- Improving LLM sensitivity to early context and reducing "lost in the middle" degradation [2505.10570].
- Automating discovery of new tools and dynamic tool-set adaptation [2410.03439].
- Enabling robust uncertainty calibration (“I don’t know” abstention) under distributional shift.
- Extending frameworks to multimodal/cross-lingual tools, persistent memory, and self-supervised agentic learning.

Multi-turn tool-calling LLMs now match or surpass closed-source models on agentic benchmarks, but further progress will depend on advances in scalable RL, efficient data synthesis, robust long-context handling, and memory-efficient architectures. These approaches are rapidly generalizing to database querying, code-to-text agents, multi-hop knowledge graph traversal, real-time enterprise orchestration, and autonomous symbolic programming.

Source: https://www.emergentmind.com/topics/multi-turn-tool-calling-large-language-models-llms