LLM-Simulated APIs

• LLM-simulated APIs are simulated environments that mimic real API behavior using large language models to support scalable agent testing and robust benchmarking.
• They leverage methods like fine-tuning on real-world request-response pairs, online simulation, and automated API construction to ensure high fidelity and diverse scenarios.
• Metrics such as BLEU-4, cosine similarity, and status-code fidelity are used to evaluate simulation accuracy, guiding improvements in agent tool integration.

LLM-simulated APIs are an emerging paradigm in which LLMs are trained or engineered to act as surrogates (“mirrors”) for external tool environments, providing programmatic, context-aware API behavior entirely in silico. These systems enable tool-augmented AI agents to operate at scale without requiring access to the real endpoints, supporting both robust benchmarking and algorithm development under realistic API complexity, high variety, and operational constraints.

1. Foundations and Motivation

LLM-simulated APIs arose from the need to test, benchmark, and train tool-using agents under conditions that reflect the complexity, scale, and noise of real-world APIs. Traditional tool-agent pipelines rely on static mocks, record-replay fixtures, or specification-driven stubs that lack behavioral diversity, dynamic context, or the rich stateful interactions seen in production APIs (Guo et al., 26 Mar 2025, Zhang, 6 Apr 2026). Simulated APIs enable (1) scalable, low-latency environments for agent development and evaluation, (2) high-fidelity modeling of real request–response semantics, errors, and workflows, and (3) the study of robustness and planning under noise, partial failures, and evolving specifications (Kim et al., 1 Jan 2026, Garcia, 20 Feb 2026).

2. Core Methodologies

2.1 API “Mirroring” via LLM Fine-Tuning

One principal methodology, exemplified in MirrorAPI (Guo et al., 26 Mar 2025), involves collecting large-scale request–response datasets (e.g., >95,000 pairs from 7,437 real-world APIs across 49 categories) and supervised fine-tuning an LLM to accurately simulate API outputs. The resulting simulator, operated in both direct SFT (supervised fine-tuning) and chain-of-thought (CoT) modes, generates outputs that closely track real documentation, schema, and behavioral patterns. Simulation fidelity is measured using BLEU-4 overlap, embedding cosine similarity, and LLM-based judgment of doc compliance.

2.2 Online Simulation for Service Dependencies

In microservice and integration testing, Mirage (Zhang, 6 Apr 2026) introduced online LLM simulation, where the LLM answers dependency calls at test runtime using as input the dependency’s source code, the caller’s code, and production traces—requiring no pre-generated mocks or static “fake” spec. State is tracked implicitly via interaction history, allowing accurate modeling of multi-step, stateful behaviors and error cascades. Formal evaluation utilizes metrics such as status-code and response-shape fidelity, defined by

$$F_{code} = \frac{|\{\,σ: \forall i,\ s_i^D = s_i^{\hat D}\}|}{|\Sigma|},\quad F_{shape} = \frac{|\{\,σ: (\forall i,\ s_i^D = s_i^{\hat D}) \wedge (\mathrm{keys}(body_i^D)=\mathrm{keys}(body_i^{\hat D}))\}|}{|\Sigma|}$$

2.3 Automated API Construction from Databases and Specifications

Data-driven approaches, such as the NL2SQL→NL2API pipeline (Elder et al., 12 Jun 2025), map SQL abstract syntax trees into sequences of API (function) calls, producing large pools of automatically constructed, invocable REST endpoints and function-call descriptions stratified by complexity (slot-filling, selection, REST-style). Benchmarking then probes how well LLMs can plan, call, and compose these APIs in the absence of explicit SQL, under varying degrees of toolset size, obfuscated names, and nested workflows.

3. Architecture and Specification: Infrastructure for Simulation

Effective LLM-simulated API systems depend on the underlying input specifications and runtime orchestration:

• Specification Formats: Standard OpenAPI docs are token- and metadata-heavy, leading to excessive context length. LAPIS (Garcia, 20 Feb 2026) provides a lightweight, LLM-native API specification format that reduces average token usage by 85.5% and encodes all semantic fields needed for planning (endpoint signatures, parameters, errors, rate limits, webhooks, flows). LAPIS is designed for direct LLM consumption and is fully convertible from OpenAPI 3.x.
• Prompt and Scenario Infrastructure: Complex scenario toggles (e.g., in WildAGTEval, 60 orthogonal complexity scenarios composed into ~32,000 test configurations) allow selective activation of specification- and execution-level perturbations, supporting isolated and cumulative evaluation of agent robustness (Kim et al., 1 Jan 2026).
• Agent–Environment Interface: Adapters route agent calls (e.g., function name, JSON args) to the simulated environment, with runtime wrappers injecting the appropriate prompts and tracking conversational history and scenario context (Guo et al., 26 Mar 2025, Zhang, 6 Apr 2026).

4. Benchmarking and Evaluation Strategies

4.1 Metrics

Simulation fidelity is assessed through multiple, domain-tailored metrics:

• Token-level/textual: BLEU-4 and embedding cosine similarity between simulated and real responses (Guo et al., 26 Mar 2025).
• Behavioral correctness: Percentage of calls with status code and shape fidelity (Zhang, 6 Apr 2026).
• Agent-centric: Core-API accuracy $\mathcal{A}_{API}$, error-handling accuracy $\mathcal{A}_{Error}$ (LLM-judge scored (Kim et al., 1 Jan 2026)), solvable pass rate (SoPR), final answer completeness (FAC).
• Functional/evaluation combinatorics: Completion rates under different pool sizes, tool/slot name obfuscation, and multi-step sequencing challenges (Elder et al., 12 Jun 2025).

4.2 Experimental Design

Key experimental protocols include:

• Isolated-complexity vs. cumulative-complexity evaluation: Isolated settings measure one-shot response quality; cumulative protocols allow error propagation and compounding (Kim et al., 1 Jan 2026).
• In/out-of-distribution tasks: Benchmarks such as MirrorAPI-Bench partition APIs to distinguish in-distribution (ID) and OOD scenarios, testing generalization (Guo et al., 26 Mar 2025).
• Direct invocation vs. ReACT/TAO agentic regimes: Both one-step prediction and interactive agent planning are probed, with metrics capturing robustness to pool size, scenario diversity, and schema variability (Elder et al., 12 Jun 2025).

5. Empirical Findings and Failure Modes

Quantitative findings across multiple benchmarks reveal key limitations and dynamics:

• In WildAGTEval, execution complexity (notably irrelevant information) causes absolute accuracy drops up to 27.3%; cumulative scenario combinations degrade agent accuracy by up to 34.3%, reaching 63.2% for individual models (Kim et al., 1 Jan 2026).
• MirrorAPI surpasses alternative methods, reaching OOD documentation compliance scores of 6.86/10 and BLEU-4 of 89.9 versus <20 for prompt baselines; embedding cosine similarity reaches 94.1% (Guo et al., 26 Mar 2025).
• In microservice simulation, Mirage achieves 99% status-code and response-shape fidelity (white-box), compared to 62%/16% for record-replay; pure dependency source code yields 100% behavioral fidelity, with costs of $0.16–$0.82 per simulation (Zhang, 6 Apr 2026).
• LLM-based agents operating over automatically constructed NL2API pools show poor task-completion rates: e.g., 7–47% in direct settings, marginally improved to 50% with ReACT agents, with substantial dropoff under obfuscation or large tool pools (Elder et al., 12 Jun 2025).
• Prominent failure modes include user-intent distortion (LLMs reporting spurious success), omission of spec-mandated calls, incorrect parameterization under ad-hoc rules, and selection of decoy/sponsored entries in noisy outputs (Kim et al., 1 Jan 2026).

6. Practical Tooling and Best Practices

• Specification compression: Use LAPIS for embedded, token-efficient API docs to maximize reasoning context (Garcia, 20 Feb 2026).
• Multi-stage pipelines: Structure agent architectures into layers: intent extraction, API selection (with heuristics based on embedding and utility), call generation, execution, and feedback (Tzachristas, 2024).
• Scenario instrumentation: Instrument backend services with OpenTelemetry or equivalent tracing to maximize coverage of behaviors, error paths, and state transitions (Zhang, 6 Apr 2026).
• Modular environments: Employ assign-and-inject mechanisms for complexity toggles (Kim et al., 1 Jan 2026), enable local macro databases (macro/function/sequence) for on-device agent deployments (Tzachristas, 2024), and use cache fine-tuning for simulated environments (Guo et al., 26 Mar 2025).

7. Limitations and Future Directions

Documented limitations include under-modeling of connectivity errors (timeouts, jitter), limitations in latency compared to real endpoints for complex APIs, and insufficient representation of stateful or safety-critical tools (Guo et al., 26 Mar 2025, Zhang, 6 Apr 2026). There is considerable room for improved planning, schema grounding, robustness to scale, and universal abstractions for function-calling and error handling—as indicated by low completion rates, brittleness under obfuscation, and qualitative error patterns in current agents (Elder et al., 12 Jun 2025, Kim et al., 1 Jan 2026).

Future research directions include:

• Simulation of adversarial or misbehaving APIs for robustness evaluation (Guo et al., 26 Mar 2025),
• Co-training agents and simulators in closed feedback loops,
• Extension to long-running session or multimodal APIs,
• Systematic framework benchmarking with new specification languages (Garcia, 20 Feb 2026),
• Standardized compositional evaluation environments for large tool pools and hierarchical workflows.

Paper Title	arXiv ID	Domain
Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexity	(Kim et al., 1 Jan 2026)	Complexity eval
StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIs	(Guo et al., 26 Mar 2025)	API mirroring
MIRAGE: Online LLM Simulation for Microservice Dependency Testing	(Zhang, 6 Apr 2026)	Microservice sim
Creating an LLM-based AI-agent: A high-level methodology towards enhancing LLMs with APIs	(Tzachristas, 2024)	Agent architecture
LAPIS: Lightweight API Specification for Intelligent Systems	(Garcia, 20 Feb 2026)	API specification
Invocable APIs derived from NL2SQL datasets for LLM Tool-Calling Evaluation	(Elder et al., 12 Jun 2025)	Tool-calling eval

These works collectively form the basis for modern LLM-simulated API research, providing the methodological, infrastructural, and empirical foundations for scalable, realistic, and robust agent evaluation and development in tool-rich environments.