Environment-free Synthetic Data Generation for API-Calling Agents

Abstract: Training API-calling LLM agents demands massive amounts of high-quality trajectories. However, collecting such data at scale typically requires fully implemented environments with executable APIs and realistic, pre-populated backend databases, creating a major bottleneck for scalability. To overcome this, we propose an environment-free synthetic data generation approach that leverages LLMs as on-the-fly digital world models. Given only API specifications, our method generates trajectories mimicking interactions between an agent and a stateful environment. Specifically, an LLM first generates diverse tasks solvable with the provided APIs. A teacher agent then iteratively solves each task while an LLM simulator generates coherent synthetic API responses conditioned on the task context and simulation history. Finally, an LLM judge filters the trajectories to ensure the quality of the resulting dataset. We evaluate our approach on the challenging AppWorld and OfficeBench benchmarks, which include both information-retrieval and state-changing tasks. Fine-tuning models on our synthetic data yields significant performance gains, demonstrating that effective supervision for API-calling agents can be generated without any executable environment. Our results establish LLM-based API simulation as a practical, scalable solution for training agents across diverse API ecosystems.

• The paper introduces ESAT, an LLM-centric pipeline that synthesizes stateful API interaction trajectories without requiring executable environments.
• It decomposes the process into task synthesis, trajectory simulation, and precise LLM-based filtering to ensure high data quality and diversity.
• ESAT-trained models achieve significant performance gains—up to 50.5% improvement—over traditional methods, demonstrating robust transfer across diverse API domains.

Environment-Free Synthetic Agentic Data Synthesis for API-Calling Agents

Motivation and Problem Statement

Training API-calling LLM agents fundamentally requires large-scale, high-fidelity multi-step agent-environment interaction data. Existing pipelines for generating such data demand executable environments with realistic API endpoints and populated backends, making both data coverage and domain adaptation expensive and slow. Direct manual collection is impractical, and synthetic approaches reliant on functional backends inherit major scalability bottlenecks due to infrastructure dependencies and the difficulty of producing diverse, state-rich environments.

ESAT Pipeline: Environment-Free Synthetic Agentic Trajectory Generation

The paper introduces ESAT, a specification-driven, LLM-centric pipeline that entirely obviates the need for executable API environments. ESAT synthesizes stateful, multi-turn agentic trajectories given only API specifications as input, using LLMs to simulate the digital world model at each trajectory step. The process is decomposed into three distinct stages:

1. Task Synthesis: An LLM generates bucketized task prompts with coverage and diversity constraints (difficulty, action type, task focus, app count, API count). Tasks are post-processed by an LLM judge for solvability, and rewritten for intent-level conciseness.
2. Trajectory Synthesis: For each task, a teacher agent LLM iteratively produces API calls; an LLM-based API simulator generates coherent, history-consistent JSON responses using current state, the input spec, user context, and trajectory history.
3. Trajectory Filtering: A high-precision LLM judge filters complete trajectories for correctness, completeness, and redundancy, using explicit verification criteria.

This paradigm is diagrammed as a sequence of LLM modules converting simple API documentation to high-quality supervision data without executable APIs or real environments.

Simulator Mechanism and Quality Assurance

The API simulator is engineered as a persistent digital world model, receiving per-app interaction histories and context to ensure consistency for both read and write API endpoints. Validation integrates:

• Deterministic input checks (parameter existence, type, constraint adherence)
• Structural output schema enforcement (strict JSON schema validation, auto-feedback error correction via in-context prompting)
• Semantic history checks using an LLM judge to rate each generated response for internal coherence and world-state consistency
• Multi-stage retry and refinement loop with bounded attempts and explicit error amnesty triggers

This layered validation pipeline maintains high response validity even at substantial trajectory depths and task complexity, moving beyond prior LLM simulators prone to history drift and schema hallucination.

Figure 1: Simulator failure rate as a function of output token length, as judged by GPT-5.1.

The simulation quality remains high ($>93\%$ valid responses per GPT-5.1), though error rates increase for extremely long output tokens, highlighting the architectural scaling path to even larger context or more capable LLMs for simulation robustness.

Dataset Synthesis, Diversity, and Coverage

ESAT is instantiated on two challenging testbeds: AppWorld (9 apps, 340–457 APIs, 10+ API/task, high compositionality) and OfficeBench (8 apps, office workflows, no public train split). ESAT generates data in three principal settings:

• AppWorld APIs (ESAT-AW7): Trajectories using only seven AppWorld apps—strict generalization to held-out targets.
• Synthetic App Suite (ESAT-S52): 52 automatically synthesized apps (1017 APIs), spanning ten functional domains beyond AppWorld, confirming transfer and compositional generalization.
• OfficeBench APIs (ESAT-OB): Coverage-focused data for cross-app office settings.

The synthesized datasets (e.g., 9000+ ESAT-AW7 trajectories; near-complete API coverage for $>$99% of APIs at all call-frequency thresholds) are empirically shown to avoid API distributional collapse, a common mode in prior tool/data generation approaches.

Figure 2: API coverage across the 340 AppWorld APIs, showing the number and percentage of APIs called in at least $N$ successful trajectories.

Downstream Performance: Supervised Fine-Tuning for Agent Models

Models trained exclusively with ESAT data (1.7B–27B Qwen3/Qwen3.5, as well as competitive baselines) achieve large and consistent performance improvements over base models, real-environment data, and prior synthetic tool-use datasets. Key results:

• On AppWorld, ESAT-trained models outperform real-environment-trained models in most settings, with gains up to $50.5\%$ (Qwen3.5-9B).
• Strong transfer from synthetic domains—ESAT-S52 consistently outperforms AppWorld data for all but the smallest models (e.g., Qwen3-4B: $+43.1\%$, Qwen3.5-9B: $+47.0\%$ over zero-shot).
• Knowledge distillation is effective: ESAT enables smaller models (Qwen3.5-9B/27B) to approach or surpass the performance of much larger LLM baselines (Nemotron-3-120B, GPT-4o).
• On OfficeBench, ESAT data yields $5.2–60.5\%$ absolute gains, with medium models surpassing models an order of magnitude larger.

These improvements are stable across model sizes and are robust to the addition of app-specific or real-environment data.

Figure 3: API coverage across our synthetic app APIs, showing the number (and percentage) of APIs called in at least $N$ successful trajectories.

Comparison with Prior Synthetic Data Pipelines

Benchmarks against state-of-the-art synthetic datasets (ToolAlpaca, ToolACE, Nemotron) demonstrate that simple tool-call datasets lacking stateful world models do not support robust agent learning in high-complexity settings. ESAT-trained agents achieve much higher downstream scores on both AppWorld and OfficeBench, with competitor methods sometimes underperforming even the base model.

Implications and Prospects

The results establish that LLMs can function as generic, stateful API environment simulators—provided world modeling, schema constraints, and trajectory-level state are carefully preserved. ESAT induces a scalable regime for agentic trajectory construction, decoupled from the need for backend instantiation, crowd data ops, or complex RL training-inference loops. Environment-free data pipelines, with semantic and schema filtering, are thus practical for bootstrapping new API domains, evaluating agent robustness, or stress-testing safety mechanisms across compositional benchmarks.

Figure 4: API coverage by frequency threshold for OfficeBench apps.

Theoretically, this suggests a unification path: large foundation models already encode actionable priors for downstream environment simulation, enabling both teacher and world functions in a single architecture. Empirically, this unlocks a rapid domain adaptation and prototyping capability—agents can be prepared for never-seen apps and APIs without infrastructure lead time. However, coverage for extreme compositionality and safety-critical edge cases may still require extensions in prompt engineering, schema verification, or stronger LLM pretraining.

Conclusion

The ESAT pipeline demonstrates that end-to-end, environment-free agentic trajectory synthesis is not only feasible but produces training data of sufficient fidelity and diversity to outperform infrastructure-heavy alternatives on canonical API-calling agent benchmarks. LLM-based simulation, coupled with structured task generation and stringent quality filtering, is an effective paradigm for scalable, specification-driven agent supervision. This reframes open challenges in agent data collection, robustness, and domain transfer, establishing new baselines for both method development and empirical evaluation in LLM agent research.