SynthTools: Synthetic Tool Ecosystems

• SynthTools is a framework that automates the creation and simulation of synthetic tool ecosystems for reproducible research in AI and creative domains.
• It utilizes a hierarchical LLM-driven process to partition fields, generate task specifications, and establish robust API contracts with validation and error handling.
• SynthTools outperforms previous benchmarks by enabling over 5000 tools across 100 domains with high simulator accuracy (94-97%) and near-perfect audit judge performance (99%).

SynthTools refers both to a family of frameworks for the scalable, automated generation and simulation of synthetic tool ecosystems for AI agent research, and to an evolving set of advanced software artifacts that enable creative, programmable, or data-driven interaction with synthesis—or with synthetic environments more broadly. SynthTools have become foundational for reproducible research in tool-using AI, audio synthesis, and interactive sonification, supporting diverse agent behaviors, experimental reproducibility, and high-dimensional creative workflows (Castellani et al., 11 Nov 2025, Liu et al., 29 Sep 2025, Esling et al., 2019, Trayford et al., 2 Apr 2025).

1. Motivation and Scope

SynthTools originated from limitations in real-world and curated synthetic tool APIs: real APIs require access keys, enforce rate limits, and change interfaces unpredictably, restricting both scale and reproducibility. Curated benchmarks such as $\tau$-Bench and ACEbench offer limited coverage—typically under 20 tools per domain—insufficient for large-scale, robust agent training and evaluation. High-fidelity service emulators are labor-intensive and remain narrow in scope. SynthTools addresses these barriers via automated hierarchical tool generation, high-fidelity simulation of input/output (I/O) behaviors—including error handling—and rigorous audit pipelines to validate tool correctness at scale (Castellani et al., 11 Nov 2025).

The term also encompasses recent advances in creative toolkits for music and sound design, including systems for disentangled synthesizer control, stepwise effect programming, and data-driven sonification pipelines (Liu et al., 29 Sep 2025, Mitcheltree et al., 2021, Trayford et al., 2 Apr 2025).

2. SynthTools Framework for Agent Toolchains

2.1 Hierarchical Tool Generation

The core methodology begins with automated construction of toolsets spanning hundreds of fields and thousands of domains. A four-stage process systematically expands from high-level fields to subdomains, tasks, tool specifications, and ultimately API contracts:

1. Field $\rightarrow$ Subdomain: partition broad categories (e.g., Finance, Healthcare) into coherent subdomains via LLM prompt.
2. Subdomain $\rightarrow$ Tasks: LLM produces natural-language task descriptions representing real-world workflows.
3. Task $\rightarrow$ Tool Specs: for each task, the LLM generates one or more API specifications, each comprising a name, description, parameter schema (types, required/optional, defaults), I/O contract (preconditions, postconditions, explicit failure modes).
4. Deduplication: Redundant tool candidates are pruned using name/body comparison and embedding-based similarity graph clustering, thresholded at cosine similarity $\tau = 0.85$ for empirically observed uniqueness (only $\approx 9\%$ pruned in evaluation) (Castellani et al., 11 Nov 2025).

2.2 Tool Simulation and Parameter Validation

Each API is instantiated as a callable simulator, splitting execution into parameter validation and response generation:

• Validation: Ensures all required parameters are present, types match declared schemas, and cross-field constraints (e.g., shape, length) hold. On violation, returns canonical error codes (e.g., HTTP 400–series mimicry).
• Response: Depending on metadata presence, operates in either data generation (arbitrary plausible responses) or information deduction mode (outputs strictly constrained by metadata). LLM prompting structures both behaviors, with iterative refinement for simulation stability.

2.3 Tool Audit and Reliability

A dedicated LLM-based judge stress-tests each tool via approximately 8 test scenarios (schema failures, constraint violations, valid new calls, consistency checks). Audit metrics include simulator accuracy $\mathrm{Acc}_{\mathrm{sim}}$ (fraction of calls yielding correct output), judge reliability (fraction of correct judgments), and false positive rate. In reported experiments, simulator accuracy is $94\%$, manual spot-checking validates $97\%$ agreement, and audit judge accuracy approaches $99\%$, outperforming prior frameworks (Castellani et al., 11 Nov 2025).

2.4 Scale and Comparative Benchmarking

SynthTools enables:

• $\rightarrow$0 domains
• Up to $\rightarrow$1 tools per domain
• $\rightarrow$2+ tools after filtering

This exceeds prior synthetic tool frameworks by an order of magnitude (e.g., $\rightarrow$3, $\rightarrow$4 compared to ACEbench) and surpasses typical LLM-based simulator accuracy ($\rightarrow$5–$\rightarrow$6) (Castellani et al., 11 Nov 2025).

2.5 Down