ToolFactory: LLM-Compatible Tool Generation

• ToolFactory is an automated system for creating executable, LLM-compatible tools from diverse references, enabling enhanced reasoning and domain adaptation.
• It employs multi-phase pipelines—including source ingestion, prompt-driven tool extraction, and semantic clustering—to aggregate and refine tool libraries.
• Empirical studies demonstrate that ToolFactory improves accuracy, scalability, and integration across various scientific and software domains.

A ToolFactory is an automated or semi-automated system for generating executable, LLM-compatible tools—typically in the form of Python functions or callable code wrappers—from diverse reference sources such as domain documents, REST API specifications, datasets with solution traces, or natural language task specifications. ToolFactory pipelines enable LLMs to augment their reasoning capacity, improve generalizability, and facilitate seamless integration with software and data resources by producing and maintaining curated libraries of tools that can be selected and invoked dynamically. Recent developments in ToolFactory methodologies address the challenges of reference grounding, scalability, tool aggregation, and robust integration with LLM agents, spanning domains from science and mathematics to RESTful API access and open-source toolkit construction.

1. Problem Definition and Motivations

The core objective of a ToolFactory is to construct and orchestrate a set of executable tools $\mathcal{T}$ from source materials $R = \{ r_1, ..., r_{|R|} \}$ (e.g., reference textbooks, API docs, or reasoning traces) to support a set of target tasks $T = \{ t_1, ..., t_{|T|} \}$ at inference time. Formally, this entails (a) Tool Creation—mapping $R$ to a toolbox $f_{\text{create}}: R \rightarrow \mathcal{T}$, and (b) Tool Utilization—applying tools $$f_{\text{use}}: (\mathcal{T}, q) \rightarrow \text{solution}$$ for query $q \in T$.

ToolFactory systems address bottlenecks in LLM reasoning:

• Domain Adaptation: LLMs lack domain-specific tools in specialized settings (e.g., scientific QA, customized REST APIs).
• Knowledge Transfer: Grounding tool extraction in external, authoritative references enables the transfer of rigorous, up-to-date methodologies into LLM-usable code.
• Scalability: As the number of tools grows, naive approaches lead to retrieval ambiguities and redundancy; ToolFactory frameworks provide organization, aggregation, and error checking.
• Automation: Reducing manual engineering effort in constructing tool libraries for LLM-augmented workflows (Liu et al., 27 May 2025, Yue et al., 9 Oct 2025, Ni et al., 28 Jan 2025, Qian et al., 2023).

Evaluation metrics for ToolFactory outputs focus on:

• Task accuracy $$\operatorname{Acc}(f_{\text{use}}(\mathcal{T}, q), \text{gold}(q))$$
• Cost metrics for construction and inference (USD, wall time)
• Coverage ratios and tool reliability for code-based tool generation (Liu et al., 27 May 2025, Ni et al., 28 Jan 2025).

2. ToolFactory Pipeline Architectures

Pipeline architectures for ToolFactory implementations typically follow multi-phase decompositions, which may include:

1. Source Ingestion/Preprocessing

• Ingestion of external materials: textbook chapters, REST API pages, QA+CoT datasets.
• Preprocessing steps: HTML de-noising, LaTeX conversion for structured documents, schema-constrained parsing (Liu et al., 27 May 2025, Ni et al., 28 Jan 2025).

2. Tool Extraction and Creation

• Prompting LLMs with domain content and templates to extract tool specifications as Python function stubs, class definitions, or JSON schemas, with testable examples.
• Validation via sandboxed code execution or agent-based testing to guarantee tools conform to the intended semantics.

3. Toolbox Organization

• Hierarchical organization: tools are grouped by topical section or semantic cluster, often reflecting the structure of the source material (Liu et al., 27 May 2025).
• Clustering algorithms: LLM-driven or embedding-based clustering yields semantically coherent groupings; aggregation within clusters collapses redundant/overlapping functionality (see Table 1).

4. Tool Aggregation and Refactoring

• Multi-agent refactoring: code agents design reusable blueprints/classes; reviewing agents validate and patch for full functional coverage (Yue et al., 9 Oct 2025).
• Automated tool aggregation reduces retrieval search space and raises throughput in large-scale tool libraries.

5. Integration and Error Diagnosis

• Automated or semi-automated validation, error-type assignment, and parameter inference based on run-time feedback or code execution results (Ni et al., 28 Jan 2025).
• Knowledge base inference recycles previously validated parameter values to address documentation sparsity in API extraction.

Pipeline Phase	Representative System	Key Mechanisms
Tool Extraction	RefTool (Liu et al., 27 May 2025), ToolLibGen (Yue et al., 9 Oct 2025)	Reference-guided LLM prompting, Chain-of-Thought abstraction
Clustering/Aggregation	ToolLibGen (Yue et al., 9 Oct 2025)	LLM clustering, semantic vector embeddings, multi-agent refactoring
Schema-Guided API Tooling	ToolFactory (Ni et al., 28 Jan 2025)	LLM extraction, knowledge-base parameter inference, code validation

3. Reference-Grounded and Code-Driven Tool Factories

Reference-guided approaches such as RefTool (Liu et al., 27 May 2025) leverage the structure and content of textbooks or authoritative documents, parsing them into chapters and sections (in LaTeX or structured text), and systematically extracting faithful code tools. Each tool is specified as a JSON object containing a description, code implementation, and worked example, validated by running the demonstration code. Tools are grouped into a hierarchical toolbox mirroring the source outline, facilitating targeted retrieval.

Chain-of-Thought-driven methods (e.g., ToolLibGen (Yue et al., 9 Oct 2025)) operate on datasets where LLM or human solutions decompose complex questions into atomic steps. Tools abstracted from these traces are clustered into a topic taxonomy, and multi-agent systems aggregate functionally similar code into scenario classes and façade functions. This automated library refactoring is validated by re-solving all source tasks with the aggregated tool code to guarantee coverage and correctness.

REST API documentations are handled by schema-guided pipelines (ToolFactory (Ni et al., 28 Jan 2025)), using soft-prompted LLMs (e.g., Llama 3 8B-Instruct) to extract endpoint, parameter, and method definitions into a unified JSON schema. Code wrappers are automatically generated and validated via dry runs and GPT-evaluated response checks, with knowledge base–driven inference for missing parameter values.

4. Tool Utilization: Selection, Planning, and Execution

ToolFactory outputs are integrated into LLM-based reasoning workflows via systematic selection, orchestration, and invocation of tools:

• Hierarchical Selection: Toolbox traversal is performed in a multi-stage LLM prompt sequence—first chapters/sections, then specific tools, as in RefTool, to limit search space and maximize relevance.
• Retrieval and Similarity Matching: Embedding-based k-NN retrieval using code or docstring representations supports scalable matching in large libraries (Yue et al., 9 Oct 2025).
• Chain-of-Solving (CoS) Planning: For task-solving, a model constructs a step-by-step plan that sequences tool calls, then executes them to synthesize final answers (Qian et al., 2023).
• API Invocation: Generated wrappers are called by agent frameworks (e.g., LangChain), enabling natural language interfaces and modular extension with new APIs or capabilities (Ni et al., 28 Jan 2025).

Empirically, hierarchical selection and semantic clustering outperform flat retrieval strategies, improving both answer accuracy and retrieval precision (Liu et al., 27 May 2025, Yue et al., 9 Oct 2025).

5. Experimental Results and Scalability

ToolFactory methodologies have been evaluated across a range of scientific and software integration benchmarks:

Reference-Grounded Tool Creation

• RefTool yields consistent accuracy gains of +11.3% over naive tool creation and retrieval-augmented baselines on causality, physics, and chemistry QA benchmarks; with 87% tool validation retention and significant cost/time reductions relative to domain-specific baselines (Liu et al., 27 May 2025).
• Hierarchical retrieval (+2.8%) and code-tools (+2.6%) outperform flat text-based or similarity-only retrieval.

Large-Scale Tool Aggregation

• ToolLibGen demonstrates state-of-the-art reasoning accuracy (70.3% seen/60.6% unseen) on science/math/medical datasets scaling to >175,000 tools, with robust retrieval (>85%) as library sizes grow. Ablations show multi-agent aggregation is essential to maintain coverage and reasoning accuracy as scale increases (Yue et al., 9 Oct 2025).

REST API Tool Generation

• ToolFactory achieves 97% structured extraction validity on semi-organized/unorganized API docs, with 52% end-to-end functional tool reliability after code generation. Knowledge base parameter inference doubles missing-value resolution compared to raw LLM inference (Ni et al., 28 Jan 2025).
• Integrates with standard agent platforms, enabling domain experts to instantiate working REST API tools in minimal time.

Open-Source Toolkit Planning and Generalization

• Toolink and LLaMA-CoS, built via the Chain-of-Solving paradigm, match or surpass closed-source LLMs (ChatGPT) in multi-step tool planning/calling on BIG-bench and generalize to novel datasets and borrowed toolkits (Qian et al., 2023).

6. Analysis, Limitations, and Future Directions

Empirical analyses reveal that grounding tool creation in authoritative references yields highly faithful and generalizable tools (≥90% faithfulness, robust transfer across datasets). Structured tool libraries—hierarchically organized or semantically clustered—facilitate navigation, reduce redundancy, and maintain high retrieval precision as library sizes increase (Liu et al., 27 May 2025, Yue et al., 9 Oct 2025).

Limitations of current ToolFactory implementations include:

• Dependence on the completeness and accuracy of source documents or QA traces—missing coverage results in incomplete toolboxes.
• Residual need for minimal domain comprehension by LLMs to parse references correctly.
• Error diagnosis and validation for code tools often requires trial executions and reference data, introducing computational overhead.

Emerging directions point toward:

• Integrating retrieval-augmented pre-training and domain adaptation to handle noisier or less-structured references (Liu et al., 27 May 2025).
• Automating correctness verification via theorem proving and advanced static/dynamic code analysis.
• Expanding knowledge-base–powered parameter/model inference for poorly documented or sparsely annotated APIs (Ni et al., 28 Jan 2025).
• Unified, domain-agnostic ToolFactory frameworks combining code generation, semantic organization, and tool aggregation for arbitrary scientific, mathematical, or professional domains (Yue et al., 9 Oct 2025).

7. Connections, Generalization, and Best Practices

ToolFactory systems are domain-agnostic by design, supporting rapid onboarding of new domains, APIs, or reasoning paradigms with minimal human supervision. The modular nature of code/calling interfaces, thin-API facades, and embedding-driven retrieval ensures LLM agents using ToolFactory outputs are resilient to domain shifts (Yue et al., 9 Oct 2025).

Best practices distilled from state-of-the-art systems include:

• Keeping tool implementations single-purpose and minimal.
• Providing human-readable documentation for each tool.
• Explicitly separating planning from execution in training and inference data.
• Applying multi-round generation/refinement and validation feedback for robust code coverage.
• Leveraging existing toolkits for compositional generalization in unseen or cross-domain settings (Qian et al., 2023).

Overall, ToolFactory frameworks have become integral enablers for advanced LLM reasoning, ensuring scalable, accurate, and extensible tool integration across a rapidly growing landscape of computational challenges and software ecosystems.