ToolLinkOS Benchmark Dataset

• ToolLinkOS is a synthetic benchmark dataset that simulates multi-step tool dependencies using a directed knowledge graph.
• It evaluates retrieval approaches by comparing lexical, vector-based, and graph-fused methods with significant improvements in mAP, recall, and nDCG.
• The dataset comprises 573 tools across 15+ industries, offering a controlled environment for testing retrieval-augmented generation and dependency resolution techniques.

ToolLinkOS is a synthetic benchmark dataset designed for evaluating large-scale tool selection in environments characterized by complex, multi-step inter-tool dependencies. Developed to accompany the Graph RAG-Tool Fusion methodology, ToolLinkOS simulates agent workflows where individual tools often require parameters or functionalities that must be obtained from other tools, thereby forming an explicit tool knowledge graph. It specifically targets the limitations inherent in conventional vector-based retrieval-augmented generation (RAG) methods when applied to highly structured tool selection tasks (Lumer et al., 11 Feb 2025).

1. Motivation and Development

The primary aim of ToolLinkOS is to establish a benchmark for tool selection scenarios involving dependencies among hundreds of tools, each potentially requiring non-trivial orchestration. ToolLinkOS supports two principal objectives: (1) to enable the study of retrieval methodologies that can identify both a queried "root" tool and all its nested dependencies; and (2) to provide a controlled, large-scale environment to measure the efficacy of hybrid retrieval systems that fuse semantic similarity and explicit knowledge graph traversal. Real-world analogs include situations where APIs or external tools have non-obvious prerequisites or sequential/parameter dependencies, such as requiring OS-level permissions or other API-derived parameters before invocation (Lumer et al., 11 Feb 2025).

2. Composition and Domain Coverage

ToolLinkOS comprises 573 fictional tools, stratified across more than 15 industries. These tools are split into two categories: 523 are domain-specific "regular" tools (e.g., "get_stock_price"), and 50 are "core" utility tools (e.g., "get_current_date"). Approximately 50 tools are represented per industry category, resulting in a balanced distribution across verticals such as Finance, E-commerce, Travel, Health, Weather, Maps, News, Social Media, Calendar, Device Control, Restaurant Reservations, Events, Product Search, Movie Search, Music Streaming, and Email. Each tool has an average of 6.3 dependencies (standard deviation ≈2.1), with most tools exhibiting between 3 and 10 direct or indirect dependencies.

3. Knowledge Graph Structure and Schema

ToolLinkOS is underpinned by a directed knowledge graph $G = (V, E)$, where $V$ is the set of tools and $E$ is the set of directed edges indicating dependency relations:

• Each tool $t \in V$ is represented by a JSON object containing fields for id, name, description, parameters (typed and described), and dependencies.
• Dependencies are expressed as edges, each possessing:
  • type (one of "tool_directly_depends_on", "tool_indirectly_depends_on", "parameter_directly_depends_on", "parameter_indirectly_depends_on"),
  • reason (the rationale for the dependency),
  • parameter_name (if applicable).

The dataset is also available in CSV format, with node and edge tables, facilitating integration with both graph-processing and tabular ML pipelines.

Schema Example

Field	Description	Example
id	Unique tool identifier	"get_current_date"
name	Human-readable tool name	"Get Current Date"
parameters	List of named, typed parameters with metadata	[ {"name": "ticker", ...} ]
dependencies	List of dependency objects	See schema above

Editor's term: "dependency edge types" refers to the four explicit dependency relations formalized in the schema.

4. Formal Evaluation Metrics

Benchmarking on ToolLinkOS leverages established retrieval metrics:

• Mean Average Precision at k (mAP@k):

$$\text{mAP}@k = \frac{1}{|Q|} \sum_{q \in Q} \left[ \frac{1}{k} \sum_{i=1}^k \text{rel}_n(i) \right]$$

where $\text{rel}_n(i) = 1$ if the $i$-th retrieved tool for query $q$ is relevant.

• Recall@k:

$$\text{Recall}@k = \frac{1}{|Q|} \sum_{q \in Q} \left[ \frac{\#\, \text{of relevant tools in top-}k}{\#\, \text{of relevant tools for } q} \right]$$

• nDCG@k: Normalized discounted cumulative gain.

$$\text{nDCG}@k = \frac{1}{|Q|} \sum_{q \in Q} \left[ \frac{\text{DCG}@k}{\text{IDCG}@k} \right]$$

with

$V$0

No train/validation/test splits are defined; all 1,569 queries are used to compute metrics. Evaluation focuses both on primary tool retrieval and on correct recovery of all dependency chains.

5. Benchmark Statistics and Example Entries

• Tool count: 573 total (523 regular, 50 core)
• Industry coverage: 15+ verticals, ≈50 tools per industry
• Mean dependencies per tool: 6.3 (standard deviation ≈2.1), range ≈3–10
• Number of evaluation queries: 1,569

Representative tool entries:

Core tool example:

$V$1 Regular tool example:

$V$2

6. Retrieval Baselines and Comparative Performance

Several retrieval paradigms were evaluated on ToolLinkOS using 1,569 queries:

Retriever	mAP@10	mAP@20	mAP@30
Lexical Search (BM25)	0.185	0.191	0.194
Naïve RAG (vector only)	0.210	0.216	0.217
Hybrid RAG (α=0.8)	0.202	0.208	0.209
Graph RAG-Tool Fusion (k=3, no rerank)	0.856	0.873	0.873
Graph RAG-Tool Fusion (k=3, rerank)	0.927	0.938	0.938

Graph RAG-Tool Fusion achieved an absolute improvement of 71.7 percentage points in mAP@10 compared to Naïve RAG (0.210 → 0.927). Detailed metrics for recall and nDCG at various cutoffs are provided in Appendix C of the source publication (Lumer et al., 11 Feb 2025).

7. Access, Licensing, and Use

ToolLinkOS is distributed under the MIT License and is available for download at https://github.com/EliasLumer/Graph-RAG-Tool-Fusion-ToolLinkOS. The dataset is intended for research purposes, particularly for evaluating retrieval methods that must address both semantic similarity and graph-structured dependency resolution in large tool knowledge bases. Since the benchmark consists of fictional tools and uses a uniform schema, it facilitates controlled studies that are decoupled from specific, potentially evolving real-world APIs.

This suggests that ToolLinkOS provides a critical reference for developing and benchmarking retrieval-augmented generation agents in scenarios where multi-step tool orchestration and explicit dependency tracking are required.