---
title: ToolLinkOS Benchmark Dataset
url: https://www.emergentmind.com/topics/toollinkos-dataset
type: topic
---

# ToolLinkOS Benchmark Dataset

ToolLinkOS is a synthetic benchmark dataset designed for evaluating large-scale tool selection in environments characterized by complex, multi-step inter-tool dependencies. Developed to accompany the Graph RAG-Tool Fusion methodology, ToolLinkOS simulates agent workflows where individual tools often require parameters or functionalities that must be obtained from other tools, thereby forming an explicit tool knowledge graph. It specifically targets the limitations inherent in conventional vector-based retrieval-augmented generation (RAG) methods when applied to highly structured tool selection tasks [2502.07223].

## 1. Motivation and Development

The primary aim of ToolLinkOS is to establish a benchmark for tool selection scenarios involving dependencies among hundreds of tools, each potentially requiring non-trivial orchestration. ToolLinkOS supports two principal objectives: (1) to enable the study of retrieval methodologies that can identify both a queried "root" tool and all its nested dependencies; and (2) to provide a controlled, large-scale environment to measure the efficacy of hybrid retrieval systems that fuse semantic similarity and explicit knowledge graph traversal. Real-world analogs include situations where APIs or external tools have non-obvious prerequisites or sequential/parameter dependencies, such as requiring OS-level permissions or other API-derived parameters before invocation [2502.07223].

## 2. Composition and Domain Coverage

ToolLinkOS comprises 573 fictional tools, stratified across more than 15 industries. These tools are split into two categories: 523 are domain-specific "regular" tools (e.g., "get_stock_price"), and 50 are "core" utility tools (e.g., "get_current_date"). Approximately 50 tools are represented per industry category, resulting in a balanced distribution across verticals such as Finance, E-commerce, Travel, Health, Weather, Maps, News, Social Media, Calendar, Device Control, Restaurant Reservations, Events, Product Search, Movie Search, Music Streaming, and Email. Each tool has an average of 6.3 dependencies (standard deviation ≈2.1), with most tools exhibiting between 3 and 10 direct or indirect dependencies.

## 3. Knowledge Graph Structure and Schema

ToolLinkOS is underpinned by a directed knowledge graph $G = (V, E)$, where $V$ is the set of tools and $E$ is the set of directed edges indicating dependency relations:

- Each tool $t \in V$ is represented by a JSON object containing fields for id, name, description, parameters (typed and described), and dependencies.
- Dependencies are expressed as edges, each possessing:
  - type (one of "tool_directly_depends_on", "tool_indirectly_depends_on", "parameter_directly_depends_on", "parameter_indirectly_depends_on"),
  - reason (the rationale for the dependency),
  - parameter_name (if applicable).

The dataset is also available in CSV format, with node and edge tables, facilitating integration with both graph-processing and tabular ML pipelines.

### Schema Example

| Field            | Description                                   | Example                          |
|------------------|-----------------------------------------------|----------------------------------|
| id               | Unique tool identifier                        | "get_current_date"               |
| name             | Human-readable tool name                      | "Get Current Date"               |
| parameters       | List of named, typed parameters with metadata | [ {"name": "ticker", ...} ]      |
| dependencies     | List of dependency objects                    | See schema above                 |

*Editor's term*: "dependency edge types" refers to the four explicit dependency relations formalized in the schema.

## 4. Formal Evaluation Metrics

Benchmarking on ToolLinkOS leverages established retrieval metrics:

- **Mean Average Precision at k (mAP@k):**
  
  $$
  \text{mAP}@k = \frac{1}{|Q|} \sum_{q \in Q} \left[ \frac{1}{k} \sum_{i=1}^k \text{rel}_n(i) \right]
  $$
  where $\text{rel}_n(i) = 1$ if the $i$-th retrieved tool for query $q$ is relevant.

- **Recall@k:**
  
  $$
  \text{Recall}@k = \frac{1}{|Q|} \sum_{q \in Q} \left[ \frac{\#\, \text{of relevant tools in top-}k}{\#\, \text{of relevant tools for } q} \right]
  $$

- **nDCG@k:** Normalized discounted cumulative gain.

  $$
  \text{nDCG}@k = \frac{1}{|Q|} \sum_{q \in Q} \left[ \frac{\text{DCG}@k}{\text{IDCG}@k} \right]
  $$
  with
  $$
  \text{DCG}@k = \sum_{i=1}^k \frac{2^{\text{rel}_n(i)} - 1}{\log_2(i+1)}
  $$

No train/validation/test splits are defined; all 1,569 queries are used to compute metrics. Evaluation focuses both on primary tool retrieval and on correct recovery of all dependency chains.

## 5. Benchmark Statistics and Example Entries

* Tool count: 573 total (523 regular, 50 core)
* Industry coverage: 15+ verticals, ≈50 tools per industry
* Mean dependencies per tool: 6.3 (standard deviation ≈2.1), range ≈3–10
* Number of evaluation queries: 1,569

Representative tool entries:

**Core tool example**:
```json
{
  "id": "get_current_date",
  "name": "Get Current Date",
  "description": "Returns today's date in YYYY-MM-DD format.",
  "parameters": [],
  "dependencies": [
    {
      "tool_id": "check_wifi_status",
      "type": "tool_directly_depends_on",
      "reason": "Require internet connectivity status to ensure correct timezone syncing",
      "parameter_name": null
    }
  ]
}
```
**Regular tool example**:
```json
{
  "id": "get_stock_price",
  "name": "Get Current Stock Price",
  "description": "Retrieves the real-time stock price for a given ticker symbol.",
  "parameters": [
    { "name": "ticker", "type": "string", "description": "Stock ticker of the company" }
  ],
  "dependencies": [
    {
      "tool_id": "get_stock_ticker",
      "type": "parameter_directly_depends_on",
      "reason": "Ticker parameter must be resolved first",
      "parameter_name": "ticker"
    },
    {
      "tool_id": "set_wifi_on",
      "type": "tool_indirectly_depends_on",
      "reason": "Optionally ensure network interface is active",
      "parameter_name": null
    }
  ]
}
```

## 6. Retrieval Baselines and Comparative Performance

Several retrieval paradigms were evaluated on ToolLinkOS using 1,569 queries:

| Retriever                             | mAP@10 | mAP@20 | mAP@30 |
|---------------------------------------|--------|--------|--------|
| Lexical Search (BM25)                 | 0.185  | 0.191  | 0.194  |
| Naïve RAG (vector only)               | 0.210  | 0.216  | 0.217  |
| Hybrid RAG (α=0.8)                    | 0.202  | 0.208  | 0.209  |
| Graph RAG-Tool Fusion (k=3, no rerank)| 0.856  | 0.873  | 0.873  |
| Graph RAG-Tool Fusion (k=3, rerank)   | 0.927  | 0.938  | 0.938  |

Graph RAG-Tool Fusion achieved an absolute improvement of 71.7 percentage points in mAP@10 compared to Naïve RAG (0.210 → 0.927). Detailed metrics for recall and nDCG at various cutoffs are provided in Appendix C of the source publication [2502.07223].

## 7. Access, Licensing, and Use

ToolLinkOS is distributed under the MIT License and is available for download at https://github.com/EliasLumer/Graph-RAG-Tool-Fusion-ToolLinkOS. The dataset is intended for research purposes, particularly for evaluating retrieval methods that must address both semantic similarity and graph-structured dependency resolution in large tool knowledge bases. Since the benchmark consists of fictional tools and uses a uniform schema, it facilitates controlled studies that are decoupled from specific, potentially evolving real-world APIs.

*This suggests that ToolLinkOS provides a critical reference for developing and benchmarking retrieval-augmented generation agents in scenarios where multi-step tool orchestration and explicit dependency tracking are required.*

Source: https://www.emergentmind.com/topics/toollinkos-dataset