---
title: IoD DeepResearch Benchmark
url: https://www.emergentmind.com/topics/iod-deepresearch-benchmark
type: topic
---

# IoD DeepResearch Benchmark

The IoD DeepResearch Benchmark is a benchmark established within the IoDResearch framework to evaluate both data representation and Deep Research capabilities on private heterogeneous data under the Internet of Data paradigm. In this setting, private heterogeneous resources are encapsulated as FAIR-compliant digital objects, refined into atomic knowledge units and knowledge graphs, and then used for three downstream task classes: Digital Object Retrieval, RAG-based Question Answering, and Structured Scientific Report Generation. The benchmark is built on a corpus of over 500 high-quality documents and papers, about 6 million tokens, plus associated resources from four domains—Chinese law, geophysical exploration, computer science, and molecular dynamics—and is intended to test multi-granularity retrieval, grounded question answering, and agentic scientific reporting over private collections rather than web-only search [2510.01553].

## 1. Definition and intended scope

The benchmark was introduced to address a specific deficiency in prevailing DeepResearch systems: most existing efforts focus primarily on web search while overlooking local private data. In the IoDResearch formulation, this leads to low retrieval efficiency for private data and poor adherence to the FAIR principles. The benchmark therefore evaluates whether a system can operate over IoD-style private corpora in a way that combines data representation quality with downstream reasoning and synthesis quality [2510.01553].

The targeted capabilities are explicitly broader than standard text-centric retrieval. They include encapsulating private heterogeneous resources as FAIR digital objects, enriching metadata, refining content into atomic knowledge and knowledge graphs, performing multi-granularity and multi-strategy search over Digital Objects, Level-2 Digital Object passages, and fine-grained knowledge, answering both single-hop and multi-hop questions, and generating structured scientific reports through an agentic pipeline. In the paper’s framing, the benchmark is not restricted to retrieval accuracy; it is also a test of whether IoD-aligned data organization improves downstream Deep Research over private collections [2510.01553].

A central organizing idea is the IoD paradigm itself: resources are assigned persistent identifiers and DOIs, integrated into domain-specific IoDs and then into a global IoD network, and exposed through standardized search tools via MCP to LLM agents. This couples data management and Deep Research evaluation in a single benchmark setting. A plausible implication is that the benchmark is as much about representational infrastructure as it is about report generation.

## 2. Corpus construction and IoD-oriented data representation

The benchmark corpus contains over 500 high-quality documents and papers, about 6 million tokens, plus associated resources from four domains: Chinese law, geophysical exploration, computer science, and molecular dynamics. The paper emphasizes multimodality and heterogeneous sources, including text documents, tables, images and audio represented through descriptive multimodal metadata, and code, logs, and datasets referenced as resources. It does not enumerate per-modality counts [2510.01553].

Each raw data entity is encapsulated as a FAIR-compliant Digital Object. This involves assigning a unique identifier and a DOI, enriching the object with metadata, and packaging it according to DOA and IoD protocols. Long documents are segmented into passages, and each chunk becomes a Level-2 Digital Object for fine-grained indexing and retrieval. Metadata enrichment extends beyond explicit curated fields: LLM-based enrichment adds content summaries, hypothetical questions, classification labels, and keywords; for non-text resources it adds descriptive multimodal metadata; and information discovered during knowledge refinement is preserved in metadata [2510.01553].

The benchmark depends on two further representational layers. The first is Atomic Knowledge Units, defined as minimal factual units distilled from Digital Objects that cannot be further decomposed. The second is a knowledge-graph layer derived from passage-level entity and relation extraction, with nodes and edges assigned canonical names, multi-keyword themes, and concise descriptions. Redundant facts and graph structures are deduplicated and merged. Together, these layers form a heterogeneous graph index that supports multi-granularity retrieval from whole Digital Objects down to AKUs and KG subgraphs [2510.01553].

The paper describes this as a three-layer IoD representation—Data Resource, Digital Object, and Knowledge Refinement. It explicitly notes that no formal mathematical graph definition or graph-scoring formula is provided. This suggests that the benchmark’s novelty lies primarily in system design and evaluation composition rather than in a formal retrieval theory.

## 3. Task structure and benchmark protocol

The benchmark is organized into three task categories built over the same IoD-processed private corpus. It is presented as an evaluation-only benchmark; the paper does not specify train, validation, or test splits [2510.01553].

| Task | Scale | Focus |
|---|---:|---|
| Digital Object Retrieval | 200 questions | DO-level retrieval with metadata-aware outputs |
| RAG-based Question Answering | 800 questions | single-domain single-hop and multi-hop, plus cross-domain multi-hop QA |
| Structured Scientific Report Generation | 60 prompts | long-form scientific synthesis over private heterogeneous sources |

Task 1 evaluates Digital Object Retrieval with 200 questions representative of common user requirements in IoD scenarios. Queries target retrieval of relevant Digital Objects, and outputs are ranked Digital Objects accompanied by metadata such as type, source, and timestamp for trust and recency filtering. Although the system supports retrieval over Digital Objects, Level-2 chunks, AKUs, and KG subgraphs, this task evaluates only DO-level retrieval. Supported retrieval strategies include keyword search, vector-based similarity, KG reasoning, and hybrid recall [2510.01553].

Task 2 evaluates RAG-based Question Answering with 800 questions. Of these, 400 are single-domain questions with a 1:1 split between single-hop and multi-hop reasoning, and 400 are cross-domain multi-hop questions. Query construction is described as manual design with assistance from human experts and RAGAS, with emphasis on multi-hop reasoning across heterogeneous sources. The benchmark evaluates generative answers rather than exact-match span extraction [2510.01553].

Task 3 evaluates Structured Scientific Report Generation with 60 prompts, divided into 30 single-domain and 30 cross-domain prompts. The goal is to test whether a system can integrate complex knowledge from private heterogeneous sources and produce structured scientific writing. The paper does not rigidly specify required sections such as abstract, methods, or results. Instead, the protocol relies on an agent architecture in which a Planner creates a stepwise plan, a Worker Team retrieves, filters, summarizes, and performs iterative refinement, and a Reporter Team writes and validates the report [2510.01553].

## 4. Evaluation methodology

The evaluation protocol is task-specific. For Digital Object Retrieval, the reported metrics are Precision, Recall, and F1. The paper reports values for these metrics but does not provide mathematical formulas. For Question Answering, the benchmark uses RAGAS metrics: Answer Accuracy, Answer Faithfulness, Answer Relevance, Context Precision, Context Recall, and Context F1. Again, the formal metric definitions are not reproduced in the paper [2510.01553].

For report generation, the benchmark uses an LLM-as-Judge protocol with Qwen-Turbo at temperature \(= 0\), with each report scored three times and averaged. The report-level dimensions are interest level, coherence and organization, relevance and focus, coverage, and breadth and depth. Human experts, operating single-blind to method identity, evaluate the same five dimensions and additionally score factual accuracy. Standard text-generation metrics such as ROUGE, BLEU, METEOR, and BERTScore are not used or reported [2510.01553].

The benchmark also operationalizes FAIR through Digital Object encapsulation and multi-granularity indexing, but it does not define an explicit FAIR compliance scoring rubric or quantitative FAIR metric. This is an important methodological boundary: FAIR is embedded as a design principle of the benchmark environment rather than as a separately scored axis [2510.01553].

The paper further notes that tasks were manually designed with assistance from human experts and RAGAS, but it does not detail per-instance gold target sets or expected citations. For QA, grounding is assessed through RAGAS context metrics; for reports, it is assessed through LLM and human judgment rather than explicit citation-format verification.

## 5. Baselines, system configuration, and reported results

All compared methods use Qwen-Turbo as the base LLM to ensure fairness. The baseline systems are Naive RAG, LightRAG, DO-RAG, and DeepSearcher, while the evaluated system under test is IoDResearch with IoDAgents. The IoDAgents configuration consists of a Planner, a Worker Team for search, reflection, iterative refinement, and tool execution, and a Reporter Team comprising a writer and a checker. Retrieval tools are exposed via MCP, and the system supports keyword, vector, KG-reasoning, and hybrid strategies over Digital Objects, Level-2 chunks, AKUs, and KG subgraphs [2510.01553].

On Task 1, Digital Object Retrieval, the reported Precision/Recall/F1 scores are: Naive RAG \(55.22 / 70.82 / 62.05\), DO-RAG \(69.51 / 84.34 / 76.21\), LightRAG \(73.15 / 85.69 / 78.93\), and IoDResearch \(76.26 / 90.18 / 82.64\). On this task, IoDResearch yields the highest values on all three reported metrics [2510.01553].

On Task 2, Question Answering, the single-domain results are reported as Answer Accuracy / Answer Faithfulness / Answer Relevance; Context Precision / Context Recall / Context F1. The scores are: Naive RAG \(70.28 / 84.53 / 87.52; 60.39 / 75.77 / 67.21\), DO-RAG \(71.45 / 84.12 / 87.34; 61.35 / 76.45 / 68.07\), LightRAG \(75.55 / 85.10 / 86.59; 64.89 / 75.95 / 69.98\), and IoDResearch \(79.98 / 87.33 / 90.20; 65.35 / 80.45 / 72.11\). For cross-domain QA, the reported scores are: Naive RAG \(42.42 / 69.98 / 66.05; 44.38 / 45.00 / 44.69\), DO-RAG \(50.32 / 70.31 / 66.58; 46.52 / 48.65 / 47.56\), LightRAG \(56.67 / 75.59 / 76.91; 50.02 / 52.50 / 51.23\), and IoDResearch \(59.40 / 78.07 / 78.18; 52.02 / 53.50 / 52.75\) [2510.01553].

On Task 3, Structured Scientific Report Generation, the reported scores are given as LLM-as-Judge / Human Expert for single-domain and cross-domain settings. Zero-shot LLM scores are \(7.61 / 7.45; 5.65 / 5.23\). LightRAG scores are \(7.95 / 7.86; 6.53 / 5.88\). IoDResearch without Agent scores are \(8.03 / 7.92; 6.56 / 5.94\). DeepSearcher scores are \(8.13 / 8.08; 6.77 / 6.02\). IoDResearch scores are \(8.31 / 8.23; 7.01 / 6.45\). The paper explicitly states an ablation result: removing the multi-agent component decreases report-writing scores, which it interprets as highlighting the importance of agent collaboration [2510.01553].

The paper does not report statistical significance testing or a separate error analysis. It does state that IoDResearch consistently surpasses representative RAG and Deep Research baselines across all tasks, with notable gains in cross-domain multi-hop QA and human-rated report quality [2510.01553].

## 6. Position within the deep-research benchmark landscape

The IoD DeepResearch Benchmark is best understood as a private-data-centric benchmark. Its defining property is that evaluation begins with FAIR Digital Object encapsulation and proceeds through multi-granularity retrieval, QA, and report writing over private heterogeneous corpora. This differs materially from enterprise, web, and report-centric benchmarks that use other naming schemes and evaluation assumptions [2510.01553].

Several adjacent benchmarks make this distinction explicit by terminology. The paper on Deep Research Bench states that “IoD” does not appear in the text and that, if “IoD DeepResearch Benchmark” is referenced elsewhere, it refers there to Deep Research Bench rather than to an IoD-specific protocol [2506.06287]. DeepResearch Bench II likewise states that the term “IoD DeepResearch Benchmark” does not appear anywhere in the paper, and that its own benchmark is distinct, with 132 grounded tasks across 22 domains and 9,430 fine-grained binary rubrics [2601.08536]. The DRACO paper similarly states that it does not mention or define an “IoD DeepResearch Benchmark,” instead presenting a cross-domain benchmark of 100 open-ended tasks derived from anonymized real-world usage patterns [2602.11685]. “Understanding DeepResearch via Reports” also states that it does not define or mention “IoD,” and instead introduces DeepResearch-ReportEval as a report-centric framework with 100 curated queries across 12 categories [2510.07861].

The contrast is not merely nominal. DRBench evaluates enterprise deep research in a self-hosted enterprise environment spanning productivity software, cloud file systems, emails, chat conversations, and the open web, with 15 tasks across 10 domains [2510.00172]. Deep Research Bench evaluates AI web research agents on 89 multi-step web research task instances in a frozen RetroSearch environment [2506.06287]. DeepResearch Bench II evaluates long-form reports with expert-derived binary rubrics [2601.08536]. DRACO emphasizes factual accuracy, breadth and depth, presentation quality, and citation quality via expert-designed weighted rubrics over 100 tasks [2602.11685]. By contrast, the IoD benchmark is anchored in FAIR Digital Objects, Level-2 chunking, AKUs, KG refinement, and private heterogeneous resources [2510.01553].

A common misconception is therefore to treat “IoD DeepResearch Benchmark” as interchangeable with web-only or rubric-only deep-research benchmarks. The available papers do not support that equivalence.

## 7. Limitations, interpretive boundaries, and likely extensions

The benchmark’s scope is meaningful but limited. Its corpus is about 500 documents and papers, about 6 million tokens, across four domains. Multimodal information is integrated through descriptive metadata rather than explicit cross-modal retrieval tasks. The paper does not provide formal mathematical definitions for the heterogeneous graph, retrieval scoring, or FAIR compliance scoring, and it does not release public code, data links, licenses, prompts, or orchestration scripts tied directly to the benchmark [2510.01553].

These omissions constrain reproducibility and comparability. The benchmark is evaluation-only, with no stated train, validation, or test split. It does not publish concrete sample queries and expected outputs, and it does not provide detailed hyperparameters beyond the use of Qwen-Turbo and the report-judge temperature. A plausible implication is that reproducing the exact benchmark environment requires substantial reconstruction of the IoD pipeline rather than simply downloading a standard benchmark package [2510.01553].

The paper also leaves several methodological gaps open. FAIR is operationalized but not quantitatively scored. Explicit multimodal tasks are not included. Error analysis and significance testing are absent. A plausible extension, consistent with the paper’s own framing, would be broader domain coverage, explicit multimodal tasks, formal FAIR metrics, and more detailed releases for reproducibility. Those extensions would move the benchmark from a strong proof of concept for private-data-centric Deep Research toward a more standardized evaluation suite [2510.01553].

Within the present literature, the benchmark’s main significance lies in making private heterogeneous data a first-class object of Deep Research evaluation. It treats data encapsulation, metadata enrichment, multi-granularity retrieval, grounded QA, and long-form scientific synthesis as a single continuum rather than as isolated tasks. That design choice distinguishes it from web-only, enterprise-app, or report-rubric benchmarks and defines its role in the emerging deep-research benchmark ecosystem.

Source: https://www.emergentmind.com/topics/iod-deepresearch-benchmark