IoD DeepResearch Benchmark
- The paper introduces IoD DeepResearch Benchmark as a novel framework that encapsulates private heterogeneous data into FAIR digital objects for enhanced retrieval and synthesis.
- The benchmark leverages a corpus of 500+ documents across diverse domains, refining content into atomic knowledge units and knowledge graphs for multi-granularity search.
- The methodology employs three key tasks—Digital Object Retrieval, RAG-based Question Answering, and Structured Scientific Report Generation—to provide comprehensive performance metrics and insights.
The IoD DeepResearch Benchmark is a benchmark established within the IoDResearch framework to evaluate both data representation and Deep Research capabilities on private heterogeneous data under the Internet of Data paradigm. In this setting, private heterogeneous resources are encapsulated as FAIR-compliant digital objects, refined into atomic knowledge units and knowledge graphs, and then used for three downstream task classes: Digital Object Retrieval, RAG-based Question Answering, and Structured Scientific Report Generation. The benchmark is built on a corpus of over 500 high-quality documents and papers, about 6 million tokens, plus associated resources from four domains—Chinese law, geophysical exploration, computer science, and molecular dynamics—and is intended to test multi-granularity retrieval, grounded question answering, and agentic scientific reporting over private collections rather than web-only search (Shi et al., 2 Oct 2025).
1. Definition and intended scope
The benchmark was introduced to address a specific deficiency in prevailing DeepResearch systems: most existing efforts focus primarily on web search while overlooking local private data. In the IoDResearch formulation, this leads to low retrieval efficiency for private data and poor adherence to the FAIR principles. The benchmark therefore evaluates whether a system can operate over IoD-style private corpora in a way that combines data representation quality with downstream reasoning and synthesis quality (Shi et al., 2 Oct 2025).
The targeted capabilities are explicitly broader than standard text-centric retrieval. They include encapsulating private heterogeneous resources as FAIR digital objects, enriching metadata, refining content into atomic knowledge and knowledge graphs, performing multi-granularity and multi-strategy search over Digital Objects, Level-2 Digital Object passages, and fine-grained knowledge, answering both single-hop and multi-hop questions, and generating structured scientific reports through an agentic pipeline. In the paper’s framing, the benchmark is not restricted to retrieval accuracy; it is also a test of whether IoD-aligned data organization improves downstream Deep Research over private collections (Shi et al., 2 Oct 2025).
A central organizing idea is the IoD paradigm itself: resources are assigned persistent identifiers and DOIs, integrated into domain-specific IoDs and then into a global IoD network, and exposed through standardized search tools via MCP to LLM agents. This couples data management and Deep Research evaluation in a single benchmark setting. A plausible implication is that the benchmark is as much about representational infrastructure as it is about report generation.
2. Corpus construction and IoD-oriented data representation
The benchmark corpus contains over 500 high-quality documents and papers, about 6 million tokens, plus associated resources from four domains: Chinese law, geophysical exploration, computer science, and molecular dynamics. The paper emphasizes multimodality and heterogeneous sources, including text documents, tables, images and audio represented through descriptive multimodal metadata, and code, logs, and datasets referenced as resources. It does not enumerate per-modality counts (Shi et al., 2 Oct 2025).
Each raw data entity is encapsulated as a FAIR-compliant Digital Object. This involves assigning a unique identifier and a DOI, enriching the object with metadata, and packaging it according to DOA and IoD protocols. Long documents are segmented into passages, and each chunk becomes a Level-2 Digital Object for fine-grained indexing and retrieval. Metadata enrichment extends beyond explicit curated fields: LLM-based enrichment adds content summaries, hypothetical questions, classification labels, and keywords; for non-text resources it adds descriptive multimodal metadata; and information discovered during knowledge refinement is preserved in metadata (Shi et al., 2 Oct 2025).
The benchmark depends on two further representational layers. The first is Atomic Knowledge Units, defined as minimal factual units distilled from Digital Objects that cannot be further decomposed. The second is a knowledge-graph layer derived from passage-level entity and relation extraction, with nodes and edges assigned canonical names, multi-keyword themes, and concise descriptions. Redundant facts and graph structures are deduplicated and merged. Together, these layers form a heterogeneous graph index that supports multi-granularity retrieval from whole Digital Objects down to AKUs and KG subgraphs (Shi et al., 2 Oct 2025).
The paper describes this as a three-layer IoD representation—Data Resource, Digital Object, and Knowledge Refinement. It explicitly notes that no formal mathematical graph definition or graph-scoring formula is provided. This suggests that the benchmark’s novelty lies primarily in system design and evaluation composition rather than in a formal retrieval theory.
3. Task structure and benchmark protocol
The benchmark is organized into three task categories built over the same IoD-processed private corpus. It is presented as an evaluation-only benchmark; the paper does not specify train, validation, or test splits (Shi et al., 2 Oct 2025).
| Task | Scale | Focus |
|---|---|---|
| Digital Object Retrieval | 200 questions | DO-level retrieval with metadata-aware outputs |
| RAG-based Question Answering | 800 questions | single-domain single-hop and multi-hop, plus cross-domain multi-hop QA |
| Structured Scientific Report Generation | 60 prompts | long-form scientific synthesis over private heterogeneous sources |
Task 1 evaluates Digital Object Retrieval with 200 questions representative of common user requirements in IoD scenarios. Queries target retrieval of relevant Digital Objects, and outputs are ranked Digital Objects accompanied by metadata such as type, source, and timestamp for trust and recency filtering. Although the system supports retrieval over Digital Objects, Level-2 chunks, AKUs, and KG subgraphs, this task evaluates only DO-level retrieval. Supported retrieval strategies include keyword search, vector-based similarity, KG reasoning, and hybrid recall (Shi et al., 2 Oct 2025).
Task 2 evaluates RAG-based Question Answering with 800 questions. Of these, 400 are single-domain questions with a 1:1 split between single-hop and multi-hop reasoning, and 400 are cross-domain multi-hop questions. Query construction is described as manual design with assistance from human experts and RAGAS, with emphasis on multi-hop reasoning across heterogeneous sources. The benchmark evaluates generative answers rather than exact-match span extraction (Shi et al., 2 Oct 2025).
Task 3 evaluates Structured Scientific Report Generation with 60 prompts, divided into 30 single-domain and 30 cross-domain prompts. The goal is to test whether a system can integrate complex knowledge from private heterogeneous sources and produce structured scientific writing. The paper does not rigidly specify required sections such as abstract, methods, or results. Instead, the protocol relies on an agent architecture in which a Planner creates a stepwise plan, a Worker Team retrieves, filters, summarizes, and performs iterative refinement, and a Reporter Team writes and validates the report (Shi et al., 2 Oct 2025).
4. Evaluation methodology
The evaluation protocol is task-specific. For Digital Object Retrieval, the reported metrics are Precision, Recall, and F1. The paper reports values for these metrics but does not provide mathematical formulas. For Question Answering, the benchmark uses RAGAS metrics: Answer Accuracy, Answer Faithfulness, Answer Relevance, Context Precision, Context Recall, and Context F1. Again, the formal metric definitions are not reproduced in the paper (Shi et al., 2 Oct 2025).
For report generation, the benchmark uses an LLM-as-Judge protocol with Qwen-Turbo at temperature , with each report scored three times and averaged. The report-level dimensions are interest level, coherence and organization, relevance and focus, coverage, and breadth and depth. Human experts, operating single-blind to method identity, evaluate the same five dimensions and additionally score factual accuracy. Standard text-generation metrics such as ROUGE, BLEU, METEOR, and BERTScore are not used or reported (Shi et al., 2 Oct 2025).
The benchmark also operationalizes FAIR through Digital Object encapsulation and multi-granularity indexing, but it does not define an explicit FAIR compliance scoring rubric or quantitative FAIR metric. This is an important methodological boundary: FAIR is embedded as a design principle of the benchmark environment rather than as a separately scored axis (Shi et al., 2 Oct 2025).
The paper further notes that tasks were manually designed with assistance from human experts and RAGAS, but it does not detail per-instance gold target sets or expected citations. For QA, grounding is assessed through RAGAS context metrics; for reports, it is assessed through LLM and human judgment rather than explicit citation-format verification.
5. Baselines, system configuration, and reported results
All compared methods use Qwen-Turbo as the base LLM to ensure fairness. The baseline systems are Naive RAG, LightRAG, DO-RAG, and DeepSearcher, while the evaluated system under test is IoDResearch with IoDAgents. The IoDAgents configuration consists of a Planner, a Worker Team for search, reflection, iterative refinement, and tool execution, and a Reporter Team comprising a writer and a checker. Retrieval tools are exposed via MCP, and the system supports keyword, vector, KG-reasoning, and hybrid strategies over Digital Objects, Level-2 chunks, AKUs, and KG subgraphs (Shi et al., 2 Oct 2025).
On Task 1, Digital Object Retrieval, the reported Precision/Recall/F1 scores are: Naive RAG $55.22 / 70.82 / 62.05$, DO-RAG $69.51 / 84.34 / 76.21$, LightRAG $73.15 / 85.69 / 78.93$, and IoDResearch $76.26 / 90.18 / 82.64$. On this task, IoDResearch yields the highest values on all three reported metrics (Shi et al., 2 Oct 2025).
On Task 2, Question Answering, the single-domain results are reported as Answer Accuracy / Answer Faithfulness / Answer Relevance; Context Precision / Context Recall / Context F1. The scores are: Naive RAG $70.28 / 84.53 / 87.52; 60.39 / 75.77 / 67.21$, DO-RAG $71.45 / 84.12 / 87.34; 61.35 / 76.45 / 68.07$, LightRAG $75.55 / 85.10 / 86.59; 64.89 / 75.95 / 69.98$, and IoDResearch $79.98 / 87.33 / 90.20; 65.35 / 80.45 / 72.11$. For cross-domain QA, the reported scores are: Naive RAG $42.42 / 69.98 / 66.05; 44.38 / 45.00 / 44.69$, DO-RAG $55.22 / 70.82 / 62.05$0, LightRAG $55.22 / 70.82 / 62.05$1, and IoDResearch $55.22 / 70.82 / 62.05$2 (Shi et al., 2 Oct 2025).
On Task 3, Structured Scientific Report Generation, the reported scores are given as LLM-as-Judge / Human Expert for single-domain and cross-domain settings. Zero-shot LLM scores are $55.22 / 70.82 / 62.05$3. LightRAG scores are $55.22 / 70.82 / 62.05$4. IoDResearch without Agent scores are $55.22 / 70.82 / 62.05$5. DeepSearcher scores are $55.22 / 70.82 / 62.05$6. IoDResearch scores are $55.22 / 70.82 / 62.05$7. The paper explicitly states an ablation result: removing the multi-agent component decreases report-writing scores, which it interprets as highlighting the importance of agent collaboration (Shi et al., 2 Oct 2025).
The paper does not report statistical significance testing or a separate error analysis. It does state that IoDResearch consistently surpasses representative RAG and Deep Research baselines across all tasks, with notable gains in cross-domain multi-hop QA and human-rated report quality (Shi et al., 2 Oct 2025).
6. Position within the deep-research benchmark landscape
The IoD DeepResearch Benchmark is best understood as a private-data-centric benchmark. Its defining property is that evaluation begins with FAIR Digital Object encapsulation and proceeds through multi-granularity retrieval, QA, and report writing over private heterogeneous corpora. This differs materially from enterprise, web, and report-centric benchmarks that use other naming schemes and evaluation assumptions (Shi et al., 2 Oct 2025).
Several adjacent benchmarks make this distinction explicit by terminology. The paper on Deep Research Bench states that “IoD” does not appear in the text and that, if “IoD DeepResearch Benchmark” is referenced elsewhere, it refers there to Deep Research Bench rather than to an IoD-specific protocol (FutureSearch et al., 6 May 2025). DeepResearch Bench II likewise states that the term “IoD DeepResearch Benchmark” does not appear anywhere in the paper, and that its own benchmark is distinct, with 132 grounded tasks across 22 domains and 9,430 fine-grained binary rubrics (Li et al., 13 Jan 2026). The DRACO paper similarly states that it does not mention or define an “IoD DeepResearch Benchmark,” instead presenting a cross-domain benchmark of 100 open-ended tasks derived from anonymized real-world usage patterns (Zhong et al., 12 Feb 2026). “Understanding DeepResearch via Reports” also states that it does not define or mention “IoD,” and instead introduces DeepResearch-ReportEval as a report-centric framework with 100 curated queries across 12 categories (Fan et al., 9 Oct 2025).
The contrast is not merely nominal. DRBench evaluates enterprise deep research in a self-hosted enterprise environment spanning productivity software, cloud file systems, emails, chat conversations, and the open web, with 15 tasks across 10 domains (Abaskohi et al., 30 Sep 2025). Deep Research Bench evaluates AI web research agents on 89 multi-step web research task instances in a frozen RetroSearch environment (FutureSearch et al., 6 May 2025). DeepResearch Bench II evaluates long-form reports with expert-derived binary rubrics (Li et al., 13 Jan 2026). DRACO emphasizes factual accuracy, breadth and depth, presentation quality, and citation quality via expert-designed weighted rubrics over 100 tasks (Zhong et al., 12 Feb 2026). By contrast, the IoD benchmark is anchored in FAIR Digital Objects, Level-2 chunking, AKUs, KG refinement, and private heterogeneous resources (Shi et al., 2 Oct 2025).
A common misconception is therefore to treat “IoD DeepResearch Benchmark” as interchangeable with web-only or rubric-only deep-research benchmarks. The available papers do not support that equivalence.
7. Limitations, interpretive boundaries, and likely extensions
The benchmark’s scope is meaningful but limited. Its corpus is about 500 documents and papers, about 6 million tokens, across four domains. Multimodal information is integrated through descriptive metadata rather than explicit cross-modal retrieval tasks. The paper does not provide formal mathematical definitions for the heterogeneous graph, retrieval scoring, or FAIR compliance scoring, and it does not release public code, data links, licenses, prompts, or orchestration scripts tied directly to the benchmark (Shi et al., 2 Oct 2025).
These omissions constrain reproducibility and comparability. The benchmark is evaluation-only, with no stated train, validation, or test split. It does not publish concrete sample queries and expected outputs, and it does not provide detailed hyperparameters beyond the use of Qwen-Turbo and the report-judge temperature. A plausible implication is that reproducing the exact benchmark environment requires substantial reconstruction of the IoD pipeline rather than simply downloading a standard benchmark package (Shi et al., 2 Oct 2025).
The paper also leaves several methodological gaps open. FAIR is operationalized but not quantitatively scored. Explicit multimodal tasks are not included. Error analysis and significance testing are absent. A plausible extension, consistent with the paper’s own framing, would be broader domain coverage, explicit multimodal tasks, formal FAIR metrics, and more detailed releases for reproducibility. Those extensions would move the benchmark from a strong proof of concept for private-data-centric Deep Research toward a more standardized evaluation suite (Shi et al., 2 Oct 2025).
Within the present literature, the benchmark’s main significance lies in making private heterogeneous data a first-class object of Deep Research evaluation. It treats data encapsulation, metadata enrichment, multi-granularity retrieval, grounded QA, and long-form scientific synthesis as a single continuum rather than as isolated tasks. That design choice distinguishes it from web-only, enterprise-app, or report-rubric benchmarks and defines its role in the emerging deep-research benchmark ecosystem.