DUETQA: Multi-hop Tool-Augmented QA Dataset

• DUETQA is a dataset for multi-hop QA that requires explicit live search and integration of evidence from at least two diverse, post-2024 sources.
• It employs a multi-agent self-play framework with search-enabled LLMs and precise formal metrics to enforce obfuscation and deep, stepwise synthesis.
• It advances the evaluation of tool-augmented language models by challenging them with non-trivial, verifiable multi-step reasoning and cross-domain evidence retrieval.

DUETQA is a rigorously constructed evaluation and training dataset for complex, verifiable, multi-hop question answering in the context of tool-augmented LLMs. It is specifically designed to probe and advance the capabilities of LLMs in long-horizon information retrieval, synthesis, and reasoning, requiring explicit reliance on live web search and evidence integration from diverse post-2024 sources. Unlike prior QA benchmarks, DUETQA enforces both genuine retrieval grounding and non-trivial multi-hop synthesis that cannot be shortcut by parametric knowledge or single-source lookups. Its data generation protocol is grounded in multi-agent self-play, systematic obfuscation, and formal validation metrics. DUETQA is currently central to advancing state-of-the-art DeepSearch and agentic systems in open-domain QA (Singh et al., 28 Sep 2025).

1. Problem Motivation and Distinguishing Properties

DUETQA addresses two main deficiencies of existing QA datasets for tool-augmented LLMs: lack of verifiability in provenance, and insufficient requirement for explicit multi-hop, heterogeneous-source reasoning. Standard benchmarks like HotpotQA, TriviaQA, or 2Wiki are often answerable by retrieving a single fact or using an LLM’s parametric memory, with minimal need for structured tool use. DUETQA introduces hard desiderata:

• Live search dependence: Each question is intractable without issuing at least one post-2024 web-search query.
• Heterogeneous domain grounding: Each question demands evidence from at least two distinct non-Wikipedia web domains.
• Obfuscated reasoning: Surface cues and obvious topic anchors are systematically removed, requiring actual tool-mediated stepwise synthesis.
• Verifiability: Correctness is guaranteed only if evidence from cited, timestamped URLs is retrieved, preventing shortcuts via memorized knowledge.

This design yields instances where successful answering necessitates non-redundant, multi-step tool invocation and explicit cross-source synthesis (Singh et al., 28 Sep 2025).

2. Multi-Agent Self-Play Generation Pipeline

DUETQA is generated through a three-agent self-supervised framework involving two independent, search-enabled LLMs (here denoted as ℳ₁ and ℳ₂, with backbones O3 and O4-mini) and one strong non-search LLM (GPT-4o). The process proceeds as follows:

• Question Synthesis:
  • Mixture-of-Themes: Randomly sample $k \in \{5,6,7\}$ themes from a curated taxonomy of 200+ domains. ℳ₁/ℳ₂ issue live search queries for recent/obscure facts, then compose an answer by chaining retrieved facts into a multi-hop scaffold.
  • Seeded-Question: Begin with a bank of 100 real-world seed questions $q_{0}$; rewrite these to incorporate at least one post-2024 fact sourced via new live search, with multi-hop scaffolding retained.
• Obfuscation: ℳ₃ paraphrases the resultant question to erase explicit anchors (e.g., dates to intervals, numbers to magnitudes, entity names to descriptors), minimizing shortcutting via fuzzy matching.
• Cross-Agent Verification: For candidate (q, a), both ℳ₁ and ℳ₂ must answer correctly under search-enabled conditions, while ℳ₃ must fail without search. This implements the formal requirement:

$$P(a | q,\mathcal{M}_{\text{search}}) \geq \delta_2\$$

• Selection: Only instances where both search-enabled agents agree and the non-search agent fails are retained, enforcing non-triviality and independence from model memory.

3. Formal Dataset Constraints and Quality Metrics

DUETQA employs precise, quantifiable constraints throughout generation:

Metric	Formal Definition	DUETQA Value
Search-Dependence Ratio	$$\mathrm{SDR} = 1/|Q| \sum_q \mathbb{1}[C(q) \geq 1]$$<br>($C(q)$ = #search steps)	1 (every q requires search)
Domain Grounding Score	$\mathrm{DGS}(q) = |\{d(u) : u \in U(q)\}|$	$\geq 2$ for all $q$ (mean ≈ 2.7)
Cross-Agent Precision (CAP)	$$\mathrm{CAP}=1/|Q|\sum_q \mathbb{1}[\mathcal{M}_1(q)=a \wedge \mathcal{M}_2(q)=a]$$	$\approx 0.97$ pre-filter

Obfuscation robustness is quantified by the drop in BLEU overlap (OR ≈ 0.82), and search dependence is empirically checked (SDM = 1.00 by design). Human spot-verification on 5% of samples confirms $q_{0}$0 for answer correctness, 100% agreement on need for web search, and absence of ungrounded hallucinations.

4. Dataset Composition, Reasoning Depth, and Representative Examples

DUETQA comprises 4,988 multi-hop QA pairs spanning over 200 science, technology, policy, business, law, and culture domains. Each question:

• Is grounded in $q_{0}$1 sampled themes
• Requires, on average, 2–4 reasoning hops (mean ≈ 2.7)
• Demands ≈3.4 web-search tool calls per question (std dev ≈1.6)
• Cites, on average, 3.1 URLs from ≈2.7 unique domains per answer

A canonical example illustrates the enforcement of these constraints: a question about a 2025 southeastern university’s synthetic fuel research requires searching for an obscure, recent partnership and extracting a performance metric from a second, independent technical source, both outside Wikipedia.

Sample answer includes timestamped facts, inline citations, and explicit tool-use traces, with all source URL domains disjoint (Singh et al., 28 Sep 2025).

5. Implications for Training and Evaluating Tool-Augmented LLMs

DUETQA provides a robust substrate for model training and evaluation in complex information-seeking settings:

• Long-horizon search: Mean web-search tool calls are ≫1, with frequent need for 4–6-step search-citation chains.
• Generalization: Enforced heterogeneity and obfuscation require agentic reasoning, robust synthesis, and resistance to shortcut learning.
• Efficient tool usage: Because the majority of existing QA datasets permit parametrically “easy” answers, DUETQA exposes models to control challenges, such as deciding breadth and depth of search via marginal utility and reward-aware scheduling (as leveraged in Fathom-DeepResearch (Singh et al., 28 Sep 2025)).
• Metric-grounded evaluation: SUITE of formal metrics (SDR, CAP, DDI, OR, SDM) enables quantitative comparison of models’ ability to reason and ground claims with tool use, not just accuracy.

Models trained on DUETQA, such as Fathom-Search-4B, demonstrate substantially improved performance on DeepSearch and DeepResearch benchmarks, including SimpleQA, WebWalker, FRAMES, MuSiQue, and novel agentic challenges.

6. SUITABILITY AND LIMITATIONS

DUETQA is explicitly unsuited for probing single-hop factual recall or parametric memorization, as all questions require citation of post-2024 facts and source diversity. It is optimized for:

• Sustained, verifiable tool-calling: Benchmarking models’ efficiency, breadth, and accuracy in extended tool-mediated QA trajectories.
• Robust agentic evaluation: Validating LLMs’ capacity to handle ambiguous, obfuscated, or novelty-framed questions in open domains.

A plausible implication is that DUETQA, by construction, will under-represent certain single-hop, common-sense QA patterns, but provide valuable signal for evaluating and improving the search, citation, and reasoning layers within tool-augmented models.

7. Dataset Access and Downstream Applications

DUETQA is directly referenced and utilized in the open-weights Fathom-DeepResearch system, which achieves state-of-the-art performance in the DeepSearch category and demonstrates cross-domain generalization to tasks including AIME-25, GPQA-Diamond, and MedQA (Singh et al., 28 Sep 2025). Its specifications—multi-agent self-play, domain-diverse citation, and enforced search-dependence—are expected to influence benchmark design for next-generation agentic QA models.

The dataset pipeline, including generation, obfuscation, and filtering logic, enables reproducibility and extension to new domains or periods as the web corpus evolves.