---
title: 'FINDER: Fine-Grained Benchmark for Deep Research Agents'
url: https://www.emergentmind.com/topics/fine-grained-deepresearch-bench-finder
type: topic
---

# FINDER: Fine-Grained Benchmark for Deep Research Agents

Fine-grained DEepResearch bench (FINDER) is an expert-driven benchmark suite established to rigorously evaluate the end-to-end research capabilities of deep research agents (DRAs) operating on complex, multi-source information tasks [2512.01948, 2506.23139, 2510.13936, 2507.05495]. FINDER combines highly structured task design, granular evaluation rubrics, diagnostic failure taxonomy, and robust annotation protocols to support reproducible, interpretable assessment of DRAs in enterprise, academic, financial, and open-domain settings.

## 1. Definition and Core Principles

FINDER is defined as a fine-grained benchmark for DRAs, extending DeepResearch Bench with 100 human-curated research tasks (50 English, 50 Chinese), 419 checklist items for granular report evaluation, and a complementary taxonomy (DEFT) for failure analysis [2512.01948]. Its core objectives are to standardize output structure, enforce analytical depth, and quantify factual grounding via explicit criteria. Unlike "question-answering" paradigms, FINDER evaluates agentic workflows that autonomously retrieve, synthesize, and report across diverse, sparsely linked sources.

Key design principles:
- **Explicit Guidance**: Each task prompt specifies required sections, format, and citation style.
- **Checklist-Driven Scoring**: Reports are evaluated against granular, per-task criteria covering structural, analytical, and factual dimensions.
- **Diagnostic Focus**: Complementary error taxonomy facilitates process-level failure analysis.
- **Reproducibility**: All components (tasks, rubrics, metrics) are public and formally defined.

## 2. Synthetic Data Pipeline and Artifact Creation

FINDER incorporates a workflow-guided, query-first synthetic data pipeline (notably for enterprise scenarios under HERB, a FINDER alias) [2506.23139]. The pipeline comprises four stages:

1. **Schema Initialization**: Organizational entities, products, employees, and customers are instantiated.
2. **Query Template Instantiation**: 41 templates generate multi-hop questions (content, people, artifact, customer types).
3. **Workflow Execution**: Nine manually crafted workflows (three per business phase) simulate authentic process artifacts under realistic noise (α ≈ 0.15).
4. **Answer & Unanswerable Generation**: Each answerable query is mapped to explicit ground-truth evidence; unanswerable cases are created by ensuring critical evidence is missing.

Artifact statistics (enterprise context):
- **Slack messages**: 33,632 (27 tokens avg.)
- **Meeting transcripts**: 321 (1,200 tokens avg.)
- **Documents**: 400 (800 tokens avg.)
- **GitHub pull requests**: 3,562
- **URLs**: 575; **Customer profiles**: 120
Total artifacts: 39,190, indexed via hybrid embedding (vector/BM25) and structured indices.

## 3. Evaluation Rubrics, Metrics, and Taxonomies

FINDER implements granular scoring frameworks tailored to domain. In finance, **HisRubric** employs a hierarchical analytical structure with a four-level competence hierarchy (Recognition, Calculation, Abstraction, Interpretation), mapped to sector-specific sections and graded via 15,808 items [2510.13936].

In research-report synthesis, FINDER's rubric includes:
- **Structural Compliance**: Section/order, format conventions, citation style adherence.
- **Analytical Depth**: Conceptual definitions, methodological rigor, cross-domain analysis.
- **Factual Grounding**: Citation accuracy, evidence verification, data source authority.

Principal scoring metrics [2512.01948, 2510.13936]:
- **RACE**: Reference-based Adaptive Criteria-driven Evaluation, scoring outputs across comprehensiveness, depth, instruction-following, and readability relative to gold-standard references.
- **FACT**: Factual Abundance and Citation Trustworthiness, quantifying supported statement–citation pairs and citation diversity.
- **Checklist Pass Rate**: Fraction of task-specific rubric items satisfied.
- **Positive Taxonomy Metric**: Success score $S_i = |D|\;\cos\!\bigl(\frac{E_i}{|D|}\frac{\pi}{2}\bigr)$ reflecting error distributions in DEFT categories.

In comparative agent evaluation, the platform Deep Research Comparator applies the Bradley–Terry model for pairwise outcome-based ranking and process-based annotation rates (step/spans) [2507.05495].

## 4. Multi-Hop Reasoning and Unanswerability Protocols

Multi-hop queries in FINDER are generated by template grammars, typically requiring 2–4 chain-of-thought reasoning steps traversing distinct entity–source pairs (e.g., locating documents, inferring authorship, mapping to organizational IDs) [2506.23139]. Each reasoning hop must be grounded in explicit artifact evidence for answerability; unanswerable queries are constructed by omitting required artifacts in the workflow.

Unanswerable handling is formalized: systems must abstain or flag "cannot answer" to score positively; performance is summarized via the Unan.% metric (percentage of correctly identified unanswerables).

## 5. Baselines, Empirical Results, and Bottleneck Analyses

A comprehensive battery of baseline systems—ranging from zero-shot LLMs to advanced retrieval-augmented generation (RAG) frameworks (hybrid, RAPTOR, GraphRAG, HippoRAG 2, PGRAG, ReAct agents)—are evaluated using FINDER [2506.23139]. Experimental highlights:

- **Best-performing standard RAG**: Hybrid (Avg = 20.61)
- **Best agentic RAG (ReAct + GPT-4o)**: Avg = 32.96, Unan.% = 23.03
- **0-shot LLM**: Avg = 4.55, Unan.% = 88.70
- **GraphRAG**: Avg = 10.31, Unan.% = 63.41

Retrieval is consistently identified as the main bottleneck; recall@20 rarely exceeds 50–60%, requiring LLMs to reason over partial contexts. Under oracle conditions (perfect evidence), best model scores rise to Avg = 85.76, yet errors persist, indicating imperfect reasoning resilience.

In report synthesis (1000 outputs over 15 agents), best proprietary system achieves RACE ≈ 50.95, FACT C.Acc. ≈ 57.1%, checklist pass ≈ 63%, and structured taxonomy scores $(S_1,S_2,S_3)\approx(89.8,97.2,89.8)$ [2512.01948].

## 6. Deep rEsearch Failure Taxonomy (DEFT): Error Typology

FINDER introduces DEFT, a taxonomy of 14 empirically-derived failure modes, constructed via multi-stage human–LLM co-annotation and validated with Krippendorff’s $\alpha\ge0.8$ [2512.01948]. DEFT spans:

- **Reasoning**: FUR (failure to understand requirements), LAD (lack analytical depth), LAS (limited scope), RPS (rigid planning)
- **Retrieval**: IIA (insufficient info acquisition), IHD (handling deficiency), IIF (integration failure), IRM (representation misalignment), VMF (verification failure)
- **Generation**: RCP (redundant content), SOD (structural dysfunction), CSD (spec deviation), DAR (deficient rigor), SCF (fabrication)

Checklist-driven evaluation is complemented by DEFT diagnosis, supporting both output-level and process-level agent assessment.

## 7. Domain Extension, Annotation, and Future Directions

FINDER’s methodology generalizes to financial (FinDeepResearch), legal, clinical, and open-domain settings by constructing domain-specific hierarchical structures, extracting fine-grained items, weighting complexity, assembling multilingual reference sets, and applying three-pronged evaluation protocols [2510.13936]. Annotation workflows leverage both human and LLM inputs, with metrics such as Cohen's κ, Fleiss’s κ, and process-level upvote rates central to aggregate reliability [2507.05495].

Recommendations for advancing FINDER include:
- Workflow-aware retrieval planners routing queries by dynamic signal.
- Multi-hop retrievers exploiting temporal, topical, and structural affinities.
- Enhanced abstention strategies and adversarial query generation for unanswerability.
- Joint models integrating retrieval and reasoning, minimizing partial evidence propagation.

A plausible implication is that transparent, checklist-based evaluation supplemented by error taxonomy will accelerate the deployment of DRAs that meet real-world demands for rigor, reliability, and explainability across heterogeneous research contexts.

---

| FINDER Component              | Description                                             | Paper Reference    |
|-------------------------------|--------------------------------------------------------|-------------------|
| Task Design                   | 100 curated research tasks, 419 checklist items        | [2512.01948]      |
| Synthetic Pipeline            | Workflow-guided artifact generation, multi-hop queries | [2506.23139]      |
| Evaluation Rubrics            | RACE, FACT, HisRubric, Checklist Pass Rate             | [2512.01948, 2510.13936] |
| Error Taxonomy (DEFT)         | 14 failure modes: reasoning, retrieval, generation     | [2512.01948]      |
| Annotation Platform           | Outcome/process annotation, BT ranking, upvote rates   | [2507.05495]      |

FINDER establishes the technical and methodological foundation for fine-grained benchmarking of deep research agents, supporting systematic analysis, robust failure diagnosis, and scalable cross-domain evaluation.

Source: https://www.emergentmind.com/topics/fine-grained-deepresearch-bench-finder