---
title: 'DiscoveryBench: Data-Driven Discovery Benchmark'
url: https://www.emergentmind.com/topics/discoverybench
type: topic
---

# DiscoveryBench: Data-Driven Discovery Benchmark

DiscoveryBench is a benchmark for evaluating whether large language models can perform **data-driven discovery** from provided datasets rather than merely answer predefined statistical questions or optimize predictive pipelines. It formalizes discovery as a multi-step process in which a system must interpret a natural-language goal, map that goal to dataset semantics, determine an appropriate analysis workflow, derive a hypothesis, and verify that the data support it. The benchmark contains a manually curated real component, **DB-Real**, and a synthetic component, **DB-Synth**, and introduces a structured representation of hypotheses in terms of **contexts**, **variables**, and **relationships**, together with a facet-based evaluation framework centered on the **Hypothesis Match Score (HMS)** [2407.01725].

## 1. Conceptualization of data-driven discovery

DiscoveryBench defines a data-driven hypothesis \(h\) as a declarative sentence whose truth can be inferred from a dataset \(D\) using a verification procedure
\[
\mathcal{V}_D:\mathcal{H} \to \{\mathrm{supported}, \mathrm{unsupported}\},
\]
where \(\mathcal{H}\) is the space of hypotheses [2407.01725]. A hypothesis may itself be composite:
\[
h := \phi(h_1, \ldots, h_n),
\]
with verification distributed over the constituent sub-hypotheses:
\[
\mathcal{V}_D(h)=\phi(\mathcal{V}_D(h_1),\ldots,\mathcal{V}_D(h_n)).
\]
This formulation treats discovery as a structured inferential object rather than a free-form answer string [2407.01725].

To make evaluation operational, DiscoveryBench rewrites hypotheses as
\[
h:= \psi(c, v, r),
\]
where \(c\) denotes **contexts**, \(v\) denotes **variables**, and \(r\) denotes **relationships** [2407.01725]. Contexts are boundary conditions or scope restrictions; variables are the interacting concepts, including a target variable and one or more independent variables; relationships are the form of interaction among variables under the specified context. In the benchmark’s wording, \(\psi(c,v,r)\) yields the declarative sentence that “under context \(c\), variables \(v\) have relationship \(r\)” [2407.01725].

This decomposition is extended through a **hypothesis semantic tree**. Internal nodes may represent intermediate derived variables, leaves correspond to non-derived independent variables, and the root corresponds to the target variable of the main hypothesis. A dataset may support multiple such trees, forming a semantic forest
\[
\mathcal{F}:=\cup_i\mathcal{T}_{h^{(i)}}
\]
[2407.01725]. This suggests that DiscoveryBench is designed to evaluate not only terminal answers but also the latent structural depth of a discovery process.

Within this formalism, a task asks the system, given one or more datasets \(D\) and a natural-language discovery goal \(G\), to “derive a hypothesis \(h = \psi(c,v,r)\) addressing \(G\) with the highest specificity for the context \(c\), variables \(v\), and relationship \(r\) supported by \(D\)” [2407.01725]. The benchmark is outcome-based: multiple workflows may be valid if they converge to a hypothesis aligned with the gold structure.

## 2. Benchmark composition and curation

DiscoveryBench consists of two components: **DB-Real** and **DB-Synth** [2407.01725].

**DB-Real** contains **264 tasks** manually derived from published scientific work across **6 domains**: **sociology, biology, humanities, economics, engineering, and meta-science** [2407.01725]. The tasks were constructed from published workflows using two curation routes. In the **data-first** route, open public datasets such as the National Longitudinal Surveys, Global Biodiversity Information Facility, and World Bank Open Data were paired with papers whose analyses could be reimplemented. Replication was labor-intensive: it took **15–40 person-hours** for each NLS-related paper and up to **90 person-hours** for GBIF datasets, with about **30%** of data-first attempts failing [2407.01725]. In the **code-first** route, scientific code repositories with available datasets were sought; among **785** Zenodo entries filtered for computational notebooks, over **85%** were unusable because of missing code, non-Python code, or inaccessible data, yielding **14** candidate repositories and only **3** final inclusions [2407.01725].

The split statistics for DB-Real are explicit. There are **25 train tasks** and **239 test tasks**, for a total of **264**; the train split contains **14 unique hypotheses** and the test split **144 unique hypotheses** [2407.01725]. Multi-dataset reasoning is frequent: **4** train tasks and **110** test tasks require more than one dataset, for **114** multi-dataset tasks in total, with up to **6 datasets** in a single task [2407.01725]. The train split covers **3 domains**, whereas the test split spans all **6** [2407.01725].

Each benchmark instance includes rich metadata. At the task level, fields can include an id, domain, workflow tags, optional domain knowledge, dataset descriptions, column descriptions, hypotheses, workflows, and one or more natural-language queries. At the query level, fields can include qid, difficulty, and the true hypothesis; in DB-Synth, relevant columns and target column may also be provided [2407.01725]. This metadata schema is central to the benchmark’s design because many tasks depend on semantic grounding over dataset columns rather than direct statistical computation alone.

Difficulty in DB-Real is approximated by **workflow length**: each workflow step adds 1 to the discovery path length [2407.01725]. The paper also posits that difficulty rises with the number of trees in the semantic forest, especially when multiple datasets are involved. This suggests a complexity model combining procedural depth and breadth of the hypothesis search space.

## 3. Synthetic benchmark generation

**DB-Synth** contains **903 tasks** across **48 domains**, generated by LLMs to mimic real-world discovery workflows while enabling controlled complexity variation [2407.01725]. Its split is **60/20/20**, instantiated as **551 train**, **153 dev**, and **200 test** tasks. Each synthetic task is assigned a **difficulty level from 1–4** [2407.01725].

The DB-Synth generation pipeline has four stages [2407.01725]. First, the system samples domains and descriptions. Second, it constructs a semantic tree \(\mathcal{T}_h\) by prompting an LLM with the domain and a sampled workflow category to generate a primary hypothesis and target variable, then recursively generating child variables, column names, descriptions, datatypes, ranges, and a pandas expression encoding the structured hypothesis \(\psi(c,v,r)\). Nodes are expanded recursively until the desired tree height is reached; with probability **0.6**, a node is further derived rather than marked as a leaf [2407.01725]. Distractor columns outside the tree are also added, yielding a synthetic semantic forest \(\mathcal{F}\).

Third, data are generated bottom-up. Leaf columns are sampled according to datatype-specific strategies, internal columns are computed by executing the generated pandas expressions, and values are probabilistically noised or dropped to introduce missing data [2407.01725]. If expression execution fails, a self-refine procedure guided by execution logs repairs the pandas expressions. Fourth, for each internal node \(h\) in the forest, the benchmark derives multiple task datasets \(D_h^{(l)}\) by varying the path length \(l\) from observed variables to the target, thereby directly controlling task difficulty [2407.01725].

Natural-language goals are generated systematically from the structured hypothesis representation. For each target hypothesis \(h=\psi(c,v,r)\), one of the three dimensions—context, variable, or relationship—is masked, and the system formulates a question asking the model to recover the missing dimension from the data [2407.01725]. The benchmark states that each discovery goal has only one intended answer, namely the target hypothesis.

This synthetic component is not merely auxiliary scale. The primary paper argues that similar scores on DB-Real and DB-Synth support the claim that the synthetic benchmark captures meaningful aspects of real discovery complexity [2407.01725]. Later work used DiscoveryBench as a downstream target for training data-driven discovery models and reported gains on its HMS metric, indicating that DB-Synth’s workflow abstraction is compatible with transfer-oriented evaluation settings [2506.08140].

## 4. Evaluation methodology and Hypothesis Match Score

DiscoveryBench evaluates outputs with an LLM-based grader rather than exact-match scoring because hypotheses are open-ended natural-language objects [2407.01725]. The original evaluator is **gpt-4-preview-0125** [2407.01725]. The scoring process first decomposes both gold and predicted hypotheses into sub-hypotheses, then compares them along the context, variable, and relationship axes [2407.01725].

The core metric is the **Hypothesis Match Score (HMS)**. Let \(M\) be the set of matched predicted/gold sub-hypothesis pairs whose contexts are judged semantically equivalent. The benchmark computes:

- \(ctx\): an F1 score for predicted versus gold contexts
- \(var\): for each matched pair, an F1 score over the interacting variable sets
- \(rel\): for each matched pair, a relationship score with heuristic values **100** for exact match, **50** when the predicted relation is broader but encompasses the gold relation, and **0** otherwise

These are combined as
\[
HMS(h^p, h^g) = ctx(h^p, h^g) \times \frac{1}{|M|} \sum_{i=1}^{|M|} \Big( var(h^p_i, h^g_i) \times rel(h^p_i, h^g_i) \Big)
\]
[2407.01725].

The multiplicative structure makes context errors especially costly. If the predicted claim is framed under the wrong subgroup or scope restriction, strong variable or relationship alignment cannot fully compensate [2407.01725]. This is a deliberate design choice: the benchmark treats correct contextualization as constitutive of scientific discovery rather than peripheral formatting.

A “successful discovery” is therefore not merely a plausible hypothesis but one whose structured decomposition aligns with the gold on contexts, variables, and relationships and is supported by the intended workflow [2407.01725]. This scoring design later became the interface through which other systems reported progress on DiscoveryBench. For example, AutoSDT evaluates generated hypotheses against gold hypotheses using HMS and reports that fine-tuning on AutoSDT-5K raises a Qwen2.5-Coder-Instruct 32B model from **6.9** to **8.1** HMS under its reproduced DiscoveryBench setup [2506.08140]. Agentics 2.0 likewise uses DiscoveryBench’s HMS-based evaluation and reports an average final score of **37.27** for its best configuration on **DB-REAL** [2603.04241].

## 5. Baselines, empirical difficulty, and observed failure modes

DiscoveryBench evaluates several LLM-based “discovery agents” built on **GPT-4o**, **GPT-4-0125-preview (GPT-4p)**, and **Llama-3-70B** [2407.01725]. The benchmarked agent frameworks are:

- **CodeGen**: generates all code in one shot, executes it, then produces a natural-language hypothesis and workflow summary
- **ReAct**: alternates thought and code generation over multiple turns
- **DataVoyager**: a multi-component discovery agent with planner, code generator, data analysis, and critic modules
- **Reflexion (Oracle)**: CodeGen extended with access to the oracle HMS score after each attempt, enabling reflection over up to 3 trials
- **NoDataGuess**: a DB-Real-only baseline that guesses the hypothesis from metadata and goal without seeing the data [2407.01725]

The benchmark’s headline empirical result is that performance is low. On **DB-Real**, the best reported score is **24.5 HMS** from **Reflexion (Oracle) + GPT-4o**; on **DB-Synth**, the best score is **23.2** from **Reflexion (Oracle) + Llama-3** [2407.01725]. Among non-oracle systems, the best DB-Real score is **16.3** from **CodeGen + GPT-4p**, and the best DB-Synth non-oracle score is **14.1** from **CodeGen + GPT-4o** [2407.01725].

The full DB-Real results are:

| Framework | GPT-4o | GPT-4p | Llama-3 |
|---|---:|---:|---:|
| NoDataGuess | 0.0 | 4.7 | 11.5 |
| CodeGen | 15.5 | 16.3 | 12.1 |
| ReAct | 15.4 | 15.6 | 13.5 |
| DataVoyager | 15.4 | 13.9 | 11.5 |
| Reflexion (Oracle) | 24.5 | 19.5 | 22.5 |

The full DB-Synth results are:

| Framework | GPT-4o | GPT-4p | Llama-3 |
|---|---:|---:|---:|
| CodeGen | 14.1 | 8.7 | 10.9 |
| ReAct | 11.6 | 7.4 | 12.0 |
| DataVoyager | 5.7 | 6.9 | 11.7 |
| Reflexion (Oracle) | 15.7 | 12.9 | 23.2 |

These numbers support several conclusions already emphasized in the primary paper [2407.01725]. More elaborate agent structure does not automatically help: **ReAct** and **DataVoyager** generally do not surpass the simpler **CodeGen** baseline. Iterative improvement can help substantially, but only under unrealistically strong oracle feedback. Similar scores on DB-Real and DB-Synth suggest that the synthetic tasks preserve important sources of difficulty.

Failure analysis is equally central to DiscoveryBench’s contribution. The benchmark reports that only **44.2%** of points for the best DB-Real model had both context and variable–relationship scores non-zero, indicating frequent failure in at least one facet [2407.01725]. Workflow complexity is a major barrier: the best non-oracle model performs mainly on simple workflow categories. The top three workflow categories are **correlation analysis (55%)**, **data selection (18%)**, and **summary statistics (18%)**, while the worst are **spatial analysis (0%)**, **pollen dating (0%)**, and **ecological modeling (0%)** [2407.01725]. Domain difficulty is similarly uneven: **biology (0%)** and **engineering (7%)** are hardest, whereas **economics (25%)** and **sociology (23%)** are easiest [2407.01725].

Task complexity scaling is explicit. In DB-Real, performance drops as workflow length increases from small (\(<10\)) to medium (\(>10,<20\)) to large (\(>20\)); in DB-Synth, performance degrades with semantic tree height [2407.01725]. The benchmark therefore measures not only whether a model can produce a discovery-like answer, but how rapidly its capability deteriorates with procedural and semantic depth.

A later collaboration study reinforced this view of DiscoveryBench as a coordination-sensitive workload. Using the archaeology subset inside Collaborative Gym, adding collaborators in an unstructured shared workspace reduced mean performance from **0.71** for a single agent to **0.63** for a three-person default team, while scaffolding with shared group memory and HITL gates raised the three-person team to **0.76** [2606.18413]. This suggests that the benchmark’s tasks stress not only analytical competence but also evidence integration and responsibility routing.

## 6. Position in the benchmark ecosystem and subsequent use

DiscoveryBench occupies a specific niche in the scientific-discovery benchmark landscape. It is narrower than end-to-end closed-loop experimentation environments, but broader than fixed statistical QA or coding benchmarks. Its focus is the post-data-collection phase of discovery: deriving and verifying hypotheses from datasets, metadata, and an open-ended natural-language goal [2407.01725].

Later work positioned DiscoveryBench as a benchmark for **hypothesis generation** in data-driven science and used it to evaluate whether scientific coding corpora improve open models’ ability to generate hypotheses after analyzing data [2506.08140]. That work describes DiscoveryBench as involving a pipeline in which, given an instruction, a description of input data schema, and a scientific query, a method is supposed to generate Python code to analyze the data and then generate scientific hypotheses [2506.08140]. Under a reproduced setup, **GPT-4o (2024-05-13)** reached **10.4 HMS**, while **AutoSDT-Coder-32B** reached **8.1**, improving over its base model’s **6.9** [2506.08140].

Agentics 2.0 used DiscoveryBench to demonstrate typed, evidence-aware agentic data workflows. In that setting, DiscoveryBench was instantiated as a two-stage typed Map-Reduce workflow in which evidence is first extracted from one or more CSV-backed sources into `IntermediateEvidence` objects and then reduced into a final `Answer` hypothesis [2603.04241]. The paper reports that its **agentics-both** configuration achieved average final score **37.27**, exceeding a previously reported score of **33.7** in the ASTABench context [2603.04241]. This suggests that DiscoveryBench is also useful as an evaluation bed for workflow architectures, not only for raw language-model prompting.

Conceptually, other benchmark papers have used DiscoveryBench as a reference point when contrasting alternative notions of scientific discovery. *ResearchBench* describes DiscoveryBench as collecting specific discovery-relevant tasks from papers, whereas it positions itself around a decomposition of hypothesis formation into inspiration retrieval, hypothesis composition, and hypothesis ranking [2503.21248]. This external characterization underscores an important distinction: DiscoveryBench does not attempt to model all of scientific discovery. It models a particular, rigorous subset—**data-driven hypothesis search and verification from datasets alone**.

The benchmark’s main limitations are also explicit in the original paper. It excludes many domains and task types that were too difficult to replicate, especially **forecasting, simulation, or highly specialized modeling**, such as species distribution, infection spread, and astrophysics equations for exoplanets [2407.01725]. It does not address very large datasets such as those from large biomedical repositories, does not cover multimodal data, and does not explicitly solve the problem of robustness against **\(p\)-hacking** [2407.01725]. Evaluation is mediated by an LLM grader rather than formal verification or human adjudication. These limitations mark DiscoveryBench as a benchmark for a specific class of discovery workflows rather than a universal benchmark of autonomous science.

Taken as a whole, DiscoveryBench established a benchmarkable formalism for data-driven discovery—hypotheses as structured objects over contexts, variables, and relationships; tasks derived from real and synthetic workflows; and facet-based evaluation through HMS—while simultaneously demonstrating that current systems remain far from reliable autonomous discovery [2407.01725]. Subsequent work has treated it as a reference benchmark for scientific coding agents, typed workflow systems, and collaborative reasoning environments, which suggests that its lasting importance lies as much in its formal task definition as in its baseline scores [2506.08140].

Source: https://www.emergentmind.com/topics/discoverybench