DiscoveryBench: Data-Driven Discovery Benchmark

• DiscoveryBench is a benchmark that formalizes data-driven discovery by defining hypotheses as declarative statements composed of contexts, variables, and relationships.
• It comprises real (DB-Real) and synthetic (DB-Synth) tasks, enabling evaluation of multi-step analytical workflows across diverse scientific domains.
• The evaluation employs a Hypothesis Match Score (HMS) to assess precise alignment in context, variables, and relationships, highlighting model performance gaps.

DiscoveryBench is a benchmark for evaluating whether LLMs can perform data-driven discovery from provided datasets rather than merely answer predefined statistical questions or optimize predictive pipelines. It formalizes discovery as a multi-step process in which a system must interpret a natural-language goal, map that goal to dataset semantics, determine an appropriate analysis workflow, derive a hypothesis, and verify that the data support it. The benchmark contains a manually curated real component, DB-Real, and a synthetic component, DB-Synth, and introduces a structured representation of hypotheses in terms of contexts, variables, and relationships, together with a facet-based evaluation framework centered on the Hypothesis Match Score (HMS) (Majumder et al., 2024).

1. Conceptualization of data-driven discovery

DiscoveryBench defines a data-driven hypothesis $h$ as a declarative sentence whose truth can be inferred from a dataset $D$ using a verification procedure

$$\mathcal{V}_D:\mathcal{H} \to \{\mathrm{supported}, \mathrm{unsupported}\},$$

where $\mathcal{H}$ is the space of hypotheses (Majumder et al., 2024). A hypothesis may itself be composite: $h := \phi(h_1, \ldots, h_n),$ with verification distributed over the constituent sub-hypotheses: $$\mathcal{V}_D(h)=\phi(\mathcal{V}_D(h_1),\ldots,\mathcal{V}_D(h_n)).$$ This formulation treats discovery as a structured inferential object rather than a free-form answer string (Majumder et al., 2024).

To make evaluation operational, DiscoveryBench rewrites hypotheses as

$h:= \psi(c, v, r),$

where $c$ denotes contexts, $v$ denotes variables, and $r$ denotes relationships (Majumder et al., 2024). Contexts are boundary conditions or scope restrictions; variables are the interacting concepts, including a target variable and one or more independent variables; relationships are the form of interaction among variables under the specified context. In the benchmark’s wording, $D$0 yields the declarative sentence that “under context $D$1, variables $D$2 have relationship $D$3” (Majumder et al., 2024).

This decomposition is extended through a hypothesis semantic tree. Internal nodes may represent intermediate derived variables, leaves correspond to non-derived independent variables, and the root corresponds to the target variable of the main hypothesis. A dataset may support multiple such trees, forming a semantic forest

$D$4

(Majumder et al., 2024). This suggests that DiscoveryBench is designed to evaluate not only terminal answers but also the latent structural depth of a discovery process.

Within this formalism, a task asks the system, given one or more datasets $D$5 and a natural-language discovery goal $D$6, to “derive a hypothesis $D$7 addressing $D$8 with the highest specificity for the context $D$9, variables $$\mathcal{V}_D:\mathcal{H} \to \{\mathrm{supported}, \mathrm{unsupported}\},$$0, and relationship $$\mathcal{V}_D:\mathcal{H} \to \{\mathrm{supported}, \mathrm{unsupported}\},$$1 supported by $$\mathcal{V}_D:\mathcal{H} \to \{\mathrm{supported}, \mathrm{unsupported}\},$$2” (Majumder et al., 2024). The benchmark is outcome-based: multiple workflows may be valid if they converge to a hypothesis aligned with the gold structure.

2. Benchmark composition and curation

DiscoveryBench consists of two components: DB-Real and DB-Synth (Majumder et al., 2024).

DB-Real contains 264 tasks manually derived from published scientific work across 6 domains: sociology, biology, humanities, economics, engineering, and meta-science (Majumder et al., 2024). The tasks were constructed from published workflows using two curation routes. In the data-first route, open public datasets such as the National Longitudinal Surveys, Global Biodiversity Information Facility, and World Bank Open Data were paired with papers whose analyses could be reimplemented. Replication was labor-intensive: it took 15–40 person-hours for each NLS-related paper and up to 90 person-hours for GBIF datasets, with about 30% of data-first attempts failing (Majumder et al., 2024). In the code-first route, scientific code repositories with available datasets were sought; among 785 Zenodo entries filtered for computational notebooks, over 85% were unusable because of missing code, non-Python code, or inaccessible data, yielding 14 candidate repositories and only 3 final inclusions (Majumder et al., 2024).

The split statistics for DB-Real are explicit. There are 25 train tasks and 239 test tasks, for a total of 264; the train split contains 14 unique hypotheses and the test split 144 unique hypotheses (Majumder et al., 2024). Multi-dataset reasoning is frequent: 4 train tasks and 110 test tasks require more than one dataset, for 114 multi-dataset tasks in total, with up to 6 datasets in a single task (Majumder et al., 2024). The train split covers 3 domains, whereas the test split spans all 6 (Majumder et al., 2024).

Each benchmark instance includes rich metadata. At the task level, fields can include an id, domain, workflow tags, optional domain knowledge, dataset descriptions, column descriptions, hypotheses, workflows, and one or more natural-language queries. At the query level, fields can include qid, difficulty, and the true hypothesis; in DB-Synth, relevant columns and target column may also be provided (Majumder et al., 2024). This metadata schema is central to the benchmark’s design because many tasks depend on semantic grounding over dataset columns rather than direct statistical computation alone.

Difficulty in DB-Real is approximated by workflow length: each workflow step adds 1 to the discovery path length (Majumder et al., 2024). The paper also posits that difficulty rises with the number of trees in the semantic forest, especially when multiple datasets are involved. This suggests a complexity model combining procedural depth and breadth of the hypothesis search space.

3. Synthetic benchmark generation

DB-Synth contains 903 tasks across 48 domains, generated by LLMs to mimic real-world discovery workflows while enabling controlled complexity variation (Majumder et al., 2024). Its split is 60/20/20, instantiated as 551 train, 153 dev, and 200 test tasks. Each synthetic task is assigned a difficulty level from 1–4 (Majumder et al., 2024).

The DB-Synth generation pipeline has four stages (Majumder et al., 2024). First, the system samples domains and descriptions. Second, it constructs a semantic tree $$\mathcal{V}_D:\mathcal{H} \to \{\mathrm{supported}, \mathrm{unsupported}\},$$3 by prompting an LLM with the domain and a sampled workflow category to generate a primary hypothesis and target variable, then recursively generating child variables, column names, descriptions, datatypes, ranges, and a pandas expression encoding the structured hypothesis $$\mathcal{V}_D:\mathcal{H} \to \{\mathrm{supported}, \mathrm{unsupported}\},$$4. Nodes are expanded recursively until the desired tree height is reached; with probability 0.6, a node is further derived rather than marked as a leaf (Majumder et al., 2024). Distractor columns outside the tree are also added, yielding a synthetic semantic forest $$\mathcal{V}_D:\mathcal{H} \to \{\mathrm{supported}, \mathrm{unsupported}\},$$5.

Third, data are generated bottom-up. Leaf columns are sampled according to datatype-specific strategies, internal columns are computed by executing the generated pandas expressions, and values are probabilistically noised or dropped to introduce missing data (Majumder et al., 2024). If expression execution fails, a self-refine procedure guided by execution logs repairs the pandas expressions. Fourth, for each internal node $$\mathcal{V}_D:\mathcal{H} \to \{\mathrm{supported}, \mathrm{unsupported}\},$$6 in the forest, the benchmark derives multiple task datasets $$\mathcal{V}_D:\mathcal{H} \to \{\mathrm{supported}, \mathrm{unsupported}\},$$7 by varying the path length $$\mathcal{V}_D:\mathcal{H} \to \{\mathrm{supported}, \mathrm{unsupported}\},$$8 from observed variables to the target, thereby directly controlling task difficulty (Majumder et al., 2024).

Natural-language goals are generated systematically from the structured hypothesis representation. For each target hypothesis $$\mathcal{V}_D:\mathcal{H} \to \{\mathrm{supported}, \mathrm{unsupported}\},$$9, one of the three dimensions—context, variable, or relationship—is masked, and the system formulates a question asking the model to recover the missing dimension from the data (Majumder et al., 2024). The benchmark states that each discovery goal has only one intended answer, namely the target hypothesis.

This synthetic component is not merely auxiliary scale. The primary paper argues that similar scores on DB-Real and DB-Synth support the claim that the synthetic benchmark captures meaningful aspects of real discovery complexity (Majumder et al., 2024). Later work used DiscoveryBench as a downstream target for training data-driven discovery models and reported gains on its HMS metric, indicating that DB-Synth’s workflow abstraction is compatible with transfer-oriented evaluation settings (Li et al., 9 Jun 2025).

4. Evaluation methodology and Hypothesis Match Score

DiscoveryBench evaluates outputs with an LLM-based grader rather than exact-match scoring because hypotheses are open-ended natural-language objects (Majumder et al., 2024). The original evaluator is gpt-4-preview-0125 (Majumder et al., 2024). The scoring process first decomposes both gold and predicted hypotheses into sub-hypotheses, then compares them along the context, variable, and relationship axes (Majumder et al., 2024).

The core metric is the Hypothesis Match Score (HMS). Let $\mathcal{H}$0 be the set of matched predicted/gold sub-hypothesis pairs whose contexts are judged semantically equivalent. The benchmark computes:

• $\mathcal{H}$1: an F1 score for predicted versus gold contexts
• $\mathcal{H}$2: for each matched pair, an F1 score over the interacting variable sets
• $\mathcal{H}$3: for each matched pair, a relationship score with heuristic values 100 for exact match, 50 when the predicted relation is broader but encompasses the gold relation, and 0 otherwise

These are combined as

$\mathcal{H}$4

(Majumder et al., 2024).

The multiplicative structure makes context errors especially costly. If the predicted claim is framed under the wrong subgroup or scope restriction, strong variable or relationship alignment cannot fully compensate (Majumder et al., 2024). This is a deliberate design choice: the benchmark treats correct contextualization as constitutive of scientific discovery rather than peripheral formatting.

A “successful discovery” is therefore not merely a plausible hypothesis but one whose structured decomposition aligns with the gold on contexts, variables, and relationships and is supported by the intended workflow (Majumder et al., 2024). This scoring design later became the interface through which other systems reported progress on DiscoveryBench. For example, AutoSDT evaluates generated hypotheses against gold hypotheses using HMS and reports that fine-tuning on AutoSDT-5K raises a Qwen2.5-Coder-Instruct 32B model from 6.9 to 8.1 HMS under its reproduced DiscoveryBench setup (Li et al., 9 Jun 2025). Agentics 2.0 likewise uses DiscoveryBench’s HMS-based evaluation and reports an average final score of 37.27 for its best configuration on DB-REAL (Gliozzo et al., 4 Mar 2026).

5. Baselines, empirical difficulty, and observed failure modes

DiscoveryBench evaluates several LLM-based “discovery agents” built on GPT-4o, GPT-4-0125-preview (GPT-4p), and Llama-3-70B (Majumder et al., 2024). The benchmarked agent frameworks are:

• CodeGen: generates all code in one shot, executes it, then produces a natural-language hypothesis and workflow summary
• ReAct: alternates thought and code generation over multiple turns
• DataVoyager: a multi-component discovery agent with planner, code generator, data analysis, and critic modules
• Reflexion (Oracle): CodeGen extended with access to the oracle HMS score after each attempt, enabling reflection over up to 3 trials
• NoDataGuess: a DB-Real-only baseline that guesses the hypothesis from metadata and goal without seeing the data (Majumder et al., 2024)

The benchmark’s headline empirical result is that performance is low. On DB-Real, the best reported score is 24.5 HMS from Reflexion (Oracle) + GPT-4o; on DB-Synth, the best score is 23.2 from Reflexion (Oracle) + Llama-3 (Majumder et al., 2024). Among non-oracle systems, the best DB-Real score is 16.3 from CodeGen + GPT-4p, and the best DB-Synth non-oracle score is 14.1 from CodeGen + GPT-4o (Majumder et al., 2024).

The full DB-Real results are:

Framework	GPT-4o	GPT-4p	Llama-3
NoDataGuess	0.0	4.7	11.5
CodeGen	15.5	16.3	12.1
ReAct	15.4	15.6	13.5
DataVoyager	15.4	13.9	11.5
Reflexion (Oracle)	24.5	19.5	22.5

The full DB-Synth results are:

Framework	GPT-4o	GPT-4p	Llama-3
CodeGen	14.1	8.7	10.9
ReAct	11.6	7.4	12.0
DataVoyager	5.7	6.9	11.7
Reflexion (Oracle)	15.7	12.9	23.2

These numbers support several conclusions already emphasized in the primary paper (Majumder et al., 2024). More elaborate agent structure does not automatically help: ReAct and DataVoyager generally do not surpass the simpler CodeGen baseline. Iterative improvement can help substantially, but only under unrealistically strong oracle feedback. Similar scores on DB-Real and DB-Synth suggest that the synthetic tasks preserve important sources of difficulty.

Failure analysis is equally central to DiscoveryBench’s contribution. The benchmark reports that only 44.2% of points for the best DB-Real model had both context and variable–relationship scores non-zero, indicating frequent failure in at least one facet (Majumder et al., 2024). Workflow complexity is a major barrier: the best non-oracle model performs mainly on simple workflow categories. The top three workflow categories are correlation analysis (55%), data selection (18%), and summary statistics (18%), while the worst are spatial analysis (0%), pollen dating (0%), and ecological modeling (0%) (Majumder et al., 2024). Domain difficulty is similarly uneven: biology (0%) and engineering (7%) are hardest, whereas economics (25%) and sociology (23%) are easiest (Majumder et al., 2024).

Task complexity scaling is explicit. In DB-Real, performance drops as workflow length increases from small ($\mathcal{H}$5) to medium ($\mathcal{H}$6) to large ($\mathcal{H}$7); in DB-Synth, performance degrades with semantic tree height (Majumder et al., 2024). The benchmark therefore measures not only whether a model can produce a discovery-like answer, but how rapidly its capability deteriorates with procedural and semantic depth.

A later collaboration study reinforced this view of DiscoveryBench as a coordination-sensitive workload. Using the archaeology subset inside Collaborative Gym, adding collaborators in an unstructured shared workspace reduced mean performance from 0.71 for a single agent to 0.63 for a three-person default team, while scaffolding with shared group memory and HITL gates raised the three-person team to 0.76 (Kotalwar et al., 16 Jun 2026). This suggests that the benchmark’s tasks stress not only analytical competence but also evidence integration and responsibility routing.

6. Position in the benchmark ecosystem and subsequent use

DiscoveryBench occupies a specific niche in the scientific-discovery benchmark landscape. It is narrower than end-to-end closed-loop experimentation environments, but broader than fixed statistical QA or coding benchmarks. Its focus is the post-data-collection phase of discovery: deriving and verifying hypotheses from datasets, metadata, and an open-ended natural-language goal (Majumder et al., 2024).

Later work positioned DiscoveryBench as a benchmark for hypothesis generation in data-driven science and used it to evaluate whether scientific coding corpora improve open models’ ability to generate hypotheses after analyzing data (Li et al., 9 Jun 2025). That work describes DiscoveryBench as involving a pipeline in which, given an instruction, a description of input data schema, and a scientific query, a method is supposed to generate Python code to analyze the data and then generate scientific hypotheses (Li et al., 9 Jun 2025). Under a reproduced setup, GPT-4o (2024-05-13) reached 10.4 HMS, while AutoSDT-Coder-32B reached 8.1, improving over its base model’s 6.9 (Li et al., 9 Jun 2025).

Agentics 2.0 used DiscoveryBench to demonstrate typed, evidence-aware agentic data workflows. In that setting, DiscoveryBench was instantiated as a two-stage typed Map-Reduce workflow in which evidence is first extracted from one or more CSV-backed sources into IntermediateEvidence objects and then reduced into a final Answer hypothesis (Gliozzo et al., 4 Mar 2026). The paper reports that its agentics-both configuration achieved average final score 37.27, exceeding a previously reported score of 33.7 in the ASTABench context (Gliozzo et al., 4 Mar 2026). This suggests that DiscoveryBench is also useful as an evaluation bed for workflow architectures, not only for raw language-model prompting.

Conceptually, other benchmark papers have used DiscoveryBench as a reference point when contrasting alternative notions of scientific discovery. ResearchBench describes DiscoveryBench as collecting specific discovery-relevant tasks from papers, whereas it positions itself around a decomposition of hypothesis formation into inspiration retrieval, hypothesis composition, and hypothesis ranking (Liu et al., 27 Mar 2025). This external characterization underscores an important distinction: DiscoveryBench does not attempt to model all of scientific discovery. It models a particular, rigorous subset—data-driven hypothesis search and verification from datasets alone.

The benchmark’s main limitations are also explicit in the original paper. It excludes many domains and task types that were too difficult to replicate, especially forecasting, simulation, or highly specialized modeling, such as species distribution, infection spread, and astrophysics equations for exoplanets (Majumder et al., 2024). It does not address very large datasets such as those from large biomedical repositories, does not cover multimodal data, and does not explicitly solve the problem of robustness against $\mathcal{H}$8-hacking (Majumder et al., 2024). Evaluation is mediated by an LLM grader rather than formal verification or human adjudication. These limitations mark DiscoveryBench as a benchmark for a specific class of discovery workflows rather than a universal benchmark of autonomous science.

Taken as a whole, DiscoveryBench established a benchmarkable formalism for data-driven discovery—hypotheses as structured objects over contexts, variables, and relationships; tasks derived from real and synthetic workflows; and facet-based evaluation through HMS—while simultaneously demonstrating that current systems remain far from reliable autonomous discovery (Majumder et al., 2024). Subsequent work has treated it as a reference benchmark for scientific coding agents, typed workflow systems, and collaborative reasoning environments, which suggests that its lasting importance lies as much in its formal task definition as in its baseline scores (Li et al., 9 Jun 2025).