HypoBench: Evaluating LLM Hypothesis Generation

• HypoBench is a systematic, multi-domain benchmark that assesses LLM-driven hypothesis generation by simulating real-world and synthetic evaluation scenarios.
• It integrates controlled difficulty parameters—such as noise, distractors, and compositional depth—to test model robustness and generalizability using metrics like HDR and FDR.
• By offering 194 datasets across diverse domains and employing multi-aspect evaluation metrics, HypoBench advances comparative research in automated hypothesis generation.

HypoBench is a systematic, multi-domain benchmark explicitly designed to support principled evaluation of LLM-driven hypothesis generation (HG) in both real-world and synthetic settings. HypoBench addresses the lack of shared definitions, datasets, and evaluation methodology in the HG field, enabling rigorous comparative research on the ability of LLM-based systems to generate, validate, and generalize explanatory hypotheses across task domains and difficulty regimes (Liu et al., 15 Apr 2025).

1. Motivation and Guiding Principles

Hypothesis generation, particularly using LLMs, has garnered significant interest, but critical questions remain unresolved: the precise criteria that make a "good" hypothesis, and the foundation for systematic, reproducible evaluation of HG methods. Existing literature frequently conflates HG with related tasks (such as ideation), often measuring only novelty rather than explanatory adequacy. The absence of standardized datasets, difficulty calibration, and robust metrics substantially impedes progress and comparability in the field.

HypoBench is governed by four central design principles:

1. Realism: Tasks and associated ground-truth hypotheses are constructed to closely reflect real scientific and everyday explanation scenarios.
2. Skill Coverage: The benchmark probes inductive reasoning, abstraction/communication, and synthesis—core HG competencies.
3. Controlled Difficulty: Synthetic task families use orthogonally varied parameters—noise, feature count, compositional depth, distractors, and textual subtlety—to facilitate granular analysis of method robustness.
4. Multi-Aspect Evaluation: Beyond discovery rate, HypoBench measures practical utility (the degree to which HG can support downstream inference), out-of-domain generalizability, and preliminary notions of "interestingness" (novelty, plausibility, clarity).

2. Benchmark Composition

HypoBench encompasses 194 datasets spanning 12 domains, split into 7 real-world and 5 synthetic task families.

Real-World Tasks

The real-world component includes 7 binary or multi-class classification tasks:

Task Domain	IND Size	OOD Size
Deception detection	1,600	640
AI-generated content	800	800
Persuasive argument	750	500
Mental stress detection	1,000	500
News headline engagement	700	453
Retweet prediction	1,000	500
Paper citation prediction	1,182	1,104

Each is provided with brief literature context ($L_Q$) and distinct in-domain (IND) and out-of-domain (OOD) splits reflecting variable sources or time periods.

Synthetic Tasks

Five synthetic task families allow fine-grained control over hypothesis complexity and distractors:

• Presidential election: Tweet to voting preference; 178,750 examples, 78 variants.
• Personality prediction: Tweet to personality; 178,750, 76 variants.
• Marine ecosystem: Environmental features to sunlight hours; 500, 1 variant.
• College admission: Student profile to admit/reject; 7,800, 26 variants.
• Shoe sales: Customer description to shoe color; 3,300, 3 variants.

Difficulty parameters are systematically modulated:

• Label noise
• Number of true features
• Compositional depth (decision-tree depth)
• Distractors (irrelevant features)
• Textual subtlety (key features explicit or embedded in free text)

The combinatorics of these variations yield 194 total datasets. For instance, "College Admission" includes combinations of 4 levels across 4 controls and counterintuitive case counterparts.

3. Evaluation Protocol and Metrics

HypoBench formalizes evaluation across quantitative and qualitative axes:

3.1 Hypothesis Discovery Rate (HDR) [Synthetic Tasks]

Given true hypothesis set $Z$, generated set $\hat{Z}$, and discovered relationships $\hat{f}$: $\mathrm{HDR} = \mathrm{FDR} \times \mathrm{RC}$

• Feature Discovery Rate (FDR):

$\mathrm{FDR} = \frac{|\hat Z \cap Z|}{|Z|}$

• Relationship Correctness (RC):

$$\mathrm{RC} = \frac{1}{|\hat Z \cap Z|} \sum_{z_i \in \hat Z \cap Z} M_r(z_i, \hat f, f)$$

where $M_r \in [0,1]$ is an LLM-based rating over direction/nature of recovered relations.

3.2 Practical Utility

$\mathrm{Accuracy}(\hat f, \hat Z, \mathbf X) = \frac1{|\mathbf X|} \sum_{(x_i,y_i)\in\mathbf X} \mathbbm{1}(y_i = M_I(x_i;\hat f,\hat Z))$

where $M_I$ is an LLM instructed to use $(\hat Z, \hat f)$ for classification on held-out samples $\mathbf X$.

3.3 Generalizability

• IND→OOD: Evaluate predictive performance (accuracy, F1) on in-domain vs. out-of-domain splits.
• Cross-Model: Generate hypotheses with one LLM, test inference using a different LLM.

3.4 Preliminary “Interestingness”

Automated rating via GPT-4o ($M_q$) on a 1–5 scale across:

• Novelty
• Plausibility
• Clarity

4. Experimental Results

Assessments reported for four state-of-the-art LLMs (GPT, Qwen, Llama, DeepSeek) and six HG approaches.

Real-World Results

OOD accuracy and F1 (averaged over tasks):

Method	GPT (Acc/F1)	Qwen	Llama	DeepSeek
Few-shot inference	65.7 / 62.7	68.9 / 68.0	72.5 / 71.2	66.9 / 64.1
IO Prompting	66.1 / 65.1	74.5 / 73.9	68.2 / 66.3	61.6 / 59.8
HypoGeniC	71.2 / 70.3	77.8 / 77.8	72.3 / 70.9	70.0 / 68.7
Literature + Data	75.3 / 75.0	77.9/77.9	76.2/75.9	74.9/74.5

Combining literature context and empirical observations (Literature+Data) yields the strongest performance across all models. Notably, Qwen+L+D achieves OOD accuracy of 77.9%; an oracle finetuned Llama-8B reaches 77.3% OOD.

Cross-model transfer (e.g., hypotheses generated by Qwen, applied via Llama) results in only a ~3.4% absolute OOD accuracy drop, indicating generalizable reasoning.

Qualitative “interestingness” (mean scores, 1–5):

Method	Novelty	Plausibility	Clarity
Zero-shot generation	2.1–2.6	4.0	3.3
Literature-only	2.0–2.2	4.1–4.2	3.4
IO Prompting	2.5–2.6	3.8	3.4
Iterative Refinement	2.7–2.9	3.5–3.8	3.1–3.3
HypoGeniC	2.5–2.7	3.9	3.1–3.4
Literature+Data	2.4–2.7	4.1	3.5–3.6

Literature-only hypotheses are most plausible but least novel; Iterative Refinement maximizes novelty at some expense of clarity and plausibility. No evaluated approach achieves simultaneous maximization of all criteria.

Synthetic Results

• Base settings (single feature, no noise/distractors): DeepSeek HDR ≈ 93.8%; Llama 87.5%; Qwen 81.3%; GPT 75.0%.
• 10% label noise: HDR drops to ≈40% (DeepSeek), ≈36% (GPT).
• Six distractors: HDR ≈38% (DeepSeek), ≈42% (GPT).
• Compositional depth: All models fall below 50% at depth ≥3; best at depth 4 is DeepSeek ≈38.8%.
• Textual subtlety: Encoding features within free text reduces HDR by 20–30 points for all models.

Best-case HDR under full difficulty is only 38.8%, exposing critical limitations in current HG systems.

5. Analysis and Domain-Specific Insights

• LLM-based HG reliably recovers simple, linear hypotheses with minimal distractors or confounders (HDR >90%).
• Performance declines sharply when confronted with moderate noise, higher feature interaction depth, or irrelevant features—characteristics common in complex real-world reasoning scenarios.
• Task domain matters: Intuitive/everyday settings such as College Admission and Shoe Sales elicit >50% HDR even under zero-shot prompting; political domains yield <20% HDR unless prompted carefully.
• Most methods exhibit a trade-off: routines tuned for plausibility (e.g., literature-only) rarely achieve high novelty, while novelty-seeking variants compromise interpretability and plausibility.

Unresolved challenges include robust abstraction from noisy text, discovery of higher-order (depth ≥3) interactions, efficient use of external knowledge (with literature context sometimes yielding insufficient marginal gain), and scalable, objective metrics for interestingness.

6. Practical Usage and Methodological Recommendations

For the benchmarking of new hypothesis-generation methods, HypoBench provides the following guidelines:

• Employ real-world task splits to evaluate practical utility and generalizability (IND versus OOD).
• Utilize synthetic families to systematically stress-test robustness, modifying one difficulty axis per experiment.
• Report HDR (with FDR and RC), accuracy/F1, and “interestingness” metrics for comprehensive evaluation.
• Include cross-model inference to demonstrate portability of generated hypotheses.

Suggested future extensions include expanding to additional domains (physical sciences, clinical trials, economics), incorporating richer background knowledge graphs, devising refined “interestingness” metrics, and introducing multi-step experimental design and causal discovery tasks.

7. Impact and Future Directions

HypoBench establishes the first multi-aspect, difficulty-controlled evaluation standard for hypothesis generation, revealing both the potential and critical current bottlenecks in LLM-driven HG. It enables quantifiable progress tracking, identification of failure regimes (e.g., composition, noise), and comparative analysis of methods’ strengths and weaknesses. Future research is directed at extending coverage to broader scientific domains, integrating richer knowledge-layer synthesis, and improving metrics for discovery and creativity in automated scientific reasoning (Liu et al., 15 Apr 2025).