HypoSpace: LLM Hypothesis Evaluation

• HypoSpace is a diagnostic framework that evaluates LLMs' ability to generate diverse hypotheses in scenarios with multiple, consistent explanations.
• It introduces three metrics—Validity Rate, Uniqueness Rate, and Recovery Rate—to measure correctness, originality, and coverage of generated hypothesis sets.
• By leveraging controlled domains with exact enumeration, HypoSpace uncovers mode collapse issues and guides improvements in LLM sampling strategies.

HypoSpace is a diagnostic framework for evaluating the set-valued generative abilities of LLMs when faced with scientific underdetermination—cases where multiple, mechanistically distinct hypotheses are consistent with the same data. Unlike standard evaluation protocols that emphasize single-answer correctness, HypoSpace treats LLMs as samplers over finite hypothesis spaces and quantifies not only appropriateness (correctness) but also diversity and coverage of their hypothesis sets. Its methodology leverages domains in which the set of all admissible hypotheses, $\mathcal H_O$, can be exactly enumerated and deterministically validated, thereby supporting precise, model-agnostic interrogation of LLM performance as hypothesis generators (Chen et al., 17 Oct 2025).

1. Scientific Underpinnings: The Role of Underdetermination

In numerous scientific contexts—including causal discovery, 3D reconstruction, and genetic interaction analysis—a fixed observation set admits a multitude of plausible, mechanism-distinct explanations. Traditional LLM benchmarks, which enforce a single “correct” output criterion, systematically overlook the critical capacity to generate or explore the entire admissible hypothesis set. HypoSpace addresses this gap by defining the LLM’s role as sampling from the space of hypotheses consistent with given observations, allowing for direct evaluation of both creativity and systematicity in hypothesis generation. By restricting attention to domains with exhaustively enumerable and exactly validated hypothesis sets, HypoSpace decouples evaluation from annotator subjectivity and incomplete answer lists.

2. Core Metrics: Validity, Uniqueness, and Recovery

HypoSpace introduces three rigorous, complementary metrics—Validity Rate (VR), Uniqueness Rate (NR), and Recovery Rate (RR)—to quantify the generative behavior of LLMs under underdetermination.

• Validity Rate (VR): Measures appropriateness as the fraction of sampled hypotheses consistent with the observations,

$$\mathrm{VR}(P) = \frac{1}{N} \left| \{ \tilde h \in P : \mathrm{val}_O(\tilde h) = 1 \} \right|,$$

where $\mathrm{val}_O(\tilde h)$ is a deterministic validator for consistency.

• Uniqueness Rate (NR): Captures originality by measuring the non-redundancy within the sampled hypotheses,

$$\mathrm{NR}(P) = \frac{1}{N} \left| \{ \tilde h \in P : \mathrm{nov}(\tilde h;A_{\tilde h}) = 1 \} \right|,$$

with $\mathrm{nov}(\tilde h;A_{\tilde h})$ indicating whether $\tilde h$ is distinct under a canonicalizer.

• Recovery Rate (RR): Measures coverage by quantifying how much of the true admissible set has been recovered,

$$\mathrm{RR}(P) = \frac{1}{| \mathcal H_O |} \left| \{ \tilde h \in P : \mathrm{val}_O(\tilde h) = 1 \wedge \mathrm{nov}(\tilde h;A_{\tilde h}) = 1 \} \right|.$$

These metrics jointly specify whether an LLM’s sampled set is (i) correct, (ii) non-redundant, and (iii) comprehensive with respect to the full admissible space.

3. Structured Domains, Validation, and Exact Enumeration

HypoSpace is instantiated on tasks that support exhaustive enumeration and deterministic validation of admissible hypothesis sets. The controlled domains are:

Domain	Hypothesis Space $(\mathcal H)$	Validation & Distinctness Criterion
Causal Graph Discovery	All DAGs on $n$ labeled nodes	Forward model for intervention; edge set identity
Gravity-Constrained 3D Voxel Stack	Bit tensors in $\{0,1\}^{K \times M \times M}$	Top-down projection, gravity constraint; voxel-wise equality
Boolean Genetic Interactions	Boolean expression trees (depth $$\mathrm{VR}(P) = \frac{1}{N} \left| \{ \tilde h \in P : \mathrm{val}_O(\tilde h) = 1 \} \right|,$$0)	Input-output consistency on observed pairs; canonicalization of expressions

• Causal Graph Discovery: Given single-node interventions and binary effects, the admissible set comprises those DAGs for which simulated interventions match the observation set. Canonicalization is exact via labeled edge sets.
• 3D Voxel Reconstruction: The admissible set consists of all gravity-constrained 3D stacks consistent with a given top-down projection. Validation includes both projection and stacking constraints.
• Boolean Genetics: The hypothesis space is all depth-bounded Boolean expression trees constructed from a given operator set, determined valid by matching all input/output pairs in the observed phenotype table. Canonicalization collapses local symmetries and algebraic equivalences.

This rigorous control allows for exact calculation of $$\mathrm{VR}(P) = \frac{1}{N} \left| \{ \tilde h \in P : \mathrm{val}_O(\tilde h) = 1 \} \right|,$$1 and thus precise ground truth for all three metrics.

4. Empirical Assessment Across LLMs

Seven LLMs were evaluated—five reasoning-focused (GPT-5, Gemini-2.5-Pro, Claude-Opus-4, DeepSeek-R1, Grok-4) and two non-reasoning baselines (GPT-4o, LLaMA-3.3-70B-Instruct)—on simple/medium/hard regimes in all three domains. Key findings include:

• Reasoning models sustained near-perfect VR (often ≈100%) across domains and difficulty, indicating robust consistency with observations.
• NR and RR decline sharply as $$\mathrm{VR}(P) = \frac{1}{N} \left| \{ \tilde h \in P : \mathrm{val}_O(\tilde h) = 1 \} \right|,$$2 increases, evidencing mode collapse: the models restrict their exploration to small subregions of the admissible hypothesis space.
• Non-reasoning baselines showed markedly inferior recovery, e.g., in hard causal graph settings ($$\mathrm{VR}(P) = \frac{1}{N} \left| \{ \tilde h \in P : \mathrm{val}_O(\tilde h) = 1 \} \right|,$$3), RR for non-reasoners was 2–7% versus ≈99.2% for GPT-5.
• Information-theoretic measures (entropy and per-sample information gain) confirmed that reasoning models maintain higher entropy across samples, reflecting more efficient exploration of admissible hypotheses, whereas baselines plateau early in both genetic and geometric settings.

Example outcome data (hard regime):

Task/Model	VR	NR	RR
Causal/GPT-5	100%	99.2%	99.2%
Causal/Non-reason	10–73%	—	2–7%
Voxel/GPT-5	100%	98.8%	98.8%
Voxel/Non-reason	10–19%	—	6–14%
Genetics/GPT-5	65%	49.9%	48.0%
Genetics/Non-reason	55–68%	—	11–29%

5. Theoretical Significance and Evaluation Paradigm

HypoSpace functions as a controlled probe, not a competitive leaderboard. Its principal strengths are:

• Elimination of subjective evaluation by relying on exact enumeration and validation in structured tasks.
• Decoupling of correctness, diversity, and coverage: exposing failure modes such as mode collapse that are not detectable by correctness-only metrics.
• Support for mechanistic studies: Enabling investigation of how decoder designs, sampling strategies (e.g., entropy-seeking), or architectural choices affect exploration of hypothesis spaces.

This suggests that set-valued evaluation, as exemplified by HypoSpace, is essential for capturing the creative and exploratory dimensions of LLM reasoning in underdetermined settings—a key desideratum for scientific workflows where multiple plausible explanations coexist.

6. Broader Implications and Future Directions

HypoSpace advances the evaluation of generative models by operationalizing the appropriateness/originality/fluency triad from creativity theory within scientific inference tasks. A plausible implication is that further refinement and expansion of HypoSpace-style benchmarks could inform the development of models and decoding algorithms better aligned with scientific discovery, by promoting coverage and diversity of hypotheses rather than mere precision. Systematic studies leveraging HypoSpace may underscore the importance of entropy-driven sampling, memory across draws, or training for proposal distribution learning in future LLM training regimes (Chen et al., 17 Oct 2025).