Discovery under Hypothesis Redundancy: A Geometric Theory of Discovery Bottlenecks

Abstract: Scientific discovery saturates when new hypotheses cease to provide independent information, even if the nominal hypothesis space remains large. We study hybrid discovery systems that combine structured local search with LLM-generated non-local proposals and pose the Search Compression Hypothesis: non-local exploration helps only when three geometric conditions co-occur: spectral compression, orthogonal escape from the explored span, and residual signal alignment with the target. We formalize these conditions, derive necessary conditions for hybrid advantage, and test the mechanism in controlled synthetic environments, large-scale A-share factor discovery, and symbolic-regression benchmarks; a public tabular operational sanity check tests the associated budget-allocation implication. Signal-planting and directed-versus-random experiments show that novelty alone is insufficient: random orthogonal jumps expand coverage but do not improve yield without predictive alignment. Across compression sweeps, real factor archives, and LLM-SRBench tasks, hybrid gains concentrate in weakly represented but target-bearing directions and vanish as the hypothesis space approaches full rank. The framework turns LLM-guided discovery from generic novelty search into a diagnostic procedure for deciding when directed non-local exploration is warranted.

• The paper presents a geometric diagnostic showing that LLM-guided non-local proposals improve discovery only when spectral compression, orthogonal escape, and residual signal alignment co-occur.
• It establishes necessary conditions and formal bounds that quantify hybrid advantage, linking the effective rank of the candidate correlation matrix to discovery yield.
• Empirical validations in synthetic, financial, and symbolic regression experiments highlight the practical impact of these diagnostics in ensuring efficient automated research.

Geometric Theory of Discovery Bottlenecks under Hypothesis Redundancy

Introduction and Framework

The paper "Discovery under Hypothesis Redundancy: A Geometric Theory of Discovery Bottlenecks" (2606.14386) presents a geometric diagnostic for the effectiveness and limitations of hybrid scientific discovery systems, specifically those combining structured (local) search with LLM-generated non-local proposals. The paper introduces and formalizes the Search Compression Hypothesis, asserting that the utility of non-local, e.g., LLM-guided, exploration vanishes unless three geometric conditions co-occur: spectral compression, orthogonal escape from the explored span, and predictive alignment of residual signal with the target.

This diagnostic is motivated by observations in automated data science, factor discovery, and symbolic regression that, as the discovery archive grows, new hypotheses rapidly become redundant—covering directions already spanned by previous candidates. The consequence is a bottleneck: despite nominally large hypothesis spaces, the effective dimension for discovery is drastically reduced, governed by the spectrum of the candidate correlation matrix.

Spectral Compression and Diminishing Returns

Efficient discovery depends on the existence of unexplored, signal-bearing directions in the hypothesis space. The effective rank $r_{\text{eff}}$ of the candidate correlation matrix quantifies the number of independent directions present. As $r_{\text{eff}}/N \rightarrow 0$ with increasing redundancy, local search yield scales linearly downward. The paper supplies formal bounds for this decline with a proposition showing that, under mild leverage and dependence assumptions, the expected number of non-redundant discoveries by local (structured) search is upper-bounded by $C \cdot q_\tau \cdot r_{\text{eff}} (1-\rho^2)$, where $q_\tau$ is the pass probability for the predictive metric and $\rho$ is the redundancy threshold.

Empirical illustrations demonstrate that, in domains like financial factor discovery, only a small fraction ($\approx$29%) of the 100-dimensional structured archive carries independent information after selection—providing a concrete measure of spectral compression.

Figure 1: Why directed exploration helps under compression. (a) Hybrid advantage vanishes as $r_{\text{eff}}/N$ approaches one. (b) The benefit maximizes when compression, escape, and RSA co-occur. Only predictive residual coverage, not coverage alone, yields discovery.

Geometric Decomposition of Hybrid Advantage

The core theoretical contribution—formalized as the Search Compression Hypothesis and Theorem 1—is that non-local (LLM-guided) discovery offers a hybrid advantage only if: (i) the archive is strongly compressed ($1 - r_{\text{eff}}/N \gg 0$), (ii) the non-local proposal escapes the span of the archive with large $d_\perp$, and (iii) the escaped direction is aligned with the target (non-trivial RSA). Formally, the necessary conditions for nontrivial hybrid gain are:

• Compression: $1-r_{\text{eff}}/N > 0$
• Escape: $r_{\text{eff}}/N \rightarrow 0$0
• Residual Signal Alignment: $r_{\text{eff}}/N \rightarrow 0$1

If any single factor is missing, the hybrid advantage $r_{\text{eff}}/N \rightarrow 0$2 collapses to zero. The empirical model supports a multiplicative structure:

$r_{\text{eff}}/N \rightarrow 0$3

This result provides a robust diagnostic rule: use LLM-guided exploration only under simultaneous, verified compression, orthogonal escape, and residual alignment.

Empirical Validation: Synthetic, Financial, and Symbolic Discovery

Synthetic and Financial Experiments

Experiments using synthetic data (with tunable correlation spectra and signal locations), large-scale A-share factor discovery (5,647 stocks), and controlled public tabular datasets validate the theoretical conditions.

Key empirical findings include:

• Local search yield and hybrid gain tightly track $r_{\text{eff}}/N \rightarrow 0$4 as predicted.
• Hybrid gain is maximized when the target signal is located in weakly represented (compressed) eigendirections. If the signal aligns with strong eigendirections, local search suffices and hybrid advantage can be negative.
• **Random non-local proposals with large $r_{\text{eff}}/N \rightarrow 0$5 but no alignment (RSA $r_{\text{eff}}/N \rightarrow 0$6) do not deliver a yield increase, differentiating the Search Compression Hypothesis from pure novelty search.

Directed vs. random seed experiments further prove that directional semantic guidance, not mere orthogonality, delivers practical discovery gain.

Symbolic Regression and Cross-Domain Stress Tests

The framework is further validated in symbolic regression (SR) tasks using the LLM-SRBench benchmark (158 equations). The hybrid method only provides an advantage on hard (compressed, adversarially transformed) equations, rescuing catastrophic failures of GP-based search in approximately half the cases where GP alone fails, but showing negligible gain on standard, "easy" synthetic equations.

Cross-domain tests in drug discovery, climate, genomics, NAS, and code generation confirm that the observed hybrid advantage tracks compression severity and vanishes in high-rank (uncompressed) settings.

Practical and Theoretical Implications

The paper establishes that the effectiveness of non-local (e.g., LLM-guided) exploration should be diagnosed rather than assumed. The geometric framework provides operational budget allocation and stopping rules for automated research agents: if any geometric condition fails, further LLM queries are wasteful.

Theoretically, the connection of exploratory discovery to the effective dimension and structure of the candidate correlation spectrum unifies diagnostic strategies across hypothesis generation paradigms, from GP and SR to LLM-based automated scientists. The findings challenge frameworks focused on pure novelty, quality-diversity, or unconstrained exploration, demonstrating that coverage only yields value in the presence of target-aligned residual signal.

Future Directions

While the framework provides strong necessary conditions for hybrid advantage and diagnostic practice, it remains a descriptive theory. Open theoretical questions include:

• Sufficient conditions for hybrid gain, beyond necessary interaction.
• Policy optimality and adaptive allocation strategies as opposed to diagnostic switching.
• Extensions to online, adversarial, and model-misspecified environments.

Conclusion

The geometric theory of discovery bottlenecks articulated in this paper provides a principled, empirically validated approach for diagnosing when hybrid (LLM-augmented) discovery transcends local search. Novelty alone is insufficient; only the intersection of compression, directed orthogonal escape, and predictive residual signal alignment constitutes the regime of useful non-local exploration. Practitioners deploying LLM-guided or hybrid research systems should use these geometric diagnostics to guide, allocate, and stop exploration, ensuring computational resources and scientific efforts target genuinely fruitful discovery prospects.