Position: Prioritize Identifying Structure, Not Complex Models, for Scientific Discovery

Abstract: Modern Machine Learning (ML) and AI models, especially LLMs, are increasingly used to generate scientific hypotheses and mechanistic explanations from observational data. This position paper argues that in the high-dimensional proxy regimes where modern ML excels, mechanistic learning is generically underdetermined: many incompatible mechanisms induce essentially the same observational relationships on the support of the data, so predictive success and coherent explanations are insufficient evidence of mechanism discovery. This underdetermination becomes uniquely hazardous with LLMs, which tend to collapse large equivalence classes of explanations into a single fluent narrative. This paper proposes concrete standards for ``mechanistic ML,'' and argues these norms are necessary if LLM-centered workflows are to support science rather than merely simulate it.

• The paper’s main finding is that imposing identifying structure, rather than model complexity, is essential for valid mechanistic discovery in proxy-rich settings.
• It shows that high-dimensional proxy measurements can yield multiple, equally plausible mechanisms, highlighting the underdetermination issue even with perfect in-domain performance.
• The work introduces the concept of 'narrative collapse' in LLMs and recommends using mechanism cards and multiplicity reporting to counter oversimplified explanations.

Prioritizing Identifying Structure Over Model Complexity in Scientific Discovery

Introduction

This paper, "Position: Prioritize Identifying Structure, Not Complex Models, for Scientific Discovery" (2606.02632), presents a systematic argument that in high-dimensional, proxy-rich scientific domains, mechanistic learning using modern ML and AI models—especially LLMs—is generically underdetermined unless explicit identifying structure is imposed. The author claims that predictive performance and post-hoc interpretability are insufficient to warrant mechanistic claims, as many incompatible mechanisms can yield identical in-domain proxy observations. Instead, the paper advocates for research priorities that focus on explicating and evaluating identifying assumptions and the discriminating regimes essential for robust scientific discovery.

The Proxy Gap and the Problem of Mechanistic Equivalence

The central technical claim addresses the "proxy gap": observable variables are typically indirect measurements (proxies) of latent scientific drivers. The mapping from latent mechanism-level variables $S$ to observable proxies $X$ via measurement channels typically induces a many-to-one relationship. Thus, for any observed proxy distribution $P_X$, there is a set $\mathcal{M}(P_X)$ of mechanism/measurement-pairings consistent with the data, forming an equivalence class at the mechanism level. This holds regardless of whether measurement is deterministic or stochastic, as formalized:

Proposition (Proxy Identifiability): For modular measurements of block-structured $S$, invertible measurement channels or blockwise reparameterizations leave $P_X$ invariant, so only features invariant under such relabeling are identified (see the formal proposition in the paper's main text and appendix).

The implication is that, absent extra structure, multiple mechanistic explanations remain compatible with any given dataset, even under infinite sample sizes and perfect knowledge of the observational distribution. This ambiguity persists despite measurement noise, parameter uncertainty, or model calibration.

High Dimensionality as a Mechanistic Obfuscator

High-dimensional proxy spaces exacerbate the problem. In modern ML workflows, proxy vectors typically lie on low-dimensional manifolds within high-dimensional ambient spaces, reflecting selection, measurement design, and institutional structure. Data thus reside on "thin supports," and the curse of dimensionality ensures that any attempt to learn unconstrained functions generalized off-support is ill-posed without further geometric or structural assumptions.

This high-dimensionality, combined with the sparsity of informative regimes and measurement design, implies that many mechanisms matching in-domain data can diverge arbitrarily outside the observed support, especially under intervention or extrapolation. Distance metrics and local pooling strategies degrade in such spaces, making it substantially easier to hide mechanistic disagreement and harder to diagnose underdetermined mechanistic claims—as illustrated by geometric arguments and references to [Bellman1957DynamicProgramming] and subsequent literature.

LLMs and Narrative Collapse

The paper introduces "narrative collapse" as a new, AI-specific hazard. LLM-centered workflows, especially those generating explanations at scale, industrialize the process of collapsing large sets of plausible mechanisms into a single, fluent narrative. This mapping from equivalence class to narrative output is fundamentally many-to-one: it produces an apparently resolved mechanistic account, even when the evidence supports only multiplicity.

Proposition (Narrative Collapse as Minimax Ambiguity): Whenever $\mathcal{M}(P_X)$ contains mechanisms $M_1, M_2$ with differing values on some query $q(M)$, any single-valued explanation function must systematically err on at least one, with minimax risk lower bounded by 0.5 under 0–1 loss.

Differentiating Identification from Other Uncertainties

The argument distinguishes structural identification uncertainty from aleatoric (irreducible noise) and epistemic (parameter/model uncertainty) uncertainty. Even with unlimited data, the absence of identifying structure means mechanistic queries remain underdetermined, and standard UQ methodologies cannot quantify the ambiguity inherent in $\mathcal{M}(P_X)$. Existing multiplicity phenomena (Rashomon effect, underspecification) refer to model-based rather than mechanism-based indeterminacy, and the proxy gap operates at a deeper level.

Empirical Illustration: Mendel’s Peas and Mechanistic Collapse

To concretize these claims, the paper simulates classical Mendelian inheritance in peas, contrasting two regimes of observation:

• Regime A (Phenotype-only): The model observes only pooled F$X$0 offspring phenotype counts (without cross labels).
• Regime B (Design): The model observes detailed cross labels and additional experimental crosses.

Even when flexible MLPs match observed phenotype distributions equally well in both regimes, their counterfactual mechanistic predictions show substantial divergence in A, but collapse to a consensus in B due to the additional identifying structure provided by design.

Figure 1: Identifying structure collapses mechanistic disagreement—mean $X$1 spread of counterfactual probabilities among MLP fits drops from 0.095 (A) to 0.029 (B) for near-optimal seeds.

Figure 2: In Regime A, both the true Mendelian mechanism and an ad hoc alternative are observationally equivalent over proxies.

Figure 3: In Regime B, only the Mendelian mechanism remains well-calibrated under labeled crosses; designed structure rules out the alternative.

Additional figures (4–6) illustrate how, in the absence of design labels, counterfactual predictions vary arbitrarily across near-optimal fits, but concentrate sharply when mechanistic structure is provided, justifying that mechanistic disagreement is not due to optimization noise:

Figure 4: Without design, counterfactual predictions show broad variability.

Figure 5: With design, the same counterfactual predictions concentrate tightly.

Figure 6: There are many near-optimal black-box fits, motivating the need to report mechanistic multiplicity rather than attributing variability to optimization.

Practical and Theoretical Implications

Research Design and Claims

The paper urges the scientific ML community to prioritize the specification and audit of identifying structure—joint restrictions on latent mechanisms, measurement processes, and discriminating (interventional or multienvironmental) data regimes. Only under such structure do mechanistic queries become identified or tractably partially identified.

Key standards suggested include:

• Mechanism Cards: Structured appendices stating mechanism-side and observation-side assumptions, and the identifying features or open equivalence classes they induce—together with explicit falsifiers.
• Multiplicity Statements: In non-identified regimes, ML systems should output the set of compatible mechanisms or answer ranges, not single narratives.
• Mechanism-Discriminating Evaluations: Claims should be tested on targets directly probing identifying structure (e.g., interventions, cross-environment invariances, or auxiliary constraints).

Implications for Scientific ML and AI

These principles have direct implications for the application of ML (particularly LLMs) in scientific discovery:

• Model complexity or predictive performance does not guarantee mechanistic discovery in proxy-rich regimes; only data and structure enabling identification or auditable partial identification do.
• LLMs and similar models should by default represent epistemic multiplicity when reporting mechanistic conclusions, particularly in open-world scientific workflows.
• Advances in autonomous discovery, equation-learning, and causal inference should be benchmarked not on predictive or narrative coherence, but on the ability to shrink equivalence classes via explicit mechanisms and discriminating regimes.

Positioning Relative to Alternative Views

The paper specifically addresses, and refutes, objections that epistemic uncertainty quantification or model scaling can resolve underdetermination in fixed observational regimes. Scale helps only insofar as it introduces new identifying structure, not by sample size alone. Simplicity criteria or Occamist selection are themselves additional identifying assumptions, and should be stated transparently.

Partial identification remains a valid and often necessary scientific goal, but must be labeled as such, with claims being presented as bounds or hypothesis classes rather than spurious point estimates.

Future Directions and Theoretical Developments

Practically, the field must invest in:

• Data collection protocols and experimental designs that inject discriminating variation—interventions, new measurement channels, or environment shifts—into proxy observations, following the successes of controlled genetics, epidemiology, and physics-informed learning.
• Formalization of mechanism cards and multiplicity reporting standards for ML-driven mechanism discovery.
• Development of new interfaces and LLM workflows that faithfully surface, rather than collapse, the mechanistic multiplicity inherent to the data and observational regime.

Theoretically, future work should:

• Characterize the minimal identifying structure necessary for query-level identification in various scientific domains.
• Clarify the boundaries between model-level multiplicity (Rashomon, underspecification) and mechanism-level indeterminacy, especially as models and data become ever more expressive and high-dimensional.

Conclusion

This position paper offers a rigorous argument that scientific discovery using ML/AI, particularly in high-dimensional proxy regimes, is only possible with explicit, auditable identifying structure. Without it, predictive and explanatory success are neither necessary nor sufficient for mechanism discovery. The scientific ML community is encouraged to redirect effort from model complexity to identification-centric design, evaluation, and reporting. Adhering to these norms is essential for AI to support, rather than merely simulate, scientific progress.