Contemporary AI lacks the imagination to diverge or negate in science

Abstract: Bold projections that artificial intelligence will accelerate scientific discovery have raced ahead of evidence from working scientists, and the field still lacks large-scale, scientist-in-the-loop tests of these claims. Here we mount the largest such evaluation to date and map what AI cannot yet do for science. We invited authors of 121,640 recent preprints across biology, medicine, chemistry, and the social sciences to judge ideas that LLMs generated from the context and puzzles of their own papers. 6,749 scientists returned 25,139 sets of ratings on novelty, empirical feasibility, probability of being true, and favorability of adoption. Three patterns emerge. First, non-reasoning LLMs collapse into a narrow "hivemind" of similar ideas; reasoning models roam a wider hypothesis space, yet no model class spontaneously proposes null hypotheses -- a move humans make more freely. Second, scientists reward ideas that resemble their own and prize probability over novelty, though social scientists tolerate risk more readily than life scientists. Senior social scientists are the harshest critics, and their skepticism is well-earned: LLMs falter most in pluralistic fields like the social sciences that demand context-aware interpretation and evolving theories. Third, automated evaluators on which the community currently relies -- LLM-as-a-judge, artificial metrics, and even state-of-the-art (SOTA) models -- agree only weakly with expert judgment, and retrieval augmentation and scientist persona prompting yield only marginal gains. A Qwen3-14B reward model we post-trained on human ratings captures field taste nuances, beats SOTA models by up to 27%, and closes the gap to the inter-rater consistency of independent peer reviewers. For all the hype, today's scientific AI still represents a collaborator whose imagination, outputs and judgment benefit from human grounding.

• The paper demonstrates that non-reasoning LLMs generate highly similar hypotheses, rarely form null hypotheses, and thus lack essential scientific imagination.
• It employs a large-scale empirical audit involving 121,640 preprints and 6,749 scientists to evaluate novelty, feasibility, and probabilistic alignment in generated ideas.
• Findings reveal that while reward models trained on expert data narrow the gap, LLMs still struggle with creative divergence and negative reasoning compared to human experts.

Evaluating AI Imagination and Divergence in Scientific Hypothesis Generation

Study Design and Methodology

The paper delivers an extensive empirical audit of the hypothesis-generation capabilities of contemporary LLMs for science, deploying the largest scientist-in-the-loop evaluation to date. 121,640 empirical preprints (post-2023) spanning biology, medicine, chemistry, and social sciences were sourced from non-arXiv platforms. The pipeline uses a reasoning model to extract the factual context, research puzzle, and explicit hypotheses from each paper, with rigorous paraphrase-based leakage detection to exclude hypothesis contamination from the input.

LLMs—including both reasoning and non-reasoning chat models, and agentic “deep research” systems from eight providers—are prompted to generate hypotheses solely from context and puzzle, without access to human hypotheses. Five semantically maximally distinguishable hypotheses are selected for each paper and sent to the original authors. 6,749 scientists participated, providing 25,139 ratings along novelty, empirical feasibility, probability, and adoption favorability after passing strict rubric comprehension checks.

Figure 1: An expert-audit pipeline for AI-generated research ideas, driving large-scale scientist-in-the-loop evaluations.

Hypothesis Space Coverage and Null Hypothesis Generation

Non-reasoning LLMs consistently collapse onto a narrow region of idea space, producing hypotheses highly similar to one another—forming the “artificial hivemind.” This is quantified via pairwise cosine similarity. In contrast, reasoning models (e.g., employing chain-of-thought or internal deliberation) generate more diverse, less aligned hypotheses, including outlier and unconventional propositions. Diversity, however, is not synonymous with scientific reach.

Critically, all LLMs—regardless of configuration—rarely formulate explicit null hypotheses (i.e., asserting no effect, relationship, or difference), a cognitive primitive in hypothesis testing and a frequent move by human researchers. Even agentic, web-accessing LLMs fail to populate the hypothesis space with absences. This deficiency stems from the severely imbalanced pretraining corpora, which overwhelmingly record positive associations while nulls are underrepresented (“file drawer problem”). Null reasoning is further attenuated by LLMs’ tendency to compress rare forms of evidence.

Figure 2: Reasoning models expand the hypothesis space, but fail to generate null hypotheses; humans outperform AI in negative reasoning.

Scientist Evaluation Behavior: Field, Seniority, and Bias

Scientists display marked biases in their evaluation of AI-generated ideas. The adoption favorability is driven primarily by similarity to their own prior ideas: within-scientist idea alignment is the strongest positive predictor (coefficient $=1.28$, $p<10^{-3}$). Further, perceived feasibility ($2.29$, $p<10^{-3}$) and probability ($3.42$, $p<10^{-3}$) rise with this similarity, but novelty is penalized ($-3.15$, $p<10^{-3}$), indicating a preference for incremental over radical divergence.

Seniority consistently lowers adoption (citation percentile $=-0.16$, $p=0.001$; log-academic age $p<10^{-3}$0, $p<10^{-3}$1), with senior scientists—particularly in social sciences—being the harshest critics. Medical researchers show the greatest receptivity, correlating with higher AI usage rates, but across-field attitudes are explained by seniority when field-by-seniority interactions are modeled.

Among the three quality dimensions, probability is the dominant driver for adoption ($p<10^{-3}$2, $p<10^{-3}$3), far outstripping novelty ($p<10^{-3}$4, $p<10^{-3}$5) and feasibility ($p<10^{-3}$6, $p<10^{-3}$7), especially in life sciences, implying a systemic aversion to risk. Social scientists are more tolerant of risky, less probable ideas.

LLMs underperform most in social sciences, which demand pluralistic, context-aware interpretation and evolving theoretical frameworks, reflected in systematically lower expert ratings of novelty, feasibility, and probability.

Figure 3: Adoption and quality judgments show strong bias: idea similarity, seniority, and field stratify evaluation, with probability dominating as a driver.

Figure 4: Idea alignment biases feasibility and probability evaluations, but seniority only impacts adoption, not perceived quality.

Automated Evaluation and Reward Model Performance

Automated evaluators—LLM-as-a-judge, artificial metrics, and SOTA reward models—fail to reliably capture scientific quality as defined by expert judgment. Direct prompting and persona injection marginally improve performance; retrieval-augmented Deep Research judges obtain the highest Pearson correlation ($p<10^{-3}$8), but all settings remain weakly aligned to human ratings. LLM judges show central tendency bias, assigning narrow, upper-mid scores and eschewing extremes routinely used by human reviewers. Matching human probability assignments is easier than novelty or feasibility.

Pairwise accuracy on held-out test sets for novelty is near random for baseline models (Skywork-Reward-V2-Qwen3-8B: $p<10^{-3}$9; Deep Research judge: $2.29$0). The reward model post-trained on expert preferences (Qwen3-14B, Bradley–Terry objective) achieves substantial gains: biology-specific accuracy for novelty $2.29$1, feasibility $2.29$2, probability $2.29$3, exceeding human–human reviewer agreement ($2.29$4 from OpenReview conferences). Domain-specific specialization is advantageous when training data are abundant; general models are preferable for limited domains.

Figure 5: Automated evaluators correlate weakly with human reviews; reward models post-trained on expert data outperform SOTA and close the gap to peer-review consistency.

Figure 6: Test accuracy trajectories for reward models; Qwen3-14B delivers optimal performance, supporting practical scalability.

Implications for Scientific Discovery and Future AI Systems

Empirical evidence contradicts popular claims that LLMs can autonomously accelerate scientific discovery. Current systems lack imagination to diverge or negate: they recombine modal patterns of their training corpora, struggle with pluralistic fields, and systematically suppress null hypothesis formulation. Automated evaluation infrastructure that substitutes for expert human judgment fails to recover nuanced, domain-specific standards of novelty, feasibility, and probability.

Three structural limitations are identified:

1. Negation and Null Formulation: The deep bias for positive findings is now inherited by AI, which cannot synthesize nulls without explicit negative evidence. Only deliberate ingestion of registered reports, replication archives, and laboratory notebooks containing failed or null results will repair this asymmetry.
2. Curiosity and Divergence: LLMs optimize for next-token prediction, not the marginal benefit of acquiring new knowledge. Encoding computational curiosity—prioritizing anomaly, contradiction, and absence—will be essential for future scientific automation.
3. Evolving Practice: Taste can be modeled; practice (the collective criteria that define what counts as progress) is deeply contingent and evolves through ongoing contestation by expert communities. Domain-specific refinement and human grounding remain essential for meaningful hypothesis generation and evaluation.

   Figure 7: Reasoning models consistently generate broader perspective distributions than non-reasoning models (UMAP replication).

Conclusion

This study demonstrates, through systematic large-scale audit, that contemporary LLMs lack essential scientific imagination in hypothesis generation, particularly in negative reasoning and creative divergence. Human experts show pronounced adoption biases and probabilistic risk aversion, especially among senior scientists in pluralistic fields. State-of-the-art automated evaluators fail to match expert judgment, but reward models trained on expert preference data can bridge the gap to human-level consistency. Fundamentally, scientific discovery requires not just generation but iterative contestation and reframing by domain experts; LLMs currently serve only as collaborators whose outputs and judgment benefit from continued human grounding. Progress in AI-driven science will depend on deliberate corrections to data asymmetries, the infusion of curiosity-driven objectives, and robust human-in-the-loop frameworks.