Measuring the Gap Between Human and LLM Research Ideas

Abstract: LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert preference. We instead ask: how far are current LLM-generated ideas from human researchers? To characterize this gap, we build a large-scale evaluation framework for ideation from high-quality human research papers. For each paper, we reverse-engineer a small set of closely related prior works that likely inspired its core idea. LLMs are then prompted to generate a new idea from the set of paper titles and summaries. We introduce a two-axis research-taste taxonomy to profile each idea by its opportunity pattern and research paradigm, and use it to quantify the divergence between human and LLM ideas. Across idea sets generated by different LLMs, we observe a consistent distributional gap: LLM ideas are disproportionately concentrated around bridge-like opportunities and synthesis methods, whereas the human paper reference distribution spreads more broadly across ways of framing gaps and constructing contributions. This result suggests that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste.

• The paper introduces a rigorous framework using a two-axis taxonomy to quantify the research-taste gap between human and LLM-generated ideas.
• It employs metrics like normalized entropy and Jensen-Shannon divergence to reveal that LLMs produce a narrower, archetype-driven range of research ideas compared to human researchers.
• The study highlights practical implications for AI-driven ideation and suggests optimizing LLM outputs to better align with the diverse approaches seen in human research.

Measuring the Research-Taste Gap Between Human and LLM Research Ideas

Introduction and Motivation

The increasing integration of LLMs into research ideation workflows raises a crucial question: do LLMs generate the same kinds of research ideas as human researchers? While prior evaluations focus on surface-level properties such as novelty, feasibility, or expert ratings, these do not capture whether LLMs replicate the distributional diversity of authentic human research ideation. This paper proposes a rigorous framework to quantitatively characterize the "research-taste gap" between LLM- and human-generated research ideas, grounded in common literature contexts. The framework decomposes each idea into its motivating gap and its methodological paradigm, allowing precise alignment and divergence analyses between human and LLM ideation outputs.

Figure 1: Overview of the research-taste gap analysis pipeline, illustrating the shared literature context, idea decomposition, taxonomy-based annotation, and comparative distributional evaluation.

Evaluation Framework and Taxonomy

Data Construction and Task

The evaluation is conducted on 11,683 papers from recent ML conferences and Nature Communications (2023–2026), covering diverse subdomains. For each human paper, the authors' core idea is extracted and mapped to its motivating gap and main methodological move. A literature context is then reconstructed by reverse-engineering the proximal prior works that likely inspired the idea. LLMs (across nine configurations) are prompted to generate new research ideas using this exact local context, requiring them to identify research gaps and construct contributions from the same pool of references as humans.

The Two-Axis Research-Taste Taxonomy

Each idea is classified via a two-axis taxonomy:

• Opportunity Pattern: the primary mechanism that motivates a new study (e.g., puzzle, explanation gap, bridge opportunity, scope mismatch).
• Method Paradigm: the high-level mode of transforming the gap into a contribution (e.g., synthesis/unification, formal derivation, artifact/system, robustification).

This taxonomy, synthesized from NSF, NIH, AHRQ, and DARPA guidance, is expert-calibrated and refined for both generalizability and discriminative power.

Main Distributional Findings

Concentration and Divergence Metrics

Comparisons are drawn using metrics such as normalized entropy, Jensen-Shannon divergence, and total variation distance (TVD). The results show that LLM-generated ideas consistently occupy a narrower region of the taxonomy space than human ideas, especially pronounced on the opportunity axis (with entropy drops from 0.92 for humans to 0.55–0.76 for LLMs).

Figure 2: Label distributions showing the strong over-representation of bridge and synthesis categories in LLM-generated ideas, contrasted with the wider human distribution across opportunity patterns and method paradigms.

Key numerical results:

• Only 12.1% of human ideas are motivated by bridge opportunities, versus 47.1%–64.2% for LLMs.
• Synthesis/unification methods are central in just 5.1% of human ideas but 22.5%–38.7% of LLM outputs.
• TVD comparing model and human distributions remains above 0.21 even for the closest LLMs, implying significant redistribution would be required for alignment.

These statistics underline a systematic narrowing and shift in the kind of research moves LLMs advance, with a strong bias towards connecting literatures and synthesizing approaches.

Diagnostic Scoring

Automated annotator-assigned diagnostics (surface stitching, bottleneck specificity, boilerplateness) reveal that LLM proposals, while syntactically fluent, often exhibit lower specificity and higher rates of generic, template-like combinations. Certain LLMs, such as Qwen variants, manifest particularly strong degradation on these diagnostics.

Reasoning Mode Ablation

Enabling explicit step-wise reasoning ("thinking mode") in LLMs amplifies distributional misalignment rather than mitigating it. Bridge- and synthesis-pattern prevalence increases, and entropy further drops (Qwen3-8B: bridge opportunities rise from 49.7% to 71.1%, entropy drops from 0.658 to 0.481).

Mechanistic Analysis

Figure 3: Mechanistic dissection reveals that LLMs disproportionately utilize generic integration/synthesis moves, whereas humans more often perform local interventions such as replacement, decoupling, or formalization.

Clustering and representation-space analysis show that model outputs converge on recognizable archetypes—typically, selecting a salient concept and integrating/unifying it with another (e.g., “integrate multi-omics with diffusion policies”), whereas humans intervene more locally (e.g., “replace brittle module”, “decouple two mechanisms”). Human proposals are more diffusely positioned relative to prior works, suggesting distinctive—rather than merely integrative—contributions.

Practical and Theoretical Implications

This research demonstrates that, irrespective of fluency and surface plausibility, current LLMs exhibit a restricted and systematically shifted research-taste profile relative to human scientists. In practice, this raises cautions for using LLMs as autonomous or semi-autonomous research ideators: overreliance is likely to reinforce integrative synthesis at the expense of hypothesis-driven, measurement-focused, or mechanism-intervention innovations. Additionally, prompt engineering, richer contexts, or reasoning augmentations do not mitigate the narrowing effect—if anything, they may exacerbate it.

Theoretically, this finding reframes the target of research ideation systems: distributional alignment with human research taste, rather than feasibility or novelty alone, should be a central evaluation axis. The results further suggest that LLMs, even strong ones, are not trivially sampling the high-level research distributions characteristic of active scientific fields; instead, they instantiate a concentrated, archetype-driven pattern largely absent in expert ideation.

Future Directions

Improvements to AI-driven ideation could incorporate direct optimization towards research-taste distributional diversity, reinforcement learning from diverse human ideations, or architectural modifications to reduce archetypal convergence. Extending the analysis to non-STEM disciplines, multi-agent ideation, and longitudinal field deployment will further clarify the boundaries of current LLM-derived creativity and its role inside human research loops.

Conclusion

This study provides a robust quantitative and mechanistic account of the persistent gap between human and LLM-generated research ideas under controlled, literature-grounded settings. LLMs remain behaviorally narrow, strongly preferring bridge and synthesis-driven research moves, whereas human scientists distribute their contributions more broadly across research gap framings and methodological innovations. The work establishes both the methodology and empirical benchmarks for measuring and closing the ideational gap in next-generation AI research assistants.