ScholarEval: Literature-Grounded Evaluation

• ScholarEval is a literature-grounded evaluation framework that systematically assesses research ideas by quantifying empirical soundness and disciplinary contribution.
• It employs a dual-module process: soundness assessment through method extraction and snippet retrieval, and contribution analysis via pairwise text comparisons.
• Empirical validations on the ScholarIdeas dataset show marked improvements in rubric coverage, reference validity, and expert user preference over conventional systems.

ScholarEval is a literature-grounded evaluation framework for assessing the quality of research ideas, introduced to explicitly address the limitations of conventional automatic evaluation systems that rely solely on LLM predictions or singular metrics. It is characterized by a retrieval-augmented, two-axis structure that systematically quantifies both the empirical validity ("soundness") and disciplinary advancement ("contribution") of research proposals, utilizing up-to-date evidence from scholarly databases and a rubric-based, multi-criteria scoring. Its design and efficacy have been validated using ScholarIdeas—a human-annotated benchmark of multi-domain research idea reviews—with substantial improvements reported over state-of-the-art agentic systems (Moussa et al., 17 Oct 2025).

1. Framework Structure and Algorithmic Workflow

ScholarEval operates as a dual-module, retrieval-augmented evaluation pipeline. The review process is decomposed into two stages:

• Soundness Assessment: Extraction of proposed methodological elements from the input idea, followed by snippet-level retrieval of relevant evidence from Semantic Scholar’s indexed literature. For each method $m_i$, up to 20 supporting snippets are collected; full texts are parsed for methods/results extraction via GROBID, and a structured, three-part review (support, contradictions, suggestions) is synthesized, each with inline citations.
• Contribution Analysis: Identification of scientific advancement axes (“dimensions”) relevant to the idea. For each dimension, seed papers are discovered and expanded via paper search, recommendations, and reference mining. Pairwise comparisons (along each dimension) between the idea and retrieved papers are performed using a combination of abstract semantic similarity (Titan Text Embedding v2) and textual comparison, producing graded (“strengths,” “weaknesses,” “suggestions”) aggregations.

The overall ScholarEval output is a two-part evaluation $\mathcal{E} = (S, C)$, with $S$ denoting soundness and $C$ contribution. The complete algorithm, in high-level pseudocode, is as follows:

$\mathcal{E} = (S, C)$4 (Moussa et al., 17 Oct 2025)

2. Formal Notation and Scoring Metrics

• Soundness Score: Let $k$ be the number of extracted methods in idea $I$. For each method, a score $s_i \in [0, 10]$ is computed from synthesized literature evidence. The aggregated soundness is:

$$\text{Soundness}(I) = S(I) = \frac{1}{k}\sum_{i=1}^k s_i$$

• Contribution Score: For the set of $\ell$ contribution dimensions, each pairwise comparison to retrieved papers supplies a score $c_j \in [-1, 1]$ (via textual claim matching and similarity). Aggregate:

$\mathcal{E} = (S, C)$0

• Automatic Coverage Metric: Given rubric set $\mathcal{E} = (S, C)$1, and evaluation $\mathcal{E} = (S, C)$2, the rubric coverage is:

$\mathcal{E} = (S, C)$3

This structure allows for both qualitative aggregation (Strengths/Weaknesses/Suggestions sections with detailed markdown citations) and quantitative reporting. Invalid reference rates are computed as the ratio of dead-link citations in the generated report.

3. Datasets, Baselines, and Protocols

• ScholarIdeas Dataset: 117 research ideas (AI, neuroscience, biochemistry, ecology), each annotated with expert rubric items, strengths, weaknesses, and suggested improvements. Reviews derive from ICLR 2025 OpenReview for AI, and eLife for others, filtered to early-stage ideation critiques.
• Baselines: Llama-3.3-70B, GPT-4.1, Claude-4 Sonnet (non-retrieval and retrieval-enabled), GPT-4o-search-preview, and o4-mini-deep-research (OpenAI's reasoning- and search-enabled agent).
• Evaluation:
  • Automatic: Coverage (mean rubric hit rate, 1–5 scale), invalid references rate, LLM-judged pairwise preference (Evidence, Actionability, Depth).
  • Expert User Study: 18 domain experts assessed citation novelty, faithfulness, focus, literature engagement, refinement, and overall usefulness on a 1–10 scale. Linear mixed-effects models were applied to test significance.

4. Empirical Results

• Automatic Rubric Coverage: ScholarEval variants achieved significantly higher rubric coverage than all agentic and LLM-only baselines (p < 1e-14, Welch’s t-test), with improvements exceeding 20% relative to o4-mini-deep-research (Moussa et al., 17 Oct 2025).
• Reference Validity: ScholarEval maintained 0% invalid citations, contrasting with 13–19% for raw LLM generations and ~1% for search-enabled baselines.
• LLM Judge Preferences: ScholarEval [Claude] was favored in 75% (Evidence), 80% (Actionability), and 78% (Depth) of pairwise comparisons with o4-mini-deep-research.
• Ablation: Removal of methods/results extraction, paper augmentation, or pairwise comparison each reduced rubric coverage by 0.3–0.4 points, indicating all stages are essential for maximal depth.
• Expert User Study: On all six dimensions, ScholarEval was preferred over o4-mini-deep-research (β = 0.60–0.81, * p < .05, *** p < .001). Notably, the number of new in-field citations supplied rose from 1.09 (baseline) to 2.66 (ScholarEval) per report.

5. Design Considerations, Limitations, and Extensions

• Strengths:
  • Dual-axis (soundness and contribution) decomposition enables multidimensional evaluation anchored in the latest accessible literature.
  • Detailed, citation-backed rationales support actionable critique and suggested refinements.
  • Discipline-agnostic architecture, demonstrated across four major scientific domains.
• Limitations:
  • Reliance on retrieval constrains evaluation fidelity for genuinely novel proposals lacking published antecedents.
  • System throughput is limited: full ScholarEval [Claude] runs cost ≈\$3.40 and ~12 min per idea.
  • Current rubric coverage is a proxy for ground-truth quality; future work is needed in citation factuality and impact metrics.
  • Dataset size (117 ideas, four fields) and absence of physics/social science validation; expansion is planned.
• Planned Extensions:
  • Integration with additional full-text sources (e.g., arXiv, PubMed) to mitigate retrieval recall limits.
  • Lightweight variants (“fast mode”) for large-scale idea triage.
  • Automated evaluation of citation relevance and factuality.
  • Systematic assessment of ScholarEval’s downstream impact on researcher productivity.

6. Significance and Context in Scholarly Evaluation

ScholarEval directly addresses recognized deficiencies in LLM-centered or scalar-metric-based research idea evaluation by embedding real-time, retrieval-augmented, multidimensional literature engagement at the core of its methodology. Its rigorous benchmarking on the ScholarIdeas dataset, along with demonstrably superior coverage, reference validity, and user preference, delineates a new standard for automated idea review systems. The methodology underpins a paradigm shift from single-score or parametric memory-based triage to evidence-rich, reproducible critique processes that align more closely with the evaluative practices of expert scientific review panels (Moussa et al., 17 Oct 2025).