SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?
Abstract: Autonomous AI research agents aim to accelerate scientific discovery by automating the research pipeline, from hypothesis generation to peer review. However, existing benchmarks rarely test a fundamental bottleneck: whether LLMs can judge the methodological viability of a research idea before expending time and computational resources. We introduce SoundnessBench, a curated benchmark of 1,099 machine-learning research proposals reconstructed from ICLR submissions, labeled with reviewer soundness sub-scores, and audited against source papers. SoundnessBench should be interpreted as a benchmark for recoverable proposal-stage soundness rather than exact prediction of full-paper review outcomes. Across 12 frontier LLMs, we find a pervasive optimism bias: under standard prompting, models frequently rate low-soundness proposals as sound, while aggressive prompting largely shifts errors from false positives to false negatives. Additional controls for public-corpus contamination, paper-identifying phrases, surface features, and human audit quality suggest that this behavior is not explained by a single confounder. Our results indicate that current LLMs are not yet reliable as standalone first-gate evaluators for scientific rigor.
Summary
- The paper presents SoundnessBench, a large-scale benchmark using curated ICLR submission data to assess LLMs' ability to judge research proposal soundness.
- The evaluation reveals that models exhibit marked optimism bias under standard prompting, with high false-positive rates for unsound proposals.
- Aggressive prompting reduces false positives but severely harms high-soundness recall, highlighting a critical trade-off in model calibration.
SoundnessBench: Benchmarking LLMs' Capacity for Proposal-Stage Scientific Judgment
Motivation and Problem Definition
Autonomous AI research agents aim to accelerate scientific investigation by automating the research cycle, but a fundamental bottleneck persists: the ability to robustly reject unsound research proposals prior to resource expenditure. Current benchmarks mainly focus on agents' execution and artifact generation, neglecting the “first-gate” decision—methodological soundness assessment based purely on proposal text. "SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?" (2605.30329) addresses this gap by presenting SoundnessBench, a large-scale, auditable benchmark for proposal-stage scientific triage in machine learning.
Benchmark Design and Dataset Construction
SoundnessBench is reconstructed from ICLR submission data spanning 2022–2026, encompassing 1,099 proposals across 16 ML subfields. Proposals are labeled using reviewer soundness sub-scores, filtered for high reviewer confidence and inter-reviewer agreement to eliminate label ambiguity. Proposals are extracted in near-verbatim format, masking all results and acceptance cues per the AI Scientist v2 protocol, and are subject to a fidelity audit employing retrieval-backed atomic-claim verification.
Figure 1: SoundnessBench pipeline: curation, labeling, extraction, verification, and dataset assembly.
Class separation is enforced by excluding ambiguous middle scores, yielding 458 low-soundness and 641 high-soundness instances. Demographically, the benchmark reflects the ICLR corpus composition—both in subfield and temporal distribution.
Figure 2: Benchmark dataset statistics and label density, showing distinct separation between low- and high-soundness proposals.
The rigorous curation pipeline ensures proposals are directly traceable to their source manuscripts and avoids result leakage. Adversarial controls and human audits further validate extraction quality and label defensibility.
Evaluation Protocol and Empirical Findings
SoundnessBench is used to evaluate 12 frontier LLMs (including GPT, Claude, Gemini, Qwen, LLaMA, Kimi) under two prompting regimes:
- Standard Prompt: Models classify proposal soundness with structured justification, absent explicit conservatism.
- Aggressive Prompt: Models are instructed to default to “low” rigor unless overwhelming evidence of methodological strength exists.
Quantitative Results: Optimism Bias
A marked optimism bias is observed: under standard prompting, the mean false-positive rate for low-soundness proposals is 74.0%, with 9 out of 12 models exceeding 70%. High-soundness recall remains elevated, indicating models are broadly permissive towards weak proposals, often failing as gatekeepers for scientific rigor.
Figure 3: Confusion matrices under standard prompting; most models are excessively optimistic, approving many unsound proposals.
Model size analysis within the Qwen3.5 family indicates that scaling increases high-soundness recall but further exacerbates optimism bias, degrading low-soundness recall.
Figure 4: Confusion matrices across Qwen3.5 model sizes, underlining that scale increases optimism rather than discrimination.
Upon switching to aggressive prompting, the optimism bias is inverted—models become over-conservative, with low-soundness false positives dropping to 19.9% but high-soundness recall collapsing to 36.1%.
Figure 5: Under aggressive prompting, optimism bias shifts to over-conservatism, severely reducing high-soundness recall.
Further, prompt sensitivity varies across model architectures and instruction-tuned variants, but instruction tuning alone does not resolve bias, as demonstrated in Qwen variants.
Figure 6: Instruction tuning fails to correct optimism bias, both base and tuned models exhibit similar prompt sensitivity.
Robustness and Confounder Controls
A battery of controls is performed:
- Contamination Analysis: An ICLR 2026-only split verifies that optimism bias persists absent public-corpus contamination.
- Identifier Removal: Masking paper-unique phrases changes results negligibly.
- Surface Feature Baselines: Simple heuristics and structural statistics over-reject high-soundness proposals, contrasting LLM over-approval.
- Adversarial Injection: Severe methodological flaws are detected, yet subtler weaknesses are missed, pointing to specific model limitations.
Slice-level analyses (year, subfield, writing quality) confirm bias is systemic and not concentrated in any data subset.
Qualitative Analysis
Representative false-positive cases display LLMs assigning high soundness to proposals with reviewer-noted fatal flaws—missing ablations, improper comparisons, overclaiming, and unsupported novelty.
Figure 7: False-positive example—LLMs label an unsound proposal as high soundness, ignoring reviewer-identified flaws.
Figure 8: Gemini 3.1 Pro response for the false-positive case.
Figure 9: GPT-5.4 Thinking response for the false-positive case.
Figure 10: Claude Opus 4.6 response for the false-positive case.
Additional examples reinforce the optimism bias; true-positive cases illustrate correct high-soundness prediction on well-justified proposals.
Figure 11: Second false-positive example—models assign high soundness to an unsound proposal despite weak empirical justification.
(Figures 12–14)
Figures 12–14: Qualitative model responses to the second false-positive, consistent optimism.
Figure 12: True-positive example—models correctly identify a sound proposal.
(Figures 16–18)
Figures 16–18: Model responses demonstrating correct high-soundness judgment.
Implications, Limitations, and Future Directions
SoundnessBench exposes a prompt-sensitive calibration trade-off in current LLMs for proposal-stage scientific assessment. Model judgments are unstable, alternating between reckless optimism and excessive skepticism based on prompt strategy. Confounder and contamination controls establish the optimism bias as a genuine capability deficit, not a dataset artifact or memorization result.
Practically, the unreliability of LLMs as standalone gatekeepers leads to misallocation of computational resources, hampering autonomous agent effectiveness. Theoretically, the findings suggest that scientific triage requires interventions beyond scale or instruction tuning, pointing to the necessity for model calibration, targeted training, or human-in-the-loop review mechanisms.
Potential future avenues include:
- Training LLMs on specifically curated scientific judgment tasks to mitigate optimism bias.
- Incorporating adversarial examples and intermediate reasoning chains for methodological flaw detection.
- Expanding SoundnessBench to cover other domains (biology, chemistry, social science) for broader applicability.
- Longitudinal studies tracking downstream execution outcomes following proposal-stage triage.
Conclusion
SoundnessBench provides an auditable, large-scale benchmark for testing LLMs' ability to discriminate research proposal viability in ML. Empirical evaluation demonstrates pervasive optimism bias and prompt sensitivity, undermining reliability as first-gate scientific critics. Robustness checks underline this as a systemic model limitation. Addressing the observed calibration deficiencies will require targeted interventions, paving the way for safer and more effective autonomous research agents.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper introduces SoundnessBench, a new test to see if AI LLMs (like ChatGPT-style tools) can tell whether a machine learning research idea is solid before anyone spends time and money running experiments. Think of it like checking a blueprint before building a house: does the plan make sense, or will it fall apart?
What questions are the researchers asking?
They focus on a simple, practical question:
- Can AI models reliably spot problems in a research plan and say “this idea and experiment design are strong” or “this is flawed” before any code is written?
They also ask:
- Do models lean too positive (approve weak ideas) or too negative (reject good ideas)?
- Are their judgments stable, or do they flip when you slightly change how you ask the question?
How did they do the study?
To test this fairly, the authors built a dataset called SoundnessBench.
Here’s the approach, explained in everyday terms:
- Start with real research: They collected thousands of submissions and reviews from ICLR, a top AI conference. From these, they carefully selected 1,099 proposals (research ideas plus experiment plans) across 16 subfields like reinforcement learning and generative models.
- Use expert signals: Each proposal was labeled “high soundness” or “low soundness” using special reviewer scores focused on “soundness” (how methodologically solid the plan is). They only kept cases where reviewers agreed, and they dropped fuzzy middle cases to keep labels clear.
- Keep it “proposal-only”: They removed results and hints like “this paper got accepted,” so the models had to judge the plan itself, not be swayed by outcomes. Imagine judging a recipe by reading it, not by tasting the dish.
- Preserve original wording and verify it: They pulled near-verbatim text from the papers (abstract, related work, risk factors, hypothesis, and experiment design). Then they ran a “fact-check” step: break each proposal into tiny claims (like ingredients in a recipe), retrieve the exact source text, and verify each claim lines up with the original paper. Only proposals that passed this audit made it into the dataset.
- Evaluate AI models: They tested 12 advanced LLMs. For each proposal, a model had to classify it as “high soundness” or “low soundness” and explain why.
- Two ways of asking:
- Standard prompt: “Judge soundness and explain your reasoning.”
- Aggressive prompt: “Be strict. Assume low soundness unless the plan is clearly strong.”
A few key terms in plain language:
- Methodological soundness: Is the plan designed in a fair, careful way that can truly test the idea?
- False positive: Calling a weak plan “good” (approving a bad idea).
- False negative: Calling a strong plan “bad” (rejecting a good idea).
- Data leakage: Accidentally using information you shouldn’t have when judging (like seeing the answers during a test).
What did they find, and why does it matter?
Main results:
- Under the standard prompt, most models were too optimistic. On average, they mislabeled about 74% of weak proposals as “sound.” They were great at recognizing strong proposals (about 92% correct) but bad at filtering out weak ones.
- Under the aggressive (stricter) prompt, the models flipped: they rejected weak proposals much better (only about 20% false positives), but they also mistakenly rejected many strong proposals, correctly recognizing only about 36% of them.
- Bigger models didn’t fix the problem. In some cases, larger models became even more approving under the standard prompt and more rejecting under the aggressive prompt.
- The pattern seems real, not a simple mistake:
- They checked for “contamination” (models memorizing public papers), removed identifying details, and tested simple style cues (like proposal length). None of these explained the behavior.
- When the authors deliberately inserted obvious flaws into strong proposals, models did catch them, which means models do read content—but they still miss many subtler, natural flaws.
Why it matters:
- If an AI “research assistant” can’t reliably reject weak plans, it may waste huge amounts of time and compute chasing bad ideas.
- If you make it stricter, it might throw away many good ideas instead.
- This shows that current models aren’t ready to be the first gatekeeper for scientific rigor on their own.
What are the implications?
- Don’t rely on AI alone for early research triage. Human oversight should stay in the loop, especially for green-lighting projects.
- Better training and calibration are needed. Models may need specialized practice and feedback to spot subtle design flaws without becoming overly strict.
- Benchmarks like SoundnessBench help diagnose weaknesses and guide improvements. In the future, versions for other fields (biology, chemistry, social science) could broaden the impact.
- Bottom line: Today’s AI can write code and papers, but deciding which ideas are worth pursuing is still a fragile skill. Careful tools, training, and human checks are needed to make AI research agents safer and more useful.
Knowledge Gaps
Unresolved gaps, limitations, and open questions
Below is a single, consolidated list of concrete knowledge gaps, limitations, and open questions that remain after this work.
- Ground-truth validity: How accurately do reviewer soundness sub-scores (assigned after reading full papers with results) reflect proposal-only methodological soundness?
- Human benchmark ceiling: What performance do domain experts achieve when judging the same results-masked proposals, and how does inter-annotator agreement compare to LLMs?
- Proposal-only recoverability: Which components of soundness are truly recoverable from proposal text alone versus only detectable post-execution?
- Label noise characterization: How much variance and bias exist in reviewer soundness sub-scores (by reviewer expertise, confidence, or area), and how do these propagate to benchmark labels?
- Ambiguous cases exclusion: What changes when the dataset includes the mid-range scores (e.g., S=2.5) using ordinal or pairwise labels instead of a binary split?
- Domain/venue generalization: Do the findings hold outside ML/ICLR (e.g., CVPR, NeurIPS, ICCV; and non-CS domains like biology, chemistry, and social science)?
- Subfield-specific failure modes: Which flaw types are most misjudged in each subfield (e.g., RL, optimization, generative modeling), and can we build a taxonomy of missed flaws?
- Style and writing-quality confounds: To what extent do rhetorical style, polish, or hedging affect judgments, even after controlling for simple surface features?
- Counterfactual controls: How do models behave on style-controlled counterfactuals (same content, different style) or content-controlled counterfactuals (same style, injected subtle flaws)?
- Leakage auditing depth: What is the residual leakage rate of results/outcomes in proposals at scale, beyond the preliminary human audit, and how does it affect scores?
- LLM-in-the-loop extraction bias: Does using an LLM (Gemini 2.5 Pro) for extraction and claim verification introduce systematic phrasing biases that advantage/disadvantage certain evaluated models?
- Verification robustness: How sensitive are pass rates and fidelity scores to verifier choice, retrieval method, and hyperparameters (tau, chunk size, k), and do non-LLM verifiers change outcomes?
- Contamination detection: Can stronger, automated near-duplicate and web-scale overlap checks more rigorously quantify public-corpus contamination and its impact?
- Dataset stability over time: As proposals and writing norms shift, do model behaviors and benchmark difficulty drift, and how often should a refreshed/hidden test set be released?
- Public-release overfitting risk: How quickly do models overfit to a public benchmark of this size (1,099 items), and what governance (e.g., rotating/hidden sets) mitigates this?
- Binary vs ranking tasks: Are models better at ranking proposals by soundness or making accept/reject decisions under budget constraints than at binary classification?
- Calibration and decision-theory: How do reliability curves, cost-sensitive metrics, or abstention strategies change conclusions compared to raw recall/F1?
- Threshold tuning and priors: Beyond two prompts, can calibrated thresholds, priors, or Bayes decision rules reduce both false approvals and false rejections simultaneously?
- Prompt/systematic intervention search: What systematic prompt-engineering, rubric-based checklists, or critique-debate protocols reliably improve both-class performance?
- Tool-augmented critics: Do retrieval-augmented evaluation, automated checklist enforcement, or static analyzers for experimental design reduce optimism without inducing over-conservatism?
- Training data interventions: Can instruction tuning or preference optimization on proposal-stage soundness judgments (e.g., using SoundnessBench-style labels) achieve stable calibration?
- Source of optimism bias: Is optimism driven by RLHF preferences, instruction-tuning objectives, safety training, or domain knowledge gaps, and which targeted training removes it?
- Scale effects across families: Do scale–bias trends generalize beyond the Qwen family, and how do data mixture and training recipe mediate optimism vs conservatism?
- Error analysis by flaw type: Which natural, subtle flaws (e.g., data leakage, metric mismatch, under-specified baselines) most often yield false positives, and can we create targeted “micro-benchmarks”?
- Multilinguality: How do models judge proposals written in other languages or translated content, and does translation introduce new artifacts?
- Agentic settings: Do multi-turn, self-critique, or tool-using agent loops improve first-gate judgments compared to single-pass answers on the same items?
- External information use: If agents are allowed to consult literature or open-source code (pre-execution), how does this change soundness judgments and biases?
- Compute-aware triage: How should decision thresholds adapt under different compute budgets and risk tolerances, and can models optimize expected utility of follow-on experiments?
- Statistical rigor: What are the confidence intervals, variance across runs, and sensitivity to decoding randomness; are differences across models statistically significant?
- Reproducibility under API drift: How stable are results across model version updates and provider-side changes, and can a standardized reproducibility protocol be established?
- Dataset balance vs prevalence: How do results change under prevalence settings that reflect real-world triage rates (potentially imbalanced), not the current high/low split?
- Expanded human audits: Can larger, cross-institution expert re-annotations quantify human–LLM gap, verify label validity at scale, and refine annotation guidelines?
- Richer modalities: How does including limited code snippets, pseudo-experiment plans, or pilot logs (without results) affect recoverable soundness and model performance?
- Longitudinal studies: How well do proposal-stage judgments predict post-execution outcomes and resource costs over time, and where do models systematically mis-calibrate this link?
- Fairness and bias: Do judgments vary with author demographics, institution, or region proxies embedded in writing, and how can such biases be measured and mitigated?
- Open-source vs closed-source differences: Which training or alignment practices explain the cross-model variation in prompt sensitivity and macro F1 stability?
- Hyperparameter sensitivity in verification pipeline: How do different retrieval algorithms (BM25 vs dense), chunking schemes, and tau thresholds alter inclusion/exclusion and downstream findings?
Practical Applications
Immediate Applications
The following near-term uses can be deployed with modest engineering, leveraging SoundnessBench’s dataset, curation pipeline, and empirical findings on optimism bias and prompt sensitivity.
- Research-agent first-gate check (Software/AI R&D)
- What: Insert a “soundness triage” step before agents run experiments, using ensemble prompts (standard + stricter) and requiring human approval when the two disagree.
- Tools/workflows: Gateway module in agent pipelines, budget gating based on low/high-soundness confidence, escalation rules.
- Assumptions/dependencies: Labels proxy methodological validity in ML only; human-in-the-loop remains necessary due to prompt sensitivity and high false positives under standard prompting.
- Vendor and model evaluation scorecards (AI model development; Procurement)
- What: Use SoundnessBench to benchmark and compare LLMs’ triage reliability (false-positive and false-negative rates, Macro F1) before adopting them as “AI scientist” components.
- Tools/workflows: Continuous evaluation dashboards; release notes tracking optimism bias; regression tests across model updates.
- Assumptions/dependencies: Dataset is public and may be partially known to models; periodic re-tests and contamination-aware splits advisable.
- Peer-review assistance, not replacement (Academic publishing)
- What: Use an LLM assistant tuned on failure modes highlighted by the benchmark to flag likely methodological flaws (e.g., missing baselines, leakage risks) as pre-rebuttal checklists for reviewers.
- Tools/workflows: Review portals with standardized “soundness checklist”; structured critique generation; audit trails.
- Assumptions/dependencies: Treat assistant output as suggestions; final judgment by human reviewers.
- Internal research-ops cost control (Industry R&D; Labs)
- What: Pre-screen internally generated project proposals to reduce wasted compute and time by filtering out high-risk, low-soundness designs.
- Tools/workflows: Proposal templates mapped to benchmark-derived rubrics; compute-grant gates tied to soundness flags; exception paths for high-novelty ideas.
- Assumptions/dependencies: Organizational buy-in for a two-tier triage (automated pre-screen + human arbitration).
- Education in research methods (Higher education; Professional training)
- What: Teach methodological rigor using high/low-soundness exemplars; run class exercises where students and LLMs assess proposals and compare rationales.
- Tools/workflows: Course modules; interactive grading rubrics; reflective assignments on optimism bias and prompt framing.
- Assumptions/dependencies: Coverage limited to ML-style proposals; instructors should contextualize limits.
- Governance for compute allocation (Research computing centers; Enterprise policy)
- What: Incorporate a “proposal-stage risk” flag into compute request workflows to surface questionable designs to governance boards.
- Tools/workflows: Risk dashboards; audit logs of AI triage vs. human decisions; post-mortems on false approvals.
- Assumptions/dependencies: Transparency and appeals processes to avoid overblocking; sensitivity to fairness concerns.
- Model diagnostics for prompt sensitivity (AI model development)
- What: Use standard vs. aggressive prompts to quantify how fragile a model’s judgment is and to calibrate default operating points.
- Tools/workflows: Prompt A/B testing harness; threshold tuning; confusion-matrix monitoring per release.
- Assumptions/dependencies: Gains from prompt tweaks are limited; deeper training or calibration likely needed.
- Retrieval-backed extraction fidelity in enterprise content (Enterprise software; Legal/compliance)
- What: Repurpose the paper’s atomic-claim verification pipeline to audit LLM-generated summaries against source documents (e.g., policies, contracts).
- Tools/workflows: BM25/embedding retrieval; claim decomposition; evidence-only verification with pass/fail thresholds.
- Assumptions/dependencies: Requires high-quality document stores; threshold tuning (e.g., τ≈0.7) per domain.
- Product experimentation pre-checks (Software/product analytics)
- What: Adapt soundness checklists to A/B test proposals (e.g., metric appropriateness, leakage, baseline adequacy) to reduce invalid tests.
- Tools/workflows: Template lints in experiment platforms; automated gating before launch.
- Assumptions/dependencies: Domain adaptation needed; benchmark is ML-centric.
- Editorial and meta-review analytics (Academic publishing; Research integrity)
- What: Apply reviewer agreement filters and soundness sub-score analysis to study consistency and to refine review guidelines.
- Tools/workflows: Reviewer dashboards; policy updates on scoring rubrics.
- Assumptions/dependencies: Access to sub-score data; reviewer training alignment.
Long-Term Applications
These opportunities require additional research, domain adaptation, or scaling beyond the current ML/ICLR scope.
- Fine-tuned “soundness critic” models (AI model development; Research tools)
- What: Train specialized models on SoundnessBench plus new human annotations to reduce optimism bias and prompt sensitivity.
- Tools/workflows: Supervised fine-tuning or DPO on critique data; calibration (temperature scaling, conformal prediction).
- Assumptions/dependencies: Larger, continuously refreshed, domain-diverse datasets; expert labels.
- Proposer–critic–arbiter multi-agent systems (Software; Autonomous research)
- What: Architect agents with explicit roles (creative proposer, skeptical critic, arbiter with cost-aware thresholds) to balance exploration vs. rigor.
- Tools/workflows: Debate protocols; disagreement-triggered human escalation; cost-sensitive decision policies.
- Assumptions/dependencies: Careful calibration to avoid degenerating into over-approval or over-rejection; compute budgets and SLAs.
- Cross-domain soundness benchmarks (Healthcare, Biotech, Materials, Social science)
- What: Extend the reconstruction and verification pipeline to clinical trial protocols, lab experiments, and field studies with domain-specific rubrics.
- Tools/workflows: Partnerships with journals/IRBs; private/sequestered test sets; modality integration (code, lab logs).
- Assumptions/dependencies: Expert availability; privacy/ethics compliance; non-ML methodological norms.
- Grant and funding pre-screening (Public policy; Philanthropy; Corporate R&D)
- What: AI-assisted triage of proposals for methodological viability to reduce reviewer load and surface high-potential projects faster.
- Tools/workflows: Two-tier screening (automated + human panel); fairness and bias audits; appeal channels.
- Assumptions/dependencies: Regulatory acceptance; transparency requirements; risk of disadvantaging unconventional designs.
- Compute-credit and cloud policy integration (Cloud providers; Research infrastructure)
- What: Gate extremely compute-intensive experiments behind soundness checks to curb waste and environmental impact.
- Tools/workflows: API hooks in job schedulers; “soundness attestations” with audit logs.
- Assumptions/dependencies: Clear consent and data-sharing policies; safeguards against stifling legitimate exploration.
- Journal policy for pre-execution artifacts (Academic publishing)
- What: Require a soundness analysis artifact with submissions; offer AI-generated preliminary reports as optional attachments.
- Tools/workflows: Submission checklists; automated linting; rebuttal hooks tied to flagged issues.
- Assumptions/dependencies: Community acceptance; avoiding overreliance on automated scores.
- Dynamic/private evaluation suites (AI evaluation ecosystem)
- What: Build rotating, private or continuously refreshed proposal sets to limit contamination and better track real capability improvements.
- Tools/workflows: Secure evaluation infrastructure; periodic domain refresh; governance for data access.
- Assumptions/dependencies: Sustainable data pipelines; stakeholder trust.
- Standards and certification for research-triage AI (Standards bodies; Regulators)
- What: Define metrics (e.g., balanced Macro F1), robustness tests (prompt variance, adversarial injection), and auditing requirements for “first-gate” AI systems.
- Tools/workflows: IEEE/ISO-like standards; third-party audits; model cards focused on triage reliability.
- Assumptions/dependencies: Multi-stakeholder process; evolving benchmarks across domains.
- Commercial products: SoundnessGuard/ProposalLint (Software; SaaS)
- What: SaaS plugins for Notion/Docs/Overleaf/Jupyter that lint research plans, PRDs, and experiment designs for methodological pitfalls before execution.
- Tools/workflows: Document add-ons; API integrations with experiment trackers (e.g., MLflow, Weights & Biases).
- Assumptions/dependencies: Domain-specific rule packs; liability and privacy considerations.
- Evidence-grounded auditing in safety-critical summarization (Healthcare, Finance, Legal)
- What: Generalize the atomic-claim, retrieve-then-verify pipeline for high-stakes summarization and compliance reporting.
- Tools/workflows: Evidence-only verifiers; per-claim support ratios; exception workflows for unsupported claims.
- Assumptions/dependencies: High-quality retrieval over regulated corpora; human oversight; documented failure modes.
- Robustness and bias-mitigation research (Academia; AI labs)
- What: Develop training objectives and evaluation protocols targeting sycophancy and framing fragility in scientific judgment (e.g., counterfactual prompting, adversarial data, risk-aware thresholds).
- Tools/workflows: Instruction-tuning on counterbalanced prompts; debate/consensus mechanisms; uncertainty-aware decisions.
- Assumptions/dependencies: Access to diverse, expert-annotated corpora; measurement alignment across domains.
Glossary
- Adversarial-injection analyses: Tests that inject adversarial content or mismatches to probe whether model judgments rely on shallow cues or are robust to perturbations. "identifier-removal, surface-feature, year/subfield/writing-quality, and adversarial-injection analyses probe whether the observed behavior is driven only by memorization or stylistic artifacts."
- Aggressive prompt: A stricter instruction that biases the model toward conservative (low-soundness) judgments unless evidence is clearly strong. "Under the tested aggressive prompt, models over-correct: while the false-positive rate drops to 19.9%, high-soundness recall collapses to 36.1%."
- Atomic claims: Minimal, verifiable statements decomposed from a proposal to enable evidence-grounded fact-checking. "we decompose each extracted hypothesis--experiment pair into atomic claims, retrieve supporting passages, and verify each claim against the source paper"
- BM25: A classic information-retrieval ranking function used to fetch relevant text passages for verification. "BM25(c,\ \mathcal{C},\ \text{top-}k=k)"
- Calibration trade-off: The tension between reducing false approvals and avoiding excessive rejections when tuning decision thresholds. "This suggests a calibration trade-off in current frontier LLMs for proposal-stage soundness judgment: stricter filtering can suppress false approvals, but it also increases the risk of rejecting promising ideas."
- Confusion matrices: Tables that summarize classification performance by comparing predicted vs. true labels across classes. "Confusion matrices under the standard prompt across 12 evaluated models."
- Data leakage: Unintended access to outcome-revealing information that can spuriously inflate evaluation performance. "such as improper baselines, data leakage, or mismatched metrics"
- Desk-rejected: Rejected by the venue before full peer review, often for reasons beyond methodological soundness. "we remove desk-rejected papers, since desk rejections may reflect factors other than scientific soundness."
- First-gate: The initial triage stage that decides whether a research idea is methodologically viable enough to proceed. "first-gate evaluators for scientific rigor."
- Identifier-removal: Eliminating paper-identifying phrases to mitigate recognition and memorization effects. "identifier-removal, surface-feature, year/subfield/writing-quality, and adversarial-injection analyses probe whether the observed behavior is driven only by memorization or stylistic artifacts."
- Information leakage: Inclusion of results or acceptance cues in inputs that compromise a fair pre-execution assessment. "information leakage: completed papers contain results and acceptance cues that can shortcut true first-gate assessment"
- Long-context LLM: A LLM capable of handling very long input sequences without truncation. "We use a strong long-context LLM (Gemini 2.5 Pro) with a task-specific prompt"
- Macro F1: The unweighted average of per-class F1 scores, treating each class equally regardless of size. "Macro F1 is the unweighted average of per-class F1 scores."
- Methodological soundness: The rigor and viability of a hypothesis and experimental design to properly test the claim. "targets recoverable proposal-stage methodological soundness rather than exact full-paper review prediction."
- Optimism bias: A systematic tendency to rate weak or flawed proposals as sound. "we observe a pervasive optimism bias: models frequently rate proposals with low reviewer-derived soundness labels as sound."
- Outcome Masking: Removing experimental outcomes and acceptance signals from proposals to prevent leakage during evaluation. "Outcome Masking to reduce leakage from experimental results or acceptance outcomes."
- Pre-execution scientific judgment: Evaluation of a research idea’s methodological viability before any experiments are run. "a large-scale benchmark for pre-execution scientific judgment in machine-learning research"
- Prompt fragility: Sensitivity of model judgments to changes in prompt wording or framing. "Prior work on sycophancy and prompt fragility suggests that LLM judgments are highly sensitive to how a question is framed"
- Public-corpus contamination: Evaluation bias arising when test content overlaps with publicly available training data. "A preliminary human audit finds low explicit leakage and broad agreement with assigned labels; an ICLR 2026 split reduces public-corpus contamination concerns"
- Results-masked research proposal: A proposal with results and outcome cues removed to simulate first-gate decision-making. "we provide a results-masked research proposal to an LLM with a fixed evaluation prompt."
- Retrieval-backed atomic-claim verification: Validating decomposed claims using retrieved evidence from the source document. "add retrieval-backed atomic-claim verification."
- Reviewer soundness sub-scores: Peer-review ratings focused specifically on methodological rigor rather than overall acceptance. "labeled with reviewer soundness sub-scores"
- Surface features: Shallow textual properties (e.g., length, counts) that can confound evaluations if models over-rely on them. "surface features, and human audit quality suggest that this behavior is not explained by a single confounder."
- Sycophancy: The tendency of models to agree with user framing instead of grounding judgments in evidence. "Prior work on sycophancy and prompt fragility suggests that LLM judgments are highly sensitive"
- vLLM: A high-throughput inference engine for efficiently serving LLMs. "we deploy model servers with vLLM on 2×NVIDIA H200 GPUs."
Collections
Sign up for free to add this paper to one or more collections.