SFBench: The SciFy Scientific Feasibility Benchmark

Abstract: We present SFBench, a benchmark dataset for evaluating systems that assess the feasibility of scientific claims. SFBench includes 197 claims in materials science, each annotated with a ground-truth feasibility score on a five-point scale along with an explanation of that assessment. The collection differs from previous collections in several important ways: 1) it defines a complex task that requires reasoning over claims of varying scientific feasibility; 2) its claims are not extracted from existing scientific publications but are created de novo, greatly reducing the chances that LLMs have trained on them; 3) claims and ground truth are established by subject matter experts, not by artificial intelligence; and 4) unlike many benchmarks that ask about question/answer pairs, provide multiple choice answers, or ask questions requiring short, fixed answers, SFBench explanations are completely open-ended. We describe the benchmark design, data creation process, and evaluation metrics, and we report baseline results using recent GPT models.

• The paper introduces SFBench, a benchmark that uses novel, SME-authored claims to assess AI's scientific feasibility judgment in materials science.
• The methodology features a dual-review annotation process with a significant improvement in inter-annotator reliability (Cohen’s kappa from 0.56 to 0.74).
• The evaluation highlights challenges in balancing information retrieval with expert reasoning, establishing a high-difficulty testbed for advanced scientific AI systems.

SFBench: A Benchmark for Scientific Feasibility Assessment

Benchmark Motivation and Domain Scope

SFBench introduces a rigorously constructed evaluation dataset for assessing the ability of AI systems to judge the scientific feasibility of claims, primarily within materials science. Unlike fact verification datasets targeting true/false or support/refute tasks, scientific feasibility involves nuanced, prospective reasoning: can a claim be accomplished with current technology, resources, and scientific knowledge? The benchmark's focus on materials science targets subfields—lithium-ion batteries, lightweight aerospace metal alloys, A15 superconductors, and thin-film semiconductors—chosen for maturity, domain complexity, and availability of expert annotators.

Claims in SFBench are authored de novo by domain experts rather than mined or edited from existing literature. This design eliminates contamination from pretraining corpora and ensures that AI models cannot rely on rote memorization, especially given the well-documented issues with information leakage affecting benchmark integrity in other datasets.

Dataset Construction and Annotation Protocol

SFBench consists of 197 scientific claims, each labeled with a five-level Likert feasibility score from -2 (extremely unlikely) to +2 (extremely likely). For each claim, annotators provide a concise, open-ended explanation, anchored in empirical, theoretical, or technical reasoning, and—when possible—evidence citations. Annotation guidelines explicitly address ambiguity reduction: claims are required to state necessary assumptions and define parameters to minimize dependence on annotator background knowledge.

The annotation process uses a dual-review loop: after initial claim authoring and self-review, a second subject-matter expert independently rates feasibility and signals issues or disagreements. Claims with divergent assessments are iteratively revised to improve agreement and clarity. This process achieved a quadratic weighted Cohen’s kappa improvement from 0.56 (first pass) to 0.74 (second pass), establishing moderate to substantial inter-annotator reliability—a critical metric for expert benchmarks.

Evaluation Metrics

SFBench is evaluated along two axes: alignment between automated and expert Likert feasibility assignments (quantified by quadratic weighted Cohen’s kappa), and explanation quality (graded by SMEs for correctness and completeness). Explanations are manually scored along detailed rubrics, capturing whether key reasoning points are present and whether claims are substantiated, free of fallacies or omissions.

Scoring is designed to penalize both random agreement and outsized error magnitude, given the ordinal nature of feasibility scoring. Explanation assessment uses a categorical-to-numeric mapping to facilitate overall correctness and completeness calculations, employing a harmonic mean ($F_1$) for aggregate reporting.

Baseline Model Performance

A set of baseline experiments was conducted using contemporary, closed LLMs (e.g., o1, o3, GPT-5). Each model receives carefully-crafted instructions and few-shot samples to standardize prompt quality and formatting. Models are tasked with producing both a Likert feasibility score and a structured, cited explanation without access to retrieval-augmented augmentation or external tools.

Empirical results demonstrate clear improvement in feasibility judgment alignment as LLM architectures progress over time, as measured by quadratic weighted kappa (Figure 1).

Figure 1: Quadratic weighted kappa scores for baseline model evaluation on SFBench, with all models operating in high reasoning effort mode. Performance trends upward with newer model bases.

Stability of model predictions is high, as evidenced by Krippendorff's $\alpha$ surpassing 0.8 for all runs. Nevertheless, even the strongest models presently fall short of inter-expert agreement ceilings, highlighting the benchmark's difficulty and its potential as a litmus test for scientific reasoning progress.

Distinctive Features and Comparative Analysis

SFBench advances the evaluation of scientific feasibility systems in several key aspects:

• Originality of Claims: All benchmark items are novel, SME-generated claims, minimizing spurious memorization.
• Prospective Feasibility Framing: Unlike support/refute tasks (e.g., SciFact [scifact_2020], CoVERt [covert]), feasibility is treated as an expert judgment—prospective and fundamentally subjective, with claims often outside established literature.
• High Complexity Reasoning and Explanation Requirement: The benchmark demands synthesis of experimental, theoretical, and technological constraints, with open-ended, multi-step justification for each score.
• Inclusion of Unsolvable Cases: The dataset explicitly includes both feasible and infeasible claims, paralleling recent unanswerability benchmarks (e.g., AbstentionBench) but targeted at scientific rather than general question answering.
• Expert-Driven Ground Truth: Direct SME involvement throughout claim creation and scoring delivers higher reliability and domain relevance compared to LLM-labeled or crowd-annotated alternatives, as seen in datasets such as Matter-of-Fact [jansen-etal-2025-matter].

Observed Challenges and Implications

Creation of Novel Feasible Claims

The construction of challenging, genuinely novel, but feasible claims proved markedly more difficult for SMEs than infeasible ones. While infeasible claims can be systematically constructed by violating empirically-supported constraints, feasible claim generation is bottlenecked by the need for agreement on reproducibility in unexplored parameter regimes—a noted limitation for expanding benchmarks into frontier scientific domains.

Specialization and Evaluation Scope

SME specialization emerged as a fundamental constraint: claims within narrow subdomains often exceeded the evaluating peer's expertise. This mirrors a critical distinction between human and frontier LLM systems; while the latter remain generalists with moderate competence, human experts demonstrate high performance but in narrower, well-defined topical slices.

Retrieval vs. Reasoning

A recurrent theme in failure analysis is the interplay between information retrieval and pure reasoning. Some claims closely parallel real, but less common, literature; an LLM with access to key sources could readily assign high-precision scores, while for other cases, scientific inference must be made absent any supporting documentation. This dichotomy exposes a fundamental trade-off in the architecture and training of feasibility assessment systems and calls for future experimental separation of retrieval and reasoning deficits.

Implications for AI Reasoning Research

SFBench's design and initial results validate the hypothesis that existing LLMs, even with best-practice prompting, have not yet closed the gap to expert-level scientific feasibility assessment. This supports the continued utility of judgment-oriented datasets for driving advances in reasoning, explanation generation, and domain adaptation. Ongoing trends in the literature—such as multi-tool systems (GAIA [gaia], R-Bench [R-Bench]), retrieval-augmented models (CoVERt [covert]), and agent-based scientific discovery pipelines [ghafarollahi2024sciagentsautomatingscientificdiscovery, baek-etal-2025-researchagent]—will likely benefit from SFBench as a high-difficulty testbed for evaluating longitudinal progress in scientific AI capabilities.

The open-ended nature of the explanation task foregrounds the need for better methods of automating explanation grading, leveraging both metric learning on expert rationales and explicit semantic matching techniques. Furthermore, the multi-dimensional claim annotation—encompassing applied vs. basic science, complexity, modalities, and tool use—provides an opportunity for granular error analysis and pathway-specific model improvement.

Conclusion

SFBench provides a challenging, SME-authored benchmark for evaluating AI-driven scientific feasibility assessment across four materials science domains. Its avoidance of common biases in dataset creation, coupled with high annotation fidelity and rigorous baseline analysis, makes it a difficult yet instructive standard for current and future LLM-based scientific reasoning systems. Improvements in feasibility scoring, the generative quality of explanations, retrieval-reasoning integration, and systematic expansion into additional domains remain fruitful avenues for subsequent research. The benchmark is poised to play a critical role in both the development and evaluation of advanced scientific AI systems.