SciCoQA: Quality Assurance for Scientific Paper--Code Alignment

Abstract: We present SciCoQA, a dataset for detecting discrepancies between scientific publications and their codebases to ensure faithful implementations. We construct SciCoQA from GitHub issues and reproducibility papers, and to scale our dataset, we propose a synthetic data generation method for constructing paper-code discrepancies. We analyze the paper-code discrepancies in detail and propose discrepancy types and categories to better understand the occurring mismatches. In total, our dataset consists of 611 paper-code discrepancies (81 real, 530 synthetic), spanning diverse computational science disciplines, including AI, Physics, Quantitative Biology, and others. Our evaluation of 21 LLMs highlights the difficulty of SciCoQA, particularly for instances involving omitted paper details, long-context inputs, and data outside the models' pre-training corpus. The best performing model in our evaluation, GPT-5, can only detect 45.7\% of real-world paper-code discrepancies.

• The paper presents SciCoQA, a dataset of 611 discrepancies that evaluates the alignment between scientific publications and their code implementations.
• It employs a novel synthetic data generation method using GPT-5 to simulate plausible paper-code mismatches across diverse computational domains.
• Evaluation of 21 LLMs shows that even the best model detects only 45.7% of real discrepancies, underscoring the need for stronger quality assurance tools.

SciCoQA: Quality Assurance for Scientific Paper--Code Alignment

Introduction and Motivation

The reproducibility crisis in scientific research, especially within computational sciences like AI, underscores the critical need to ensure that published code faithfully implements the methodologies described in scientific papers. The paper "SciCoQA: Quality Assurance for Scientific Paper--Code Alignment" (2601.12910) addresses this issue by presenting the SciCoQA dataset, designed to detect discrepancies between scientific publications and their codebases. This dataset is constructed from real-world data sourced from GitHub issues and reproducibility papers, complemented by synthetically generated data to enhance its scale and coverage across various computational domains.

Data Collection and Dataset Construction

The SciCoQA dataset consists of 611 paper-code discrepancies, divided into 81 real and 530 synthetic instances. Real discrepancies are sourced from reproducibility papers and issues reported in GitHub repositories. The synthetic discrepancies are generated using a novel data generation method employing GPT-5, which constructs plausible paper-code mismatches in AI and other computational domains.

Figure 1: Overview of the data collection process of SciCoQA.

The dataset aims to capture semantic conflicts where the code does not faithfully reproduce the paper's reported method. The discrepancies are categorized into three types: differences (e.g., logical deviations like L1 vs. L2 normalization), paper omissions (where essential details are missing in the paper), and code omissions (where steps described in the paper are absent in the code). This classification enables a detailed analysis of the mismatches occurring in scientific research.

Evaluation and Analysis

The evaluation conducted with 21 LLMs, including state-of-the-art models like GPT-5, highlights the difficulty of the SciCoQA task. The best performing model, GPT-5, could detect only 45.7% of real-world discrepancies, indicating that current models need significant improvements in understanding complex scientific text and code.

Figure 2: Results of the top 8 best performing models (sorted by average recall on the real and synthetic data) on the discrepancy dataset by different analyses.

Further analysis shows a strong correlation between model performance on real and synthetic data, validating the inclusion of synthetic data as a reliable proxy for real-world discrepancies. The dataset includes papers from diverse computational domains, making it a comprehensive resource for developing models that can generalize across different disciplines.

Implications and Future Directions

SciCoQA provides a benchmark for evaluating LLMs' ability to ensure the reliability of scientific research at scale. As the field advances towards automation with AI systems generating scientific outputs, the need for robust quality assurance tools becomes paramount. SciCoQA can serve as a foundation for developing such tools, ensuring that AI-driven scientific discovery remains trustworthy.

Figure 3: Quantitative analysis of synthetic code modifications, highlighting the precision of generated discrepancies.

Future developments could focus on enhancing model capabilities to align paper descriptions with code implementations more accurately. Also, expanding the dataset to cover more languages and computational fields could further improve model generalizability.

Conclusion

The SciCoQA dataset represents a significant step forward in addressing the reproducibility crisis in computational sciences by providing a means to evaluate the alignment between scientific papers and their codebases. Despite the promising insights gained, current LLMs are insufficient for autonomous verification of scientific validity, underscoring a need for continued research and development in this critical area. The findings from SciCoQA highlight the challenges faced by LLMs and set the stage for future innovations in automating quality assurance in scientific research.