Review the Code, Not the Story: A Vision and Protocol for Code-First Peer Review

Abstract: Peer review in computational fields remains centered on author-written manuscripts, even though the decisive evidence for many claims resides in executable code, data, configurations, and experiment pipelines. This manuscript-first workflow gives authors substantial control over narrative framing while leaving reviewers with limited time to inspect implementation details, reproduce results, or detect unsupported claims. This vision and protocol paper proposes code-first peer review: authors submit executable research artifacts and minimal claim manifests; a venue-controlled AI system builds the environment, executes experiments, audits code paths, maps claims to evidence, and generates a standardized Review Package for human reviewers. The goal is not to replace reviewers or to give authors an automatic writing assistant. Instead, AI serves as review infrastructure that shifts the target of peer review from polished narratives to executable evidence. We formalize a claim-evidence contract, define the Generated Review View and Review Package abstractions, give a worked example, outline a system architecture, and analyze evaluation and governance challenges including AI bias, prompt injection, model instability, auditability, and author appeal.

• The paper introduces a code-first peer review protocol that emphasizes executable artifact validation and AI-mediated reproducibility over traditional narrative reviews.
• The protocol employs a claim-evidence contract and standardized review packages to ensure transparency, defect detection, and fairness.
• Practical implications include enhanced auditability, reduced narrative bias, and potential shifts in research dissemination despite current AI limitations.

Code-First Peer Review: Formalizing Artifact-Driven, Venue-Controlled Scientific Evaluation

Motivation and Problem Formulation

The traditional peer review paradigm in computational research focuses on the evaluation of author-written manuscripts, relegating source code, data, and executable environments to supplementary status. This process introduces a structural misalignment: decisive empirical evidence for computational claims typically resides in artifacts, yet evaluation is predominantly narrative-centric. Authors maintain narrative control, enabling selective disclosure and editorial weighting of claims, baselines, limitations, and failures. Furthermore, while artifact evaluation exists (e.g., ACM badging), it lacks the scalability and integration to supplant narrative as the basis for review.

Recent advances in AI for code analysis and reproducibility verification have not achieved full scientific autonomy—replication rates of leading agents on complex computational research remain below 25% [starace2025paperbench] [siegel2024corebench]—but AI can mechanize substantial portions of evidence extraction and artifact auditing. The compelling institutional opportunity lies in re-engineering peer review's interface: shifting evidentiary focus from the author-crafted manuscript to standardized, venue-generated reports grounded in executable artifacts.

The proposed protocol, termed "code-first peer review," formalizes this shift. Authors submit an executable package $S$—including source code, data (or access logic), environment specification, experiment scripts, baselines, and a structured claim manifest—instead of a polished manuscript. The venue, via controlled AI agents, audits, executes, and analyzes the submission, producing a Review Package $R$ comprising a Generated Review View (GRV), reproducibility report, claim-evidence matrix, code audit, baseline fairness analysis, limitations, and provenance. The authors cannot directly modify the GRV; all changes must traverse artifact or manifest revision, each triggering re-execution and explicit logging.

Technical Design: Protocol and Architecture

Claim-Evidence Contract and Manifest

A minimal claim manifest structures each scientific claim with explicit evidence scripts, metrics, baseline definitions, expected metric directions, hardware/compute requirements, and enumerated limitations. This contract grounds claims in declared empirical evidence and constrains narrative embellishment.

Generated Review View and Review Package

The GRV is a venue-controlled, AI-generated, manuscript-like document that exposes: abstract, summary of methods and experiments, main results, explicit claim-evidence references, system-detected limitations, and detailed reproducibility status. Each empirical statement in the GRV must be anchored in code, configuration, executable script, log, or hash; unsupported claims are visually and textually isolated.

The Review Package (RP) systematically documents:

• Execution logs, container/environment specifications, hardware metadata, resource usage, random seeds, and all output hashes
• Baseline comparisons with code and execution path audits
• Defect detection (e.g., hardcoded results, data leakage, baseline/fairness discrepancies)
• Explicit mapping of claims to supporting (or negating) evidence
• Limitation report with provenance and manifest traceability

System Components

• Repository Analyzer: Parses submissions to build dependency graphs, enumerate experiment entry points, and map configurations.
• Build and Execution Agent: Reconstructs environments, launches experiments, enforces resource quotas, and maintains strong sandboxing.
• Result Extraction Agent: Harvests and verifies metrics, logs, and outputs, supporting both deterministic and stochastic regimes.
• Claim Verification Agent: Constructs and maintains the claim-evidence matrix with status and caveat fields.
• Fairness and Security Monitor: Enforces metadata blinding, detects prompt injection, and audits for adversarial submission content or compute/time tampering.
• Human Reviewer Interface: Reviewer interacts with the Review Package, tracing claims to evidence and surfacing unsupported or partially supported results.

Evaluation and Governance

Evaluation Criteria

The paper stipulates a rigorous multi-faceted evaluation methodology:

• Faithfulness: Measures hallucinated-claim rate, unsupported generalization, missing limitations, and failure to link claims/evidence.
• Reproducibility: Assesses environment build/run success, reproduction error, dependency repair frequency, and provenance precision.
• Defect Detection: Evaluates agent precision/recall on synthetic and real injected defects.
• Reviewer Utility: Benchmarks time to review, richness of evidence-backed critique, and reproducibility-driven confidence.
• Fairness and Robustness: Uses counterfactual audits to quantify sensitivity to author/institution metadata and model-version drift, as per $$\Delta_{meta} = \max_{m_i,m_j} | s(S,m_i) - s(S,m_j) |$$.
• Human Judgment Boundary: Enshrines human authority for judgments of novelty, significance, and field advancement.

Baseline comparisons include status quo manuscript-only review, artifact review sans AI, and author-controlled AI-generated papers.

Governance and Threat Model

The protocol embeds safeguards against:

• Narrative Manipulation: No direct author edits to venue-generated texts.
• Prompt Injection: Submission text is sanitized and scanned for hidden instructions.
• Metadata Bias: Blinding with counterfactual fairness audits (e.g., LLM acceptance rate differences with author/institution permutations [vonwedel2024affiliation]).
• Model Drift: Model versioning, calibration suites, and redundancy in adjudication.
• Resource Inequality: Recorded budgets and partial verification regimes for computationally intensive submissions.
• Appeals: Structured author appeals for AI misjudgment, logged and version-controlled.

Theoretical and Practical Implications

Theoretical Implications

The code-first peer review model reorients peer review theory: narrative persuasion yields to auditability, provenance, and empirical grounding. The notion of the "claim-evidence contract" recasts the epistemology of computational science, offering formal traceability and bounding the risk surface of unsupported claims. This protocol also aligns with reproducibility movements but exceeds artifact badging by instantiating artifact execution, evidence extraction, and standardized reporting as the basis of review, not supplements to it.

Practical Implications

Adoption trajectories include optional artifact checklists (Level 1), pre-review reproducibility reports (Level 2), and ultimately, venue-generated Review Packages as the review object's primary basis (Level 3/4). The pilot application to communication systems research highlights both the feasibility (e.g., standardized topologies, channel models, benchmarks) and the necessity for confidential/private artifact handling in proprietary deployments.

The protocol is compatible with diverse post-acceptance publication strategies: dual publication of narrative and evidence package, executable articles, or hybrid communication+evidence artifacts, depending on venue policy and community standards.

AI-mediated review infrastructure introduces new attack surfaces—prompt injection, environment tampering, and model bias—but the protocol's design asserts that these can be governed more easily than narrative-driven fraud, provided strong auditability and continuous red-teaming are prioritized.

Limitations and Future Directions

Notable limitations include current limits on AI's competency for fully autonomous evidence extraction and build automation, the infeasibility of full-scale reruns for resource-heavy experiments, potential inequities for theoretical or non-executable work, and evolving challenges in model governance and red-teaming. The proposal's effect on longitudinal norms of publication remains undetermined and will require field trials with iterative calibration.

Future directions include formal specification of claim-evidence contracts for new research fields, expansion of build/execution infrastructure to handle confidential and proprietary artifacts, continuous benchmarking of AI reproducibility agents, and collaborative standardization with artifact badge-granting bodies.

Conclusion

This paper advances a formal protocol and reference architecture for code-first, venue-controlled peer review in computational research, positing the executable artifact and structured claim manifest as the primary substrate for evaluation. Venue-mediated artifact execution and evidence extraction serve to standardize, de-bias, and increase the auditability of scientific claims, shifting the reviewer burden from narrative parsing to empirical validation. Robust governance and institutional adaptation remain essential to harnessing the benefits of this transition while mitigating emergent risks in AI-involved peer review.

---

References

• "Review the Code, Not the Story: A Vision and Protocol for Code-First Peer Review" (2606.07683)
• "PaperBench: Evaluating AI's Ability to Replicate AI Research" (Starace et al., 2 Apr 2025)
• "CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark" (Siegel et al., 2024)
• "Affiliation Bias in Peer Review of Abstracts by a LLM" [vonwedel2024affiliation]
• "Hidden Prompts in Manuscripts Exploit AI-Assisted Peer Review" (Lin, 8 Jul 2025)
• "Automated Article Generation" (Harper, 2024)
• "The AI Scientist" (Lu et al., 2024)
• "Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning" (Seo et al., 24 Apr 2025)
• and others as detailed in the source document.