Code-First Peer Review

• Code-First Peer Review is a paradigm that shifts evaluation from traditional manuscripts to executable research artifacts like code, data, and experiment scripts.
• It introduces standardized claim–evidence contracts and structured review packages that integrate static analysis, dynamic testing, and performance metrics.
• The approach leverages automation and AI to enhance reproducibility and auditability while addressing challenges such as metadata bias and reviewer calibration.

Code-first peer review is a review paradigm in which executable research artifacts—code, data-access logic, environment specifications, experiment scripts, baselines, and minimal claim manifests—become the primary object of evaluation, rather than a manuscript narrative alone. In computational mathematics and adjacent computational fields, the paradigm is motivated by the observation that reviewers will seldom look at attached code under conventional workflows, even when the computational component is decisive for the result; recent work therefore argues for software peer-reviewing processes, structured claim–evidence interfaces, and venue-controlled systems that generate standardized evidence packages for human reviewers (Hanselman, 3 Mar 2025, Chen, 5 Jun 2026).

1. Motivation, scope, and the shift away from manuscript-first review

The immediate rationale for code-first peer review is the increasing dependence of research on large-scale computation. In mathematics, this dependence now includes “terabytes of data, complex algorithms, and automated proof assistants,” and without proper software review, computational proofs can become “black boxes” whose outputs cannot be independently verified. The argument is stated sharply in the mathematical software-review literature: a claim without accessible and reproducible code is “tantamount to a theorem without a written proof” (Hanselman, 3 Mar 2025).

A related critique appears in the protocol literature for computational fields more generally. Traditional peer review is characterized there as a “manuscript-first” workflow in which authors perform experiments, write a prose narrative, optionally upload code and data, and reviewers read the narrative before attempting to inspect the evidence. This produces a “review-interface gap”: the narrative is author-controlled and polished, while the underlying code, configurations, logs, and data remain comparatively opaque. Code-first peer review inverts that ordering through “artifact-first submission,” “venue-controlled processing,” and a standardized “Generated Review Package” for reviewers (Chen, 5 Jun 2026).

The scope of the idea is broader than formal publication venues alone. The same emphasis on executable evidence appears in mathematical software review, in large-scale peer review of code-based assignments, in educational systems that require reviewers to clone and execute peer projects, and in automated code-review pipelines. This suggests a family of practices unified by the proposition that the code, not only the story told about the code, should be directly reviewable.

2. Formalization: executable submissions, claim–evidence contracts, and review packages

A fully specified code-first protocol has been formalized as an executable submission

$S \;=\; (C,\; D,\; E,\; X,\; B,\; M)$

where $C$ is source code, $D$ is data or data-access logic, $E$ is environment specification, $X$ is experiment scripts and configurations, $B$ is baselines, and $M$ is a minimal claim manifest. The venue system transforms this submission into a review package

$R \;=\; (G,\;Q,\;V,\;A,\;F,\;L,\;P)$

where $G$ is the Generated Review View, $Q$ the reproducibility report, $C$0 the claim–evidence matrix, $C$1 the code audit report, $C$2 the baseline fairness report, $C$3 the limitation report, and $C$4 the provenance log (Chen, 5 Jun 2026).

The minimal claim manifest is central because it converts free-form claims into auditable units. Each claim $C$5 is defined as

$C$6

linking a statement to an evidence script, metric, baselines, and an expected direction such as “lower_is_better.” The protocol then defines a claim–evidence mapping: $C$7 This converts review from a largely narrative judgment into a traceable relation between claims, scripts, outputs, seeds, hashes, and caveats (Chen, 5 Jun 2026).

The “Generated Review View” is defined as a venue-controlled, AI-generated “manuscript” that the author cannot directly edit. Its structure includes an abstract-style summary of problem and contributions, a methods summary linked to code modules and key functions, an experiment summary, main results with explicit claim-to-evidence references, limitations, and reproducibility status. Every empirical statement is tagged with a link to the exact command, log file, plot hash, or output CSV that supports it. The Review Package supplements this with build logs, audit reports, fairness checks, and a structured rebuttal form, making the review interface itself standardized rather than author-shaped (Chen, 5 Jun 2026).

3. Review workflow, reproducibility checks, and assessment metrics

In its most concrete operational form, code-first peer review has been described as a six-step workflow for reviewers of mathematical papers with code. The steps are: metadata survey; static analysis; environment setup and reproducibility checks; dynamic testing; results verification; and documentation and reproducibility report. The metadata survey records title, authors, repository URL, code version or download date, and file categories. Static analysis checks for LICENSE, README, and CITATION files, runs a linter such as flake8 or pylint, and inspects code structure. Environment setup follows installation instructions verbatim while logging OS, CPU architecture, RAM, and interpreter or compiler versions. Dynamic testing runs the test suite, measures code coverage with tools like coverage.py, and executes canonical examples from the paper. Results verification compares outputs to mathematical claims, for example by checking $C$8 and measuring $C$9, or by fitting empirical timing data to a claimed $D$0 scaling through $D$1. The final report is structured under four headings: Metadata & Role, Reproducibility, Reliability & Correctness, and Readability (Hanselman, 3 Mar 2025).

The criteria and metrics used in this workflow are correspondingly explicit. Code Quality includes percentage of functions with docstrings, cyclomatic complexity, and linter-reported issues. Correctness includes pass rate of the test suite and consistency of outputs with published tables. Performance includes benchmark runtimes versus claimed scalings and memory consumption. Documentation includes the presence of README, API reference, example notebooks, and usage instructions. These measures do not remove expert judgment, but they make the computational component inspectable in a repeatable way (Hanselman, 3 Mar 2025).

A concrete illustration is provided by the ANTS XVI software review outcomes:

Stage	Reported outcomes	Summary indicator
Submitted	77.6% easy installs, 73.1% readable, 59.3% correct	50.8% both ran & correct
Accepted→Re-review	97.2% easy installs, 82% readable, 82.8% correct	77.1% both ran & correct

The same source describes this as “clear evidence that actionable review feedback raises code quality by ~25 %,” and reports that, in the ANTS and LuCaNT experiments cited by Hanselman, introducing software peer review increased code availability from 89.3 % to 92.3 % and improved installability and correctness rates by over 20 % (Hanselman, 3 Mar 2025).

4. Reviewer obligations and author-facing software standards

Code-first peer review imposes obligations not only on reviewers but also on authors who submit code as a scholarly artifact. For reviewers, recommended practices include enforcing consistent naming such as compute_zeta() rather than cz(), checking directory layout such as /src, /tests, /docs, and /examples, and verifying license compatibility when code imports GPL- versus MIT-licensed packages. Test suites should cover edge cases such as zero-dimensional inputs and random stress tests. Profiling with cProfile or line_profiler is recommended when a paper makes asymptotic claims, since quadratic loops $D$2 may contradict a claimed $D$3 algorithm. Reviewers are also encouraged to inspect commit history, recommend CI services such as GitHub Actions or GitLab CI, and use coverage badges in README files as signals of test completeness (Hanselman, 3 Mar 2025).

For authors, the guidelines are correspondingly concrete. Code should be modularized, for example by grouping core algorithms in src/algorithms and utility routines in src/utils. Authors are advised to follow a style guide such as PEP8 for Python or Google C++ style, and to adopt Test-Driven Development by writing tests/test_<module>.py before implementing functionality. Documentation standards include a detailed README with sections on Installation, Usage, Examples, and Citation; an environment file such as requirements.txt or environment.yml, or a Dockerfile; and inline docstrings in Google or NumPy style. Long-term access requires an open-source license compatible with dependencies and archival of a release on Zenodo to mint a DOI that can be referenced in the paper (Hanselman, 3 Mar 2025).

These prescriptions matter because one of the recurring obstacles to reproducibility is not algorithmic subtlety but infrastructural incompleteness: missing installation instructions, undocumented dependencies, broken links, absent metadata, and unlicensed code. In the code-first framework, such omissions are not peripheral inconveniences; they directly undermine the evidentiary status of computational results.

5. Empirical variants in large-scale and educational peer review

Empirical work on code-centered peer review shows that making code reviewable does not automatically make peer judgments reliable. In a large MOOC study based on the Coursera course “Getting and Cleaning Data,” 22,448 enrollments produced 51,648 total final-project submissions. After excluding invalid links or missing required files and restricting attention to students who received at least one peer grade, the final analytic cohort was 11,222 students. Among 8,741 students with at least two submissions, only 505 received at least two peer grades. Within that group, 92% saw their score increase on a later graded submission, 4% stayed the same, and 4% decreased; the mean first-grade was approximately $D$4, the mean last-grade approximately $D$5, and paired $D$6-tests yielded $D$7. Yet the association between score change and code change was weak: $D$8 for commits and $D$9 for line changes, while a “NO” group with no intervening GitHub commits still had mean $E$0, compared with $E$1 in the “YES” group; the difference was not statistically significant $E$2. The authors conclude that massive scale peer-review scores are highly variable and that allowing unlimited re-submissions boosts average scores but does not guarantee better code (Wang et al., 2021).

A different educational model appears in Socratique, a gamified peer-assessment platform with GenAI assistance for a Functional Programming course. Students upload code or related artifacts, each submission is automatically assigned to three mandatory reviewers on Day D+1, and students may opt into up to three additional optional reviews per session after completing mandatory reviews. Reviewers draft comments in a dashboard where “Mr. Pepper” can analyze a draft against an internal rubric of clarity, relevance, and specificity, using OpenAI’s ChatGPT-3.5-turbo API. The system returns a rubric-based score from 0 to 9 and natural-language rewrite suggestions. In a randomized controlled experiment with 34 Master’s students, the treatment group had full gamification and GenAI UI, while the control group had GenAI assistance hidden and no game elements. Reported outcomes were: Quantity of Feedback, Control $E$3 and Treatment $E$4, with $E$5, $E$6, $E$7; Overall Quality, Control $E$8 and Treatment $E$9, with $X$0, $X$1, $X$2; no significant Condition effect for clarity $X$3; and significant effects for relevance $X$4 and specificity $X$5. The significant Session×Condition interaction for relevance and specificity indicates a learning effect in the control group over time (Wlodarski et al., 3 Apr 2025).

A third pedagogical implementation emphasizes anonymized rubric-based peer assessment before instructor grading. In a large introductory programming course, teams of three developed a 2D maze-runner game over ten weeks, and an automated algorithm anonymously assigned each team two peer projects to review. Reviewers cloned and executed peer projects, played through the levels, evaluated gameplay and code/documentation using a fixed rubric, and submitted both numeric scores and written justifications through a Confluence form accessible only to instructors. Peer assessments aligned moderately with instructor grades: for Peer Review 1, $X$6, $X$7, and $X$8; for Peer Review 2, $X$9, $B$0, and $B$1. Post-review surveys of 47 teams reported that 100% believed their peer evaluations were fair, 83% enjoyed acting as evaluators, and 82% accurately predicted whether peer scores would be higher or lower than instructor grades. The design recommendations that follow include structured, comprehensive rubrics, anonymized automated assignments, dual reviews per submission, mandatory justifications, calibration and training, and transparent aggregation policy (Berrezueta-Guzman et al., 28 May 2025).

Taken together, these studies show that code-first peer review in educational and large-scale settings has at least two distinct empirical dimensions: the requirement that reviewers inspect executable artifacts, and the calibration problem of making peer judgments track substantive code quality rather than resubmission dynamics, surface cues, or uncontrolled reviewer variance.

6. Automation, AI infrastructure, and governance

Automation research has approached code review as three sub-tasks: Review Necessity Prediction (RNP), Review Comment Generation (RCG), and Code Refinement (CR). A recent empirical study proposes a privacy-aware federated model, FedLLM, based on LLaMA-3 8B and Federated Learning, with a multi-task objective

$B$2

where $B$3 is binary cross-entropy for RNP and $B$4 are sequence cross-entropy losses for RCG and CR. The study compares five multi-task strategies: FedTOC, FedCOT, FedCAT, FedCFT, and FedCFT-reg. Sequential methods and the parallel aggregation method suffered catastrophic forgetting; FedTOC’s final $B$5 lost $B$6 and $B$7 capabilities, and FedCOT and FedCAT were unstable. By contrast, the cumulative approach FedCFT-reg matched or exceeded the best individual federated models on all tasks, with approximately $B$8 $B$9 on $M$0, $M$1-BLEU approximately $M$2, METEOR approximately $M$3, and ROUGE-L approximately $M$4 on $M$5, and $M$6-BLEU approximately $M$7, METEOR approximately $M$8, and ROUGE-L approximately $M$9 on $R \;=\; (G,\;Q,\;V,\;A,\;F,\;L,\;P)$0; a Wilcoxon test reported $R \;=\; (G,\;Q,\;V,\;A,\;F,\;L,\;P)$1. The paper therefore recommends cumulative multi-task FL and separate adapters for classification versus generation when tasks differ in nature (Kumar et al., 2024).

In publication-oriented code-first review, AI is assigned a different role: not reviewer replacement, but review infrastructure. The venue-controlled protocol specifies eight stages: submission ingestion; environment reconstruction; static and dynamic audit; experiment execution; result extraction; claim–evidence mapping; generation of the Review Package; and human review plus revision. The system architecture includes a Repository Analyzer, Build & Execution Agent, Result Extractor, Code Audit Agent, Claim Verifier, Fairness & Security Monitor, and Human Reviewer Interface. Reviewers inspect the standardized output rather than reconstructing experiments themselves, and revisions trigger re-execution and an updated Review Package (Chen, 5 Jun 2026).

The governance questions are substantial. The protocol literature identifies narrative manipulation, artifact gaming, prompt injection, metadata bias, model-version drift, compute inequality, false accusation, and over-standardization as explicit threats. Corresponding safeguards include venue-controlled generation with no direct author editing, sandboxed execution with randomization and static plus dynamic audit, adversarial treatment of submission text, metadata blinding and counterfactual audits, model-version logging and calibration suites, reduced verification suites or sampled full audits when compute is limited, human review of flags with an author appeal form, and reviewer override with separate significance review. Final accept or reject decisions remain human decisions. A common misconception is therefore directly rejected by the protocol itself: code-first peer review is not proposed as a way to replace reviewers with AI, but as a way to shift review from polished narrative toward executable evidence while auditing the AI system for bias, reliability, and security (Chen, 5 Jun 2026).

This combination of formal claim–evidence contracts, reproducibility-oriented software review, reviewer calibration, and automation-assisted evidence generation indicates that code-first peer review is not a single method but a developing review regime. Its unifying principle is that computational claims should be reviewable through the artifacts that produce them, with narrative subordinated to executable evidence rather than treated as the primary object of trust.