- The paper demonstrates that response-only loss masking boosts prediction accuracy by 18.5 percentage points in revision cycle classification.
- It introduces a comprehensive dataset of 3,668 multi-round peer-review dialogues across diverse scientific domains.
- FIRSTPASS enables pre-submission critique and optimized editorial triage by grounding predictions in real-world editorial outcomes.
Authoritative Summary of "FirstPass: Grounding AI Scientific Judgment in Multi-Round Editorial Outcomes" (2606.20769)
Context and Motivation
Peer review is increasingly strained by surging submission volumes and stagnating reviewer pools, particularly in high-impact scientific journals. Existing AI approaches to automated peer review have largely been restricted to CS/ML venues, employ static, one-off review generation, and evaluate primarily by stylistic mimicry rather than true editorial judgment. This creates three systemic failures: domain narrowness, disregard for iterative dialogue, and evaluation on proxy metrics.
Dataset and Methodological Innovations
The authors introduce FIRSTPASS, a comprehensive dataset comprising 3,668 multi-round peer-review dialogues from Nature Communications, spanning five domains: biology, chemistry, neuroscience, physics, and earth science. Leveraging Nature Communications' mandatory transparent peer review, the dataset achieves 100% verified content integrity, with detailed audits showing negligible noise in reviewer dialogue extraction.
FIRSTPASS addresses three key tasks:
- Review Generation: Produce detailed peer reviews resembling expert evaluations.
- Reviewer Updating: Simulate reviewer update post author rebuttal.
- Outcome Prediction: Classify manuscripts as requiring STANDARD (2 rounds) or EXTENDED (3+ rounds) revision cycles—a proxy for editorial outcome rigor.
Crucially, outcome labels are grounded in actual editorial decisions, not ratings of generated text, aligning evaluation with real-world scientific judgment. The training protocol utilizes Qwen2.5-7B-Instruct as the base LLM, with LoRA fine-tuning targeting all projection matrices (not just attention), and applies a structured curriculum with response-only loss masking to preserve the classification signal against long input sequences.
Response-Only Loss Masking: Empirical Findings
A decisive methodological result is the empirical demonstration that response-only loss masking is not an optimization but a required architectural principle for long-context, short-output classification tasks. Without masking, model accuracy collapses to 62.0%, below the majority-class baseline (65.4%). With masking, accuracy reaches 80.5%, yielding an 18.5 percentage point swing. This finding is robust across all domains and task types, and has broader implications for LLM fine-tuning regimes where input token volume far exceeds output signal.
Experimental Results
Editorial Outcome Prediction
FIRSTPASS achieves 80.5% accuracy and F1-macro 78.2% on predicting manuscript revision cycle, outperforming Gemini-3.1-flash-lite-preview by 10.4 percentage points (McNemar p < 0.001). Zero-shot Qwen2.5-7B posts an F1-macro of 73.3% (p = 0.185 vs. FIRSTPASS), indicating strong prior capability but with explicit fine-tuning yielding deployability and improved consistency. Few-shot and other zero-shot baselines either regress to the majority baseline or suffer significant performance collapse, especially without loss masking.
Domain-level breakdown shows high consistency (76.9% to 83.8% accuracy), confirming generalizable signals of unresolved reviewer concern across disciplines.
Peer Review Generation
FIRSTPASS generates reviews averaging 1,187 words (closest to human 2,155-word reference), with ROUGE-L 0.154 (p < 0.001 vs baselines). Llama-3-8B achieves higher absolute ROUGE-L (0.164), but this is attributed to formulaic, short outputs with elevated TTR, lacking technical depth. FIRSTPASS reviews contain specific section references, critique aligned with reviewer dialogue, and domain-appropriate technical rigor. Per-domain ROUGE-L confirms robust cross-disciplinary performance.
Practical and Theoretical Implications
Practically, deploying FIRSTPASS as a pre-submission anticipatory reviewer enables authors to simulate expert critique and predict revision cycle outcomes before journal submission. This creates a workflow wherein manuscripts can be strengthened upstream, reducing rebuttal cycle latency and improving overall author experience. For editorial offices, accurate EXTENDED predictions allow triage and assignment of senior reviewers, focusing editorial resources on problematic manuscripts.
Theoretically, the response-only loss masking finding generalizes to any long-input, short-output classification scenario, cautioning against standard token-level loss computation in such regimes. The architecture and training principle presented is portable to a wide class of scientific and medical LLM tasks.
Limitations and Future Directions
FIRSTPASS is limited by the absence of rejected manuscripts and figures/supplementary data, constraining its ability to predict publishability and technical concerns arising from non-text content. The STANDARD/EXTENDED label is a round-count proxy, not a direct measurement of reviewer concern severity. Prospective validation—tracking revision outcomes post FIRSTPASS prediction in real author workflows—remains outstanding. Transferability to other journals with different editorial cultures (e.g., eLife, PLOS ONE) has not yet been validated.
Planned extensions include multimodal document processing, integration of rejected samples once accessible, human annotation of concern severity, and cross-journal validation.
Conclusion
FIRSTPASS establishes a new standard for AI-driven peer review: outcome-grounded, domain-agnostic, multi-round editorial judgment, operationalized in a 7B-scale, open-weight LLM. The model demonstrates that anticipatory scientific judgment is achievable at scale when trained on authentic multi-round dialogues and evaluated against real editorial outcomes. Its key methodological result—response-only loss masking—has immediate applicability beyond peer review, spanning any task involving lengthy scientific documents and concise decision signals. FIRSTPASS occupies the boundary between tool and scientific co-author, setting the stage for further empirical evaluation and deployment in research governance.