EchoReview: Citation-Driven Peer Review

• EchoReview is a citation-context-driven framework that automates peer review by extracting evaluative signals from long-term citation histories.
• It employs a multi-stage data synthesis pipeline to create the EchoReview-16K dataset, producing structured, evidence-backed review samples.
• The EchoReviewer-7B model, fine-tuned on this dataset, achieves state-of-the-art performance in providing comprehensive and consistent automated reviews.

EchoReview

EchoReview is a citation-context-driven framework for large-scale, automated peer review. It systematically mines evaluative signals from the long-term citation history of published scientific papers and transforms these signals into structured review data suitable for supervised fine-tuning of automated reviewers. EchoReview is instantiated via the EchoReview-16K dataset and the EchoReviewer-7B model, which demonstrate that leveraging citation contexts yields state-of-the-art automated reviews with high evidence support and comprehensiveness. This paradigm proposes a scalable alternative to direct human review data by distilling the implicit, collective judgments of the scientific community into machine-consumable supervision (Zhang et al., 31 Jan 2026).

1. Data Synthesis Pipeline: Mining Review Data from Citations

EchoReview employs a multi-stage pipeline to generate review-style data using citation contexts as the primary supervisory signal. The salient pipeline stages are as follows:

1. Paper Collection & Preprocessing: The input corpus encompasses all accepted papers from ACL, EMNLP, ICLR, ICML, and NeurIPS (2020–2022). Metadata and citation counts are retrieved from Semantic Scholar. Only citing papers indexed on ArXiv and referencing the target paper via explicit BibTeX keys are retained. Paper PDFs are converted to Markdown for downstream processing.
2. Citation Context Extraction: For each citation in a citing paper's LaTeX file, EchoReview extracts a window composed of the sentence containing the citation plus its immediate context (one preceding and one following sentence).
3. Implicit Signal Enhancement: Each citation context is processed via LLMs (e.g., GPT-4o) to classify its polarity (strength, weakness, or neutral), answer up to nine diagnostic evaluation questions (categorical: “Yes”, “No”), and convert positive/negative contexts into concise review-style comments. Semantic deduplication is performed so each unique evaluative point is listed once.
4. Chain-of-Thought and Evidence Construction: For each distilled comment, relevant evidence snippets (1–3 verbatim supporting excerpts from the cited paper) are automatically extracted, feeding these into LLMs to generate “Evidence → Analysis → Conclusion” chains. These chains are further audited for faithfulness (citation validity, logical coherence, explanatory quality) using a secondary model, filtering for high-scoring instances.
5. Final Sample Formatting: Each SFT sample contains the cited paper's full text, a set of structured strengths/weaknesses, and accompanying evidence-justified chains-of-thought for each point.

This method utilizes no manual annotation, but rather relies on the community's collective judgment as manifested in the literature, aiming to encode robust, scale-invariant evaluation signals that persist beyond the subjectivity of single-source review data (Zhang et al., 31 Jan 2026).

2. EchoReview-16K: Dataset Construction and Statistics

EchoReview-16K is the concrete instantiation of the EchoReview paradigm, produced entirely via the aforementioned pipeline. Key dataset properties:

• Scale and Coverage: 16,306 review samples covering five conferences across three years (2020–2022). Each entry synthesizes 2–3 strengths and 2–3 weaknesses per paper, each with chain-of-thought rationale.
• Sampling and Filtering: Papers are required to have ≥20 citations, and citing contexts must fall within 1,000 days of publication to focus on active, relevant community engagement. Only ArXiv-indexed citations are included.
• Composition: NeurIPS accounts for 39.4% of samples, with the remainder distributed among ICML, ICLR, EMNLP, and ACL. Yearly distribution is approximately uniform.
• Data Quality Assurance: Each sample must pass a faithfulness audit (logic score ≥8/10, citation validity checks).

A concise summary table:

Metric	Value
Number of samples	16,306
# Conferences	5
Mean strengths/paper	2–3
Mean weaknesses/paper	2–3

EchoReview-16K thus provides a cross-venue, cross-year evidence-grounded review dataset far exceeding the scale and diversity of prior human-annotated review corpora (Zhang et al., 31 Jan 2026).

3. Supervised Reviewer Training: EchoReviewer-7B

EchoReviewer-7B is a 7B-parameter LLM fine-tuned via LoRA on EchoReview-16K, taking as input the full text of a target paper and outputting highly structured JSON reviews. Major technical details:

• Architecture: Qwen2.5-7B-Instruct serves as the backbone, LoRA rank = 8, context window = 8K tokens.
• Input: Full paper text in Markdown plus an explicit review formatting instruction.
• Output: JSON with Strengths and Weaknesses, each containing a “comment” and its chain-of-thought explanation.
• Optimization: Cross-entropy loss, AdamW optimizer (lr=5×10⁻⁵, weight decay=0.01), 3 epochs, early stopping on validation perplexity.
• Training regime: Gradient accumulation yields an effective batch of 32; dropout 0.1 is used.

Through this training architecture, EchoReviewer-7B is conditioned to generate reviews richly grounded in paper text and well-calibrated to the long-term evaluative trajectory of a paper's reception in the field (Zhang et al., 31 Jan 2026).

4. Evaluation Methodology and Results

Model evaluation relies on both held-out test sets and LLM-as-Judge paradigms. Review quality is scored along four axes: Comprehensiveness, Specificity, Evidence Support, and Consistency (0–10 scale per dimension).

• Benchmarks: EchoReview-Bench comprises Test A (general: 1,398 papers) and Test B (human reference: 233 ICLR papers with OpenReview reviews).
• Review Quality: EchoReviewer-7B outperforms previous SFT models (CycleReviewer-8B, DeepReviewer-7B) in Evidence Support (+0.3 absolute) and Comprehensiveness (+0.2). Its overall review quality is 5.65 (mean), close to prompt-engineered GPT-based models (5.7–5.9).
• Research Issue Overlap: When compared to human reviews, EchoReviewer-7B exposes complementary “ModelOnly” issues (0.553 vs. overlapping issues 0.252). This suggests EchoReview identifies long-term, usage-driven concerns less likely to be flagged in initial peer review (Zhang et al., 31 Jan 2026).
• Citation Time Window Analysis: Restricting to citations within 500 days increases Specificity (59.6% win rate) and Consistency (56.2%), but modestly reduces Comprehensiveness and Evidence Support.
• Cross-Paradigm SFT: Fine-tuning EchoReviewer-7B on human review data further improves Specificity and OverallQuality while retaining strong evidence grounding, confirming that hybridization with human-curated supervision is synergistic.

5. Limitations, Biases, and Future Directions

Several critical limitations are recognized:

• Citation Bias: As citation distributions reflect systemic factors (venue prestige, author reputation, disciplinary trends), EchoReview may under-sample novel, uncited, or non-mainstream work.
• Epistemic Homogenization: Aggregation over citation contexts risks alignment toward dominant paradigms, potentially marginalizing unconventional research contributions.
• Fragmented Coherence: As citation contexts are inherently local, global organizational consistency in the generated reviews is limited; Consistency scores lag behind evidence support.
• Human-in-the-Loop Need: Automated reviews are not suitable for publication decisions without accompanying expert human oversight.

Planned mitigations include diverse sampling, public code/dataset release for bias audits, inclusion of multi-modal community signals (e.g., social media), adaptive context weighting, and interactive reviewer UIs (Zhang et al., 31 Jan 2026).

6. Significance and Implications

EchoReview demonstrates that the distributed, persistent judgments of a research discipline, as encoded in citation networks and context, can serve as a robust supervisory signal for scalable automated reviewing. EchoReviewer-7B delivers reviews with high evidence attribution and broad scholarly context, revealing long-term strengths and weaknesses that may not be captured by single-iteration peer review. Integrating this approach with human curation could improve transparency, reduce reviewer load, and enhance the factual foundation of scientific evaluation. However, it is essential to continually audit for entrenched biases, ensure balanced representation, and foreground domain expertise in decision-making (Zhang et al., 31 Jan 2026).