EchoReview-16K: Citation-Driven Review Dataset

• EchoReview-16K is a citation-context-driven review dataset that systematically transforms long-term academic citation signals into structured, evidence-based review samples.
• It utilizes a multi-stage data synthesis pipeline to extract, enhance, and organize citation contexts with explicit chain-of-thought rationale for both strengths and weaknesses.
• The dataset underpins automated peer review benchmarks and is integral to fine-tuning LLM reviewers like EchoReviewer-7B, showcasing improved evidence support and specificity.

EchoReview-16K is a large-scale, citation-context-driven scientific review dataset created through the EchoReview framework for automated peer review and meta-scientific research. It systematically transforms long-term community judgment, as expressed in citation contexts spanning multiple years and venues, into structured review data augmented with evidence and chain-of-thought rationales. The dataset serves as both a resource for benchmarking peer review automation and a foundation for training LLMs that emulate detailed, evidence-supported scientific review behavior (Zhang et al., 31 Jan 2026).

1. Data Synthesis Pipeline

The EchoReview-16K dataset is produced by a multi-stage data synthesis pipeline designed to convert raw academic citation signals into review-style samples with explicit supporting evidence and stepwise reasoning. The pipeline comprises four principal stages:

A. Paper Collection & Preprocessing

• Retrieves all accepted papers from five major conferences—ACL, EMNLP, ICLR, ICML, and NeurIPS—for 2020–2022.
• Each cited paper is mapped to its Semantic Scholar record, annotated with paperId, publication date, and citation count.
• For every cited paper, all ArXiv-cited papers are sourced, along with associated LaTeX and BibTeX files.
• Cited paper PDFs are converted (via MinerU) to Markdown, omitting author details, acknowledgments, references, and appendices.

B. Citation Context Extraction

• Each citing paper is scanned for citation marks referencing a given cited paper.
• For each citation mark, a three-sentence window ($\{s_{-1}, s_0, s_{+1}\}$) is extracted to ensure logical and polarity preservation.
• Contexts are associated with citation-paper publication lag ($\Delta t$) and raw text.

C. Implicit Signal Enhancement

• Each citation context is processed via LLMs (e.g., GPT-4o) and rule-based filters for:
  • Polarity classification ($\ell_\text{strength}$, $\ell_\text{weakness}$, $\ell_\text{neutral}$).
  • Style conversion to reviewer-suitable “Strength”/“Weakness” commentary.
  • Diagnostic probing across nine axes (e.g., adoption, critique) to surface deeper evaluative content.
  • Semantic deduplication to retain the most comprehensive, non-redundant insights.

D. Training Data Construction with Chain-of-Thought (CoT)

• For each review comment, supporting verbatim evidence is extracted from the cited paper, paired with succinct justifications.
• GPT-4o composes chain-of-thought stepwise rationales, each quoting and analyzing evidence, culminating in a strength or weakness conclusion.
• Qwen-max cross-validates each sample for citation fidelity, logical coherence, and explanatory quality, retaining only those above a faithfulness threshold.
• Final data are packaged as JSON objects comprising system prompt, paper text, and Strength/Weakness statements with CoT.

2. Dataset Statistics and Structure

EchoReview-16K is characterized as the first cross-conference, cross-year, citation-driven review corpus. Its scope and distribution are as follows:

• Total samples: 16,306 (each consisting of a single Strength or Weakness comment plus CoT).
• Conference coverage (2020–2022):
  • ACL: 12.3% (~2,005 samples)
  • EMNLP: 13.8% (~2,252)
  • ICLR: 14.3% (~2,334)
  • ICML: 20.2% (~3,293)
  • NeurIPS: 39.4% (~6,422)
• Temporal distribution:
  • 2020: 19.7% (~3,212)
  • 2021: 31.2% (~5,089)
  • 2022: 49.1% (~7,995)
• Average contexts per cited paper: ~7.4
• Review dimension breakdown (by automated LLM labeling):
  • Evidence Support–oriented: ~45%
  • Comprehensiveness–oriented: ~30%
  • Novelty/Originality–oriented: ~25%

This design yields broad coverage across the machine learning and NLP domains, with a tilt toward highly cited and actively discussed topics (Zhang et al., 31 Jan 2026).

3. Model Architecture and Training Paradigm

EchoReview-16K is the primary supervised fine-tuning (SFT) dataset for EchoReviewer-7B, an LLM-based automated reviewer. EchoReviewer-7B is produced via LoRA-based fine-tuning of Qwen2.5-7B-Instruct on the EchoReview-16K set.

Key architectural and training details:

• Base Model: Qwen2.5-7B-Instruct (7B parameter transformer)
• LoRA Configuration: rank 8, α=16, dropout=0.1
• Hardware: 2 × NVIDIA RTX A6000 (48 GB GPUs)
• Framework: LLaMA-Factory
• Data Split: 80% train (13,044 samples), 10% validation (1,631), 10% test (1,631)
• Hyperparameters: batch size 32/GPU, learning rate 3×10⁻⁵ (linear warmup, cosine decay), 3 epochs, max sequence 4,096 tokens

The workflow explicitly packages each review sample with stepwise rationales anchored in paper evidence, facilitating evidence-based review generation.

4. Evaluation Metrics and Experimental Results

EchoReviewer-7B is benchmarked against prior automated reviewers and frontier LLMs. Evaluation covers core review dimensions—Comprehensiveness, Specificity, Evidence Support, and Consistency—scored 0–10 per axis by automatic evaluation (Gemini-2.5-Pro, 3 runs).

Summary of main findings (Test Set A, 1,398 papers):

Model	Comp.	Spec.	Ev. Sup.	Cons.	Overall
Claude 4.5	5.4	5.8	5.2	6.0	5.6
DeepSeek-R1	5.6	6.1	5.1	6.3	5.8
Gemini 3	5.8	6.2	5.3	6.4	6.0
GPT-5	6.0	6.4	5.4	6.5	6.1
CycleReviewer 8B	5.2	5.6	5.0	5.8	5.4
DeepReviewer 7B	5.5	6.0	5.5	6.1	5.8
EchoReviewer-7B	6.2	6.3	6.1	6.2	6.2

• EchoReviewer-7B uniquely leads on Evidence Support (+0.6 margin).
• Weakness coverage analysis (Test Set B, 233 ICLR papers):
  • Overlap with human reviews: 0.252
  • “Human-only” unique issues: 0.195
  • “Model-only” unique issues: 0.553

This suggests that EchoReviewer-7B systematically discovers long-term, usage-driven limitations overlooked by human reviewers, highlighting the complementarity of citation-driven automated critique.

5. Comparative Analysis and Citation Temporal Effects

Pairwise ablation studies probe the impact of citation time span (“NoTimeFilter” vs. “TimeFilter”) on review quality dimensions and overall breadth:

Dimension	NoTimeFilter wins	TimeFilter wins
Comprehensiveness	59.3%	40.7%
Specificity	40.4%	59.6%
Evidence Support	53.8%	46.2%
Consistency	43.8%	56.2%
Overall	52.7%	47.3%

Longer-horizon citation contexts improve breadth and evidence support but slightly diminish focus and logical coherence, a dynamic inherent to dataset construction from multi-year citation chains (Zhang et al., 31 Jan 2026).

Fine-tuning EchoReviewer with DeepReview-13K further increases specificity (+0.2) without degradation in other metrics, suggesting that incremental fusion with conventional review corpora is synergistic.

6. Qualitative Review Examples and Data Characteristics

EchoReview-16K review samples are characterized by explicit evidence anchoring (references to equations, tables), long-term community focus (scalability, robustness), and actionable feedback (e.g., requests for ablation studies). For instance:

• Strength: “The paper formalizes slot consistency in Eq. (2) … aligning directly with the ERR metric in Eq. (15).”
• Weakness: “It is unclear how much of the gains arise from KNN retrieval vs. the IRN network; Table 4 isolates IRN on Laptop only. Suggest adding KNN-only vs. IRN-only ablations across all domains.”
• Weakness: “The ERR metric treats slots as set membership; it fails to capture slot-value misalignment after lexicalization. Recommend a value-level accuracy metric.”

Such samples demonstrate the dataset’s focus on reproducibility, fine-grained analysis, and experiment-driven suggestions, aligning with high standards of scientific review.

7. Limitations, Biases, and Implications

EchoReview-16K and its associated modeling paradigm experience several limitations and biases:

• Citation-driven data disproportionately represent well-cited venues and dominant research paradigms, risking under-coverage for low-visibility or interdisciplinary contributions.
• Aggregation of long-term community sentiment may encode collective biases, potentially diluting recognition for disruptive or high-risk research.
• Fragmentation of citation contexts necessitates sophisticated coherence planning; residual inconsistencies may persist at the sample level.
• The intrinsic reliance on accepted papers and their citation chains constrains coverage of borderline, controversial, or rapidly changing research topics.

A plausible implication is that EchoReview-16K serves as a scalable complement—rather than a replacement—to traditional peer review, extending coverage to latent, community-wide judgments and evidence-supported meta-critique (Zhang et al., 31 Jan 2026).