--- title: 'DeepReview-13K: Structured Review Dataset' url: https://www.emergentmind.com/topics/deepreview-13k-dataset type: topic --- # DeepReview-13K: Structured Review Dataset DeepReview-13K is a curated benchmark for multi-stage, structured scientific paper reviewing, pairing complete academic manuscripts with exhaustive review information, fine-grained reviewer annotations, and explicit chain-of-thought reasoning traces. This dataset consists of 13,378 samples—each capturing an ICLR 2024 or 2025 submission with its full structured review data—intended to train and evaluate LLMs in expert-level, evidence-based paper assessment and meta-review [2503.08569]. ## 1. Dataset Composition and Scope DeepReview-13K comprises 13,378 unique paper–review instances. Each sample includes the following elements: - Full paper text in parseable Markdown (from arXiv source or PDF converted with MinerU) - A review set $R$, an array of reviewer entries, formally: - Strengths, weaknesses, and questions (free text) - Author rebuttal excerpts (dialogue text) - Three fine-grained scores per reviewer: soundness $\in \{1,2,3,4\}$, presentation $\in \{1,2,3,4\}$, contribution $\in \{1,2,3,4\}$ - Overall rating $R \in [1, 10]$ - Accept/reject outcome (binary) - Meta-review and final meta-rating ($\in [1,10]$), meta-decision (accept/reject) - Three explicit “thinking” steps (intermediate rationales): - $z_1$: Novelty verification (summary of literature comparison) - $z_2$: Multi-dimension review (reconstruction integrating reviewer commentary and rebuttal) - $z_3$: Reliability verification (validation against the manuscript, with confidence attribution) This structure is designed to provide a comprehensive, end-to-end view of the peer review decision process, including both human and model-synthesized rationale [2503.08569]. The average combined length per sample (paper plus review) is approximately 10,178 tokens. ## 2. Collection and Quality Control Pipeline Samples are sourced by cross-linking ICLR 2024 and 2025 submission records from arXiv PDFs/LaTeX (converted to Markdown) and reviews, rebuttals, and meta-reviews from OpenReview. Papers are included only if all the following are present: - Parsed Markdown manuscript - At least one full reviewer entry (with required categories and scores) - Author rebuttal text - Published meta-review and final decision For each included paper, automated agents (LLMs) generate $z_1$, $z_2$, $z_3$, and a finalized meta-review using explicit system prompts. After generation, all samples undergo automated QC with Qwen-2.5-72B-Instruct: 1. **Logical Consistency**: Sanity-checking for contradictions across the $z_1$–$z_3$ chain and final scores. 2. **Completeness**: Filtering out missing fields or incomplete entries. Samples failing these gates are excluded, ensuring high internal dataset quality [2503.08569]. ## 3. Annotation Schema and Rationale Modeling Each review in DeepReview-13K is decomposed into the following annotation fields: - *Strengths*, *Weaknesses*, *Questions*: Reviewer-written free-form text. - *Soundness*, *Presentation*, *Contribution*: Integer ratings, each $\in \{1,2,3,4\}$. - *Overall Rating*: $R \in [1,10]$ (floating/integer). - *Accept/Reject*: Final recommendation per review. - *Author Rebuttal*: Excerpted dialogue. - *Meta-review, Meta-rating, Meta-decision*: Synthesis by area chairs or model agent. - *Chain-of-Thought* (z₁, z₂, z₃): Explicit fields capturing structured intermediate reasoning: - $z_1$ (Novelty Verification): Systematic literature retrieval and originality analysis. - $z_2$ (Multi-dimension Review): Professional reformulation and enrichment, integrating reviewer/author dialogue, adhering to requirements for depth, actionable feedback, and technical rigor. - $z_3$ (Reliability Verification): Stepwise evidence validation for each reviewer point, referring back to the manuscript and scoring comment strength. The review process is formalized as follows: $$ p(a|q) \propto \int p(a|z_{1:3},q) \; \prod_{t=1}^{3} p(z_t|z_{