NAIDv2: Peer-Review Dataset Benchmark

• NAIDv2 is a large-scale, structured peer-review dataset that mitigates reviewer inconsistencies using domain-year grouping and confidence-aware aggregation.
• It aggregates 24,276 ICLR submissions from 2021 to 2025, including comprehensive metadata, parsed content, and semantic domain assignments.
• NAIDv2 supports the NAIPv2 framework through pairwise comparisons and a continuous review tendency signal for robust automated paper quality estimation.

NAIDv2

NAIDv2 is a large-scale, structured peer-review dataset specifically constructed to support the development and evaluation of debiased, efficient paper quality estimation systems. It is introduced in conjunction with NAIPv2, a pairwise learning framework for automated scientific paper assessment, and constitutes both an operational benchmark and a research resource (Zhao et al., 29 Sep 2025). NAIDv2 is motivated by the shortcomings of existing pointwise regression approaches to review-score prediction, which are hampered by inter-domain, inter-year, and reviewer-confidence inconsistencies in peer-review labels. It provides the first method-aligned dataset designed to address these issues through domain-year grouping and confidence-aware aggregation within a systematically curated ICLR corpus.

1. Dataset Construction and Scope

NAIDv2 aggregates peer-review data from ICLR (International Conference on Learning Representations) submissions spanning five years (2021–2025), comprising 24,276 papers. Each entry is enriched with comprehensive metadata, parsed document structure, reviewer scores and confidences, semantic domain assignments, and derived supervision signals. The dataset is constructed using a structured multi-stage ingestion and normalization pipeline:

• Submission retrieval and review linkage from OpenReview.net, including full review text and scores.
• Structured storage in an SQL-backed engine, with dedicated controller, service, and DAO layers.
• Paper PDF acquisition and parsing via MinerU to extract detailed content.
• Semantic embedding of title and abstract text using Qwen3-Embedding-4B and hierarchical clustering to assign latent research domains.
• Normalization of review scores and confidences to standardized continuous scales.

For experimental integrity, NAIDv2 employs a temporal leakage-aware split: the test set is composed exclusively of ICLR 2025 submissions (1,029 samples), while the remaining 23,247 papers comprise the training and internal validation partition. This split ensures minimal overlap with known LLM pretraining corpora, addressing concerns raised with earlier benchmarks such as PeerRead and Re².

2. Data Schema and Content

The NAIDv2 schema reflects its focus on multi-level annotation and review structure. Two principal relational tables, Papers and Reviews, capture the following (partial fields):

Entity	Field	Type	Description
Papers	id	BIGINT	Submission identifier
	title	VARCHAR(255)	Paper title
	abstract	TEXT	Abstract
	external_id	VARCHAR(255)	OpenReview ID
	openreview_venue	VARCHAR(255)	Venue name
	scores	JSON	Reviewer scores
	confs	JSON	Reviewer confidences
	pub_year	INT	Publication year
	embedding	JSON	Semantic embedding
	cluster_cat	VARCHAR(64)	Assigned domain label
Reviews	id	BIGINT	Review identifier
	paper_id	BIGINT	Foreign key to Papers.id
	conf_score	FLOAT	Reviewer confidence value
	rec_score	FLOAT	Reviewer recommendation
	review_summary	TEXT	Structured review text
	strengths	TEXT	Claimed strengths
	weakness	TEXT	Claimed weaknesses

Each submission may have multiple review records, each with potentially varying recommendations and confidences. Further fields capture reply threading, rebuttal linkage, and full review text. Parsed PDF structure is retained beyond what is used in the main modeling experiments.

3. Labeling and Supervision: The Review Tendency Signal (RTS)

The primary supervised target for NAIDv2 is the Review Tendency Signal (RTS), a continuous, probabilistically aggregated label computed from raw reviewer scores and confidence levels. Under the RTS construction,

$$p(s_i \mid x, c_i) = \mathcal{N}(s_i \mid x, \sigma(c_i)^2), \qquad \mathrm{RTS} = \mathbb{E}[x \mid s_{1:n}, c_{1:n}]$$

where $s_i$ is a review score, $c_i$ is a reviewer confidence (normalized to $[0,1]$), $x$ is the latent paper quality, and $\sigma(c)$ is a non-increasing variance function of confidence (e.g., $\sigma_{\text{smooth}}(c) = 0.2(1-c) + 0.05$). This construction treats reviewer confidence as an inverse uncertainty weight in the aggregation. Extensive ablations show that RTS outperforms mean, median, mode, and simple weighted averages as a training target for automatic quality estimation systems.

NAIDv2 also provides binary accept/reject labels when available, but its methodological core is the provision of a continuous, debiased supervision target encompassing reviewer uncertainty.

4. Debiased Pairwise Learning and Domain-Year Grouping

A defining property of NAIDv2 is its explicit encoding of both semantic domains and publication years via hierarchical clustering of title/abstract embeddings and metadata fields. This enables pairwise preference generation under the critical restriction that only submissions within the same domain-year group are compared, mitigating issues with inconsistent scoring scales across different research areas and temporal cohorts.

For any valid paper pair $(a, b)$ within the same domain-year group, NAIDv2 forms a binary comparison label:

$$\mathrm{RTS}_{ab} = \mathbb{I}[\,\mathrm{RTS}_a > \mathrm{RTS}_b\,]$$

Pairs with small RTS gaps can be downsampled or excluded to control labeling noise, with a default minimum difference of $0.05$ empirically optimal.

This pairwise data structure is foundational for NAIPv2’s training: the model is optimized to order papers correctly within comparable subpopulations, rather than attempting incoherent global scale fitting.

5. Benchmarking, Model Use, and Evaluation Protocols

NAIDv2 serves as both a training resource and a public benchmark for automated peer-review systems, particularly the NAIPv2 framework. NAIPv2 is trained using pairwise comparisons induced from NAIDv2 but deployed as a pointwise regressor for efficiency. Core inputs are title and abstract; inclusion of full-structured content does not improve performance in mainline evaluations.

The benchmark supports a range of evaluation protocols:

• Leakage-aware temporal test split: The primary ICLR evaluation is on papers from 2025, with training from prior years.
• Cross-venue transfer: NAIDv2-trained models are evaluated on a 13,223-sample NeurIPS set to test generalization.
• Metrics: Area Under Curve (AUC) for acceptance prediction, Spearman’s $s_i$0 for ranking correlation, pairwise accuracy for ordinal comparisons, F1, and NDCG@20 for top-$s_i$1 relevance.
• Debiasing ablations: Grouping by hierarchical cluster and year outperforms time, keyword, or ungrouped pair construction, confirming the efficacy of the dataset’s design for removing systemic scale bias.

The dataset saturates in training efficacy at approximately $s_i$2 pairwise comparisons.

6. Biases, Limitations, and Practical Considerations

While NAIDv2 is constructed to address many of the confounders in prior peer-review datasets, several residual challenges are noted:

• Domain and selection bias: As a single-venue dataset, NAIDv2 reflects ICLR’s research and review culture, limiting immediate applicability to other scientific disciplines.
• Reviewer noise: RTS models confidence-uncertainty but does not remove the subjectivity of human review; label noise is not eliminated.
• Temporal drift: Grouping by year reduces but does not eradicate the impact of evolving community standards.
• Information leakage: Test set isolation to 2025 mitigates, but cannot guarantee, exclusion from LLM pretraining data.
• Rich content not default: Although structured full paper content is stored, the main modeling protocols use only titles and abstracts.
• Ethical limitations: Released for research purposes only, with an explicit caution against using model outputs as objective measures of scientific merit.

7. Comparative and Community Impact

Relative to earlier resources such as PeerRead and Re², NAIDv2 offers superior recency, scale, leakage awareness, domain-aware organization, and confidence-integrated labeling. Unlike prior datasets with fixed accept/reject or pointwise review-score targets, NAIDv2’s pairwise, domain-normalized, confidence-weighted structure enables the training of models whose inferences are robust to reviewer and domain inconsistencies, and whose efficiency is suitable for both large-scale benchmarking and real-world deployment pipelines.

Access is provided via sway.cloud.microsoft/Pr42npP80MfPhvj8 (Zhao et al., 29 Sep 2025). The dataset forms the empirical bedrock for the current state-of-the-art in peer-review prediction benchmarks and is designed to be extensible for future methodological innovation in scientific content assessment.