---
title: 'NAIDv2: Peer-Review Dataset Benchmark'
url: https://www.emergentmind.com/topics/naidv2
type: topic
---

# NAIDv2: Peer-Review Dataset Benchmark

NAIDv2

NAIDv2 is a large-scale, structured peer-review dataset specifically constructed to support the development and evaluation of debiased, efficient paper quality estimation systems. It is introduced in conjunction with NAIPv2, a pairwise learning framework for automated scientific paper assessment, and constitutes both an operational benchmark and a research resource [2509.25179]. NAIDv2 is motivated by the shortcomings of existing pointwise regression approaches to review-score prediction, which are hampered by inter-domain, inter-year, and reviewer-confidence inconsistencies in peer-review labels. It provides the first method-aligned dataset designed to address these issues through domain-year grouping and confidence-aware aggregation within a systematically curated ICLR corpus.

## 1. Dataset Construction and Scope

NAIDv2 aggregates peer-review data from ICLR (International Conference on Learning Representations) submissions spanning five years (2021–2025), comprising 24,276 papers. Each entry is enriched with comprehensive metadata, parsed document structure, reviewer scores and confidences, semantic domain assignments, and derived supervision signals. The dataset is constructed using a structured multi-stage ingestion and normalization pipeline:

- Submission retrieval and review linkage from OpenReview.net, including full review text and scores.
- Structured storage in an SQL-backed engine, with dedicated controller, service, and DAO layers.
- Paper PDF acquisition and parsing via MinerU to extract detailed content.
- Semantic embedding of title and abstract text using Qwen3-Embedding-4B and hierarchical clustering to assign latent research domains.
- Normalization of review scores and confidences to standardized continuous scales.

For experimental integrity, NAIDv2 employs a temporal leakage-aware split: the test set is composed exclusively of ICLR 2025 submissions (1,029 samples), while the remaining 23,247 papers comprise the training and internal validation partition. This split ensures minimal overlap with known LLM pretraining corpora, addressing concerns raised with earlier benchmarks such as PeerRead and Re².

## 2. Data Schema and Content

The NAIDv2 schema reflects its focus on multi-level annotation and review structure. Two principal relational tables, Papers and Reviews, capture the following (partial fields):

| Entity  | Field            | Type           | Description                  |
|---------|------------------|----------------|------------------------------|
| Papers  | id               | BIGINT         | Submission identifier        |
|         | title            | VARCHAR(255)   | Paper title                  |
|         | abstract         | TEXT           | Abstract                     |
|         | external_id      | VARCHAR(255)   | OpenReview ID                |
|         | openreview_venue | VARCHAR(255)   | Venue name                   |
|         | scores           | JSON           | Reviewer scores              |
|         | confs            | JSON           | Reviewer confidences         |
|         | pub_year         | INT            | Publication year             |
|         | embedding        | JSON           | Semantic embedding           |
|         | cluster_cat      | VARCHAR(64)    | Assigned domain label        |
| Reviews | id               | BIGINT         | Review identifier            |
|         | paper_id         | BIGINT         | Foreign key to Papers.id     |
|         | conf_score       | FLOAT          | Reviewer confidence value    |
|         | rec_score        | FLOAT          | Reviewer recommendation      |
|         | review_summary   | TEXT           | Structured review text       |
|         | strengths        | TEXT           | Claimed strengths            |
|         | weakness         | TEXT           | Claimed weaknesses           |

Each submission may have multiple review records, each with potentially varying recommendations and confidences. Further fields capture reply threading, rebuttal linkage, and full review text. Parsed PDF structure is retained beyond what is used in the main modeling experiments.

## 3. Labeling and Supervision: The Review Tendency Signal (RTS)

The primary supervised target for NAIDv2 is the Review Tendency Signal (RTS), a continuous, probabilistically aggregated label computed from raw reviewer scores and confidence levels. Under the RTS construction,

\[
p(s_i \mid x, c_i) = \mathcal{N}(s_i \mid x, \sigma(c_i)^2), \qquad \mathrm{RTS} = \mathbb{E}[x \mid s_{1:n}, c_{1:n}]
\]

where $s_i$ is a review score, $c_i$ is a reviewer confidence (normalized to $[0,1]$), $x$ is the latent paper quality, and $\sigma(c)$ is a non-increasing variance function of confidence (e.g., $\sigma_{\text{smooth}}(c) = 0.2(1-c) + 0.05$). This construction treats reviewer confidence as an inverse uncertainty weight in the aggregation. Extensive ablations show that RTS outperforms mean, median, mode, and simple weighted averages as a training target for automatic quality estimation systems.

NAIDv2 also provides binary accept/reject labels when available, but its methodological core is the provision of a continuous, debiased supervision target encompassing reviewer uncertainty.

## 4. Debiased Pairwise Learning and Domain-Year Grouping

A defining property of NAIDv2 is its explicit encoding of both semantic domains and publication years via hierarchical clustering of title/abstract embeddings and metadata fields. This enables pairwise preference generation under the critical restriction that only submissions within the same domain-year group are compared, mitigating issues with inconsistent scoring scales across different research areas and temporal cohorts.

For any valid paper pair $(a, b)$ within the same domain-year group, NAIDv2 forms a binary comparison label:

\[
\mathrm{RTS}_{ab} = \mathbb{I}[\,\mathrm{RTS}_a > \mathrm{RTS}_b\,]
\]

Pairs with small RTS gaps can be downsampled or excluded to control labeling noise, with a default minimum difference of $0.05$ empirically optimal.

This pairwise data structure is foundational for NAIPv2’s training: the model is optimized to order papers correctly within comparable subpopulations, rather than attempting incoherent global scale fitting.

## 5. Benchmarking, Model Use, and Evaluation Protocols

NAIDv2 serves as both a training resource and a public benchmark for automated peer-review systems, particularly the NAIPv2 framework. NAIPv2 is trained using pairwise comparisons induced from NAIDv2 but deployed as a pointwise regressor for efficiency. Core inputs are title and abstract; inclusion of full-structured content does not improve performance in mainline evaluations.

The benchmark supports a range of evaluation protocols:

- Leakage-aware temporal test split: The primary ICLR evaluation is on papers from 2025, with training from prior years.
- Cross-venue transfer: NAIDv2-trained models are evaluated on a 13,223-sample NeurIPS set to test generalization.
- Metrics: Area Under Curve (AUC) for acceptance prediction, Spearman’s $\rho$ for ranking correlation, pairwise accuracy for ordinal comparisons, F1, and NDCG@20 for top-$k$ relevance.
- Debiasing ablations: Grouping by hierarchical cluster and year outperforms time, keyword, or ungrouped pair construction, confirming the efficacy of the dataset’s design for removing systemic scale bias.

The dataset saturates in training efficacy at approximately $10^4$ pairwise comparisons.

## 6. Biases, Limitations, and Practical Considerations

While NAIDv2 is constructed to address many of the confounders in prior peer-review datasets, several residual challenges are noted:

- **Domain and selection bias:** As a single-venue dataset, NAIDv2 reflects ICLR’s research and review culture, limiting immediate applicability to other scientific disciplines.
- **Reviewer noise:** RTS models confidence-uncertainty but does not remove the subjectivity of human review; label noise is not eliminated.
- **Temporal drift:** Grouping by year reduces but does not eradicate the impact of evolving community standards.
- **Information leakage:** Test set isolation to 2025 mitigates, but cannot guarantee, exclusion from LLM pretraining data.
- **Rich content not default:** Although structured full paper content is stored, the main modeling protocols use only titles and abstracts.
- **Ethical limitations:** Released for research purposes only, with an explicit caution against using model outputs as objective measures of scientific merit.

## 7. Comparative and Community Impact

Relative to earlier resources such as PeerRead and Re², NAIDv2 offers superior recency, scale, leakage awareness, domain-aware organization, and confidence-integrated labeling. Unlike prior datasets with fixed accept/reject or pointwise review-score targets, NAIDv2’s pairwise, domain-normalized, confidence-weighted structure enables the training of models whose inferences are robust to reviewer and domain inconsistencies, and whose efficiency is suitable for both large-scale benchmarking and real-world deployment pipelines.

Access is provided via sway.cloud.microsoft/Pr42npP80MfPhvj8 [2509.25179]. The dataset forms the empirical bedrock for the current state-of-the-art in peer-review prediction benchmarks and is designed to be extensible for future methodological innovation in scientific content assessment.

Source: https://www.emergentmind.com/topics/naidv2