NAIPv2: Debiased Paper Quality Estimation

• NAIPv2 is a debiased and efficient framework that automates scientific paper quality estimation through a novel pairwise ranking approach.
• It employs a domain–year grouping strategy to mitigate scale inconsistencies in reviewer scores using the innovative Review Tendency Signal.
• The framework achieves state-of-the-art accuracy and linear-time inference efficiency, validated on the large-scale NAIDv2 dataset with over 24,000 ICLR submissions.

NAIPv2 is a debiased and efficient framework for automated scientific paper quality estimation. Employing a pairwise learning approach within domain–year clusters and introducing the Review Tendency Signal (RTS) for principled score aggregation, NAIPv2 achieves state-of-the-art accuracy while maintaining linear-time inference efficiency. The system is trained and evaluated on the large-scale NAIDv2 dataset, comprising 24,276 ICLR submissions enriched with metadata, structured content, and reviewer annotations (Zhao et al., 29 Sep 2025).

1. Motivation and Background

The exponential increase in AI-related publications (approximately 242,000 in 2023) presents significant challenges for both human and machine assessment of scientific quality. Traditional early-stage quality signals—including citation counts and venue prestige—are unreliable or discouraged for automated evaluation due to initiatives such as DORA and the Leiden Manifesto. Existing LLM-based qualitative assessment frameworks (e.g., DeepReview, CycleReviewer) use autoregressive inference requiring substantial computational resources and yielding high latency (on the order of minutes per paper). Conversely, direct score regression enables fast evaluation but is hampered by scale inconsistencies across diverse domains and years and ignores reviewer confidence information.

NAIPv2 addresses these deficiencies by leveraging debiased pairwise learning and constructing a probabilistic review aggregation signal, enabling both performance and scalability advantages over prior approaches.

2. Methodology: Domain–Year Pairwise Learning

The central methodological innovation in NAIPv2 is the use of pairwise ranking within tightly defined domain–year groups to mitigate scale inconsistencies in reviewer scoring. The specific workflow is as follows:

• Each paper’s title and abstract are embedded via Qwen3-Embedding-4B, and hierarchical agglomerative clustering yields latent domain labels.
• Domain labels are combined with the publication year to define discrete group keys.
• Only pairs of submissions $(a, b)$ within the same domain–year cluster are sampled for comparative training, ensuring that comparisons are homogeneous across field and era.

For each pair, the Review Tendency Signals $RTS_a, RTS_b \in [0, 1]$ are computed (see Section 3), and the ground truth ordering is $z_{ab} = \mathbb{I}[RTS_a > RTS_b]$. The model $f(\cdot~;~\theta)$ (LLaMA-3 backbone) produces scores $\hat{y}_a = f(x_a), \hat{y}_b = f(x_b)$. The predicted preference is $\hat{z}_{ab} = \sigma(\hat{y}_a - \hat{y}_b)$, using the sigmoid activation. The pairwise binary cross-entropy loss:

$$L(a, b) = -\left[z_{ab} \log \sigma(\hat{y}_a - \hat{y}_b) + (1 - z_{ab}) \log(1 - \sigma(\hat{y}_a - \hat{y}_b))\right]$$

drives learning. Curriculum sampling is employed, where pairs are bucketed by $\Delta_{ab} = |RTS_a - RTS_b|$, and training progresses from easy (large $\Delta$) to hard pairs.

This pairwise loss, restricted to domain–year clusters, provides an effective debiasing mechanism by preventing comparisons across spurious scales.

3. Review Tendency Signal (RTS)

The Review Tendency Signal is a probabilistically weighted aggregation of reviewer scores and confidences, designed to handle rater uncertainty and variance:

• Each review $s_i$ (normalized to $RTS_a, RTS_b \in [0, 1]$0) has an associated confidence $RTS_a, RTS_b \in [0, 1]$1 (also $RTS_a, RTS_b \in [0, 1]$2).
• Reviews are modeled as noisy measurements of a latent true quality $RTS_a, RTS_b \in [0, 1]$3, with

$RTS_a, RTS_b \in [0, 1]$4

where $RTS_a, RTS_b \in [0, 1]$5 is a decreasing function of confidence (i.e., higher confidence implies reduced variance).

• The posterior for $RTS_a, RTS_b \in [0, 1]$6 is proportional to $RTS_a, RTS_b \in [0, 1]$7, and the RTS is defined as

$RTS_a, RTS_b \in [0, 1]$8

Key properties of RTS:

• It guarantees $RTS_a, RTS_b \in [0, 1]$9 and provides a smooth interpolation between review scores.
• Principally incorporates reviewer uncertainty.
• Outperforms alternatives (mean, median, weighted average) in ablation studies:

| Signal Type | AUC | |---------------|--------| | RTS (ours) | 0.782 | | Mean | 0.753 | | Median | 0.757 | | Weighted Avg. | 0.754 |

4. NAIDv2 Dataset Construction and Characteristics

NAIDv2 is a large-scale dataset built to support NAIPv2 development and evaluation:

• Data scale: 24,276 ICLR submissions from 2021–2025, extracted via the OpenReview API.
• Test split: Strict evaluation is performed on a disjoint 2025 test partition (1,029 papers).
• Content and metadata: Each entry contains title, abstract, authors, year, cluster category, OpenReview ID, reviewer scores/confidences, textual reviews (including summaries, strengths, weaknesses, rebuttals), parsed PDF structure (via MinerU), and key figure/table annotations.
• Processed fields: All paper embeddings (Qwen3-Embedding-4B) are included as JSON vectors for clustering/domaining. The debias pipeline consists of embedding+clustering, normalization+RTS computation, and export of RTS and cluster keys for pairwise training.

This comprehensive dataset furnishes the necessary structure for robust, debiased learning and supports broad empirical validation.

5. Model Training and Inference Efficiency

NAIPv2 employs a LLaMA-3 8B-parameter model, adapted via LoRA and 8-bit quantized for memory efficiency. Training is executed on 4×A40 GPUs and requires approximately 1 hour for up to 23,000 pairwise samples (batch size 8, learning rate $z_{ab} = \mathbb{I}[RTS_a > RTS_b]$0, 1 epoch).

Training is driven by domain–year bucketed BCE loss with curriculum learning. At inference, the same backbone operates in pointwise scoring mode: for any given paper, its scalar score $z_{ab} = \mathbb{I}[RTS_a > RTS_b]$1 is predicted independently. The inference algorithm exhibits linear time complexity $z_{ab} = \mathbb{I}[RTS_a > RTS_b]$2 in the number of candidates, supporting throughput greater than 1,000 papers/s on an RTX 3090 (approx. 1 ms/paper).

6. Empirical Evaluation and Generalization

NAIPv2 reports state-of-the-art performance on the ICLR 2025 test set and exhibits robust generalization to NeurIPS submissions. Major findings include:

Method	Accuracy	F1	AUC	NDCG	Spearman’s ρ
Random Lower Bound	0.514	0.410	0.527	0.525	0.002
“Upper Bound” (info)	0.819	0.757	0.894	0.995	0.984
ChatGPT API (pt)	0.644	0.427	0.654	0.702	0.315
DeepReviewer (14B)	0.689	0.623	–	–	0.408
NAIPv1 (8B)	0.545	0.472	0.605	0.629	0.183
NAIPv2 (ours)	0.706	0.609	0.782	0.771	0.432

Ablation studies reveal that the time+hierarchical grouping strategy yields the highest AUC (0.782) compared to no grouping (0.739) or naive keyword-based assignment (0.556). RTS consistently outperforms mean, median, and weighted averages as the review signal. Curriculum learning and the plain BCE loss also provide best-in-class results.

Cross-venue transfer evaluation on 13,223 NeurIPS submissions (2021–2024) demonstrates monotonic score increases across decision tiers (Rejected $z_{ab} = \mathbb{I}[RTS_a > RTS_b]$3 Oral), mirroring reviewer-based decision distributions.

7. Discussion, Limitations, and Future Directions

NAIPv2’s principal contributions include scalable debiasing via domain–year pairwise training, the probabilistic incorporation of reviewer confidence (RTS), and order-of-magnitude efficiency improvements versus chain-of-thought LLM inference.

Limitations include dependence on access to historical peer review data (scores and confidences), the inability to produce textual justifications or explanation chains, and a focus strictly on review-based “quality” rather than predictive citation impact.

Potential future extensions include the integration of listwise or hybrid ranking strategies to model groupwise preferences, dynamic quality modeling with citation or altmetric signals, simulation-based extension to double-blind venues, and the fusion of multimodal content (full text, figures, and code) to further enrich representation capabilities (Zhao et al., 29 Sep 2025).