GraphReview: LLM-Driven Graph Evaluation

• GraphReview is a graph-based framework that evaluates scientific papers by integrating intrinsic text quality scores with pairwise LLM-based comparisons.
• It employs Personalized PageRank to diffusely aggregate node-level quality priors and edge-level comparative signals for ranking, decision prediction, and review generation.
• Experimental results on ICLR/ICML datasets show significant accuracy and ranking gains over traditional methods, despite limitations in scope and model size.

GraphReview is an LLM-based, graph-based framework for scientific paper evaluation that treats reviewing as review-signal message passing over a semantic paper graph rather than as isolated manuscript scoring. It is designed for settings in which evaluation depends simultaneously on intrinsic paper quality, relations among contemporaneous submissions, and relations to prior literature. In GraphReview, LLMs estimate node-level quality priors and generate edge-level comparative evidence through pairwise paper comparisons, and Personalized PageRank integrates these signals for quality ranking, decision prediction, and review generation (Zheng et al., 26 May 2026).

1. Semantic paper graph and problem formulation

GraphReview represents the evaluation corpus as a semantic paper graph $G=(V,E)$, where the nodes $V$ are papers and the edges $E$ are review-signal links (Zheng et al., 26 May 2026). The node set contains both the $n$ concurrent submissions under review and the $N-n$ historical papers used as prior context. This construction makes contemporaneous comparison and historical grounding part of the same computational object rather than separate auxiliary modules.

The graph contains two types of undirected edges. Synchronic links connect contemporaneous submissions within the same review cycle. Diachronic links connect a new submission to relevant prior literature. This joint representation is central to the framework’s claim that scientific paper evaluation often involves not only assessing a manuscript itself, but also relating it to contemporaneous research and prior literature (Zheng et al., 26 May 2026).

Edge selection is defined by the Sequential 2-Factor Matching (S2FM) algorithm. At iteration $t$, S2FM solves

$$\max \sum_{u,v} h_u^\top h_v \cdot \Delta c^{(t)}_{uv}$$

subject to

$$\forall u:\ \sum_v \Delta c^{(t)}_{uv}=2,\qquad \Delta c^{(t)}_{uv}=\Delta c^{(t)}_{vu}\in\{0,1\},\qquad \Delta c^{(t)}_{uv}\le 1-c^{(t-1)}_{uv}.$$

After $T$ rounds, the binary adjacency is $C^{(T)}=(c^{(T)}_{uv})$, with

$V$0

and the pairwise weight may be viewed as

$V$1

or equivalently $V$2 (Zheng et al., 26 May 2026).

A common simplification is to treat paper evaluation as either pointwise scoring or pairwise comparison alone. GraphReview explicitly combines both, which suggests a hybrid formulation in which local quality assessment and relational evidence are intended to be mutually corrective rather than mutually exclusive.

2. Node-level quality priors

For each paper $V$3 with text $V$4, GraphReview prompts an LLM backbone $V$5 with $V$6 to output a distribution over $V$7 discrete score anchors $V$8 (Zheng et al., 26 May 2026). In the paper notation,

$V$9

This output is converted into a scalar estimate

$E$0

GraphReview then forms a normalized prior vector, denoted in the summary as $E$1, by

$E$2

where $E$3 ensures positivity. In practice, $E$4 serves as the teleport prior in Personalized PageRank (Zheng et al., 26 May 2026).

This stage operationalizes intrinsic quality. The framework does not rely solely on pairwise relational evidence; it first assigns each paper a prior mass based on LLM judgment of the paper text itself. A plausible implication is that GraphReview is intended to remain informative even when graph connectivity is sparse or when some papers have limited comparative coverage, because the teleport prior remains well-defined.

The design also fixes the quality estimation problem to a discrete-anchor distribution rather than unrestricted scalar regression. That choice is important for the later training objective, where continuous reviewer scores are converted into soft targets over the same anchor set.

3. Edge-level comparative evidence

For each connected pair $E$5 with $E$6, GraphReview prompts the same LLM backbone with both papers and a comparison prompt $E$7, producing a binary preference distribution

$E$8

(Zheng et al., 26 May 2026). This is the edge-level comparative evidence layer.

The comparison output is then encoded as a directed adjacency update. If the model judges $E$9, the edge $n$0 is removed by setting $n$1 and keeping $n$2; the reverse occurs if $n$3. The same information can be written as a signed comparative score matrix $n$4 with entries

$n$5

Thus, although the semantic paper graph is constructed with undirected review-signal links, GraphReview turns comparative judgments into directed evidence for downstream propagation (Zheng et al., 26 May 2026).

This mechanism distinguishes GraphReview from methods that model intrinsic signals and pairwise signals separately. Here the pairwise comparison is not merely an auxiliary reranker; it directly rewrites the graph evidence used in the final propagation step. A common misconception is that pairwise comparison in such systems only provides local ordering constraints. In GraphReview, the pairwise outputs are subsequently diffused over the graph, so a comparison contributes beyond the two papers explicitly queried.

4. Integration by Personalized PageRank

After node priors and edge comparisons are obtained, GraphReview builds a directed transition matrix $n$6 by column-normalizing $n$7 (Zheng et al., 26 May 2026). Let

$n$8

then

$n$9

The global score vector $N-n$0 is the stationary distribution satisfying

$N-n$1

which is computed by power iteration. Final ranking is obtained by $N-n$2 over the $N-n$3 submissions, and the top $N-n$4, where $N-n$5 is the acceptance rate, are predicted as “Accept,” with the remainder predicted as “Reject” (Zheng et al., 26 May 2026).

Personalized PageRank is therefore the actual integration operator in GraphReview. The graph component is not only structural bookkeeping: it is the medium through which review signals are propagated and aggregated into final decisions. This suggests that GraphReview’s “message passing” is implemented through graph diffusion over LLM-derived priors and preferences rather than through a learned graph neural network layer.

The same propagated signal supports three outputs named in the abstract: quality ranking, decision prediction, and review generation (Zheng et al., 26 May 2026). The first two are explicitly specified in the formulation above. The third indicates that the resulting graph evidence is also used to support generated review text rather than only scalar prediction.

5. Reward-induced maximum likelihood training

To improve graph evidence quality, GraphReview introduces reward-induced maximum likelihood (RIML) objectives for training the LLM backbones (Zheng et al., 26 May 2026). Training alternates between two stages: a cold-start supervised fine-tuning stage using human-annotated “review-signal tokens,” and RIML fine-tuning aligned with both node and edge rewards.

For the node-level loss, the ground-truth continuous score $N-n$6 is soft-approximated by the anchor set $N-n$7. A distance-based reward is defined as

$N-n$8

and converted into a soft target distribution

$N-n$9

The objective is

$t$0

For the edge-level loss, the training set contains pairs $t$1 with $t$2. The label is $t$3 if $t$4, otherwise $t$5; the one-hot target is $t$6, and the weight is $t$7. The objective is

$t$8

These objectives align the node scorer with scalar review quality and the pairwise comparator with score differences (Zheng et al., 26 May 2026).

A plausible implication is that GraphReview treats calibration and discrimination as distinct but coupled problems: node-level RIML calibrates absolute quality priors, while edge-level RIML sharpens relative preference evidence.

6. Experimental results, generalization, and limitations

GraphReview is evaluated on ICLR 2025 submissions, with approximately $t$9 train/validation instances and $$\max \sum_{u,v} h_u^\top h_v \cdot \Delta c^{(t)}_{uv}$$0 test instances, using average human reviewer scores as labels. Its external knowledge base consists of all accepted ICLR, ICML, and NeurIPS papers from 2023–24, approximately $$\max \sum_{u,v} h_u^\top h_v \cdot \Delta c^{(t)}_{uv}$$1 papers. Additional test sets come from ICLR 2026 and ICML 2025. The reported backbone is Qwen2.5-7B-Instruct with LoRA (Zheng et al., 26 May 2026).

The evaluation metrics are divided into decision and ranking metrics. Decision metrics are Accuracy, Macro-F1, and AUC. Ranking metrics are Spearman’s $$\max \sum_{u,v} h_u^\top h_v \cdot \Delta c^{(t)}_{uv}$$2, Kendall’s $$\max \sum_{u,v} h_u^\top h_v \cdot \Delta c^{(t)}_{uv}$$3, and NDCG@10. Baselines include classic GNNs such as GCN, GAT, and GraphSAGE; naïve LLMs including GPT-5-Mini, Gemini-2.5-Flash, and DeepSeek; Review Agents; and comparative systems including PairReview, CNPE-7B, and NAIPv2 (Zheng et al., 26 May 2026).

On the main results table, GraphReview achieves Accuracy $$\max \sum_{u,v} h_u^\top h_v \cdot \Delta c^{(t)}_{uv}$$4 versus $$\max \sum_{u,v} h_u^\top h_v \cdot \Delta c^{(t)}_{uv}$$5 for NAIPv2, Spearman’s $$\max \sum_{u,v} h_u^\top h_v \cdot \Delta c^{(t)}_{uv}$$6 $$\max \sum_{u,v} h_u^\top h_v \cdot \Delta c^{(t)}_{uv}$$7 versus $$\max \sum_{u,v} h_u^\top h_v \cdot \Delta c^{(t)}_{uv}$$8, and average performance $$\max \sum_{u,v} h_u^\top h_v \cdot \Delta c^{(t)}_{uv}$$9 versus $$\forall u:\ \sum_v \Delta c^{(t)}_{uv}=2,\qquad \Delta c^{(t)}_{uv}=\Delta c^{(t)}_{vu}\in\{0,1\},\qquad \Delta c^{(t)}_{uv}\le 1-c^{(t-1)}_{uv}.$$0. The reported relative gains are $$\forall u:\ \sum_v \Delta c^{(t)}_{uv}=2,\qquad \Delta c^{(t)}_{uv}=\Delta c^{(t)}_{vu}\in\{0,1\},\qquad \Delta c^{(t)}_{uv}\le 1-c^{(t-1)}_{uv}.$$1 in Accuracy, $$\forall u:\ \sum_v \Delta c^{(t)}_{uv}=2,\qquad \Delta c^{(t)}_{uv}=\Delta c^{(t)}_{vu}\in\{0,1\},\qquad \Delta c^{(t)}_{uv}\le 1-c^{(t-1)}_{uv}.$$2 in Spearman’s $$\forall u:\ \sum_v \Delta c^{(t)}_{uv}=2,\qquad \Delta c^{(t)}_{uv}=\Delta c^{(t)}_{vu}\in\{0,1\},\qquad \Delta c^{(t)}_{uv}\le 1-c^{(t-1)}_{uv}.$$3, and $$\forall u:\ \sum_v \Delta c^{(t)}_{uv}=2,\qquad \Delta c^{(t)}_{uv}=\Delta c^{(t)}_{vu}\in\{0,1\},\qquad \Delta c^{(t)}_{uv}\le 1-c^{(t-1)}_{uv}.$$4 on average. The abstract further states that GraphReview consistently outperforms the strongest baseline on decision and ranking metrics, produces higher-quality review texts, and generalizes effectively across time periods and conference venues (Zheng et al., 26 May 2026).

Ablation studies confirm that each of the three information sources—intrinsic, synchronic, and diachronic—and both training stages, SFT and RIML, contribute substantially. This supports the framework’s central claim that paper evaluation benefits from combining manuscript-internal evidence with relational evidence over a paper graph (Zheng et al., 26 May 2026).

The limitations are explicit. All experiments are on computer science conference papers; other domains remain untested. The model still underperforms expert human reviewers in open-ended settings. Only a single 7B model is trained, and larger models may yield further gains. The training data may encode human bias, so fairness auditing and mitigation remain open requirements. The listed future directions are more advanced graph architectures, cross-discipline evaluation, stronger backbones, and bias mitigation. The code is available at the repository linked in the abstract (Zheng et al., 26 May 2026).