GraphReview: Scientific Paper Evaluation via LLM-Based Graph Message Passing

Abstract: Scientific paper evaluation often involves not only assessing a manuscript itself, but also relating it to contemporaneous research and prior literature. However, existing LLM-based methods typically model these signals separately and lack a unified mechanism for propagating review evidence across papers. We propose $\textbf{GraphReview}$, a graph-based LLM framework that formulates paper evaluation as review-signal message passing over a semantic paper graph. The graph jointly captures intrinsic quality, synchronic links among contemporaneous papers, and diachronic links to prior work. LLMs are used to estimate node-level quality priors and generate edge-level comparative evidence through pairwise paper comparisons, while Personalized PageRank integrates review signals for quality ranking, decision prediction, and review generation. To produce higher-quality graph evidence, we propose reward-induced maximum likelihood objectives for training the LLM backbones. Experiments show that GraphReview consistently outperforms the strongest baseline, achieving average improvements of 29.7% on decision and ranking metrics, including gains of 23.7% in Accuracy and 57.6% in Spearman's $ρ$. It also produces higher-quality review texts and generalizes effectively across time periods and conference venues. The code is available at https://github.com/ECNU-Text-Computing/GraphReview.

• The paper introduces a graph-based LLM framework that integrates intrinsic, synchronic, and diachronic evidence through message passing for comprehensive paper evaluation.
• It employs a Sequential 2-Factor Matching algorithm and Personalized PageRank to merge node and edge signals, achieving substantial improvements over existing baselines.
• The approach demonstrates significant gains in decision and ranking metrics, offering a scalable, interpretable, and robust method for automated peer review.

GraphReview: LLM-Based Graph Message Passing for Scientific Paper Evaluation

Motivation and Problem Statement

Scientific paper evaluation requires integrating diverse sources of evidence: intrinsic manuscript quality, competitive standing among contemporaneous submissions, and relevance or advancement over prior literature. Existing LLM-driven approaches typically model these signals in isolation, lacking explicit integration and propagation of comparative evaluative evidence across submissions and the broader literature. This fragmentation limits their fidelity to real-world peer review, where both context- and evidence-aware assessment is central.

GraphReview Framework

GraphReview addresses these deficits through a graph-based LLM framework, representing paper evaluation as review-signal message passing over a semantic graph encapsulating intrinsic, synchronic, and diachronic links among papers.

• Intrinsic Quality: Manuscript-level properties (originality, clarity, significance).
• Synchronic Links: Connections among submissions in the same review cycle, capturing direct competition.
• Diachronic Links: Temporal/inheritance-based relationships to prior literature, supporting judgments about novelty and scholarly impact.

The framework operationalizes three phases (Figure 1): (i) Message—LLMs generate pairwise comparative evidence and quality priors, (ii) Aggregation—evidence from related papers is accumulated, and (iii) Update—signals are merged via Personalized PageRank (PPR) to obtain global ranking and editorial decisions.

Figure 1: The overall process of GraphReview, including message passing, aggregation, and update, with input papers ranked by quality.

This design allows explicit, iterative integration of review-relevant information flows, moving beyond unstructured LLM context windows (Figure 2).

Figure 2: Unlike prior methods that isolate information sources, GraphReview integrates multiple evidence types via graph-based message passing.

Methodological Details

Graph Construction and Message Passing

The Sequential 2-Factor Matching (S2FM) algorithm efficiently builds a sparse, evidence-rich graph by incrementally selecting informative paper pairs for comparison, controlling computation at $O(TN)$ (with $T\ll N$ iterations).

Node priors—direct assessments of a paper's standalone merit—are predicted using LLMs fine-tuned with reward-induced maximum likelihood objectives. For each edge, LLMs perform targeted pairwise comparisons, producing both a comparative preference signal and rationales.

Personalized PageRank serves as the global aggregator, combining node-level priors and edge-level pairwise constraints into stable, interpretable ranking scores. The damping factor $\lambda$ controls the balance between prior and propagated evidence.

Training Objectives

GraphReview’s LLM backbones are trained in a two-stage procedure:

• Supervised Fine-Tuning (SFT): The model learns to predict node scores and provide accompanying rationales from ground-truth data, with prompts optimized via an iterative self-evolution approach.
• Reward-Induced Maximum Likelihood (RIML): Node- and edge-level predictions receive soft or one-hot targets from empirical score distributions and comparative judgments, emphasizing both absolute accuracy and ordinal relationships.

This scheme departs from expensive and often misaligned reinforcement learning on sparse rewards, exploiting the holistic and evidence-aligned nature of expert review.

Evaluation Pipeline

As illustrated in Figure 3, paper embeddings, graph construction, two-stage LLM training, and prediction/generation are tightly integrated.

Figure 3: Pipeline of dataset construction (left) and the two-stage model training process (right).

Experimental Analysis

Main Results

Across decision (Accuracy, F1, AUC) and ranking (Spearman ρ, Kendall τ, NDCG@10) metrics, GraphReview outperforms all strong baselines—GNNs, LLM agents, and pairwise comparison models—by substantial margins. Compared to the best competitive baseline, NAIPv2, GraphReview yields a 29.7% average improvement, with 23.7% and 57.6% relative gains in Accuracy and Spearman's ρ, respectively. Textual review outputs show consistently higher win rates against major baselines across all qualitative dimensions.

Ablation Studies

Information-fusion ablations demonstrate the critical contribution of each signal: exclusion of synchronic links produces the largest degradation (12.4% drop), substantiating the necessity of context-aware competition modeling. Both SFT and RIML stages provide complementary benefits, with performance losses observed upon removing either.

Hyperparameter and Generalization Analysis

Model performance converges after $T=5$ message-passing rounds; best results are achieved with $\lambda=0.20$, indicating task-specific optimal propagation depth and reliance on local evidence (Figure 4).

Figure 4: Performance convergence with increasing message-passing steps (left) and optimal performance at a low damping factor (right).

Cross-cycle and cross-venue benchmarks (ICLR 2026, ICML 2025) show high discriminative power between rejected and accepted papers, with statistically significant gaps in predicted rankings maintained across evaluation domains (Figure 5).

Figure 5: Distribution of normalized ranking scores by paper quality level and venue, with significant separation between rejected and accepted groups.

Theoretical and Practical Implications

GraphReview demonstrates that evidence-aware, contextually grounded, and structurally organized evaluation frameworks are decisively superior to both uninformed LLM prediction and isolated comparison methods for automated paper assessment. This paradigm:

• Advances Graph-LLM Integration: It validates graph-based evidence propagation as a foundation for complex evaluative tasks, differentiating itself from previous retrieval-augmented or memory-centric graph applications.
• Enables Attributive, Interpretable Review: The modular architecture separates and preserves rationale at both node and edge levels, improving attributions required for peer review transparency.
• Sets a Template for Multi-source Reasoning: The explicit integration of multiple relational evidence types via message passing generalizes to domains beyond peer review, such as scientific discovery, grant selection, or collaborative intelligence systems.
• Unifies Prior Paradigms: Ablations and modularization experiments demonstrate that GraphReview’s graph-based message passing subsumes earlier pointwise and pairwise objective formulations, supporting robust fusion of diverse LLM expertise.

Practically, GraphReview signals a direction where automated reviewers become reliable, data-driven assistants, augmenting (rather than replacing) human experts, with risk analyses and mitigation strategies embedded in deployment.

Future Directions

Important avenues remain, including:

• Extending domain coverage beyond computer science to other disciplines with distinct evaluation cultures.
• Scaling model and graph capacity, exploring larger LLMs or more semantically rich graphs.
• Enhancing bias mitigation, interpretability, and robustness, especially against known human or model-driven artifacts.
• Developing techniques for deeper evidence traceability within review rationales, further aligning automated systems with real-world attribution standards.

Conclusion

GraphReview establishes a state-of-the-art, theoretically-motivated, and empirically validated framework for scientific paper evaluation, framed as LLM-based graph message passing. It demonstrates the benefits of fusing intrinsic, synchronic, and diachronic evidence sources, and provides a scalable, extensible paradigm for automated evaluation tasks where multifaceted evidence integration is paramount.