PeeriScope: Automated Peer Review Evaluation
- PeeriScope is a modular platform for automated, multi-dimensional evaluation of peer review quality using structured metrics, LLM-based judgments, and supervised prediction.
- It integrates interpretable signals from review text and reviewer metadata to assist in self-assessment, editorial triage, and large-scale auditing.
- The framework employs a hybrid architecture that combines rubric-guided evaluations with supervised regressors, ensuring scalability, transparency, and model-agnostic flexibility.
Searching arXiv for the PeeriScope paper and closely related peer-review-quality work. PeeriScope is a modular platform for automated, multi-dimensional evaluation of peer review quality. It addresses the problem that peer review quality is difficult to assess at scale, particularly as review volumes rise, review quality remains highly variable, and LLM-assisted or LLM-generated reviews can appear polished while lacking depth, factual grounding, or actionable guidance. The framework combines interpretable structured signals, rubric-guided LLM judgments, and supervised prediction into a deployable analytics system with both a public interface and a documented API (Ebrahimi et al., 27 Apr 2026).
1. Problem setting and system scope
PeeriScope is introduced as infrastructure for systematic assessment of review quality in scholarly publishing. Its stated motivation is that peer review is central to science, yet most venues lack standardized, scalable, and interpretable ways to assess whether a review is actually good. The system is framed not as a replacement for human oversight, but as a modular, post-hoc assessment platform that supports reviewer self-assessment, editorial triage, and large-scale auditing (Ebrahimi et al., 27 Apr 2026).
The paper emphasizes three design requirements: scalability, transparency and interpretability, and compatibility and openness. This suggests that the system is intended simultaneously for operational deployment and for methodological extension. A plausible implication is that PeeriScope is meant to occupy a middle ground between narrowly targeted review analytics and opaque end-to-end scoring systems.
In operational terms, the platform takes as input the paper title, paper abstract, full review text, and optionally an OpenAlex identifier for the reviewer. From these inputs it produces multiple dimension-level scores, structured diagnostics, and an overall quality estimate, exposed through either an interactive dashboard or JSON-formatted API outputs (Ebrahimi et al., 27 Apr 2026).
2. Hybrid architecture and processing workflow
The framework evaluates a review through three complementary sources of evidence: structured metrics extracted from the review text and optionally reviewer metadata, rubric-guided LLM-based metrics for abstract review qualities, and a supervised overall quality estimator that combines these signals into a single predicted overall quality score (Ebrahimi et al., 27 Apr 2026).
This layered design is central to the platform. Surface-level structured features are presented as interpretable but incomplete; LLM judgments can capture richer qualitative dimensions but are imperfect; supervised prediction is then used to calibrate and integrate the heterogeneous evidence. The paper explicitly describes the platform’s combination strategy as a sequence of steps: compute structured review and reviewer signals, compute rubric-guided LLM judgments, feed these signals into a supervised overall estimator, and expose both the interpretable dimensions and the aggregated score (Ebrahimi et al., 27 Apr 2026).
The workflow includes input ingestion, feature extraction, optional reviewer-profile augmentation, local LLM scoring, supervised prediction, and presentation. Intermediate outputs include text-level feature values, reviewer metadata-derived indicators, dimension-level LLM scores, and overall quality predictions. The dashboard is designed to foreground these intermediate dimensions rather than collapsing everything into a single opaque scalar (Ebrahimi et al., 27 Apr 2026).
A plausible implication is that the platform’s architecture is meant to support both human interpretability and modular replacement of subcomponents, especially because the paper describes it as modular and model-agnostic.
3. Structured metrics and reviewer metadata
The structured component of PeeriScope groups features into writing style and readability, tone and reviewer attitude, review content and engagement, and optional reviewer-based metadata (Ebrahimi et al., 27 Apr 2026).
Writing-style and readability features include review length, hedging, lexical diversity, and readability. Review length is used as a proxy for thoroughness. Hedging is measured using a cue-based neural detector. Lexical diversity is measured as type-token ratio. Readability is measured using the Flesch Reading Ease score (Ebrahimi et al., 27 Apr 2026).
Tone and reviewer-attitude features include politeness, sentiment polarity, and similarity between the review and the paper. The similarity signal is interpreted as reflecting the reviewer’s general domain proximity to the manuscript. Review-content and engagement features include mentions of manuscript structure such as references to figures or sections, mentions of citations, and the presence of questions. For questions, the system uses a fine-tuned classifier trained on interrogative forms that indicate substantive reviewer intent (Ebrahimi et al., 27 Apr 2026).
Reviewer-based metadata can be incorporated through OpenAlex. These optional features include topical alignment between the submission and the reviewer’s publication history, computed using SPECTER embeddings, as well as citation count and scholarly tenure. These are intended as proxies for expertise, seniority, continuity, visibility, and influence (Ebrahimi et al., 27 Apr 2026).
The paper explicitly notes that it does not provide a complete formal feature table, exact extraction formulas for all metrics, classifier architectures for every submodule, or threshold values. Consequently, the feature families and their conceptual roles are specified, but not all implementation details are recoverable from the published description (Ebrahimi et al., 27 Apr 2026).
4. Rubric-guided LLM evaluation
PeeriScope uses LLM-as-a-judge scoring to evaluate qualities that structured signals do not directly capture, including fairness, constructiveness, and objectivity. The main deployed evaluator is Qwen-3-8B, used as an instruction-tuned LLM that is locally deployed for privacy and fast inference, although the system is described as model-agnostic (Ebrahimi et al., 27 Apr 2026).
For each review, the LLM is given the paper title, the paper abstract, and the review text. It is prompted to assign scores across multiple qualitative dimensions using a rubric. In the empirical alignment study, the LLM assigns scores on a five-point ordinal scale. The paper states that the rubric is adapted from editorial guidelines and prior annotation schemes, but does not reproduce the exact prompt text or full point-by-point scoring anchors (Ebrahimi et al., 27 Apr 2026).
The rubric covers thirteen dimensions:
| Aspect | Description |
|---|---|
| Overall Quality | Holistic evaluation of the review's usefulness and professionalism |
| Comprehensiveness | Covering all key aspects of the paper |
| Actionability | Helpfulness of the review in suggesting clear next steps |
| Sentiment Polarity | Overall sentiment conveyed by the reviewer |
| Constructiveness | Whether the review suggests improvements rather than only criticism |
| Use of Technical Terms | Using domain-specific vocabulary |
| Objectivity | Presence of unbiased, evidence-based commentary |
| Alignment | Relevance of the review to the scope of the paper |
| Vagueness | Degree of ambiguity or lack of specificity in the review |
| Fairness | Perceived impartiality and balance in judgments |
| Politeness | Tone and manner of the review language |
| Clarity and Readability | Ease of understanding the review, including grammar and structure |
| Factuality | Accuracy of the statements made in the review |
These rubric dimensions define the platform’s clearest taxonomy of review quality (Ebrahimi et al., 27 Apr 2026). This suggests that PeeriScope operationalizes review quality as a multi-axis evaluative construct rather than as a single latent variable.
5. Supervised overall quality estimation and empirical results
PeeriScope includes a supervised predictor that outputs a single overall quality score intended to approximate expert human judgment. The paper motivates this module by arguing that editorial decisions often require a synthesized judgment, even when underlying evidence remains multidimensional (Ebrahimi et al., 27 Apr 2026).
Two classes of supervised models are used. The first class consists of lightweight regressors over structured and LLM-derived features: linear regression, random forest, and a two-layer multilayer perceptron (MLP). The second class is an end-to-end LLM-based predictor built from LLaMA3-8B, fine-tuned to regress directly from title, abstract, and review text to the human overall quality score using parameter-efficient fine-tuning via low-rank adaptation (LoRA) with 8-bit weight quantization (Ebrahimi et al., 27 Apr 2026).
The evaluation dataset contains 753 peer reviews from 200 papers drawn from OpenReview, F1000 Research, and the Semantic Web Journal. Each review was paired with the paper title and abstract, and was independently annotated by trained graduate students across the thirteen quality dimensions. An additional overall quality score was assigned using a continuous rubric designed to reflect editorial standards (Ebrahimi et al., 27 Apr 2026).
The paper uses two main validation setups. For LLM-human alignment on the rubric dimensions, it reports Kendall’s rank correlation for GPT-4o, Phi-4, and Qwen-3. For overall quality estimation, it evaluates the supervised estimators using 10-fold cross validation, again measured with Kendall’s (Ebrahimi et al., 27 Apr 2026).
The reported LLM-human alignment results are as follows:
| Aspect | GPT-4o | Phi-4 | Qwen-3 |
|---|---|---|---|
| Overall Quality | 0.359 | 0.241 | 0.252 |
| Comprehensiveness | 0.476 | 0.374 | 0.338 |
| Actionability | 0.411 | 0.279 | 0.314 |
| Sentiment Polarity | 0.407 | 0.397 | 0.428 |
| Constructiveness | 0.343 | 0.259 | 0.211 |
| Use of Technical Terms | 0.327 | 0.254 | 0.176 |
| Objectivity | 0.298 | 0.215 | 0.186 |
| Alignment | 0.295 | 0.204 | 0.105 |
| Vagueness | 0.189 | 0.175 | 0.078 |
| Fairness | 0.163 | 0.186 | 0.139 |
| Politeness | 0.128 | 0.053 | 0.106 |
| Clarity and Readability | 0.124 | 0.038 | 0.117 |
| Factuality | 0.115 | 0.006 | 0.089 |
The paper draws several direct conclusions from these values: GPT-4o has the best alignment on most dimensions, absolute agreement is still modest, the best dimension overall is Comprehensiveness at $0.476$ for GPT-4o, and some dimensions are especially hard, including Factuality, Clarity and Readability, Politeness, Fairness, and Vagueness. A notable exception is that Qwen-3 slightly outperforms GPT-4o on Sentiment Polarity, with $0.428$ versus $0.407$ (Ebrahimi et al., 27 Apr 2026).
For overall quality prediction, the paper states that structured-feature regressors outperform all zero-shot LLMs and also outperform the fine-tuned LLaMA3-8B model, and that the two-layer MLP achieves the highest agreement among the regressors. However, the exact numeric Kendall’s values for these supervised models are not given in the provided text (Ebrahimi et al., 27 Apr 2026).
6. Implementation, interfaces, and deployment
PeeriScope is implemented as a public platform with both a live interface and API access. The backend uses FastAPI, the frontend uses React 18 with TypeScript, Vite, Axios, Recharts, and Redux Toolkit, and deployment is via Docker containers. Qwen-3-8B is hosted locally via vLLM. Reviewer metadata comes from OpenAlex and is stored in MongoDB (Ebrahimi et al., 27 Apr 2026).
The paper notes a specific privacy design decision: requests are processed in-memory and discarded after computation, so no review or paper content is stored beyond the computation window. This is intended to minimize privacy exposure and facilitate lightweight compliance (Ebrahimi et al., 27 Apr 2026).
The API exposes three REST endpoints for different review analysis tasks, although the endpoint names and payload schemas are not enumerated in the paper. It can return structured JSON outputs for single reviews or batches, enabling large-scale automated auditing (Ebrahimi et al., 27 Apr 2026). The system is publicly available through a live demo and repository-linked API services (Ebrahimi et al., 27 Apr 2026).
The paper highlights three concrete use cases. In reviewer self-assessment, a reviewer can submit a title, abstract, and review text to receive interpretable scores and an overall quality estimate. In editorial triage, editors or area chairs can identify reviews that may need attention, such as those with very low comprehensiveness, very high vagueness, low fairness, low actionability, or weak overall quality. In large-scale auditing, venues can use the API and batch outputs to compare review quality across tracks or venues, monitor reviewer pools, evaluate interventions, and inspect trends in LLM-assisted reviewing (Ebrahimi et al., 27 Apr 2026).
7. Limitations, risks, and significance
The paper presents the system cautiously and does not claim that peer review can be fully automated. Its own results show that even the best zero-shot LLM judgments are only moderately aligned with humans, which means the system should not be treated as ground truth (Ebrahimi et al., 27 Apr 2026).
Several limitations are explicit. Peer review quality is inherently subjective and context-dependent, especially for dimensions such as fairness, objectivity, clarity, and factuality. The annotated dataset, while publicly released for research use, contains 753 reviews and is therefore modest for broad cross-domain deployment. LLMs perform weakly on several dimensions, especially factuality, politeness, and clarity/readability. Reviewer metadata features such as citation count and scholarly tenure may be informative, but they also raise fairness concerns because they may correlate with status and seniority. The paper therefore positions the system as more appropriate for feedback, triage, auditing, and decision support than for punitive or fully automated gatekeeping (Ebrahimi et al., 27 Apr 2026).
The paper also does not provide formal ablations on feature families, reviewer metadata inclusion, or LLM-score removal, and it does not include robustness stress tests, adversarial analyses, or domain-shift experiments. Its evidence for interpretability is largely architectural: exposure of individual metrics in a dashboard, use of structured features with clear semantics, and preference for hybrid prediction over opaque end-to-end scoring (Ebrahimi et al., 27 Apr 2026).
Within research on computational assessment of scholarly review, PeeriScope’s principal significance lies in treating review-quality evaluation as a hybrid systems problem rather than a single-model prediction task. It provides a multi-dimensional quality framework, a supervised calibration layer that reportedly outperforms both zero-shot LLM judging and direct LLM fine-tuning, and a deployable dashboard-plus-API infrastructure for real editorial and research workflows (Ebrahimi et al., 27 Apr 2026). This suggests a conception of peer-review analytics in which automation is used to structure, prioritize, and audit evaluative work rather than to replace expert judgment.