PeerPrism: Benchmarking LLM Detection

• PeerPrism is a benchmark that evaluates LLM detection by separating evaluative reasoning (idea provenance) from surface text generation to highlight hybrid authorship.
• It organizes 20,690 reviews into multiple regimes—fully human, fully synthetic, and hybrid—to stress-test traditional detection methods.
• Empirical findings reveal that conventional LLM detectors often fail under hybrid settings, prompting a shift toward multidimensional provenance quantification.

PeerPrism is a benchmark for evaluating LLM detection in scientific peer review under conditions where evaluative reasoning and surface realization do not necessarily have the same provenance. Introduced in "PeerPrism: Peer Evaluation Expertise vs Review-writing AI" (Sadeghian et al., 16 Apr 2026), it is designed around the observation that existing peer-review LLM detection methods largely treat authorship as a binary problem—human vs. AI—without accounting for hybrid workflows in which the ideas may be human-origin while the final wording is AI-generated. The benchmark therefore targets the idea–text provenance gap and frames authorship as a multidimensional construct spanning semantic reasoning and stylistic realization.

1. Motivation and problem formulation

Peer review sits at the heart of scientific quality control. As LLMs such as GPT-5 or Claude enter reviewers’ toolkits for drafting, rewriting, expansion, and refinement, the boundary between “human-written” and “machine-written” reviews becomes difficult to interpret in a simple binary sense (Sadeghian et al., 16 Apr 2026). Traditional AI-text detectors assume that surface style faithfully tracks intellectual provenance. PeerPrism is built to test that assumption directly.

The central question is whether a detector identifies the origin of the surface text or the origin of the evaluative reasoning. In practical review workflows, these can diverge. A reviewer’s critique may be entirely human in substance while the prose is substantially AI-assisted; conversely, an LLM may generate both the ideas and the wording. PeerPrism operationalizes this distinction by disentangling idea provenance from text provenance, allowing stress-testing of detection methods under controlled hybrid regimes.

A common misconception, directly challenged by the benchmark, is that authorship in peer review can be reduced to a single label. The reported findings indicate that this framing fails when reasoning and realization are decoupled. This suggests that any detector trained or evaluated only on fully human versus fully synthetic text may be measuring only one dimension of authorship while implicitly claiming to measure both.

2. Dataset construction and provenance design

PeerPrism comprises 20,690 reviews over 160 seed papers (80 from ICLR and 80 from NeurIPS, sampled equally over 2021–2024) (Sadeghian et al., 16 Apr 2026). The dataset is organized into six regimes spanning fully human, fully synthetic, and provenance-controlled hybrids.

Data Partition	Idea Origin / Text Origin	Count
Human	Human / Human	674
Fully Synthetic	AI / AI	3,840
Rewritten	Human / AI	4,044
Extract & Regenerate	Human / AI	4,044
Expanded	Mixed / AI	4,044
Hybrid	Mixed / AI	4,044

The Human regime contains 674 genuine reviews scraped from OpenReview for the selected papers. The Fully Synthetic regime contains 3,840 reviews generated by 6 frontier LLMs (GPT-5, o4-mini, Gemini-2.5, Claude-Haiku-4.5, DeepSeek-R1, Llama-4-Scout), where each model produces four persona-driven reviews per paper (Conservative, Highly-Detailed, Lazy, Nitpicky).

The four hybrid regimes are constructed from the human reviews and are explicitly designed to separate idea provenance from text provenance. In Rewritten, the model rewrites the original human review, preserving informational content but replacing style. In Extract & Regenerate, the system first extracts a structured list of strengths and weaknesses from the human review plus manuscript, then regenerates a review from this structured “idea sketch,” with evaluative points remaining human-origin while the text is fresh AI output. In Expanded, the model receives the human review plus manuscript and elaborates it, introducing some new AI-generated points while preserving the core human critique. In Hybrid Augmentation, the model merges one human review with four independently LLM-generated reviews of the same paper, synthesizing a meta-review that blends human insights and AI suggestions.

Each review instance is annotated as

$$D_i = \bigl\{\,T,\; O_{\text{idea}},\; O_{\text{text}},\; M,\; G,\;\text{Meta}\bigr\}$$

where $T$ is the review text, $$O_{\text{idea}} \in \{\text{Human}, \text{AI}, \text{Mixed}\}$$, $O_{\text{text}} \in \{\text{Human}, \text{AI}\}$, $M$ is the generating model or N/A, $G$ is the generation regime, and Meta contains $$\{\text{venue}, \text{year}, \text{decision outcome}\}$$. By construction, the benchmark disentangles whether evaluative reasoning originated from a human or LLM versus whether the final wording was human-typed or AI-generated.

3. Benchmark tasks and evaluation metrics

PeerPrism defines two primary evaluation regimes (Sadeghian et al., 16 Apr 2026). The first is Binary Detection, where reviews are classified as “Human” versus “AI” using the original human reviews and the fully synthetic reviews. The second is Robustness under Hybrid Regimes, where the same classifiers are applied to Rewritten, Expanded, Extract & Regenerate, and Hybrid, and the shifts in predictions are examined.

The reported metrics are the standard binary classification measures for the “AI” class: $$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN},$$

$$\text{Precision} = \frac{TP}{TP + FP}, \quad \text{Recall} = \frac{TP}{TP + FN},$$

$$\mathrm{F1} = 2 \cdot \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}.$$

The benchmark also reports macro-averaged F1 across both classes.

The choice of tasks is central to the benchmark’s contribution. The binary setting measures conventional detector performance under the cleanest human-versus-AI contrast. The hybrid setting measures whether the same detectors remain interpretable when idea provenance and text provenance no longer align. A plausible implication is that a detector that appears strong under the binary task may not provide a semantically principled judgment once the review-writing process becomes collaborative.

4. Detector families and empirical behavior under hybridization

PeerPrism benchmarks seven off-the-shelf detectors, each representing a distinct paradigm (Sadeghian et al., 16 Apr 2026): GLTR (Likelihood-based), Binoculars (Likelihood-ratio), DetectGPT, Fast-DetectGPT, Lastde++ (Perturbation-based), RADAR (Supervised classifier), and Anchor (Context-aware embedding similarity).

On the binary Human vs. Fully Synthetic task, the reported results are:

Method	Overall Acc.	Overall F1
Anchor	90.39%	84.75%
Binoculars	30.99%	30.96%
DetectGPT	53.25%	46.97%
Fast-DetectGPT	89.45%	82.72%
GLTR	30.59%	27.55%
Lastde++	52.25%	48.99%
RADAR	25.47%	25.40%

The benchmark identifies Anchor and Fast-DetectGPT as high performers on the clean binary task, both at approximately 90% accuracy. It also reports that Likelihood-based methods such as GLTR, ratio-based methods such as Binoculars, and the supervised method RADAR collapse near chance on modern, highly fluent LLM outputs, especially GPT-5, Gemini, and Claude. By contrast, Perturbation methods and embedding methods show greater robustness across generator families.

The hybrid regimes produce the benchmark’s most consequential result. On the Rewritten set, GLTR flags approximately 65% as AI, Fast-DetectGPT flags approximately 26% as AI, and Anchor flags approximately 18% as AI. On Expanded reviews, Fast-DetectGPT flags 74% as AI, Anchor 18% as AI, and GLTR 28% as AI. The paper concludes that no two detectors agree consistently under these mixed-provenance conditions. Some systems appear to latch onto machine-like phrasing, while others appear to follow semantic lineage. This suggests that different detector families are responding to different latent signals while outputting the same nominal label.

5. Stylometric and semantic analyses

To account for detector disagreement, the benchmark includes stylometric and semantic analyses (Sadeghian et al., 16 Apr 2026). The reported stylometric features compare human and synthetic reviews using mean values. Lexical Diversity (Type-Token Ratio) is .55 for Human vs. .61 for Synthetic. Readability (Flesch Ease) is 37.8 for Human vs. 14.0 for Synthetic. First-Person Pronouns occur 5.04 per review for Human vs. 0.37 for Synthetic. Questions occur 2.34 for Human vs. 3.68 for Synthetic. External Citations are 0.95 for Human vs. 0.31 for Synthetic. Manuscript References are 1.57 for Human vs. 1.23 for Synthetic.

The transformed reviews are described as lying in between these endpoints: style shifts toward AI, with fewer first-person pronouns and harder reads, but citation and referencing remain human-like or are even amplified. This distinction is important because it provides a mechanism for disagreement among detectors. A style-sensitive detector may treat reduced first-person language and lower readability as evidence of AI involvement, while a detector more aligned to content may still register continuity with the original human review.

Semantic similarity is reported using gte-multilingual embeddings. The average cosine scores are: Human–Human (same paper): 0.83, Synthetic–Synthetic (same paper): 0.92, Human–Transformed: 0.92, Synthetic–Transformed: 0.88, Human–Manuscript: 0.82, and Synthetic–Manuscript: 0.86. The benchmark interprets these values as showing that LLM critics form an “echo chamber,” converging on similar semantic points, whereas human critiques are more diverse. It also reports that transformed reviews inherit human semantic cores while shifting stylistically toward AI.

These analyses directly support the benchmark’s central claim: the same review can remain close to human reasoning in semantic space while looking machine-like in surface form. A plausible implication is that a single scalar authorship label discards structurally important information about the review-generation process.

6. Authorship model, limitations, and research directions

The principal conclusion is that binary detectors conflate surface style with idea origin, and that authorship is fundamentally multidimensional (Sadeghian et al., 16 Apr 2026). A detector that flags AI style as “AI” may mislabel human ideas behind AI-rewritten text, while a detector that tracks semantics may miss AI-origin ideas expressed in more human-like style. Under hybrid regimes, the benchmark reports that detector consensus fractures and that the binary Human-vs.-AI framing fails whenever reasoning and realization are decoupled.

The authors therefore recommend moving from binary classification to provenance quantification. The proposed direction separates semantic contribution estimation from stylistic realization estimation. The paper gives, as an example, a vector

$T$0

to indicate degrees of human versus machine authorship in reasoning and expression. This is presented as a future research direction rather than an implemented benchmark component.

The benchmark also delineates several limitations. Its domain scope is restricted to ICLR and NeurIPS reviews, English, ML domain. The model set is focused on high-capacity LLMs, leaving smaller or more specialized models untested. The four controlled pipelines do not exhaust the continuum of human–LLM collaboration. The reported detection thresholds use off-the-shelf settings or held-out calibration, and fine-tuning on PeerPrism might alter results. Finally, the paper notes ethical considerations, including the risk of misuse through over-flagging benign LLM assistance and the need for governance beyond technical benchmarks.

Within these constraints, PeerPrism establishes a benchmark definition of peer-review authorship in which what was thought and how it was worded are analytically distinct. The benchmark’s significance lies in making that distinction measurable, and in showing that detector performance under conventional binary evaluation does not resolve the more difficult question of provenance in realistic hybrid review workflows.