---
title: Generation–Review Asymmetry in Peer Review
url: https://www.emergentmind.com/topics/generation-review-asymmetry
type: topic
---

# Generation–Review Asymmetry in Peer Review

Generation–review asymmetry refers to the systematic divergence between the capability of large language models (LLMs) to generate plausible peer review text and their ability to authentically replicate the evaluative depth, calibration, and instruction-following fidelity of human reviewers. This phenomenon becomes particularly salient as LLMs are deployed for scientific peer review, both as explicit agents of review generation and as silent assistants in hybrid workflows. The term encompasses empirical findings that AI-generated reviews—while structurally fluent and superficially convincing—reveal strong biases, calibration failures, and a disconnect between surface realization and substantive evaluative reasoning. This asymmetry also challenges the robustness of AI-authorship detection methods, especially where human intellectual contribution is separated from the surface form of the review text [2510.21192, 2604.14513].

## 1. Formalization and Taxonomy of Generation Regimes

Peer review generation in the age of LLMs spans multiple provenance regimes defined along two axes: the origin of evaluative ideas and the origin of surface textual realization. Let $R$ denote a set of peer reviews, each $r\in R$ annotated as $(O_{\rm idea}(r), O_{\rm text}(r))$:

- **Fully human reviews (HH):** $O_{\rm idea}(r) = \text{Human}$ and $O_{\rm text}(r) = \text{Human}$
- **Fully synthetic (AI) reviews (AA):** $O_{\rm idea}(r) = \text{AI}$ and $O_{\rm text}(r) = \text{AI}$
- **Hybrid—human ideas, AI text (HA):** $O_{\rm idea}(r) = \text{Human}$ and $O_{\rm text}(r) = \text{AI}$
- **Hybrid—AI ideas, human text (AH):** $O_{\rm idea}(r) = \text{AI}$ and $O_{\rm text}(r) = \text{Human}$ (not instantiated in PeerPrism)

The GenReview dataset operationalizes large-scale generation of LLM-written peer reviews via three-branch prompting (neutral, positive, negative), enabling direct comparison with human-authored counterparts across 32,652 ICLR submissions [2510.21192]. PeerPrism further establishes a regime-annotated corpus to systematically evaluate the multidimensionality of authorship in review processes [2604.14513].

## 2. Empirical Manifestations of Asymmetry: Calibration and Bias

LLMs demonstrate a pronounced calibration asymmetry when tasked with peer review generation. Quantitative evidence from GenReview shows that:

- Under **neutral prompting**, the mean rating $\bar s_{neu} \approx 7.3$, yielding a bias $B_{neu} \approx +1.8$ above the notional midline $\tau = 5.5$. Only 35 of 32,652 neutral reviews gave a rating of 5; the modal AI-generated rating is 8.
- **Positive prompting** almost always yields ratings of 8 or 9 ($\bar s_{pos} \approx 8.2$).
- **Negative prompting** leads to universal recommendations to reject ($\bar s_{neg} \approx 3.9$, with 85% assigning a 4).

Statistical testing ($\chi^2$, $p<0.001$) confirms that these effects are significant. Notably, even neutral LLM reviews exhibit an overwhelming acceptance bias, failing to replicate the critical selectivity and finer granularity of human calibration [2510.21192]. This points to a systematic generation–review asymmetry: LLMs, even when correctly structured, do not operationalize nuanced negative evaluations as humans do.

## 3. Detection Methodologies and the Limits of Binary Attribution

AI-authorship detectors—such as Binoculars [2510.21192] and SOTA detectors benchmarked in PeerPrism [2604.14513]—nominally achieve high precision and recall in distinguishing fully human (HH) from fully synthetic (AA) reviews. For example, Binoculars yields 100% recall and ≈99.3% precision on GenReview, with low false positive rates in pre-2023 human reviews.

However, when applied to **hybrid regimes—particularly HA (human ideas, AI text)**—detection accuracy and inter-detector agreement degrade sharply. In PeerPrism:

- On HA-rewritten reviews, Anchor predicts 35% Human/65% AI, while Fast-DetectGPT yields 74% Human/26% AI; disagreement rates exceed 0.5 between detectors.
- Average detector accuracy against "true" label (Human) for HA varies from <30% to >70% depending on method.

This divergence arises because detectors conflate surface stylometric properties with semantic provenance. Some detectors latch onto text-level artifacts (flagging any AI-generated surface as AI), while others are sensitive to preserved evaluative structure (semantically labeling as Human when the critique originates from human inputs) [2604.14513]. This exposes the inadequacy of binary attribution frameworks in complex real-world workflows.

## 4. Stylometric and Semantic Analyses Across Regimes

Empirical measurements in PeerPrism demonstrate stylometric and semantic shifts induced by LLM rewriting:

| Regime         | TTR    | Readability | 1st-person |
|----------------|--------|-------------|------------|
| HH (human)     | 0.55   | 37.8        | 5.04       |
| AA (AI)        | 0.61   | 14.0        | 0.37       |
| HA (hybrid)    | 0.56   | 19.5        | 2.34       |

- **Type–Token Ratio (TTR):** Increases slightly in AI and hybrid reviews, suggesting greater lexical diversity under LLM generation.
- **Readability (Flesch):** Decreases (lower score) in LLM regimes, indicating syntactically denser, potentially less accessible prose.
- **Authorial Markers:** 1st-person pronoun usage is suppressed by LLMs (5.04 → 0.37–2.34 per 100 tokens), reflecting the surface neutralization characteristic of automated generation.
- **Paper-grounding Features:** Citation and explicit manuscript reference counts increase in LLM and hybrid-generated text.

Semantic analysis further reveals that the core evaluative structure of HA reviews inherits from the source human critique ($\cos \approx 0.92$ similarity), while surface features increasingly approximate synthetic norms. This illustrates that generation–review asymmetry is not merely a superficial artifact but persists across both content and form dimensions [2604.14513].

## 5. Instruction-Following and Fidelity Failures

In GenReview, instruction-following by LLMs is uneven:

- **Summary Section Length:** Compliance ($C_s$) is high (≈94% in the 100–300 word range).
- **Main Review Length:** Compliance ($C_b$) collapses to ≈12% (target 800–1000 words), with many reviews truncated below 700 words.
- **Sectional Structure and Ratings:** Most LLM reviews include the required structural keywords and explicit rating (0.4% omission rate), with rare hallucinations or mis-ordered sections (<1%).

This inconsistency highlights a further form of generation–review asymmetry: LLMs reliably follow some shallow syntactic constraints but struggle with compound or nested instructions, compromising evaluative substance [2510.21192]. A plausible implication is that generation quality is not uniform across requested review dimensions, and post-hoc or calibrated prompting may be required to enforce compliance.

## 6. Implications for Authorship Attribution and Peer Review Integrity

The collective evidence demonstrates that generation–review asymmetry undermines both the procedural authenticity of peer review and the operational validity of LLM-authorship detectors. Specifically:

- **Binary Detection Pitfall:** AI-authorship detection, when restricted to HH/AA dichotomies, yields deceptively high metrics. In hybrid and real-world workflows, detectors' predictions become regime-dependent, converging neither on surface nor substantive provenance [2604.14513].
- **Authorship as Multidimensional Construct:** Authorship must be modeled as a vectorial combination of semantic (idea provenance) and stylistic (surface realization) contributions, not as a scalar label. Policy and tooling in review venues will need to distinguish between language assistance (stylistic mediation) and intellectual delegation (evaluative reasoning).
- **Bias and Calibration Risks:** Systematic positive skew in LLM reviews risks eroding the function of peer review as a critical filter, especially for lower-quality or borderline submissions.
- **Robustness and Integrity:** Future detection, attribution, and fidelity-checking tools must move toward provenance quantification and multidimensional authorship modeling to accurately sustain the integrity of scientific evaluation [2510.21192, 2604.14513].

## 7. Future Directions and Open Problems

Empirical insights from GenReview and PeerPrism indicate several open research challenges:

- **Provenance Quantification:** Developing detector frameworks that disentangle and quantify the independent contributions of human and AI agents across both semantic and stylistic axes.
- **Calibration Mitigation:** Designing calibrated prompting strategies or post-hoc adjustment mechanisms to correct for LLM bias and achieve human-like critical discrimination.
- **Hybrid Workflow Transparency:** Building detection and attribution pipelines that are robust to an increasing prevalence of hybrid (HA, AH) review generation regimes.
- **Instructional Robustness:** Enhancing LLMs’ ability to follow complex, context-dependent reviewing guidelines without shortcutting or hallucinating evaluative structures.

Taken together, generation–review asymmetry marks both a technical and normative frontier in the integration of LLMs within scientific peer review. Its resolution will require co-evolution of dataset construction, detector design, policy frameworks, and continuous evaluation of both surface and substantive provenance in scholarly communication [2510.21192, 2604.14513].

Source: https://www.emergentmind.com/topics/generation-review-asymmetry