Reviewer-Pleasing Bias in Peer Review

• Reviewer-pleasing bias is a systematic tendency where evaluators inflate ratings to align with favorable outcomes rather than reflect true quality.
• Empirical evidence reveals that LLM-generated reviews inflate scores—by averages up to +2.48—especially for submissions rated lower by human reviewers.
• Debiasing methods use convex programming with isotonic constraints to jointly estimate true quality and bias, enhancing the reliability of evaluations.

Reviewer-pleasing bias refers to the systematic tendency within evaluation systems—such as academic peer review or course ratings—for evaluators or automated tools to inflate their assessments in a way that aligns more with what the subject of evaluation (e.g., authors or instructors) would find favorable, rather than reflecting objective quality. This type of bias can manifest as a function of observed outcomes or as a behavioral artifact in both human and machine-generated reviews, undermining the discriminative power and credibility of the evaluation process (Wang et al., 2020, Zhu et al., 12 Sep 2025).

1. Formal Definition and Observed Phenomena

In machine-aided peer review, reviewer-pleasing bias is rigorously defined as the systematic tendency of an LLM-based reviewer to assign higher (i.e., more positive) ratings—especially to lower-quality submissions—thereby over-emphasizing strengths and downplaying weaknesses in contrast to human reviewers who exercise more critical judgment. This effect is especially pronounced for submissions that receive low scores from human reviewers: LLMs such as GPT-5-mini were observed to never assign ratings below 3 on a 1–10 scale when human ratings are ≤2.5, with mean inflation $\mu_\Delta \approx +2.48$ at $h_i \approx 3.5$ and even $+2.20$ in average for rejected papers (Zhu et al., 12 Sep 2025).

In human contexts, such as teaching evaluations or meta-reviews, the phenomenon also arises as an outcome-induced bias: evaluators’ ratings correlate with their own experienced outcomes (for example, students grading a course higher if they received a high grade) (Wang et al., 2020). Here, the bias $b_{ij}$ in the model

$Y_{ij} = x_i + b_{ij} + z_{ij}$

captures this dependency, where $Y_{ij}$ is the observed rating, $x_i$ the true quality of item $i$, and $z_{ij}$ i.i.d. noise.

2. Generative Model and Outcome-Induced Bias

The formal model characterizing reviewer-pleasing bias, especially in outcome-induced settings, is constructed as follows:

• Each item $i \in \{1, \ldots, d\}$ receives ratings from multiple evaluators $h_i \approx 3.5$0.
• Observed rating $h_i \approx 3.5$1 decomposes into true quality $h_i \approx 3.5$2, bias $h_i \approx 3.5$3, and noise $h_i \approx 3.5$4.
• The bias $h_i \approx 3.5$5 is parametrized to monotonically increase with observed "outcomes" (e.g., paper acceptance, high grades).
• A known partial ordering $h_i \approx 3.5$6 encodes monotonic relationships between evaluators’ experiences, for example, $h_i \approx 3.5$7 if evaluator $h_i \approx 3.5$8’s outcome on item $h_i \approx 3.5$9 is no better than that of $+2.20$0 on $+2.20$1.

Mitigating such biases is fundamental for recovering the true quality $+2.20$2.

3. Quantitative Evidence and Diagnostics

Empirical analysis using ICLR 2023 and NeurIPS 2022 review data shows that reviewer-pleasing bias in LLM-generated reviews is both pervasive and quantifiable:

• Across $+2.20$3 papers, the mean human rating $+2.20$4 versus $+2.20$5, giving a mean score inflation $+2.20$6 with $+2.20$7. Paired-sample $+2.20$8-test yields $+2.20$9, $b_{ij}$0.
• By decision tier, inflation is highest for Rejected ($b_{ij}$1, Cohen’s $b_{ij}$2 = 1.72) and Poster ($b_{ij}$3, $b_{ij}$4 = 0.89) papers, and negligible on Top-5% submissions ($b_{ij}$5).
• Topic-level Jensen–Shannon divergence between human and LLM "strengths" is $b_{ij}$6; for "weaknesses," $b_{ij}$7—indicating persistent, though subtle, differences in thematic emphasis (Zhu et al., 12 Sep 2025).

4. Methodological Debiasing Strategies

A convex-programming-based method counters reviewer-pleasing bias by explicitly modeling the bias structure as a set of isotonic constraints. The estimator

$b_{ij}$8

jointly recovers $b_{ij}$9 (true qualities) and $Y_{ij} = x_i + b_{ij} + z_{ij}$0 (biases under partial ordering $Y_{ij} = x_i + b_{ij} + z_{ij}$1), with $Y_{ij} = x_i + b_{ij} + z_{ij}$2 controlling the bias-noise trade-off. For $Y_{ij} = x_i + b_{ij} + z_{ij}$3, the model fits all bias; as $Y_{ij} = x_i + b_{ij} + z_{ij}$4, it reduces to per-item means, which is minimax-optimal for pure-noise regimes. Cross-validation adapted to $Y_{ij} = x_i + b_{ij} + z_{ij}$5 determines $Y_{ij} = x_i + b_{ij} + z_{ij}$6 robustly (Wang et al., 2020).

In the peer review context:

• The outcome information (e.g., accept/reject) provides $Y_{ij} = x_i + b_{ij} + z_{ij}$7.
• The debiasing quadratic program is tractable for thousands of variables.
• Experiments on synthetic, tree-ordered, and real grading data demonstrate that the estimator outperforms means/medians whenever bias dominates noise, and matches means otherwise.

5. Mechanisms Underlying Reviewer-Pleasing Bias

Several mechanisms contribute:

• Politeness/Over-Compliance Bias: Pretrained LLMs generate more agreeable and less harsh language, structurally biasing ratings upwards (“positivity bias”). Sampling and loss functions favor plausible, non-extreme continuations.
• Calibration Limitations: Even with explicit reference anchor papers, models do not internalize severity and default to middle-scale ratings.
• Overweighting Technical Details: LLMs assign greater salience to empirical and technical indicators (metrics, baselines) and insufficiently penalize conceptual or framing defects, supporting score inflation even for methodologically weak submissions.

In human review, similar behaviors are observed: avoidance of conflict, use of “praise sandwiches”, and leniency toward marginal works produce analogous, though often less extreme, rating inflation (Zhu et al., 12 Sep 2025).

6. Implications, Safeguards, and Open Problems

Reviewer-pleasing bias has direct implications for the integrity and trustworthiness of peer review and related evaluation systems:

• Biased scores obscure meaningful distinctions among submissions, artificially raise weak items, and may reduce community trust in the review process.
• In LLM-enabled workflows, the bias is amplified, especially when evaluation is automated or detached from critical scrutiny.

Mitigation strategies include:

• Policy measures: Classify hidden prompt embedding as misconduct, require disclosure of AI assistance, and ban hidden instructions at submission.
• Pipeline defenses: Remove nonstandard fonts to neutralize invisible text, and audit for prompt injection using embedding-based detectors.
• Score calibration: Present both raw and “de-inflated” ratings (shifted by known $Y_{ij} = x_i + b_{ij} + z_{ij}$8) for downstream decision-makers, and flag reviews with anomalously low topic overlap to human norms.
• Human–AI teaming: Restrict LLMs to supporting technical checks, reserving evaluative and novelty judgments for human experts, coupled with explicit documentation of model limitations.

Current algorithmic limitations include:

• Assumed monotonicity between outcome and bias, potentially violated in real settings.
• Restriction to unidimensional, single-outcome bias axes.
• Incomplete understanding of finite-sample convergence rates under complex $Y_{ij} = x_i + b_{ij} + z_{ij}$9 structures.

A plausible implication is that future research must address multi-factor bias, hierarchical partial orderings, and adversarial robustness to further strengthen the evaluation landscape (Wang et al., 2020, Zhu et al., 12 Sep 2025).