Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews

Abstract: Peer review plays a central role in the NLP publication process, but is susceptible to various biases. Here, we study language-of-study (LoS) bias: the tendency for reviewers to evaluate a paper differently based on the language(s) it studies, rather than its scientific merit. Despite being explicitly flagged in reviewing guidelines, such biases are poorly understood. Prior work treats such comments as part of broader categories of weak or unconstructive reviews without defining them as a distinct form of bias. We present the first systematic characterization of LoS bias, distinguishing negative and positive forms, and introduce the human-annotated dataset LOBSTER (Language-Of-study Bias in ScienTific pEer Review) and a method achieving 87.37 macro F1 for detection. We analyze 15,645 reviews to estimate how negative and positive biases differ with respect to the LoS, and find that non-English papers face substantially higher bias rates than English-only ones, with negative bias consistently outweighing positive bias. Finally, we identify four subcategories of negative bias, and find that demanding unjustified cross-lingual generalization is the most dominant form. We publicly release all resources to support work on fairer reviewing practices in NLP and beyond.

• The paper introduces the LOBSTER corpus to empirically quantify language-of-study bias in NLP peer reviews.
• It benchmarks several LLMs, with Gemini 3.1 Pro achieving 87.37 Macro F1, demonstrating robust bias detection capabilities.
• Results show non-English papers have up to 40-fold higher bias rates, highlighting the need for fair and transparent review practices.

Language-of-Study Bias in NLP Peer Review: Systematic Evidence and Implications

Introduction

This paper introduces the first comprehensive and empirical characterization of language-of-study (LoS) bias in NLP peer review, providing quantitative and qualitative evidence that review outcomes often depend not just on scientific merit but also on the specific human languages investigated. Despite prior anecdotal attention and explicit warnings in reviewing guidelines, LoS bias has remained poorly studied, with no formal taxonomy or large-scale measurements distinguishing between negative and positive forms. The authors construct the LOBSTER corpus, the first expert-annotated dataset focused specifically on LoS bias, and present robust LLM-based methodologies for detection and analysis across a large multi-venue corpus. The results show non-English papers are substantially more likely to be subject to both negative and positive biases—especially demands for generalizability and English-centric standards.

Dataset Construction and Annotation Methodology

A two-stage sampling and annotation strategy underpins the creation of LOBSTER, addressing both efficiency (surface likely bias candidates using LLMs) and coverage (ensuring label balance across languages and contribution types). Review segments, rather than entire reviews, are the classification unit to detect multiple potential biases per review. The dataset comprises 529 segments labeled as negative bias, positive bias, or no bias, each annotated by at least three NLP experts using full paper context. Substantial inter-annotator agreement (Cohen's kappa: mean 0.786) demonstrates reliability and label clarity.

The explicit operational definitions for negative and positive LoS bias are a major methodological contribution. Negative bias captures cases where reviewers penalize work for studying non-English (or low-resource) languages disproportionately, e.g., by demanding unwarranted cross-lingual generalization or treating English as a required validity check. Positive bias, though rarer, arises when reviewers praise language choice without engaging with methodologies or evidence—a pattern that can also distort scientific evaluation.

Bias Detection: Model Benchmarking and Performance

The paper benchmarks six contemporary LLMs (Gemini 3.1 Pro, Claude Opus 4.6, Grok 4.1, GPT 5.2, DeepSeek V3.2, Llama 4 Maverick) on LOBSTER, using a rigorously standardized prompt. Gemini 3.1 Pro achieves the best results with 87.37 Macro F1 and 93.60 Weighted F1, outperforming both random and majority-class baselines by over 50 points.

(Figure 1)

Figure 1: Gemini-3.1-Pro-Preview confusion matrix for bias classification, showing the conservative tendency to default to 'No Bias' in ambiguous cases.

The confusion matrix reveals that false negatives dominate, largely reflecting the prevalence of nuanced or lexically indirect bias types that challenge both humans and models. These robust results enable reliable corpus-scale extension of the analysis.

Quantitative Analysis: Rates, Polarity, and Scope

Applying the top-performing model to 15,645 reviews from six major NLP venues yields strong quantitative evidence that language-of-study bias is systematic and non-trivial:

• Bias Prevalence: Reviews of single non-English papers show a collective bias rate of 14.8%, versus only 0.4% for English-only papers—a 40-fold increase. Specified multilingual papers likewise exhibit higher rates (4.2%) than language-agnostic or unspecified multilingual papers, which are close to the English baseline.

  Figure 2: Bias rate (%) by polarity across languages with at least 20 reviews, showing dramatically higher rates for non-English and specified multilingual papers.

• Polarity: Negative biases consistently outweigh positive ones across all strata. For single-language non-English papers, 71 reviews show negative bias versus 31 positive (2.3:1). For Chinese, the imbalance is 31 negative to 6 positive (over 5:1), yet the absolute rate is much higher than for English. Positive bias, while present, manifests more for certain low-resource languages and in multilingual settings.

Cross-Language, Contribution, and Temporal Factors

The prevalence and form of LoS bias are not homogeneous:

• Per-language Variation: Top non-English studied languages (Chinese, German, Arabic, etc.) all show elevated bias rates relative to English, but with variance reflecting field focus and reviewer familiarity.

  Figure 3: Top 20 non-English studied languages in the full analysis corpus, highlighting the diversity of language focus but overwhelming dominance by Chinese.

• Contribution Type: LoS bias correlates with paper type. Data/benchmarking and linguistic analysis papers (those intrinsically tied to language choice) are most affected (~2% bias rates), whereas modeling papers—which tend to frame contributions as language-agnostic—exhibit the lowest (0.66%).

  Figure 4: Distribution of contribution types in the full analysis corpus, with most frequent types in applications, data, and analytical studies.

• Temporal Stability: Negative bias rates are highest in the earliest review cycles, with a possible slight decrease over time, though venue sampling differences preclude strong conclusions about historical trends.

Qualitative Analysis: Subcategories of Negative Bias

Manual and automated subcategory analysis unpacks the qualitative landscape of how bias manifests, revealing four major patterns:

• A—Generalizability Demand (62.16%): Penalizing lack of cross-lingual results irrespective of stated scope.
• B—English as Gold Standard (9.46%): Requiring English-based validation as an implicit universal benchmark.
• C—Language Choice Interrogation (12.16%): Questioning the motivation for focusing on certain languages.
• D—Dismissing Impact (16.22%): Downplaying work based on perceived community size or novelty.

  Figure 5: Distribution of negative bias subcategories (A–D) across language scope, illustrating increased pattern diversity for non-English and multilingual papers.

English-only papers overwhelmingly receive pattern A, while non-English and multilingual papers encounter a wider repertoire, including impact dismissal and explicit language-choice interrogation.

Implications and Future Directions

The findings demonstrate a structural pattern of linguistic inequality in NLP scientific evaluation. For reviewers, institutional actors, and policymakers, the evidence supports immediate interventions:

• Reviewing Practices: Guidelines should be restructured to require reviewers to explicitly justify criticisms about language scope with respect to the paper's stated claims, rather than implicit field-wide standards. The documented patterns provide concrete negative exemplars to avoid.
• Automated Oversight: The high F1 achieved by LLM-based detectors suggests feasibility for automated screening tools that flag candidate bias for further editorial review.

Practically, these results signal that non-English NLP research may face heightened publication barriers disconnected from methodological quality—potentially skewing scientific progress and resource allocation. Theoretically, these patterns represent a case study in how ingrained field norms and reviewer priors interact with peer evaluation, underlining the need for more sophisticated and context-sensitive fairness frameworks.

A key avenue for future work involves expanding the analysis to rejected papers (currently underrepresented due to data availability), non-NLP fields, and broader temporal horizons. Further, exploring the interplay between LoS bias and intersectional biases (gender, institutional prestige, topic selection) presents a rich domain for AI-driven meta-scientific research.

Conclusion

This work establishes, with both empirical and methodological rigor, that language-of-study bias is a systematic phenomenon in NLP peer review, affecting non-English work at rates orders of magnitude higher than field baselines and manifesting in both overt and subtle forms. The LOBSTER corpus and the associated classification pipeline create a foundation for scalable and reproducible research on bias in peer evaluation. Addressing these biases is critical for promoting linguistic equity, ensuring scientific quality, and broadening the scope of NLP research.

The essay is based on "Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews" (2604.07119).