Language-of-Study Bias

• Language-of-Study Bias is a phenomenon where the language of presentation affects model behavior and outcomes despite equivalent underlying content.
• It spans settings such as automated grading, peer review, and scientific NLP, revealing penalties for non-native phrasing and style deviations.
• Operational measures include score differentials and accuracy gaps across languages, underscoring the need for multilingual calibration and fair evaluation protocols.

Language-of-Study Bias denotes systematic variation in model behavior, evaluation outcomes, or knowledge access that arises from the language in which a task is written, prompted, studied, or reviewed, rather than solely from the underlying content or scientific merit. Across recent work, the term has been used to describe style-induced penalties in automated grading, cross-lingual shifts in stereotype bias, unequal treatment of papers because of the language they study, English-centric distortions in scientific NLP pipelines, and performance disparities across languages in educational LLM deployments (Jadhav et al., 19 Mar 2026, Liang et al., 17 Dec 2025, Barkhordar et al., 8 Apr 2026, Ebrahimi et al., 2024). Taken together, these studies treat Language-of-Study Bias not as a single benchmark artifact but as a recurrent dependence of outputs, judgments, or representations on linguistic presentation, language choice, or multilingual evaluation design.

1. Conceptual scope and boundary conditions

In the grading literature, Language-of-Study Bias refers to systematic score penalties applied to responses whose surface form reflects the writer’s linguistic background or instructional context, such as non-native English phrasing or informal register, even when the underlying content is equally correct. In multilingual bias evaluation, the same term denotes the dependence of measured bias on the language in which a fixed model is prompted and assessed. In peer review, it denotes differential evaluation of a paper because of the human language it studies rather than its scientific claims. In scientific document processing, it denotes distortions introduced when English-only models, tokenizers, and benchmarks are applied to multilingual corpora (Jadhav et al., 19 Mar 2026, Liang et al., 17 Dec 2025, Barkhordar et al., 8 Apr 2026, Ebrahimi et al., 2024).

This usage is adjacent to, but distinct from, several related constructs. It is not identical to publication language bias, which concerns the sociological dominance of English in publishing; nor is it identical to indexing bias, which concerns metadata coverage. It also differs from national bias and cultural bias, although these can interact with language-conditioned behavior. In multilingual evaluation methodology, the term further extends to conclusions that change because the evaluated language set is typologically imbalanced or aggregated with naive averages, thereby privileging dominant language clusters (Pikuliak et al., 2023).

The scope of the phenomenon is correspondingly broad. It includes bias in content scoring, stereotype elicitation, positional processing, recommendation quality, peer review discourse, citation choice in multilingual RAG, and scientific retrieval or summarization. A plausible implication is that Language-of-Study Bias is best regarded as a family of content-noninvariant linguistic dependencies rather than a single failure mode.

2. Operationalization and measurement

The literature operationalizes Language-of-Study Bias by holding task semantics or relevance fixed while varying linguistic realization. In automated grading, the core statistic is the paired score difference $$\Delta = \text{Base Score} - \text{Perturbed Score}$$, where $\Delta > 0$ indicates a penalty induced by grammar errors, informal language, or non-native phrasing despite preserved reasoning and conclusions (Jadhav et al., 19 Mar 2026). In multilingual bias benchmarking, translated prompts and word lists are used to compare explicit bias in BBQ and implicit bias in prompt-based association tests across languages, while educational advising work measures language-conditioned divergence in score distributions with Jensen–Shannon Divergence and targeted local-language uplift with Mean Divergence (Liang et al., 17 Dec 2025, Liu et al., 25 Feb 2025).

In multilingual evaluation methodology, Language-of-Study Bias is also quantified at the aggregate level. The naive average

$\mu = \frac{1}{|L|} \sum_{l \in L} s(l)$

is contrasted with family macro-averaging $$\mu_{\text{family}} = \frac{1}{|F|} \sum_{f \in F} \left( \frac{1}{|L_f|} \sum_{l \in L_f} s(l) \right)$$, and the gap $\Delta = \mu - \mu_{\text{family}}$ is treated as evidence that typological imbalance distorts multilingual conclusions (Pikuliak et al., 2023). In multilingual RAG, the preference for English or for the query language is measured through citation-token prediction accuracy gaps, $$\Delta(\ell_{\mathrm{target}}) = \mathbf{Acc}^{(\ell_{\mathrm{target}})} - \mathbf{Acc}^{(\mathrm{en})}$$, under contexts where relevance is controlled by parallel evidence (Ki et al., 17 Sep 2025).

Setting	Operationalization	Representative metric
Automated grading	Base vs. style-perturbed matched responses	$$\Delta = \text{Base Score} - \text{Perturbed Score}$$
Cross-lingual bias evaluation	Same benchmark translated across languages	Bias scores, accuracy, implicit association score
Multilingual advising/RAG	Same decision or citation context across languages	JSD, Mean Divergence, citation accuracy gap
Multilingual evaluation design	Re-aggregation across language families	$\mu$, $\mu_{\text{family}}$, $\Delta$
Peer review	Human-annotated review segments	Bias rate, negative/positive polarity, Macro F1

The measurement regimes are heterogeneous. Some studies report paired $\Delta > 0$0-tests, Cohen’s $\Delta > 0$1, confidence intervals, or nested $\Delta > 0$2-tests, while others remain descriptive and explicitly omit inferential statistics or multiple-comparison corrections. This methodological variation is itself important: the field has converged on the existence of language-conditioned disparities more quickly than on a unified inferential standard (Jadhav et al., 19 Mar 2026, Liang et al., 17 Dec 2025, Zhou et al., 20 Jan 2026).

3. Educational assessment and tutoring

Educational use cases provide some of the clearest controlled evidence. In "Implicit Grading Bias in LLMs: How Writing Style Affects Automated Assessment Across Math, Programming, and Essay Tasks" (Jadhav et al., 19 Mar 2026), a controlled dataset of 180 perturbed student responses plus 60 base answers was graded by LLaMA 3.3 70B Instruct and Qwen 2.5 72B Instruct on a 1–10 scale under explicit anti-bias instructions. Essay/Writing tasks showed statistically significant penalties across both models and all perturbation types, with effect sizes ranging from medium to very large; informal language received the heaviest penalty, with mean deductions of 1.90 points for LLaMA and 1.20 for Qwen, while non-native phrasing was penalized by 1.35 and 0.90 points respectively. Mathematics and Programming showed minimal or inconsistent bias, which the paper links to clearer objective correctness.

Related multilingual educational evaluations show that the language of interaction affects not only grading but also misconception detection, feedback selection, tutoring, and solution quality. "Multilingual Performance Biases of LLMs in Education" (Gupta et al., 24 Apr 2025) evaluates GPT-4o, Gemini 2.0 Flash, Claude 3.7 Sonnet, Llama 3.1 405B, Mistral Large 2407, and Command-A across English, Hindi, Arabic, Farsi, Telugu, Ukrainian, and Czech. Averaging across tasks other than translation grading, English achieves 70.9% versus Hindi 63.1%, Czech 55.3%, Ukrainian 67.8%, Telugu 49.7%, Farsi 66.2%, and Arabic 67.4%. The same study reports that English prompts generally outperform translated prompts, with an average of 71.6% versus 66.2% across tasks except tutoring, and that complex tutoring degrades sharply in Telugu and Czech for several models.

A more narrowly curricular study, "Investigating Bias: A Multilingual Pipeline for Generating, Solving, and Evaluating Math Problems with LLMs" (Mahran et al., 22 Sep 2025), translates 628 German K–10 mathematics exercises into English, German, and Arabic and evaluates step-by-step solutions with a held-out panel of LLM judges. English consistently receives the highest proportion of “Best” rankings, Arabic often the highest “Worst” rankings, and German generally lies between them. The asymmetry is especially strong for GPT-4o-mini, where English solutions receive 493 “Best” rankings and Arabic 38. The paper’s qualitative analysis attributes English preference to more explicit pedagogical structure, while Arabic solutions are often judged less segmented or less pedagogically complete.

These studies jointly establish that educational Language-of-Study Bias is not restricted to raw correctness. It affects rubric-following, explanatory depth, tutoring interaction, and judged pedagogical quality. A common pattern is that explicit instructions to ignore style or translated prompt parity do not reliably remove the bias.

4. Multilingual model behavior, evaluation, and training effects

Cross-lingual benchmarking shows that Language-of-Study Bias is not confined to educational tasks. "Cross-Language Bias Examination in LLMs" (Liang et al., 17 Dec 2025) evaluates GPT-4 in English, Chinese, Arabic, French, and Spanish using BBQ and a prompt-based Implicit Association Test. Arabic and Spanish show the highest overall explicit stereotype levels, while Chinese and English are relatively lower; by contrast, implicit bias is highest for age across all languages, with all scores above 0.75 and Arabic “almost 1.00.” The study’s central result is that explicit and implicit bias can diverge sharply by evaluation language.

Other work shows that language conditions not only bias scores but also internal processing. "Position of Uncertainty: A Cross-Linguistic Study of Positional Bias in LLMs" (Mikhail et al., 22 May 2025) finds that positional bias is model-driven but language-sensitive: Qwen2.5-7B-Instruct tends to prefer later positions in English, Russian, German, and Hindi, while Llama3-8B-Instruct prefers early positions across all five tested languages. Prompted relevance scores do not help on average, and the “All-Zero” setting can catastrophically reduce accuracy. The broader implication is that even conclusions about where a model “looks” in context can be language-of-study dependent.

Pairwise evaluation systems exhibit a related asymmetry. "Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judge" (Zhou et al., 20 Jan 2026) reports large same-language judging disparities across 15 languages, with European languages consistently outperforming African languages, and shows that inter-language judging usually favors English answers. The answer language matters more than the question language, and low-perplexity bias explains only part of the effect. The paper’s regressions and nested $\Delta > 0$3-tests indicate that language identity adds significant variance beyond perplexity, especially for low-resource languages.

In retrieval-augmented generation, the same dynamic appears as citation preference. "Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG" (Ki et al., 17 Sep 2025) uses controlled parallel evidence to show that models preferentially cite English sources when queries are in English, with larger drops for lower-resource languages such as Swahili and Bengali and stronger effects for mid-context documents. In a two-document setup, accuracy decreases when the relevant document is in the target language and the distractor is in English, demonstrating that models can trade off relevance for language preference.

Training interventions do not produce a single monotonic pattern. "Do Multilingual LLMs Mitigate Stereotype Bias?" (Nie et al., 2024) finds that, under a controlled five-language decoder-only setup, multilingual training reduces CrowS-Pairs stereotyping preference and improves overall BBQ accuracy relative to monolingual training. By contrast, "Comparing Biases and the Impact of Multilingual Training across Multiple Languages" (Levy et al., 2023) finds that multilingual finetuning on sentiment analysis often amplifies across-group dispersion relative to monolingual finetuning, even when multilingual pretraining has mixed effects. This contrast indicates that multilinguality can mitigate or amplify Language-of-Study Bias depending on architecture, task, benchmark, and intervention stage.

5. Peer review, scientific communication, and information infrastructures

Peer review studies show that Language-of-Study Bias operates at both the author-background level and the paper-scope level. "You Cannot Sound Like GPT: Signs of language discrimination and resistance in computer science publishing" (Lepp et al., 12 May 2025) analyzes 76,453 ICLR reviews on 20,827 submissions and finds that higher shares of authors from Asia, China, or TOEFL-required countries are associated with fewer sentences praising clarity, more clarity critiques, and lower ratings before ChatGPT; these effects attenuate after ChatGPT but often persist. Interviews in the same study show that reviewers infer language background from grammar, “GPT style,” and non-linguistic cues, then link those cues to scientific quality.

"Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews" (Barkhordar et al., 8 Apr 2026) isolates the bias more directly at the level of the studied language. It introduces the LOBSTER dataset, reports 87.37 Macro F1 for detection, and applies the detector to 15,645 reviews over 7,906 papers. English-only papers have a bias rate of 0.37%, while single non-English papers have a bias rate of 14.79%, approximately 40 times higher; negative bias exceeds positive bias, and the dominant negative subtype is unjustified generalizability demand, accounting for 62.16% of negative instances in the annotated set. The paper thereby distinguishes Language-of-Study Bias from generic weak reviewing.

The same English-centric tendency appears in scientific NLP infrastructure. "Since the Scientific Literature Is Multilingual, Our Models Should Be Too" (Ebrahimi et al., 2024) shows, on 28.58 million high-confidence language-identified Semantic Scholar abstracts, that English accounts for 85.11% but the next five languages collectively account for 10.37%, with 50 languages detected overall. English-only SciBERT produces extreme unknown-token rates for several non-Latin scripts, while XLM-R’s maximum unknown-token percentage remains below 2.98%. The paper also documents user-facing summarization failures on non-English scientific papers, including unrelated or harmful hallucinations.

At the level of knowledge organization, "Multiple Texts as a Limiting Factor in Online Learning: Quantifying (Dis-)similarities of Knowledge Networks across Languages" (Mehler et al., 2020) demonstrates that Wikipedia’s language editions build different item networks for the same subject areas. Across 35 languages and 25 topics, cross-language similarities in hyperlink structure and topical content are often low, and there is no universal lingua franca. This suggests that Language-of-Study Bias can emerge even before model inference, through the language-conditioned structure of the information landscape itself.

Methodological work further shows that evaluation protocols can manufacture apparent multilingual competence. "Average Is Not Enough: Caveats of Multilingual Evaluation" (Pikuliak et al., 2023) argues that naive averaging over typologically imbalanced language sets privileges dominant clusters such as Germanic–Italic–Slavic languages and can reverse or obscure practical conclusions. In this sense, Language-of-Study Bias can reside in the evaluator’s aggregation scheme as much as in the model under study.

6. Mechanisms, mitigation, and unresolved issues

The literature attributes Language-of-Study Bias to several interacting mechanisms. Recurrent explanations include asymmetries in pretraining data and alignment coverage, tokenization and subword segmentation differences across scripts and morphology, prompt and translation artifacts, and task-specific reliance on stylistic cues when objective correctness is hard to verify (Liang et al., 17 Dec 2025, Ebrahimi et al., 2024, Zhou et al., 20 Jan 2026). In grading, the style penalty is strongest in essays and weakest in mathematics or programming, which supports the claim that subjective or holistic judgments create more room for linguistic proxies (Jadhav et al., 19 Mar 2026). In peer review, reviewer ideologies and indexical inference connect perceived “good English” to perceived “good science,” and these inferences adapt rather than disappear after new tools such as ChatGPT enter the workflow (Lepp et al., 12 May 2025).

Mitigation proposals are correspondingly multi-level. Automated grading work recommends perturbation-based audits, style-neutral rubrics, structured outputs, post-hoc correction factors, style-normalizing preprocessing, ensemble judges, and human oversight for essays before institutional deployment (Jadhav et al., 19 Mar 2026). Cross-lingual bias work recommends reporting explicit and implicit bias per language, auditing both ambiguous and disambiguated settings, and using language-specific calibration, multilingual debiasing, and culturally aware datasets rather than relying on English-centric fairness checks (Liang et al., 17 Dec 2025). Educational deployment studies recommend validating each model on the target language and task before use, preferring models with demonstrated multilingual consistency, and treating prompt language as a first-order variable rather than a cosmetic translation choice (Gupta et al., 24 Apr 2025, Mahran et al., 22 Sep 2025).

In scientific communication and evaluation, proposed remedies include rubric reforms that separate clarity support from scientific merit, reviewer training on LoS bias, automated screening of review text, multilingual publishing pathways, and per-language reporting rather than undifferentiated averages (Barkhordar et al., 8 Apr 2026, Lepp et al., 12 May 2025, Pikuliak et al., 2023). Scientific NLP work adds infrastructural remedies: multilingual encoders, language detection, MT fallback, hybrid text-graph retrieval, parity-oriented evaluation, and explicit language coverage reporting (Ebrahimi et al., 2024).

Several unresolved issues remain. Many studies rely on machine translation without back-translation or cultural localization; some use LLM judges without human calibration; several report descriptive patterns without inferential statistics; and per-language sample sizes are often small outside the dominant languages (Liang et al., 17 Dec 2025, Mahran et al., 22 Sep 2025, Barkhordar et al., 8 Apr 2026). The most important unresolved theoretical point is that multilinguality itself is not a sufficient remedy: multilingual pretraining can reduce bias in one setting while multilingual finetuning amplifies it in another (Nie et al., 2024, Levy et al., 2023). This suggests that Language-of-Study Bias is not merely a data-balance problem, but a broader interaction among linguistic representation, training objective, evaluation design, and institutional norms.