StylisticBias: Bias from Surface Style

• StylisticBias is defined as the systematic dependence on surface style cues, such as writing style and appearance, rather than underlying semantic content.
• It spans various domains like media analysis, information retrieval, automated grading, and dialogue, highlighting measurable distortions based solely on stylistic variations.
• Research utilizes controlled experiments and benchmarks—including counterfactual data augmentation and MLLM evaluations—to quantify and mitigate these style-induced biases.

StylisticBias denotes systematic dependence of judgments, rankings, scores, or generated outputs on surface form, writing style, or appearance cues rather than on underlying semantic content alone. Across recent work, the term covers lexical and framing bias in media, style-conditioned distortions in retrieval and preference modeling, grading penalties for informal or non-native phrasing, user-preference effects in dialogue, and appearance-driven social judgments in multimodal models. It is also the name of a controlled benchmark for evaluating attribute-level social bias in multimodal LLMs (MLLMs) by holding identity fixed and changing one visual attribute at a time (Cruickshank et al., 2023, Cao, 2024, Bharadwaj et al., 5 Jun 2025, Kolli et al., 18 Jun 2026).

1. Conceptual scope

In media analysis, a central distinction is between topical bias—what topics or events are covered—and writing-style bias—how those topics or events are described. Within writing-style bias, the literature isolates lexical bias (word choice bias) and framing bias, including “information bias,” where side information changes how a central event is perceived (Cruickshank et al., 2023). In document analysis, an allied notion is subjective bias, defined as language that “should be neutral and fair” but is “skewed by feeling, opinion, or taste,” which makes stylistic bias a problem of wording rather than factual correctness (Suresh et al., 2023).

This scope has widened in recent work. In preference modeling, stylistic bias appears as overreliance on “length, structure, jargon, sycophancy and vagueness,” even when these features are only weakly related to human preference (Bharadwaj et al., 5 Jun 2025). In information retrieval, the same term refers to systematic performance differences between semantically equivalent texts that differ only in writing style, such as formal, concise, data-driven prose versus informal, emotive, or anecdotal variants (Cao, 2024). In dialogue, the relevant distinction is between subjective stylistic similarity, as perceived by the user, and objective stylistic similarity, as assigned by third-party annotators; these capture different phenomena and lead to different judgments of conversational quality (Numaya et al., 15 Jul 2025).

A recurring theme is that style is not a single variable. The xSLUE benchmark explicitly organizes 15 styles into four groups—figurative, personal, affective, and interpersonal—including formality, politeness, humor, sarcasm, metaphor, sentiment, offense, emotion, romance, age, ethnicity, gender, education level, country, and political view (Kang et al., 2019). Taken together, these formulations suggest that StylisticBias is best understood as a multidimensional dependence structure rather than a single scalar distortion.

2. Operationalization and measurement

The literature operationalizes StylisticBias differently across tasks, but most settings share a counterfactual logic: hold content fixed and vary style, then measure the resulting shift in model behavior.

Setting	Unit of comparison	Core measure
Media writing bias	Same-event articles and sentences	Sentence similarity, sentiment filtering, article/domain networks (Cruickshank et al., 2023)
Information retrieval	Same semantics, 10 document styles	Average rank and unfairness score (Cao, 2024)
Document bias scoring	Pairs of revisions of the same article	Bradley–Terry-style pairwise bias score (Suresh et al., 2023)
Automated grading	Base vs perturbed answer	$$\Delta = \text{Base Score} - \text{Perturbed Score}$$ (Jadhav et al., 19 Mar 2026)
MLLM social judgment	Base face vs single-attribute variation	$\phi_i(x)$, $\Delta_i(x_v)$, and $\mathrm{SBS}(x_v)$ (Kolli et al., 18 Jun 2026)

In information retrieval, one formalization ranks an original document and nine style rewrites for the same query: $$R_i = Rank\left(\left[ \text{Cosine}(\text{Emb}(q_i),\ \text{Emb}(d)) \ \text{for}\ d \in D_i \right]\right)$$ and then averages those ranks: $\overline{R} = \frac{1}{N} \sum_{i=1}^{N} R_i$ A style-fair model should assign similar average ranks across styles. The paper summarizes dispersion with the unfairness score

$$Score = \big( \max(\overline{R}) - \min(\overline{R}) \big) \times \text{std}(\overline{R})$$

where larger values indicate stronger stylistic bias (Cao, 2024).

For document-level bias scoring, a Bradley–Terry-style formulation assigns each document a scalar bias score $s_i$ and models pairwise preference as

$P(i \succ j) = \frac{e^{s_i}}{e^{s_i} + e^{s_j}}$

with the document score written as

$$s_i = \widehat{\mathbf{t}}_i^{\top} \mathbf{W}\,\widehat{\mathbf{t}}_i + \mathbf{b}^{\top} \widehat{\mathbf{t}}_i$$

This yields an interpretable decomposition from document representations to bias scores and, through derived word scores, to lexically bias-inducing terms (Suresh et al., 2023).

In grading, the core quantity is explicitly counterfactual: $\phi_i(x)$0 where the perturbation changes grammar, register, or phrasing but preserves content correctness. Positive $\phi_i(x)$1 indicates penalization attributable to style (Jadhav et al., 19 Mar 2026).

The named multimodal benchmark “StylisticBias” defines, for scenario $\phi_i(x)$2 and image $\phi_i(x)$3, a preference score

$\phi_i(x)$4

then measures the shift induced by a single visual attribute edit: $\phi_i(x)$5 and aggregates these shifts as Signed Bias Shift: $\phi_i(x)$6 This separates identity-level disparity from attribute-level sensitivity (Kolli et al., 18 Jun 2026).

3. Behavior in retrieval, evaluation, grading, and dialogue

In embedding-based retrieval, stylistic bias is widespread. Many universal text embedding models prefer the human-written original and Style‑2: clear, simple language, while Style‑3: informal, emojis, slang, Style‑6: motivational, Style‑7: friendly, anecdotal, and Style‑8: expressive, emotive narrative are less favored. Many models also show style matching, in which the style of the query influences the style of retrieved documents, while some models retain a fixed preference for a single style regardless of the query (Cao, 2024). The same study further reports that most text embedding models are biased toward LLM answer styles when used as evaluation metrics for answer correctness, so equal correctness can still yield different automatic scores if answer style differs (Cao, 2024).

Preference models exhibit a closely related pathology. Using controlled counterfactual pairs, one study finds that preference models favor responses with magnified biases in more than 60% of instances, with model preferences showing high miscalibration (~40%) compared to human preferences. The same work reports that bias features show only mild negative correlations with human preference labels, with mean $\phi_i(x)$7, but moderately strong positive correlations with labels from a strong reward model, with mean $\phi_i(x)$8, indicating overreliance on spurious cues (Bharadwaj et al., 5 Jun 2025).

Automated grading shows that these effects can become materially consequential. A controlled study of 180 student responses across Mathematics, Programming, and Essay/Writing reports statistically significant grading bias in Essay/Writing tasks across both models and all perturbation types, with effect sizes ranging from $\phi_i(x)$9 to $\Delta_i(x_v)$0. Informal language received the heaviest penalty, with LLaMA deducting an average of 1.90 points and Qwen deducting 1.20 points on a 10-point scale; non-native phrasing was penalized 1.35 and 0.90 points respectively. Mathematics and Programming showed minimal bias by comparison (Jadhav et al., 19 Mar 2026).

Dialogue introduces another layer: stylistic bias can be user-relative rather than third-party-observable. In the DUO dataset, subjective stylistic similarity correlates strongly with user preference, with $\Delta_i(x_v)$1 in EmpatheticDialogues and $\Delta_i(x_v)$2 in Wizard of Wikipedia, whereas objective stylistic similarity assigned by third-party annotators does not meaningfully correlate with user preference (Numaya et al., 15 Jul 2025). This suggests that stylistic bias in dialogue evaluation depends not only on text but also on who is judging.

4. Media, documents, and cultural corpora

In news and media studies, StylisticBias is often approached as variation in wording and framing across articles that cover the same event. An unsupervised framework for vaccine-mandate coverage uses sentence embeddings from distiluse-base-multilingual-cased-v2, VADER sentiment, a semantic similarity threshold $\Delta_i(x_v)$3, and a sentiment threshold $\Delta_i(x_v)$4 to construct article-to-article and domain-to-domain similarity networks. Its results show that standard outlet-level bias labels align only weakly with actual event-specific writing patterns: article-level Adjusted Rand Index values are 0.062673, 0.00213, and 0.02405 across the three events studied, which the authors interpret as evidence that a single bias label per outlet is insufficient (Cruickshank et al., 2023).

A second line of work scores textual bias directly from wording. Using pairs of Wikipedia revisions linked to NPOV-related edits, an interpretable pairwise model achieves 76.8% accuracy in its static quadratic form and 77.6% in its contextual quadratic form, compared with a human annotator at ~74% on the same task. Its highest-scoring words include “impressive,” “finest,” “superb,” “wonderful,” and “brilliant,” while legal texts emerge as the least biased domain and news as the most biased (Suresh et al., 2023). This supports a view of stylistic bias as a measurable property of evaluative, promotional, or loaded wording.

For hyperpartisan news, a masking study directly separates style from topic by masking either the high-frequency lexicon or the content lexicon. On the cleaned BuzzFeed-Webis Fake News Corpus 2016, the best unmasked baseline reaches macro $\Delta_i(x_v)$5, the topic-based model reaches 0.66, and the style-based model 0.57, confirming that topic-related features outperform stylistic ones while still showing that style alone contains substantial signal (Sánchez-Junquera et al., 2019). The same paper argues that simply including higher-length n-grams can already yield competitive results, implying that some purported stylistic bias detection may still be driven by narrow lexical fragments (Sánchez-Junquera et al., 2019).

Cultural corpora show the same pattern in different forms. One large-scale study of song lyrics analyzes more than half a million songs spread over five decades, characterizing style in terms of vocabulary, length, repetitiveness, speed, and readability, and reports that popular songs differ significantly from other songs. Using distributed representations and the WEAT test, it also finds gender and racial biases that correlate with prior results on human subjects (Barman et al., 2019). In multimodal caption datasets, “Stereotyping and Bias in the Flickr30K Dataset” identifies linguistic bias and unwarranted inferences in categories such as activity, ethnicity, event, goal, relation, and status/occupation, showing that stylistic bias can arise even in ostensibly descriptive ground truth (Miltenburg, 2016).

5. “StylisticBias” as a multimodal benchmark

The benchmark named “StylisticBias” introduces a controlled evaluation regime for MLLMs. It generates 500 photorealistic base faces and about 50 single-attribute variations per face, producing about 25K images, and evaluates six MLLMs across 25 binary social judgment scenarios (Kolli et al., 18 Jun 2026). The design fixes identity and changes one visual attribute at a time, so that appearance effects can be separated from identity differences.

The main empirical result is concentration. The benchmark finds that age and body type dominate identity-level effects, while fashion style and other visual cues drive the largest attribute-level shifts. It further reports that about 15 attributes account for nearly 80\% of the total variation, which means that a relatively small set of visual cues explains most measured bias (Kolli et al., 18 Jun 2026). Sensitivity is strongest in judgments that are semantically aligned with appearance, “especially socioeconomic and style-related judgments,” rather than in traits such as honesty or loyalty that have weaker visual correlates (Kolli et al., 18 Jun 2026).

The reported group-level disparities also show that demographic bias is not distributed evenly across protected attributes. Average Variation Strength is 0.075 for age, 0.069 for body type, 0.038 for ethnicity, and 0.030 for gender, which indicates that age and body type are the strongest identity-level drivers in this setup (Kolli et al., 18 Jun 2026). At the attribute level, negative cues are often stronger than positive counterparts. The paper reports, for example, that the median $\Delta_i(x_v)$6 for worn/distressed clothing is 0.167, compared with 0.121 for formal/business clothing, and that messy hair has median $\Delta_i(x_v)$7 versus 0.0098 for slicked-back hair (Kolli et al., 18 Jun 2026). This suggests a strong negativity asymmetry in appearance-conditioned social judgments.

6. Mitigation, attack surface, and unresolved issues

Mitigation strategies for StylisticBias fall into at least four families: counterfactual data augmentation, style-obfuscating rewriting, direct model editing, and evaluation redesign. In preference modeling, counterfactual data augmentation with synthesized contrastive examples reduces average miscalibration from 39.4% to 32.5% and average absolute skew difference from 20.5% to 10.0%, while maintaining overall RewardBench performance (Bharadwaj et al., 5 Jun 2025). This establishes that style-sensitive preference models can be debiased without full retraining from scratch.

A second family rewrites text to obfuscate style. “Style Pooling” defines two central styles: an intersection style that “effectively intersects the various styles seen in training,” and a union style that “seeks to obfuscate by adding stylistic features of all sensitive attributes to text.” Its stated use case is improving the fairness of downstream classifiers through automatic rewriting (Mireshghallah et al., 2021). “SimpleStyle” similarly treats attribute-controlled rewriting as a mechanism for “regulating attributes and biases of textual training data and a machine generated text,” combining controlled denoising and output filtering, while an MLM-based debiasing approach uses latent content encoding plus explicit keyword replacement to improve content preservation in neutralization tasks (Bandel et al., 2022, Tokpo et al., 2022).

A third family changes the model itself. “BiasEdit” uses lightweight editor networks to generate parameter updates that remove stereotypical bias while preserving language modeling through a retention loss. On StereoSet and CrowS-Pairs it is reported to be effective, efficient, and robust, with “little to no impact” on general capabilities (Xu et al., 11 Mar 2025). This line of work treats stylistic bias not only as a dataset or prompting issue but also as an editable internal representation.

At the same time, recent work shows that stylistic bias is an attack surface. “Turning Bias into Bugs” introduces BITE, which casts the selection of semantics-preserving stylistic edits as a contextual bandit problem and uses a LinUCB policy to exploit judge preferences for verbosity, formatting, tone, and related cues. The attack achieves an attack success rate exceeding 65% and raises scores by 1–2 points on a 9-point scale, while preserving semantic equivalence (Yang et al., 24 May 2026). This makes stylistic bias a reliability and security problem, not merely a fairness issue.

Several unresolved issues recur across the literature. First, style-content disentanglement is incomplete: many studies explicitly caution that correlation does not imply causation, and that topic, demographic signal, and style may remain entangled (Cruickshank et al., 2023, Kang et al., 2019). Second, evaluation perspective matters: user-perceived similarity and third-party similarity can diverge sharply in dialogue (Numaya et al., 15 Jul 2025). Third, debiasing can itself homogenize style, marginalize non-dominant communication norms, or erase legitimate identity expression, which is why style-obfuscation papers warn against converging uncritically toward a majority style (Mireshghallah et al., 2021). A plausible implication is that StylisticBias is not a single problem with a single remedy, but a family of bias mechanisms that must be analyzed at the level of task, representation, evaluator, and social context.