StylisticBias: A Few Human Visual Cues Drive Most Social Biases in MLLMs

Abstract: Multimodal LLMs (MLLMs) are increasingly deployed in personally and societally consequential settings, yet the visual cues that shape how these models judge people remain poorly understood. Prior work often compares different (groups of) individuals, making it difficult to separate appearance effects from identity differences. We introduce StylisticBias, a controlled benchmark for evaluating attribute-level social bias in MLLMs. We generate 500 photorealistic base faces and create about 50 single-attribute variations per face, producing about 25K images. This design keeps identity fixed and changes one visual attribute at a time. It lets us measure how specific cues shift model judgments. We evaluate six MLLMs across 25 binary social judgment scenarios. We find that age and body type dominate identity-level effects, while fashion style and other visual cues drive the largest attribute-level shifts. We further find that about 15 attributes account for nearly 80\% of the total variation, showing that bias is concentrated in a small set of visual cues. Sensitivity is strongest in judgments that are semantically aligned with appearance, especially socioeconomic and style-related judgments. We release StylisticBias as a benchmark for fine-grained bias evaluation in multimodal models. Code and dataset: https://github.com/timo-cavelius/StylisticBias and https://hf.co/datasets/shaghayegh/stylistic-bias-dataset.

• The paper demonstrates that a small subset of visual cues, about 15 attributes, drives nearly 80% of the observed bias in MLLMs.
• It employs a controlled synthetic benchmark of around 15,700 images to isolate the effect of single-attribute perturbations on social judgments.
• Findings show that appearance-based judgments, notably in style and socioeconomic status, highlight actionable targets for debiasing efforts in AI.

StylisticBias: Deconstructing the Visual Roots of Social Bias in MLLMs

Introduction

"StylisticBias: A Few Human Visual Cues Drive Most Social Biases in MLLMs" (2606.20527) offers a comprehensive and systematic analysis of how specific human visual cues systematically affect the social judgments rendered by multimodal LLMs (MLLMs). Leveraging a tightly controlled synthetic image benchmark, this work isolates the effects of individual appearance attributes—decoupled from confounding identity perturbations—on a broad set of social perception tasks spanning warmth, competence, socioeconomic status, and behavioral traits. The principal empirical finding is that a small and interpretable subset of visual cues drives the vast majority of observed biases in these models, with significant implications for future benchmark design, safety auditing, and mitigation efforts in AI systems interfacing with human-facing imagery.

Methodology and Benchmark Construction

The paper operationalizes social bias in MLLMs as a systematic shift in binary forced-choice judgments induced by controlled variations of a single visual attribute, where all other facets—including depicted identity, pose, lighting, and background—are held constant. To facilitate this, the authors constructed the StylisticBias benchmark, encompassing 500 photorealistic base faces evenly spanning a Cartesian product of four major demographic axes: age, gender, ethnicity, and body type. For each base face, approximately 50 single-attribute perturbations were generated, yielding $\sim$25,000 images and, after filtering for visual and semantic clarity, a final set of 15,726 unique images for evaluation.

This design allows for fine-grained attribution of model sensitivities: across six current open-source MLLMs, each image is subjected to 25 binary forced-choice social judgment scenarios and multiple prompt randomizations—amounting to 4.72 million judgment calls per model.

Figure 1: Benchmark construction and evaluation pipeline, highlighting the generation of base faces via demographic attributes, single-attribute variations, and forced-choice evaluation across scenarios.

Figure 2: Example base faces (top: female, bottom: male) and representative visual attribute variations; labels indicate controlled attribute and value.

Core Findings on Attribute-Driven Bias

Concentration of Bias in a Few Visual Cues

A critical empirical result is that bias is not uniformly distributed across all visual features. Approximately 15 attributes account for nearly 80% of the total aggregate predictive shift (as measured by absolute signed bias shift, $|\mathrm{SBS}|$), indicating a substantial Pareto concentration in the susceptibility of MLLMs to visual perturbations.

Figure 3: Cumulative $|\mathrm{SBS}|$ by attribute, sorted by effect magnitude; 15 attributes explain ~80% of total variation in model bias.

Key drivers of social judgment shifts include fashion style, facial hair, makeup, and eyewear—attributes that are culturally interpreted as deliberate self-presentation signals. In contrast, features such as skin irregularities, hair color, and non-salient accessories contribute minimally. These findings are robust across all six MLLMs evaluated and are especially pronounced in judgment domains semantically aligned with visual appearance (e.g., socioeconomic or style-related traits).

Amplification and Moderation Effects

Appearance-driven bias is disproportionately expressed in judgments of style and socioeconomic status (e.g., Stylish vs. Unstylish and Wealthy vs. Poor), which show large effect sizes (SBS ≈ 0.244 and 0.114, respectively). By contrast, attributions for internal psychological and interpersonal traits (e.g., Honesty, Trustworthiness) remain largely invariant to visual perturbations.

Figure 4: Mean SBS for all 25 scenarios, sorted; appearance- and status-related judgments show highest sensitivity.

Figure 5: Joint plot of direction (mean SBS) and magnitude (mean $|\mathrm{SBS}|$) per judgment scenario, with appearance-related inferences occupying the upper-right quadrant.

Consistent negativity bias is also observed, with unfavorable cues (worn/distressed clothing, messy hair) producing stronger adverse shifts than the uplift induced by positive cues (formal attire, neat grooming), introducing a methodological risk: evaluations restricted to “positive” variations substantially underestimate practical bias magnitude in deployment.

Demographic context further moderates attribute interpretation. For example, facial tattoos and multiple piercings produce positive shifts for female faces, negative or null for males. The same visual cue (e.g., long hair) can thus reverse its social valence depending on the perceived gender—a critical nuance for accurate bias audits.

Demographic Effects and Bias Hierarchies

Analysis of the base faces (i.e., before attribute variation) reveals that body type and age are the dominant demographic drivers of social judgment divergence ($\mathrm{VS}=0.069$ and $0.075$, respectively), while ethnicity and gender yield substantially weaker effects ($\mathrm{VS}=0.038$ and $0.030$). This ranking is consistent with findings from social psychology on the cultural linkage of status and competence attributions to these characteristics.

Architectural Invariance

Despite differences in overall magnitude, the structure of attribute sensitivity is largely invariant across the tested MLLMs. Larger or more conservative models (e.g., Qwen3) attenuate the absolute effect size but preserve the relative ranking of the most bias-inducing cues. Paired comparisons within model families (e.g., Gemma-3 vs. Gemma-4) demonstrate that core patterns of semantic alignment and attribute importance persist across generational improvements.

Figure 6: Scenario-level predictive shift comparison between Gemma-3 and Gemma-4; slope and $r$ indicate structural preservation with moderate magnitude attenuation.

Theoretical and Practical Implications

Implications for Benchmark Design and Model Auditing

This work demonstrates that fine-grained, attribute-level analysis is essential for precise measurement and mitigation of social bias in MLLMs. Benchmarks or audits that only compare global demographic categories, or that average over shifting contextual interactions, will systematically obscure the true structure and magnitude of appearance-driven bias. The attribute-by-group interaction effects uncovered here suggest that reporting only averaged signed shifts can mask subgroup harms or failures—risking deployment of systems that are ostensibly "fair" at an aggregate level but systematically biased within or across sensitive contexts.

Mitigation and Correction

The clear, interpretable concentration of appearance-driven bias in MLLMs establishes actionable targets for mitigation: auditing and correction efforts should prioritize self-presentation cues (clothing, grooming, etc.) over attempts to debias non-salient biological signals. Given the architectural invariance in attribute sensitivity, such interventions are likely to generalize across model families.

On the other hand, the persistence of semantic alignment bias—in which model sensitivity is greatest whenever the queried judgment is culturally associated with visible appearance—raises important limit questions about the learnability and suppression of such biases. It is unclear whether current alignment or adversarial training paradigms can effectively modulate these inferential patterns without either significant utility loss or the introduction of new pathologies.

Future Research Directions

The controlled, identity-preserving design of StylisticBias sets a methodological precedent for further work on disentangling latent causes of social bias in multimodal and generative models. Extensions to real-world imagery (subject to privacy and ethical limits), expansion of the attribute space, and integration with causal inferencing methods will be necessary to fully characterize both the origins and the practical manifestations of bias under realistic deployment regimes. This work also suggests opportunities for dataset curation and synthetic data generation strategies aimed at counterbalancing or desensitizing models to the high-leverage cues identified herein.

Conclusion

By precisely quantifying how a focused subset of human visual cues—principally self-presentation attributes—drives the overwhelming majority of social bias in MLLMs, this work underscores the necessity of attribute-level bias analysis and the limitations of demographic-only audits. StylisticBias establishes a robust, scalable benchmark that enables controlled, repeatable measurement of attribute-induced model behavior and provides a solid foundation for both diagnosing and mitigating practical harms in real-world AI deployments. As MLLMs proliferate in applications with consequential social judgments, such benchmarks will be critical to achieving both transparency and safety.