AesBiasBench: MLLM Bias & Aesthetic Alignment

• AesBiasBench is a benchmark that operationalizes the evaluation of multimodal language models in personalized image aesthetic assessment by distinguishing stereotype bias from human-preference alignment.
• It measures performance on three subtasks—Aesthetic Perception, Assessment, and Empathy—using structured prompts and metrics like IFD, NRD, and AAS to capture demographic disparity and alignment.
• Empirical findings reveal that smaller models display higher bias whereas larger models better align with human judgments, with identity conditioning often intensifying stereotype bias.

Searching arXiv for the benchmark paper and closely related methodological references. AesBiasBench is a benchmark for evaluating multimodal LLMs (MLLMs) in Personalized Image Aesthetic Assessment (PIAA) under a dual lens of stereotype bias and human-preference alignment. It was introduced to address a specific ambiguity in subjective vision-language tasks: demographic variation in model outputs may reflect either unjustified stereotyping or legitimate alignment with group-specific human aesthetic preferences. To make that distinction explicit, AesBiasBench evaluates MLLMs on three subtasks—Aesthetic Perception, Aesthetic Assessment, and Aesthetic Empathy—and introduces structured metrics to quantify both demographic disparity and alignment with human judgments (Li et al., 15 Sep 2025).

1. Definition and conceptual framing

AesBiasBench is formulated around the setting in which an MLLM produces an output conditioned on an image $i$, a task $t$, and an identity descriptor $g$. The paper writes this as

$M(i, t, g) = k,$

where $k \in O$ is a discrete output label, $g \in G$ is the identity condition, and $$t \in \{\text{Aesthetic Perception}, \text{Aesthetic Assessment}, \text{Aesthetic Empathy}\}$$ (Li et al., 15 Sep 2025).

The benchmark’s central premise is that fairness analysis in PIAA cannot be reduced to demographic invariance alone. In an objective task, demographic sensitivity is often treated as prima facie evidence of bias. In image aesthetics, however, different demographic groups may genuinely exhibit different preference distributions. AesBiasBench therefore separates two evaluation dimensions. The first is stereotype bias, defined operationally through demographic shifts in model outputs. The second is human-preference alignment, defined through similarity between model outputs and actual human judgments from corresponding demographic groups (Li et al., 15 Sep 2025).

This dual structure places AesBiasBench in a methodological space adjacent to benchmark-design work arguing that “bias” is not a primitive quantity but the outcome of a full measurement stack comprising task definition, operating condition, base metric, disparity function, and aggregation rule (Hutiri et al., 2024). It also resembles adaptable fairness benchmarking approaches that treat benchmark configuration itself as part of the experimental object rather than as a fixed background assumption (Defrance et al., 2024). A plausible implication is that AesBiasBench should be read not only as a dataset-and-metric package, but also as a proposal about how subjective multimodal evaluation should be operationalized.

2. Benchmark construction and demographic structure

The benchmark is built primarily on the Personalized Image Aesthetics Database with Rich Attributes (PARA), which contains 31,220 images annotated by 438 human raters and includes metadata suitable for personalized image aesthetic assessment (Li et al., 15 Sep 2025). For supplementary generalization experiments, the paper also uses LAPIS, which contains 11,723 artistic images, ratings on a 0–100 scale, an average of 24 annotators per image, and 552 participants total (Li et al., 15 Sep 2025).

AesBiasBench uses three demographic categories: $$\Phi = \{\text{age}, \text{gender}, \text{education}\}.$$ The identity groups are defined as follows (Li et al., 15 Sep 2025).

Category	Groups
Age	18–21, 22–25, 26–29, 30–34, 35–40
Gender	male, female
Education	junior high school, technical secondary school, senior high school, university, junior college

For image-conditioned disparity analysis, the benchmark also uses image-type categories

$$\Omega = \{\text{portrait}, \text{animal}, \text{plant}, \text{scene}, \text{building}, \text{still life}, \text{night scene}, \text{indoor}, \text{others}\},$$

with $m \in \Omega$ (Li et al., 15 Sep 2025).

The paper explicitly limits its identity coverage to age, gender, and education. It notes that other potentially important factors such as culture, race, and religion are not included (Li et al., 15 Sep 2025). This suggests that the benchmark is identity-aware but not exhaustive as a sociotechnical representation of aesthetic subjectivity.

3. Tasks, prompting protocol, and label construction

AesBiasBench evaluates three subtasks, each with a fixed output space and a fixed response format (Li et al., 15 Sep 2025).

Subtask	Focus	Output space
Aesthetic Perception	technical quality, color balance, lighting, sharpness, exposure, contrast, overall visual impact	$t$0
Aesthetic Assessment	composition, color harmony, visual balance, overall attractiveness	$t$1
Aesthetic Empathy	emotion conveyed or induced by the image	$t$2

The benchmark uses two prompt variants. In the default condition, no identity information is provided. In the personalized condition, an identity cue is inserted using the pattern “As a [demographic], please ...” while keeping the image, task wording, and response format fixed (Li et al., 15 Sep 2025). For reproducibility, the prompts require a one-label output in a keyed format such as perception: positive, aesthetic: normal, or empathy: awe, and they instruct the model not to mention that it is using additional context (Li et al., 15 Sep 2025).

For Aesthetic Perception and Aesthetic Assessment, human ratings are continuous in the source datasets, whereas model outputs are categorical. The benchmark therefore discretizes human scores into three classes using equidistant intervals: $t$3 where

$t$4

and

$t$5

The paper specifies $t$6 for PARA and $t$7 for LAPIS (Li et al., 15 Sep 2025).

The benchmark’s fairness criterion is grounded in demographic parity. An unbiased model would satisfy

$t$8

For image-type-conditioned disparity, the paper defines the unbiased case as

$t$9

These equations provide the formal background for the benchmark’s two bias metrics, IFD and NRD (Li et al., 15 Sep 2025).

4. Metrics for bias and alignment

AesBiasBench uses three named metrics—IFD, NRD, and AAS—and an intermediate similarity score $g$0 derived from Jensen–Shannon divergence (Li et al., 15 Sep 2025).

IFD: Identity Frequency Disparity

IFD measures how much the label distribution for a specific identity differs from the aggregate distribution over all identities in the same demographic category. The paper prints the formula as

$g$1

and defines

$g$2

The manuscript notes, and the paper’s technical summary reiterates, that the printed IFD equation contains a formatting issue; the intended meaning is an average absolute deviation of $g$3 from $g$4 across identity groups and outputs (Li et al., 15 Sep 2025).

The paper states that $g$5, with $g$6 only under perfect demographic parity (Li et al., 15 Sep 2025). Lower IFD therefore indicates lower demographic disparity.

NRD: Normalized Representation Disparity

NRD is intended to capture conditional demographic disparity given image type and output class. The paper does not provide the full aggregate NRD equation in the manuscript; an equation slot is present, but the formula itself is omitted. What is provided is the component

$g$7

together with the unbiased target

$g$8

Accordingly, NRD is described as measuring deviation of $g$9 from equal representation across identities, aggregated over image types and outputs (Li et al., 15 Sep 2025).

Because the main aggregate formula is missing, any more explicit reconstruction would exceed what is stated in the paper. The safe interpretation is that NRD is the benchmark’s image-type-conditioned demographic-disparity metric.

Similarity score and AAS

To quantify alignment with human judgments, the paper defines a similarity score using Jensen–Shannon divergence between one-hot-encoded model and human output distributions: $M(i, t, g) = k,$0 with

$M(i, t, g) = k,$1

and

$M(i, t, g) = k,$2

The similarity score is then

$M(i, t, g) = k,$3

Higher $M(i, t, g) = k,$4 means stronger alignment between the model and human judgments for identity $M(i, t, g) = k,$5 (Li et al., 15 Sep 2025).

The Aesthetic Alignment Score is defined relatively: $M(i, t, g) = k,$6 where $M(i, t, g) = k,$7 is the mean similarity score across all identities in the same demographic category (Li et al., 15 Sep 2025). Positive AAS indicates above-average alignment for a specific demographic group; negative AAS indicates below-average alignment.

This metric design is methodologically notable. In contrast to broad LLM bias test suites that bundle bias, fairness, ethics, and factuality into a large evaluation surface (Abhishek et al., 31 Mar 2025), AesBiasBench restricts itself to a smaller metric set specifically tailored to subjective multimodal judgment. A plausible implication is that AesBiasBench privileges interpretability of the bias–alignment tradeoff over maximal breadth of responsible-AI coverage.

5. Evaluated models and empirical findings

The benchmark evaluates 19 MLLMs, including proprietary systems—GPT-4o, Claude-3.5-Sonnet, and Gemini-2.0-Flash—and open-source models including the InternVL2.5 family, the Qwen2.5-VL family, LLaVA variants, Llama-3.2-Vision-11B, mPLUG-Owl3-7B, Mono-InternVL-2B, Phi-3.5-Vision-4B, GLM-4V-9B, and DeepSeek-VL2-7B (Li et al., 15 Sep 2025).

The main empirical conclusions are threefold. First, stereotype bias is widespread across evaluated MLLMs. Second, smaller models show stronger stereotype bias, while larger models align more closely with human preferences. Third, incorporating identity information often exacerbates bias, particularly in emotional judgments (Li et al., 15 Sep 2025).

The strongest family-level scaling result is reported for InternVL2.5. On LAPIS, IFD decreases monotonically with model size:

• InternVL2.5-2B: 0.4651
• InternVL2.5-4B: 0.4437
• InternVL2.5-8B: 0.3560
• InternVL2.5-26B: 0.2857
• InternVL2.5-38B: 0.2020 (Li et al., 15 Sep 2025)

The paper also highlights that Qwen2.5-VL-7B exhibits a notably high IFD of 1.1518 on LAPIS, whereas larger InternVL2.5 variants are comparatively fairer by that metric (Li et al., 15 Sep 2025).

Without identity information, the benchmark reports strong default alignment regularities. Across all three tasks, 17 out of 19 models align most strongly with female preferences (Li et al., 15 Sep 2025). For the no-identity condition, the dominant groups are:

• Perception: female (17 models), age 22–25 (12 models), technical secondary school (7 models)
• Assessment: female (17 models), age 22–25 (17 models), university (17 models)
• Empathy: female (17 models), age 18–21 (8 models), junior (7 models) (Li et al., 15 Sep 2025)

With explicit identity conditioning, those patterns become less stable, and in empathy they reverse sharply. In the identity-conditioned setting, the dominant alignment pattern for empathy becomes male for 17 models, producing what the paper describes as a “complete reversal” from the no-identity case (Li et al., 15 Sep 2025). The corresponding table entries are:

• Perception: female (15 models), age 22–25 (10 models), junior (7 models)
• Assessment: female (14 models), age 22–25 (15 models), university (10 models)
• Empathy: male (17 models), age 30–34 (8 models), university (6 models) (Li et al., 15 Sep 2025)

The empathy task is consistently the most sensitive to demographic cueing. The paper highlights a qualitative failure mode in which models disproportionately predict fear for younger people, females, and less-educated individuals, treating this as evidence of stereotype-like emotional attribution (Li et al., 15 Sep 2025). It also quantifies gender-conditioned empathy shifts using

$M(i, t, g) = k,$8

with large positive values indicating that adding gender information moves the model toward male preference patterns more than toward female patterns. Reported values include 0.4054 for Gemini-2.0-Flash, 0.2346 for Claude-3.5-Sonnet, and 0.1143 for GPT-4o (Li et al., 15 Sep 2025).

An important nuance is that empathy can appear comparatively fairer by IFD in the default setting while still being less aligned by AAS. The paper interprets this as evidence that lower measured disparity does not automatically imply better modeling of genuine human preferences (Li et al., 15 Sep 2025). That observation is consistent with broader measurement literature showing that different base metrics and aggregation schemes can support contradictory fairness conclusions even when each is internally coherent (Hutiri et al., 2024).

6. Interpretation, methodological significance, and limitations

AesBiasBench’s main contribution is conceptual as much as empirical. It argues that in subjective multimodal evaluation, demographic sensitivity should not be treated as a unitary failure mode. Some variation is stereotype-driven; some may reflect real human-preference heterogeneity. The benchmark operationalizes that distinction through a paired bias-and-alignment design rather than through a single fairness score (Li et al., 15 Sep 2025).

This design choice is closely related to a broader methodological concern in bias evaluation: benchmark conclusions depend strongly on the chosen metric family and aggregation strategy. Work on fairness evaluation in speaker verification shows that changing the base metric, disparity function, or meta-measure can reverse which groups appear advantaged or disadvantaged (Hutiri et al., 2024). Work on adaptable fairness benchmarking makes a parallel point by treating intervention stage, sensitive-feature composition, fairness notion, and output regime as benchmark configuration variables rather than fixed background assumptions (Defrance et al., 2024). AesBiasBench can be read as the multimodal-aesthetics analogue of that perspective.

The paper also identifies several limitations. The demographic space is restricted to age, gender, and education; other salient identity variables such as culture, race, and religion are not modeled (Li et al., 15 Sep 2025). The benchmark does not report formal inferential statistical tests, confidence intervals, or p-values (Li et al., 15 Sep 2025). The prompting protocol is clearly templated, but the manuscript does not provide full decoding hyperparameters such as temperature or top-$M(i, t, g) = k,$9 (Li et al., 15 Sep 2025). In addition, the main NRD equation is absent from the manuscript, leaving that metric only partially specified (Li et al., 15 Sep 2025).

These limitations do not negate the benchmark’s significance. Rather, they define its current scope. AesBiasBench is best understood as an identity-aware benchmark for subjective vision-language evaluation that formalizes a distinction often collapsed in fairness auditing: the difference between demographic disparity as stereotype bias and demographic variation as possible human-preference alignment. Within that scope, it provides a concrete task suite, prompt protocol, and metric framework for analyzing how MLLMs behave when asked not merely to describe or classify images, but to judge them aesthetically on behalf of different people (Li et al., 15 Sep 2025).