---
title: 'AesBiasBench: MLLM Bias & Aesthetic Alignment'
url: https://www.emergentmind.com/topics/aesbiasbench
type: topic
---

# AesBiasBench: MLLM Bias & Aesthetic Alignment

Searching arXiv for the benchmark paper and closely related methodological references.
AesBiasBench is a benchmark for evaluating multimodal large language models (MLLMs) in Personalized Image Aesthetic Assessment (PIAA) under a dual lens of stereotype bias and human-preference alignment. It was introduced to address a specific ambiguity in subjective vision-language tasks: demographic variation in model outputs may reflect either unjustified stereotyping or legitimate alignment with group-specific human aesthetic preferences. To make that distinction explicit, AesBiasBench evaluates MLLMs on three subtasks—Aesthetic Perception, Aesthetic Assessment, and Aesthetic Empathy—and introduces structured metrics to quantify both demographic disparity and alignment with human judgments [2509.11620].

## 1. Definition and conceptual framing

AesBiasBench is formulated around the setting in which an MLLM produces an output conditioned on an image \(i\), a task \(t\), and an identity descriptor \(g\). The paper writes this as
\[
M(i, t, g) = k,
\]
where \(k \in O\) is a discrete output label, \(g \in G\) is the identity condition, and \(t \in \{\text{Aesthetic Perception}, \text{Aesthetic Assessment}, \text{Aesthetic Empathy}\}\) [2509.11620].

The benchmark’s central premise is that fairness analysis in PIAA cannot be reduced to demographic invariance alone. In an objective task, demographic sensitivity is often treated as prima facie evidence of bias. In image aesthetics, however, different demographic groups may genuinely exhibit different preference distributions. AesBiasBench therefore separates two evaluation dimensions. The first is stereotype bias, defined operationally through demographic shifts in model outputs. The second is human-preference alignment, defined through similarity between model outputs and actual human judgments from corresponding demographic groups [2509.11620].

This dual structure places AesBiasBench in a methodological space adjacent to benchmark-design work arguing that “bias” is not a primitive quantity but the outcome of a full measurement stack comprising task definition, operating condition, base metric, disparity function, and aggregation rule [2408.13614]. It also resembles adaptable fairness benchmarking approaches that treat benchmark configuration itself as part of the experimental object rather than as a fixed background assumption [2409.16965]. A plausible implication is that AesBiasBench should be read not only as a dataset-and-metric package, but also as a proposal about how subjective multimodal evaluation should be operationalized.

## 2. Benchmark construction and demographic structure

The benchmark is built primarily on the Personalized Image Aesthetics Database with Rich Attributes (PARA), which contains **31,220 images** annotated by **438 human raters** and includes metadata suitable for personalized image aesthetic assessment [2509.11620]. For supplementary generalization experiments, the paper also uses LAPIS, which contains **11,723 artistic images**, ratings on a **0–100** scale, an average of **24 annotators per image**, and **552 participants total** [2509.11620].

AesBiasBench uses three demographic categories:
\[
\Phi = \{\text{age}, \text{gender}, \text{education}\}.
\]
The identity groups are defined as follows [2509.11620].

| Category | Groups |
|---|---|
| Age | 18–21, 22–25, 26–29, 30–34, 35–40 |
| Gender | male, female |
| Education | junior high school, technical secondary school, senior high school, university, junior college |

For image-conditioned disparity analysis, the benchmark also uses image-type categories
\[
\Omega = \{\text{portrait}, \text{animal}, \text{plant}, \text{scene}, \text{building}, \text{still life}, \text{night scene}, \text{indoor}, \text{others}\},
\]
with \(m \in \Omega\) [2509.11620].

The paper explicitly limits its identity coverage to age, gender, and education. It notes that other potentially important factors such as culture, race, and religion are not included [2509.11620]. This suggests that the benchmark is identity-aware but not exhaustive as a sociotechnical representation of aesthetic subjectivity.

## 3. Tasks, prompting protocol, and label construction

AesBiasBench evaluates three subtasks, each with a fixed output space and a fixed response format [2509.11620].

| Subtask | Focus | Output space |
|---|---|---|
| Aesthetic Perception | technical quality, color balance, lighting, sharpness, exposure, contrast, overall visual impact | \(\{\text{positive}, \text{normal}, \text{negative}\}\) |
| Aesthetic Assessment | composition, color harmony, visual balance, overall attractiveness | \(\{\text{positive}, \text{normal}, \text{negative}\}\) |
| Aesthetic Empathy | emotion conveyed or induced by the image | \(\{\text{amusement}, \text{excitement}, \text{contentment}, \text{awe}, \text{disgust}, \text{sadness}, \text{fear}, \text{neutral}\}\) |

The benchmark uses two prompt variants. In the default condition, no identity information is provided. In the personalized condition, an identity cue is inserted using the pattern “As a [demographic], please ...” while keeping the image, task wording, and response format fixed [2509.11620]. For reproducibility, the prompts require a one-label output in a keyed format such as `perception: positive`, `aesthetic: normal`, or `empathy: awe`, and they instruct the model not to mention that it is using additional context [2509.11620].

For Aesthetic Perception and Aesthetic Assessment, human ratings are continuous in the source datasets, whereas model outputs are categorical. The benchmark therefore discretizes human scores into three classes using equidistant intervals:
\[
L(s) = l_j
\]
where
\[
r + \frac{j-1}{3} \times (R - r) < s \leq r + \frac{j}{3} \times (R - r),
\]
and
\[
\{l_j \mid_{j=1}^3 \} = \{ \text{negative}, \text{normal}, \text{positive}\}.
\]
The paper specifies \(r=1, R=5\) for PARA and \(r=0, R=100\) for LAPIS [2509.11620].

The benchmark’s fairness criterion is grounded in demographic parity. An unbiased model would satisfy
\[
P\!\big(M(i, t, g)=k\big) = P\!\big(M(i, t)=k\big).
\]
For image-type-conditioned disparity, the paper defines the unbiased case as
\[
P(g \mid M(i,t)=k,\, \text{type}(i)=m) = \tfrac{1}{n_G}.
\]
These equations provide the formal background for the benchmark’s two bias metrics, IFD and NRD [2509.11620].

## 4. Metrics for bias and alignment

AesBiasBench uses three named metrics—IFD, NRD, and AAS—and an intermediate similarity score \(S(g)\) derived from Jensen–Shannon divergence [2509.11620].

### IFD: Identity Frequency Disparity

IFD measures how much the label distribution for a specific identity differs from the aggregate distribution over all identities in the same demographic category. The paper prints the formula as
\[
\text{IFD}(t) = \frac{1}{n_\Phi\times n_{O} \sum_{k \in O} \sum_{G \in \Phi}^{} \sum_{g \in G} {\left| p_{g,k} - p_{G,k} \right|} ,
\]
and defines
\[
p_{g,k} = \frac{n(M(i, t, g)=k)}{\sum_{r=1}^{n_O} n(M(i, t, g)=r)}.
\]
The manuscript notes, and the paper’s technical summary reiterates, that the printed IFD equation contains a formatting issue; the intended meaning is an average absolute deviation of \(p_{g,k}\) from \(p_{G,k}\) across identity groups and outputs [2509.11620].

The paper states that \(\text{IFD}(t)\ge 0\), with \(\text{IFD}(t)=0\) only under perfect demographic parity [2509.11620]. Lower IFD therefore indicates lower demographic disparity.

### NRD: Normalized Representation Disparity

NRD is intended to capture conditional demographic disparity given image type and output class. The paper does not provide the full aggregate NRD equation in the manuscript; an equation slot is present, but the formula itself is omitted. What is provided is the component
\[
q_{g,k,m} = \frac{n(M(i, t, g)=k | m)}{\sum_{h=1}^{n_G} n(M(i, t, h)=k | m)},
\]
together with the unbiased target
\[
P(g \mid M(i,t)=k,\, \text{type}(i)=m) = \tfrac{1}{n_G}.
\]
Accordingly, NRD is described as measuring deviation of \(q_{g,k,m}\) from equal representation across identities, aggregated over image types and outputs [2509.11620].

Because the main aggregate formula is missing, any more explicit reconstruction would exceed what is stated in the paper. The safe interpretation is that NRD is the benchmark’s image-type-conditioned demographic-disparity metric.

### Similarity score and AAS

To quantify alignment with human judgments, the paper defines a similarity score using Jensen–Shannon divergence between one-hot-encoded model and human output distributions:
\[
\text{JS}(E_g \! \parallel \! E_h) = \frac{1}{2} \left[ \text{KL}(E_g \! \parallel \! \bar{E}) \! + \! \text{KL}(E_{h} \! \parallel \! \bar{E}) \right],
\]
with
\[
\bar{E} = \frac{E_g + E_h}{2},
\]
and
\[
\text{KL}(E \parallel \bar{E}) = \sum_{j} E(j) \log \left( \frac{E(j)}{\bar{E}(j)} \right).
\]
The similarity score is then
\[
S(g) = 1 - \text{JS}(E_g  \parallel  E_h).
\]
Higher \(S(g)\) means stronger alignment between the model and human judgments for identity \(g\) [2509.11620].

The Aesthetic Alignment Score is defined relatively:
\[
\text{AAS}(g) = S(g)-\bar{S},
\]
where \(\bar{S}\) is the mean similarity score across all identities in the same demographic category [2509.11620]. Positive AAS indicates above-average alignment for a specific demographic group; negative AAS indicates below-average alignment.

This metric design is methodologically notable. In contrast to broad LLM bias test suites that bundle bias, fairness, ethics, and factuality into a large evaluation surface [2503.24310], AesBiasBench restricts itself to a smaller metric set specifically tailored to subjective multimodal judgment. A plausible implication is that AesBiasBench privileges interpretability of the bias–alignment tradeoff over maximal breadth of responsible-AI coverage.

## 5. Evaluated models and empirical findings

The benchmark evaluates **19 MLLMs**, including proprietary systems—**GPT-4o**, **Claude-3.5-Sonnet**, and **Gemini-2.0-Flash**—and open-source models including the **InternVL2.5** family, the **Qwen2.5-VL** family, **LLaVA** variants, **Llama-3.2-Vision-11B**, **mPLUG-Owl3-7B**, **Mono-InternVL-2B**, **Phi-3.5-Vision-4B**, **GLM-4V-9B**, and **DeepSeek-VL2-7B** [2509.11620].

The main empirical conclusions are threefold. First, stereotype bias is widespread across evaluated MLLMs. Second, smaller models show stronger stereotype bias, while larger models align more closely with human preferences. Third, incorporating identity information often exacerbates bias, particularly in emotional judgments [2509.11620].

The strongest family-level scaling result is reported for InternVL2.5. On LAPIS, IFD decreases monotonically with model size:
- **InternVL2.5-2B**: **0.4651**
- **InternVL2.5-4B**: **0.4437**
- **InternVL2.5-8B**: **0.3560**
- **InternVL2.5-26B**: **0.2857**
- **InternVL2.5-38B**: **0.2020** [2509.11620]

The paper also highlights that **Qwen2.5-VL-7B** exhibits a notably high IFD of **1.1518** on LAPIS, whereas larger InternVL2.5 variants are comparatively fairer by that metric [2509.11620].

Without identity information, the benchmark reports strong default alignment regularities. Across all three tasks, **17 out of 19 models** align most strongly with **female** preferences [2509.11620]. For the no-identity condition, the dominant groups are:

- **Perception**: female (**17** models), age **22–25** (**12** models), technical secondary school (**7** models)
- **Assessment**: female (**17** models), age **22–25** (**17** models), university (**17** models)
- **Empathy**: female (**17** models), age **18–21** (**8** models), junior (**7** models) [2509.11620]

With explicit identity conditioning, those patterns become less stable, and in empathy they reverse sharply. In the identity-conditioned setting, the dominant alignment pattern for empathy becomes **male** for **17 models**, producing what the paper describes as a “complete reversal” from the no-identity case [2509.11620]. The corresponding table entries are:

- **Perception**: female (**15** models), age **22–25** (**10** models), junior (**7** models)
- **Assessment**: female (**14** models), age **22–25** (**15** models), university (**10** models)
- **Empathy**: male (**17** models), age **30–34** (**8** models), university (**6** models) [2509.11620]

The empathy task is consistently the most sensitive to demographic cueing. The paper highlights a qualitative failure mode in which models disproportionately predict **fear** for younger people, females, and less-educated individuals, treating this as evidence of stereotype-like emotional attribution [2509.11620]. It also quantifies gender-conditioned empathy shifts using
\[
\Delta = \Delta S_E(M) - \Delta S_E(F),
\]
with large positive values indicating that adding gender information moves the model toward male preference patterns more than toward female patterns. Reported values include **0.4054** for **Gemini-2.0-Flash**, **0.2346** for **Claude-3.5-Sonnet**, and **0.1143** for **GPT-4o** [2509.11620].

An important nuance is that empathy can appear comparatively fairer by IFD in the default setting while still being less aligned by AAS. The paper interprets this as evidence that lower measured disparity does not automatically imply better modeling of genuine human preferences [2509.11620]. That observation is consistent with broader measurement literature showing that different base metrics and aggregation schemes can support contradictory fairness conclusions even when each is internally coherent [2408.13614].

## 6. Interpretation, methodological significance, and limitations

AesBiasBench’s main contribution is conceptual as much as empirical. It argues that in subjective multimodal evaluation, demographic sensitivity should not be treated as a unitary failure mode. Some variation is stereotype-driven; some may reflect real human-preference heterogeneity. The benchmark operationalizes that distinction through a paired bias-and-alignment design rather than through a single fairness score [2509.11620].

This design choice is closely related to a broader methodological concern in bias evaluation: benchmark conclusions depend strongly on the chosen metric family and aggregation strategy. Work on fairness evaluation in speaker verification shows that changing the base metric, disparity function, or meta-measure can reverse which groups appear advantaged or disadvantaged [2408.13614]. Work on adaptable fairness benchmarking makes a parallel point by treating intervention stage, sensitive-feature composition, fairness notion, and output regime as benchmark configuration variables rather than fixed background assumptions [2409.16965]. AesBiasBench can be read as the multimodal-aesthetics analogue of that perspective.

The paper also identifies several limitations. The demographic space is restricted to age, gender, and education; other salient identity variables such as culture, race, and religion are not modeled [2509.11620]. The benchmark does not report formal inferential statistical tests, confidence intervals, or p-values [2509.11620]. The prompting protocol is clearly templated, but the manuscript does not provide full decoding hyperparameters such as temperature or top-\(p\) [2509.11620]. In addition, the main NRD equation is absent from the manuscript, leaving that metric only partially specified [2509.11620].

These limitations do not negate the benchmark’s significance. Rather, they define its current scope. AesBiasBench is best understood as an identity-aware benchmark for subjective vision-language evaluation that formalizes a distinction often collapsed in fairness auditing: the difference between demographic disparity as stereotype bias and demographic variation as possible human-preference alignment. Within that scope, it provides a concrete task suite, prompt protocol, and metric framework for analyzing how MLLMs behave when asked not merely to describe or classify images, but to judge them aesthetically on behalf of different people [2509.11620].

Source: https://www.emergentmind.com/topics/aesbiasbench