MIST: Mitigating Intersectional Bias in Diffusion Models

• MIST is a method that reduces both single-attribute and intersectional biases in text-to-image diffusion models by fine-tuning the cross-attention key/value projections.
• It leverages the CLIP <EOS> token as a global representation to disentangle prompt semantics, eliminating the need for retraining or manual preservation sets.
• MIST achieves improved balance and content preservation on benchmarks like WinoBias compared to methods such as TIME, UCE, and ITI-GEN.

MIST, short for Mitigating Intersectional Bias with Disentangled Cross-Attention Editing in Text-to-Image Diffusion Models, is a method for reducing social bias in pretrained text-to-image diffusion models—specifically Stable Diffusion 1.5—by editing their cross-attention projections in a targeted, disentangled way. Its defining claim is that the same mechanism can mitigate not only familiar single-attribute biases such as gender or race bias, but also intersectional bias, where bias appears at the combination of multiple social identities such as race $\times$ gender or gender $\times$ age $\times$ eyeglasses. The method is designed to avoid several burdens of earlier debiasing approaches: it requires no retraining from scratch, no manual preservation list, and no reference-image set, and it aims to preserve the original quality for unaltered concepts (Yesiltepe et al., 2024).

1. Bias setting and design objective

MIST addresses a now well-known problem in text-to-image generation: models trained on large internet-scale datasets inherit and amplify correlations in those datasets (Yesiltepe et al., 2024). For prompts such as “a CEO,” “a nurse,” or “a doctor,” image generators may overproduce one demographic group, such as men for CEO or women for nurse. The paper treats this as a distributional bias problem in generated outputs.

The central motivation is that this problem becomes more difficult in the intersectional setting. The model may not merely be biased on each attribute independently; it may encode coupled patterns across attributes. Correcting gender bias alone can still leave the model preferring white women over women of color, and changing race may alter gender balance in undesirable ways. The paper’s argument is therefore that debiasing one attribute at a time is inadequate when interactions between attributes create compounded bias.

This design goal is framed against shortcomings in prior methods. TIME edits cross-attention by mapping source prompts to target prompts, but because it updates projections using all prompt token embeddings, it can unintentionally alter related concepts beyond the intended target. UCE adds preservation terms, but requires a manually curated preservation set of concepts that must remain unchanged and still struggles with interdependent multi-attribute effects. ITI-GEN learns inclusive tokens, but needs reference image sets for each attribute category, limiting practicality (Yesiltepe et al., 2024).

Method	Characterization in the paper	Limitation emphasized
TIME	edits cross-attention by mapping source prompts to target prompts	can alter related concepts beyond the intended target
UCE	adds preservation terms	requires a manually curated preservation set
ITI-GEN	learns inclusive tokens	needs reference image sets for each attribute category
MIST	edits cross-attention projections in a disentangled manner	avoids preservation lists and reference-image sets

A common misconception is that MIST is simply another one-attribute debiasing method. The paper explicitly positions it instead as a method for single and intersectional attributes, with a formulation intended to edit several attribute directions simultaneously rather than sequentially.

2. Diffusion backbone and cross-attention setting

MIST is implemented on Stable Diffusion 1.5 and operates inside the standard latent diffusion pipeline (Yesiltepe et al., 2024). The paper reviews the diffusion process as

$$p_\theta(x_{1:T}) = p(x_T)\prod_{t=T}^{1} p_\theta(x_{t-1}\mid x_t)$$

with the original text-to-image training objective

$$\mathbb{E}_{x,c,\epsilon,t}\left[\lVert \epsilon_\theta(x_t,t,c)-\epsilon\rVert_2^2\right].$$

At inference time, classifier-free guidance is used:

$$\tilde{\epsilon}_\theta(x_t,t,c) = \epsilon_\theta(x_t,t,\emptyset) + \gamma\left[\epsilon_\theta(x_t,t,c)-\epsilon_\theta(x_t,t,\emptyset)\right],$$

where $\gamma \ge 1$.

Text prompts are encoded by CLIP into token embeddings

$$\mathcal{C}=\{c_{\text{[SOS]}},c_1,\ldots,c_N,c_{\text{[EOS]}}\}.$$

Within each U-Net cross-attention layer, visual features are projected to queries with $W_q$, and text embeddings are projected to keys and values with $W_k$ and $\times$0. For token $\times$1,

$\times$2

and over all tokens,

$\times$3

MIST edits the key and value projection matrices in these cross-attention layers. The method does not retrain the whole diffusion model and does not introduce time-dependent masks or per-step attention replacement schedules. Once the edited projection weights are installed, generation proceeds as in a normal diffusion model, with the edits affecting conditioning throughout denoising.

3. Disentangled <EOS>-based cross-attention editing

The core technical idea is the paper’s claim that the CLIP text encoder’s end-of-sentence token, <EOS>, acts as a global representation of the prompt (Yesiltepe et al., 2024). On that basis, MIST replaces tokenwise editing with editing based only on the prompt’s global <EOS> representation. This is what the paper means by disentangled cross-attention editing.

The contrast with prior objectives is important. TIME is restated as

$\times$4

where $\times$5 and $\times$6 are source and destination embeddings and $\times$7. UCE separates edited concepts $\times$8 and preserved concepts $\times$9:

$\times$0

MIST instead takes a source prompt $\times$1, such as “a CEO,” and a guidance prompt $\times$2, such as “a female CEO,” and fine-tunes the projection matrix $\times$3 by minimizing

$\times$4

Here $\times$5 is the <EOS> embedding for the guidance concept, $\times$6 is the <EOS> embedding for the source concept, and $\times$7 is the original pretrained projection matrix. The practical interpretation is that MIST pushes the cross-attention projections so that the global representation of the biased source prompt aligns with that of a debiased guidance prompt, without directly altering all individual tokens.

The paper describes this as injecting the <EOS> token from the guidance into the source embedding. Qualitative examples are used as intuition: “a woman” with guidance “a woman with lipstick” changes the lipstick attribute while keeping the rest of the face stable, and a prompt such as “Asian male person with eyeglasses” suggests that the <EOS> token can carry multiple attributes jointly.

This design matters because the paper’s argument is not merely that <EOS> is compact. It is that tokenwise edits can entangle the target attribute with neighboring semantic content, whereas <EOS>-only editing provides a more global but more isolated control handle.

4. Intersectional formulation and optimization procedure

For intersectional debiasing, MIST extends the single-direction edit to multiple guidance directions collected through the <EOS> token (Yesiltepe et al., 2024). It defines

$\times$8

and for $\times$9 guidance directions solves

$$p_\theta(x_{1:T}) = p(x_T)\prod_{t=T}^{1} p_\theta(x_{t-1}\mid x_t)$$0

The paper’s interpretation is that several attribute directions can be edited simultaneously in one optimization, rather than by sequentially applying one debiasing operation after another. In practice, the target is a uniform balance over the specified categories: 50/50 for binary gender, and equal proportions over joint categories for multi-category race or combinations such as gender $$p_\theta(x_{1:T}) = p(x_T)\prod_{t=T}^{1} p_\theta(x_{t-1}\mid x_t)$$1 race.

The optimization procedure is deliberately lightweight. The reported implementation uses Stable Diffusion 1.5, runs on a single L40 GPU, and uses learning rate 0.5. For the multi-attribute objective, the regularization is set to

$$p_\theta(x_{1:T}) = p(x_T)\prod_{t=T}^{1} p_\theta(x_{t-1}\mid x_t)$$2

where $$p_\theta(x_{1:T}) = p(x_T)\prod_{t=T}^{1} p_\theta(x_{t-1}\mid x_t)$$3 is the number of concepts being debiased.

The stopping rule is not derived from a closed-form update or a separate validation loss. During optimization, the method periodically generates $$p_\theta(x_{1:T}) = p(x_T)\prod_{t=T}^{1} p_\theta(x_{t-1}\mid x_t)$$4 images and computes a CLIP-based attribute score to test whether a uniform distribution across attributes has been achieved. Fine-tuning stops when the sample distribution over attributes is balanced according to that CLIP-based estimate.

The paper does not introduce additional masking terms, decomposition penalties, or explicit orthogonality constraints beyond the central use of the <EOS> token plus regularization toward the old weights. A practical implication is that MIST is best understood as fine-tuning cross-attention weights, not as retraining the model or adding inference-time control modules.

5. Evaluation protocol, fairness metrics, and empirical results

The experimental focus is occupational prompts, where diffusion models exhibit strong demographic stereotypes (Yesiltepe et al., 2024). For single-attribute debiasing, the paper uses the WinoBias dataset, which contains 36 professions; one section reports 35 professions, while the supplement provides the full table. Evaluated attributes include gender, race, age, and eyeglasses, with multi-attribute combinations such as gender $$p_\theta(x_{1:T}) = p(x_T)\prod_{t=T}^{1} p_\theta(x_{t-1}\mid x_t)$$5 race, gender $$p_\theta(x_{1:T}) = p(x_T)\prod_{t=T}^{1} p_\theta(x_{t-1}\mid x_t)$$6 eyeglasses, gender $$p_\theta(x_{1:T}) = p(x_T)\prod_{t=T}^{1} p_\theta(x_{t-1}\mid x_t)$$7 race $$p_\theta(x_{1:T}) = p(x_T)\prod_{t=T}^{1} p_\theta(x_{t-1}\mid x_t)$$8 age, and gender $$p_\theta(x_{1:T}) = p(x_T)\prod_{t=T}^{1} p_\theta(x_{t-1}\mid x_t)$$9 race $$\mathbb{E}_{x,c,\epsilon,t}\left[\lVert \epsilon_\theta(x_t,t,c)-\epsilon\rVert_2^2\right].$$0 eyeglasses. In the intersectional experiments, race is treated as six categories: Indian, Asian, African, European, Latino, and Middle Eastern.

Three metrics are defined. The biasedness metric $$\mathbb{E}_{x,c,\epsilon,t}\left[\lVert \epsilon_\theta(x_t,t,c)-\epsilon\rVert_2^2\right].$$1 measures deviation from the ideal ratio:

$$\mathbb{E}_{x,c,\epsilon,t}\left[\lVert \epsilon_\theta(x_t,t,c)-\epsilon\rVert_2^2\right].$$2

Here $$\mathbb{E}_{x,c,\epsilon,t}\left[\lVert \epsilon_\theta(x_t,t,c)-\epsilon\rVert_2^2\right].$$3 is perfectly debiased, $$\mathbb{E}_{x,c,\epsilon,t}\left[\lVert \epsilon_\theta(x_t,t,c)-\epsilon\rVert_2^2\right].$$4 means complete bias toward one category, and for $$\mathbb{E}_{x,c,\epsilon,t}\left[\lVert \epsilon_\theta(x_t,t,c)-\epsilon\rVert_2^2\right].$$5 balanced categories $$\mathbb{E}_{x,c,\epsilon,t}\left[\lVert \epsilon_\theta(x_t,t,c)-\epsilon\rVert_2^2\right].$$6. The deviation of ratios $$\mathbb{E}_{x,c,\epsilon,t}\left[\lVert \epsilon_\theta(x_t,t,c)-\epsilon\rVert_2^2\right].$$7 measures preservation on non-edited concepts:

$$\mathbb{E}_{x,c,\epsilon,t}\left[\lVert \epsilon_\theta(x_t,t,c)-\epsilon\rVert_2^2\right].$$8

with signed $$\mathbb{E}_{x,c,\epsilon,t}\left[\lVert \epsilon_\theta(x_t,t,c)-\epsilon\rVert_2^2\right].$$9. The Average Pixel Shift (APS) measures content preservation relative to Stable Diffusion under the same seed:

$$\tilde{\epsilon}_\theta(x_t,t,c) = \epsilon_\theta(x_t,t,\emptyset) + \gamma\left[\epsilon_\theta(x_t,t,c)-\epsilon_\theta(x_t,t,\emptyset)\right],$$0

The protocol uses 1000 images per profession for single-attribute profession debiasing, 500 randomly generated samples for intersectional quantitative experiments, and 250 images for each non-debiased occupation in preservation experiments.

On a subset of nine professions, average biasedness drops from $$\tilde{\epsilon}_\theta(x_t,t,c) = \epsilon_\theta(x_t,t,\emptyset) + \gamma\left[\epsilon_\theta(x_t,t,c)-\epsilon_\theta(x_t,t,\emptyset)\right],$$1 for Stable Diffusion to $$\tilde{\epsilon}_\theta(x_t,t,c) = \epsilon_\theta(x_t,t,\emptyset) + \gamma\left[\epsilon_\theta(x_t,t,c)-\epsilon_\theta(x_t,t,\emptyset)\right],$$2 for MIST. The corresponding baseline values reported are 0.28 for UCE, 0.38 for TIME, 0.63 for Concept Algebra, and 0.60 for Debias-VL (Yesiltepe et al., 2024).

Occupation / Summary	Stable Diffusion	MIST
Average biasedness on nine professions	$$\tilde{\epsilon}_\theta(x_t,t,c) = \epsilon_\theta(x_t,t,\emptyset) + \gamma\left[\epsilon_\theta(x_t,t,c)-\epsilon_\theta(x_t,t,\emptyset)\right],$$3	$$\tilde{\epsilon}_\theta(x_t,t,c) = \epsilon_\theta(x_t,t,\emptyset) + \gamma\left[\epsilon_\theta(x_t,t,c)-\epsilon_\theta(x_t,t,\emptyset)\right],$$4
Teacher	0.36	0.04
Nurse	1.00	0.26
Librarian	0.92	0.04
Housekeeper	1.00	0.10
Developer	0.98	0.14
CEO	0.96	0.20
Doctor	0.64	0.12

On content preservation, MIST yields lower APS than TIME, Concept Algebra, UCE, and ITI-GEN on the reported occupations. For example, for CEO the APS relative to Stable Diffusion is 54.86 $$\tilde{\epsilon}_\theta(x_t,t,c) = \epsilon_\theta(x_t,t,\emptyset) + \gamma\left[\epsilon_\theta(x_t,t,c)-\epsilon_\theta(x_t,t,\emptyset)\right],$$5 45.53 for MIST, versus 111.62 for TIME, 102.30 for Concept Algebra, and 99.58 for UCE. For Nurse, MIST reports 86.71 versus 126.35, 142.72, and 114.84 for those same baselines. For Teacher, MIST reports 97.71 versus 127.47, 143.96, and 146.77.

For intersectional bias, the paper reports that MIST achieves near-uniform distributions in the majority of cases. The qualitative figures show race $$\tilde{\epsilon}_\theta(x_t,t,c) = \epsilon_\theta(x_t,t,\emptyset) + \gamma\left[\epsilon_\theta(x_t,t,c)-\epsilon_\theta(x_t,t,\emptyset)\right],$$6 gender and triple combinations such as gender $$\tilde{\epsilon}_\theta(x_t,t,c) = \epsilon_\theta(x_t,t,\emptyset) + \gamma\left[\epsilon_\theta(x_t,t,c)-\epsilon_\theta(x_t,t,\emptyset)\right],$$7 age $$\tilde{\epsilon}_\theta(x_t,t,c) = \epsilon_\theta(x_t,t,\emptyset) + \gamma\left[\epsilon_\theta(x_t,t,c)-\epsilon_\theta(x_t,t,\emptyset)\right],$$8 eyeglasses, and the quantitative plots indicate that one-shot multi-attribute editing can approach the intended uniform output distribution.

6. Interpretation, limitations, and place in the literature

The central conceptual contribution of MIST is not only that it debiases Stable Diffusion, but that it proposes the <EOS> token as a disentangled global editing handle for cross-attention (Yesiltepe et al., 2024). A plausible implication is that the paper is also making a representational claim about CLIP prompt encoding: the global prompt summary may be a more stable intervention point than tokenwise prompt directions when multiple attributes interact.

The paper is careful, however, about what its results do and do not establish. First, MIST inherits the limitations of Stable Diffusion and CLIP, both trained on biased data. Because CLIP is used both for prompt encoding and for evaluation/classification of attributes, debiasing is itself shaped by those same biases; race categorization is explicitly noted as particularly sensitive. Second, MIST aims for distributional balance, not a formal fairness guarantee. Equal output ratios over prompts do not ensure equitable or non-stereotyped representation in every semantic or cultural sense. Third, the method depends on the textual definitions of source and guidance prompts and does not claim invariance to prompt wording. Fourth, some professions still show nontrivial residual bias or imperfect balancing. Fifth, the work focuses on perceived binary gender because of classifier limitations, even though the paper explicitly notes that gender is not binary.

A second common misconception is that lower pixel shift alone proves better fairness. The paper does not make that claim. APS is a preservation metric: it supports the argument that <EOS>-based editing changes surrounding image content less than competing methods, but it is distinct from the fairness metrics $$\tilde{\epsilon}_\theta(x_t,t,c) = \epsilon_\theta(x_t,t,\emptyset) + \gamma\left[\epsilon_\theta(x_t,t,c)-\epsilon_\theta(x_t,t,\emptyset)\right],$$9 and $\gamma \ge 1$0.

In practical terms, MIST is a method for a practitioner who wants to debias a pretrained text-to-image diffusion model at the level of cross-attention weights, especially when multiple interacting attributes must be addressed without retraining or collecting curated preservation or reference-image datasets. Its stated advantages are tightly scoped: it edits only cross-attention key/value projections of an existing Stable Diffusion model, uses the prompt’s <EOS> token as a global control handle, debiases single and intersectional attributes in one process, avoids UCE-style preservation lists, avoids ITI-GEN-style reference images, and tends to preserve unrelated concepts better as measured by lower pixel shift.