EquiSteer: Debiasing in Diffusion Models

• The paper introduces EquiSteer, a training-free approach that edits cross-attention activations to achieve per-sample fairness without retraining.
• It precomputes attribute steering vectors from contrastive prompts, removing intrinsic bias and injecting target demographic directions to improve parity metrics by up to 87%.
• The method employs prompt-aware gating, orthogonalisation, and adaptive injection to preserve explicit attribute prompts while enforcing demographic neutrality.

EquiSteer is a training-free, inference-time debiasing method for text-to-image diffusion models that operates by editing cross-attention activations so that attribute-neutral prompts are redistributed toward a prescribed target distribution over protected attributes, while attribute-specific prompts are left unchanged (Gaintseva et al., 1 Jul 2026). It is formulated for demographic and related attributes including gender, race, age, body type, and eyeglasses, and is evaluated on Stable Diffusion 1.5, Stable Diffusion 2.1, SDXL, and SANA. The central technical claim is that demographic information is sufficiently encoded in cross-attention outputs to permit per-sample fairness control without retraining: EquiSteer precomputes attribute directions from contrastive prompts, detects whether a prompt is already attribute-specific, removes existing attribute components from cross-attention outputs for neutral prompts, and then injects a sampled target attribute direction.

1. Problem formulation and fairness objective

EquiSteer addresses a specific failure mode of text-to-image diffusion models: when prompts are attribute-neutral, such as profession prompts, the generated images often reflect demographic regularities of the training data rather than a balanced distribution implied by the prompt (Gaintseva et al., 1 Jul 2026). The paper defines a concept $X$ with discrete attributes $\{a_i\}_{i=1}^n$ and seeks an inference-time intervention that makes generated images follow a target distribution $p(a_1,\dots,a_n)$, with the practical choice

$p = U(a_1,\dots,a_n),$

that is, uniform sampling over attributes. The intervention must satisfy two constraints simultaneously: it should debias neutral prompts, and it should preserve explicit user intent for attribute-specific prompts.

The attribute families studied are gender as a binary concept $(male,female)$; race as a 5-way concept $(White, Black, Asian, Indian, Latino)$; age as a 3-way concept $(young, middle\text{-}aged, elderly)$; body type as a 3-way concept $(slim, average\ build, heavy)$; and eyeglasses as a binary concept $(with\ eyeglasses,\ without\ eyeglasses)$. Gender is the primary concept in the main paper, while race, age, body type, and eyeglasses are studied in the main additional-concept section and the supplementary material.

The method is positioned against several limitations in prior debiasing approaches. Some methods require retraining or finetuning; others operate at the batch level rather than per sample; some rely on prompt-specific embedding manipulations that are sensitive to phrasing; and some depend on model-specific constraints or external classifiers. EquiSteer is instead defined as a training-free, per-sample, inference-time intervention on cross-attention activations. In this formulation, fairness is operationalized statistically: for neutral prompts the empirical attribute marginal should approach the chosen target distribution, typically parity.

2. Cross-attention steering as the intervention mechanism

The method is built around the proposition that cross-attention is the relevant locus for fairness intervention because it is text-conditioned, spatially aware, accessible at inference time, and empirically shown to encode attribute information. EquiSteer therefore edits the output of every cross-attention layer across the denoising trajectory rather than altering only text embeddings or model weights.

Its starting point is a base steering rule inherited from cross-attention steering: $$ca^{out\_new}_{lt} = ca^{out}_{lt} + \alpha s^X_{lt},$$ where $\{a_i\}_{i=1}^n$0 is the output of cross-attention layer $\{a_i\}_{i=1}^n$1 at denoising step $\{a_i\}_{i=1}^n$2, $\{a_i\}_{i=1}^n$3 is a steering vector for concept $\{a_i\}_{i=1}^n$4, and $\{a_i\}_{i=1}^n$5 is the steering strength. For debiasing toward a sampled target attribute $\{a_i\}_{i=1}^n$6, this becomes

$\{a_i\}_{i=1}^n$7

The cross-attention output has shape $\{a_i\}_{i=1}^n$8, and the same steering vector is added to each image token. For stability, the paper renormalizes the edited activation: $\{a_i\}_{i=1}^n$9

The paper reports that this basic intervention fails in two characteristic ways. First, it can overwrite explicit prompts such as “male doctor,” which is incompatible with intent preservation. Second, if pre-existing attribute signal remains in the activation, the result can be mixed or ambiguous attributes rather than clean redistribution. EquiSteer therefore augments basic cross-attention steering with three additional mechanisms: prompt-aware gating, orthogonalisation, and adaptive injection magnitude.

Steering vectors are precomputed offline from contrastive prompt pairs for every layer-step pair $p(a_1,\dots,a_n)$0. For gender, prompt pairs have the form $p(a_1,\dots,a_n)$1, giving $p(a_1,\dots,a_n)$2 prompt pairs for each direction. For race, age, body type, and eyeglasses, the construction uses $p(a_1,\dots,a_n)$3 pairs per attribute. This design makes the runtime procedure independent of any online retraining stage and confines calibration to statistics derived from contrastive prompts and activation traces.

3. Prompt-aware gating, orthogonalisation, and adaptive injection

The prompt-aware gate is designed to distinguish neutral prompts from attribute-specific prompts using internal activation geometry rather than a text parser (Gaintseva et al., 1 Jul 2026). For image token $p(a_1,\dots,a_n)$4, layer $p(a_1,\dots,a_n)$5, step $p(a_1,\dots,a_n)$6, and attribute direction $p(a_1,\dots,a_n)$7, the relevant score is the tokenwise dot product $p(a_1,\dots,a_n)$8. Because attribute evidence may be spatially localized, EquiSteer uses the maximal token response

$p(a_1,\dots,a_n)$9

For each attribute $p = U(a_1,\dots,a_n),$0, the threshold is defined as the midpoint between empirical means on attribute-specific and neutral calibration prompts: $p = U(a_1,\dots,a_n),$1 The gate is evaluated once per generation at timestep $p = U(a_1,\dots,a_n),$2 and a model-specific gate layer $p = U(a_1,\dots,a_n),$3; if

$p = U(a_1,\dots,a_n),$4

the prompt is treated as attribute-specific and the model falls back to vanilla generation. The selected gating layers are $p = U(a_1,\dots,a_n),$5 for SD-1.5, $p = U(a_1,\dots,a_n),$6 for SD-2.1, $p = U(a_1,\dots,a_n),$7 for SDXL, and $p = U(a_1,\dots,a_n),$8 for SANA.

For neutral prompts, EquiSteer first removes the current attribute component by projecting the cross-attention output out of the span of all attribute steering vectors for the concept. If $p = U(a_1,\dots,a_n),$9 is an orthonormal basis for $(male,female)$0, arranged as columns of $(male,female)$1, then

$(male,female)$2

The matrices $(male,female)$3 and the projection operators $(male,female)$4 are precomputed offline. This orthogonalisation stage is intended to reduce mixed-attribute artifacts by erasing whatever demographic residual is already present in the activation.

After projection removal, the method injects a sampled target attribute direction with an adaptive magnitude calibrated from attribute-specific prompts. Defining

$(male,female)$5

the paper precomputes

$(male,female)$6

and sets

$(male,female)$7

The edited activation is then

$(male,female)$8

Because $(male,female)$9 is orthogonal to the attribute subspace, the reinjected component is intended to carry only the selected attribute direction.

The full runtime algorithm is structurally simple. The gate is run once at $(White, Black, Asian, Indian, Latino)$0. If the prompt is neutral, a target attribute is sampled uniformly,

$(White, Black, Asian, Indian, Latino)$1

and then, for every denoising step, every cross-attention layer, and every image token, EquiSteer applies projection removal, adaptive target injection, and optional renormalization before writing the edited activation back into the denoiser. The intervention therefore spans the entire cross-attention computation rather than a small subset of locations.

The eyeglasses concept is treated specially. The method learns only the positive direction $(White, Black, Asian, Indian, Latino)$2. If the sampled target is eyeglasses, standard addition is used. If the sampled target is no eyeglasses, EquiSteer performs a CASteer-style erasure: $(White, Black, Asian, Indian, Latino)$3 with fixed erasure strength $(White, Black, Asian, Indian, Latino)$4. The negative case is therefore implemented by removing alignment with the positive eyeglasses direction rather than learning a separate “no eyeglasses” vector from negation prompts.

4. Experimental protocol, calibration, and evaluation methodology

EquiSteer is evaluated on Stable Diffusion 1.5, Stable Diffusion 2.1, SDXL, and SANA using profession prompts in both neutral and attribute-specific forms (Gaintseva et al., 1 Jul 2026). The neutral template is “A photo of a {profession},” and the attribute-specific template is “A photo of a {attribute} {profession}.” The professions are CEO, doctor, pilot, technician, fashion designer, librarian, teacher, and nurse. The evaluation uses 1,000 images per neutral prompt or profession and 300 images per attribute-specific prompt or profession.

Calibration prompts used for steering-vector construction and threshold estimation are disjoint from the test prompts. For threshold fitting, each calibration prompt is rendered with 10 seeds, and the threshold is estimated from neutral and specific means. For gender gating, the neutral professions used in calibration are not the test professions; examples given are cleaner and counselor.

The principal fairness metric is parity gap. For binary settings, the paper writes

$(White, Black, Asian, Indian, Latino)$5

with $(White, Black, Asian, Indian, Latino)$6 and $(White, Black, Asian, Indian, Latino)$7 the observed attribute ratio, and the supplementary tabulates the equivalent form

$(White, Black, Asian, Indian, Latino)$8

For 3-way and 5-way settings, the corresponding averaged absolute deviations from $(White, Black, Asian, Indian, Latino)$9 and $(young, middle\text{-}aged, elderly)$0 are used. Lower $(young, middle\text{-}aged, elderly)$1 is better, and $(young, middle\text{-}aged, elderly)$2 corresponds to perfect parity. For attribute-specific prompts, the target is $(young, middle\text{-}aged, elderly)$3 for the requested attribute, and the paper reports preservation or recall.

Attribute classification is itself model mediated. The evaluation uses CLIP ViT-L/14 zero-shot for gender, race, age, and body type, and BLIP-VQA capfilt-large for eyeglasses because CLIP underestimates eyeglasses. The supplementary also uses GPT-4o as a human-aligned oracle. Image quality is measured with CLIPScore for text-image alignment and CMMD for image fidelity, both computed on 30,000 images generated from MS-COCO 2014 validation prompts.

This evaluation protocol is important to the interpretation of the results. EquiSteer is not assessed through human demographic annotation at scale; instead, its fairness and preservation claims are operationalized through classifier outputs, parity metrics, and prompt-conditioned recall. The paper treats that choice as practical, but also notes the resulting limitations.

5. Empirical results, ablations, and transferability

Across SD-1.5, SD-2.1, SDXL, and SANA, EquiSteer reduces average parity gap by up to $(young, middle\text{-}aged, elderly)$4 relative to vanilla generation and by up to $(young, middle\text{-}aged, elderly)$5 over the strongest baseline (Gaintseva et al., 1 Jul 2026). For gender debiasing on SD-1.5, the average $(young, middle\text{-}aged, elderly)$6 falls from $(young, middle\text{-}aged, elderly)$7 for vanilla to $(young, middle\text{-}aged, elderly)$8 for TEI and $(young, middle\text{-}aged, elderly)$9 for EquiSteer. Profession-specific gender ratios move correspondingly toward parity: CEO shifts from $(slim, average\ build, heavy)$0 to $(slim, average\ build, heavy)$1, doctor from $(slim, average\ build, heavy)$2 to $(slim, average\ build, heavy)$3, and nurse male ratio from $(slim, average\ build, heavy)$4 to $(slim, average\ build, heavy)$5. For SDXL, average gender parity gap changes from $(slim, average\ build, heavy)$6 to $(slim, average\ build, heavy)$7, and for SANA from $(slim, average\ build, heavy)$8 to $(slim, average\ build, heavy)$9.

A central result is that explicit user intent is largely preserved. On prompts such as “A photo of a male nurse,” the supplementary reports average male-specified recall of $(with\ eyeglasses,\ without\ eyeglasses)$0 and average female-specified recall of $(with\ eyeglasses,\ without\ eyeglasses)$1 on SD-1.5. This preservation behavior is the primary functional role of the gate: unconditional redistribution would improve neutral-prompt parity but would also overwrite explicit demographic specification.

The paper reports small quality trade-offs relative to vanilla generation. On SD1.5, CLIPScore changes from $(with\ eyeglasses,\ without\ eyeglasses)$2 for vanilla to $(with\ eyeglasses,\ without\ eyeglasses)$3 for EquiSteer, while CMMD changes from $(with\ eyeglasses,\ without\ eyeglasses)$4 to $(with\ eyeglasses,\ without\ eyeglasses)$5. On SDXL, CLIPScore changes from $(with\ eyeglasses,\ without\ eyeglasses)$6 to $(with\ eyeglasses,\ without\ eyeglasses)$7, while CMMD changes from $(with\ eyeglasses,\ without\ eyeglasses)$8 to $(with\ eyeglasses,\ without\ eyeglasses)$9. The paper interprets these shifts as minimal quality cost, and in some settings CLIPScore is slightly improved.

The method extends beyond gender. For race, $$ca^{out\_new}_{lt} = ca^{out}_{lt} + \alpha s^X_{lt},$$0 on SDXL changes from $$ca^{out\_new}_{lt} = ca^{out}_{lt} + \alpha s^X_{lt},$$1 to $$ca^{out\_new}_{lt} = ca^{out}_{lt} + \alpha s^X_{lt},$$2, and on SANA from $$ca^{out\_new}_{lt} = ca^{out}_{lt} + \alpha s^X_{lt},$$3 to $$ca^{out\_new}_{lt} = ca^{out}_{lt} + \alpha s^X_{lt},$$4. For age, SDXL changes from $$ca^{out\_new}_{lt} = ca^{out}_{lt} + \alpha s^X_{lt},$$5 to $$ca^{out\_new}_{lt} = ca^{out}_{lt} + \alpha s^X_{lt},$$6, and SANA from $$ca^{out\_new}_{lt} = ca^{out}_{lt} + \alpha s^X_{lt},$$7 to $$ca^{out\_new}_{lt} = ca^{out}_{lt} + \alpha s^X_{lt},$$8. For body type, SDXL changes from $$ca^{out\_new}_{lt} = ca^{out}_{lt} + \alpha s^X_{lt},$$9 to $\{a_i\}_{i=1}^n$00, and SANA from $\{a_i\}_{i=1}^n$01 to $\{a_i\}_{i=1}^n$02. For eyeglasses, SDXL changes from $\{a_i\}_{i=1}^n$03 to $\{a_i\}_{i=1}^n$04 under BLIP evaluation, and SANA from $\{a_i\}_{i=1}^n$05 to $\{a_i\}_{i=1}^n$06. The strongest reported additional-attribute reductions are approximately $\{a_i\}_{i=1}^n$07–$\{a_i\}_{i=1}^n$08 for race and $\{a_i\}_{i=1}^n$09 for age on SANA.

EquiSteer also supports sequential multi-concept debiasing. In the joint-4 setting combining gender, race, age, and body type, SDXL changes from gender $\{a_i\}_{i=1}^n$10 to $\{a_i\}_{i=1}^n$11, race $\{a_i\}_{i=1}^n$12 to $\{a_i\}_{i=1}^n$13, age $\{a_i\}_{i=1}^n$14 to $\{a_i\}_{i=1}^n$15, and body $\{a_i\}_{i=1}^n$16 to $\{a_i\}_{i=1}^n$17. On SANA, the joint-4 results are gender $\{a_i\}_{i=1}^n$18 to $\{a_i\}_{i=1}^n$19, race $\{a_i\}_{i=1}^n$20 to $\{a_i\}_{i=1}^n$21, age $\{a_i\}_{i=1}^n$22 to $\{a_i\}_{i=1}^n$23, and body $\{a_i\}_{i=1}^n$24 to $\{a_i\}_{i=1}^n$25. These results indicate that the method is not confined to one protected attribute at a time, although the paper retains separate thresholds, steering vectors, and attribute subspaces for each concept.

The transferability study uses the same steering vectors and thresholds without retuning on paraphrased, long-context, and compositional prompts. Under that constraint, EquiSteer reduces gender parity gap by $\{a_i\}_{i=1}^n$26–$\{a_i\}_{i=1}^n$27. Examples include SDXL paraphrased prompts changing from $\{a_i\}_{i=1}^n$28 to $\{a_i\}_{i=1}^n$29, SANA long contextual prompts changing from $\{a_i\}_{i=1}^n$30 to $\{a_i\}_{i=1}^n$31, and SANA compositional prompts changing from $\{a_i\}_{i=1}^n$32 to $\{a_i\}_{i=1}^n$33. The weakest transfer case is SANA on multi-subject prompts, which the paper attributes to the fact that the gate is image-level rather than entity-specific.

The ablation results isolate the contribution of each component. On SD-1.5 gender, add only yields neutral $\{a_i\}_{i=1}^n$34; add plus erase yields $\{a_i\}_{i=1}^n$35 but poor male-specific preservation; add plus erase plus gate yields $\{a_i\}_{i=1}^n$36 together with strong explicit-prompt preservation. The gate itself is strongly separable in activation space: supplementary AUROC is at least $\{a_i\}_{i=1}^n$37 on every evaluated backbone-by-attribute cell, and many cells reach $\{a_i\}_{i=1}^n$38. The threshold-multiplier sweep further shows a trade-off: larger global threshold multiplier $\{a_i\}_{i=1}^n$39 causes the gate to fire less often, increasing debiasing but degrading attribute-specific recall; the default is $\{a_i\}_{i=1}^n$40, while long prompts sometimes benefit from stronger thresholds, approximately $\{a_i\}_{i=1}^n$41 for SANA and $\{a_i\}_{i=1}^n$42 for SDXL.

6. Limitations, ethical interpretation, and significance

The paper explicitly identifies several practical limitations of EquiSteer (Gaintseva et al., 1 Jul 2026). Runtime overhead is modest but nonzero because the intervention is applied at every cross-attention layer, every denoising step, and every image token, with projection and optional renormalization. Calibration remains concept-specific: thresholds $\{a_i\}_{i=1}^n$43, mean magnitudes $\{a_i\}_{i=1}^n$44, and gate-layer choice must be determined per concept and backbone. Gate reliability is imperfect in both directions, with false positives causing neutral prompts to be skipped and false negatives causing explicit prompts to be debiased accidentally.

A more structural limitation is that the gate is image-level rather than entity-level. Mixed prompts with multiple people and richer compositional prompts are therefore harder. This limitation is visible in the weaker multi-subject transfer results. The paper also notes that several target attributes are represented as discrete, often binary or low-cardinality classes. That representation simplifies real demographic variation, and the uniform-parity target need not be appropriate in every context. The method therefore operationalizes one specific fairness notion—statistical parity over predefined categories—rather than resolving the broader normative problem of demographic fairness in generative models.

Evaluation is also classifier mediated. CLIP is described as conservative or unreliable on some attributes, especially race, body type, and eyeglasses. The paper mitigates this partly with BLIP and GPT-4o, but the reported fairness gains remain downstream of automated attribute recognition. Ethical concerns follow directly from that setup: overcorrection, simplified identity categories, deployment in contexts where demographic balancing may be inappropriate, and unintended distortions of user intent if gating fails.

Within those bounds, EquiSteer’s main technical significance is that it reframes fairness intervention as direct manipulation of cross-attention activations rather than weight editing, finetuning, or batch-level output balancing. Its operative insight is not merely that cross-attention is interpretable, but that it is controllable in a way that supports training-free, per-sample, prompt-aware demographic redistribution. The method’s three defining properties are therefore architectural rather than purely empirical: no weight updates, per-sample sampling of target attributes, and abstention on attribute-specific prompts. Empirically, the reported reductions in parity gap, the preservation of explicit prompts, and the small changes in CLIPScore and CMMD make it a representative example of activation-space fairness control for text-guided image generation.