FailureAtlas: Auditing T2I Model Failures

• FailureAtlas is a framework that actively audits text-to-image models by exploring a structured semantic space of entity-attribute pairs to identify minimal failure-inducing conditions.
• It employs a combinatorial search with rule-based pruning and prediction-based prioritization to efficiently uncover detailed error slices in generated images.
• The framework also analyzes the correlation between discovered failure slices and training-data scarcity, providing actionable insights for model improvement.

FailureAtlas is a framework for auditing text-to-image (T2I) models by actively exploring the smallest semantic conditions under which they fail, rather than passively measuring performance on a fixed prompt set. It treats evaluation as a structured search over an entity-attribute prompt space, identifies minimal failure-inducing concepts, and records them as error slices whose generation success rate falls below a threshold. In experiments on Stable Diffusion models, it uncovers large numbers of fine-grained, previously unknown failure slices and uses the same semantic structure to study whether these failures align with training-data scarcity (Chen et al., 26 Sep 2025).

1. Conceptual scope and diagnostic objective

FailureAtlas is designed as a complement to static T2I benchmarks. The underlying claim is that benchmarks such as GenEval, T2I-CompBench, HRS-Bench, and TIFA are useful for comparison but limited as diagnostic instruments because they use imbalanced prompt distributions, provide poor diagnostic resolution when multiple semantic factors co-occur in one prompt, and offer weak support for root-cause analysis. FailureAtlas therefore replaces passive scoring with active exploration of a model’s failure landscape: the distribution of errors across a structured semantic space of prompts (Chen et al., 26 Sep 2025).

The semantic space is built from combinations of entities and attributes. A failure is not defined as a generic quality defect or aesthetic shortcoming, but as insufficient semantic faithfulness to the intended entity and attributes. The framework seeks minimal failure-inducing concepts. If a model already fails on an entity such as screwdriver, then more specific descendants such as red screwdriver or metal screwdriver are not treated as more fundamental discoveries. This induces a hierarchy of semantic conditions and makes the output diagnostic rather than merely enumerative.

An error slice is the central object of analysis. It denotes a semantic subset of prompts, represented by one entity with zero, one, or more attributes, for which the model’s generation success rate falls below a threshold. The significance of this formulation is methodological: it isolates failure conditions at a granularity that static prompt sets usually do not expose.

2. Formal search problem and failure criterion

FailureAtlas defines a structured combinatorial search over an entity set $E$ and an attribute set $A$, with nodes of the form

$x = (e,S), \quad e \in E,\; S \subseteq A.$

The search is layered. Layer 1 contains entities only, $x=(e,\emptyset)$; layer 2 adds one attribute; layer 3 adds two attributes. In the reported experiments, the search depth is capped at three layers, so each node contains at most two attributes. There is also a category constraint: at most one attribute from the same category may appear in a node, so if one size attribute is already present, another from the same category is excluded (Chen et al., 26 Sep 2025).

Each node is rendered into a prompt and evaluated by generating

$K=25$

images. An MLLM-based evaluator, Qwen2-VL-72B, answers multiple-choice questions about the entity and the target attributes. The generation success rate is defined as the proportion of evaluator predictions aligned with the intended labels. In the notation used in the source description, if $Y_x$ is the set of queried semantic labels for node $x$, then the success rate has the form

$$\mathrm{Succ}(x) = \frac{1}{K \cdot |Y_x|} \sum_{i=1}^{K} \sum_{j \in Y_x} \mathbf{1}\!\left[\hat y_{i,j} = y_j\right].$$

A node is classified as an error slice iff

$\mathrm{Succ}(x) < \tau,$

with

$\tau = 0.8.$

The same formal scaffold is used for attribution analysis. For a discovered slice $A$0, let $A$1 denote its estimated frequency in training data, and let $A$2 denote the average slice frequency at layer $A$3. A discovered error slice is attributed to data scarcity if

$A$4

where $A$5 is varied during analysis. This does not define a causal mechanism; it defines a frequency-based attribution rule over the discovered semantic slices.

3. Corpus construction and semantic coverage

FailureAtlas makes the prompt space tractable by replacing unconstrained language with a structured entity-attribute corpus. The corpus is built in three stages. First, an LLM initializes a vocabulary using general world knowledge. Second, the vocabulary is refined and expanded by mining COCO Captions and T2I-CompBench; unmatched frequent terms are added and the vocabulary is manually curated to remove redundancy and form hierarchies. Third, an LLM annotates the semantic validity of entity-attribute pairs so that implausible compositions are excluded from search (Chen et al., 26 Sep 2025).

The resulting corpus contains 758 entities and 437 attributes. The appendix statistics report 5 main categories and 25 subcategories for entities, and 2 main categories and 29 subcategories for attributes. The corpus favors general concepts over proper nouns or highly specific named entities, so it is intended as a broad semantic scaffold rather than a catalog of all possible prompts.

The paper evaluates semantic coverage by mapping benchmark concepts into this corpus. Reported coverage is 88.2% entity and 93.7% attribute for COCO Captions, 88.5% and 96.0% for T2I-CompBench, 89.6% and 90.6% for HRS-Bench, and 92.0% and 90.8% for TIFA. These values are presented as evidence that the corpus spans a benchmark-relevant semantic space while also supporting systematic compositional auditing.

Prompt instantiation is modular. A node is converted into text using a base description, an action description, and a background description. Qwen2.5-14B-Instruct performs only minor grammatical correction, such as pluralization, article insertion, tense consistency, and smoother conjunctions. The stated intention is to avoid uncontrolled prompt expansion while keeping the semantic specification consistent.

4. Exploration pipeline and acceleration mechanisms

FailureAtlas performs breadth-first search across entity trees, layer by layer. For each node, it instantiates a prompt, generates 25 images with the target T2I model, evaluates entity and attribute consistency using the MLLM-based multiple-choice protocol, computes the success rate, and then either records the node as an error slice or expands its descendants. In the reported configuration, the full search space up to depth three still contains about 7.36M nodes, so tractability is a central design problem (Chen et al., 26 Sep 2025).

The first acceleration mechanism is rule-based pruning. It relies on the monotonicity-inspired assumption that generating images with more attributes is generally more challenging than generating images with any subset of those attributes. If a node fails, its descendants are skipped. This dramatically reduces search cost and biases the output toward more fundamental failure concepts. For SD1.5, the paper reports that pruning reduces the third layer to 4.2% of its full size, and the extrapolated fourth layer to 0.4%.

The second acceleration mechanism is prediction-based prioritization. Within a layer, nodes are independent, so their order can be optimized. FailureAtlas uses T5 text embeddings of the entity and attributes and a lightweight transformer decoder to estimate a node’s success rate $A$6. Missing attributes at shallower depths are zero-padded, and training uses an $A$7 loss,

$A$8

The predictor is retrained online every 10,000 explored nodes and evaluated on the next 10,000 nodes. The reported effect is roughly a $A$9 speed-up in failure discovery.

These mechanisms produce a search process that is both hierarchical and budget-aware. The framework is not merely enumerating failures; it is ranking likely failures, pruning semantically dominated descendants, and preserving a minimality bias in the resulting map.

5. Experimental results and discovered failure structure

FailureAtlas is evaluated primarily on Stable Diffusion 1.5 and Stable Diffusion XL Turbo, with additional case studies on SD3.5 Large Turbo and Flux.1-dev. The main experiments use search depth 3, $x = (e,S), \quad e \in E,\; S \subseteq A.$0 images per node, Qwen2-VL-72B as evaluator, T5 embeddings for prioritization, and $x = (e,S), \quad e \in E,\; S \subseteq A.$1 as the failure threshold (Chen et al., 26 Sep 2025).

Model	Explored nodes	Error slices
SD1.5	439,467	247,430 (56.3%)
SDXL Turbo	1,040,415	439,942 (42.3%)

The layer-wise counts further show that failures are abundant at different granularities. For SD1.5, layer 1 yields 162 error slices over 758 explored nodes, layer 2 yields 113,418 over 134,816, and layer 3 yields 133,850 over 303,893. For SDXL Turbo, the corresponding numbers are 91 over 758, 108,111 over 150,170, and 331,740 over 889,487. The paper explicitly warns that total counts should not be compared directly across models because pruning changes how much of the tree is explored; weaker models fail earlier and thus trigger more aggressive pruning.

The discovered slices include both broad and highly specific weaknesses. Reported SD1.5 examples include incorrect assignment of colors to the background and clock, with only 57% success; failure to depict a person bending down; difficulty with entities such as screwdriver even without attributes; and appendix examples such as dusty chopstick, sitting zebra, and distorted badminton scenes where the racket or shuttlecock is missing. Reported SDXL Turbo examples include wooden table where the material appears but the entity is wrong or absent, replacement of paper helicopter with paper airplane, and turning an oval-shaped cake into a circular one. Case studies on SD3.5 Large Turbo report failures on rubber bands, pink tomato, lighting and style controls, and exact tool counts; Flux.1-dev shows failures on bent keys, square camera, and a gray camera despite other colors working.

The computational scale is substantial. For SD1.5 alone, FailureAtlas explores 439,467 nodes and generates about 11 million images. The paper therefore emphasizes budgeted use cases as well: in SD1.5 third-layer search, discovering 10K error slices requires evaluating about 12.5K nodes, and discovering 50K error slices requires about 65K nodes.

6. Attribution analysis, limitations, and methodological significance

FailureAtlas uses the discovered slices to examine whether failures align with training-data scarcity. For SD1.5, the analysis is conducted against LAION-2B-en, using a random sample of 1M training instances. Entities and attributes are extracted, aligned to the FailureAtlas vocabulary, and compared against layer-wise average frequencies. The qualitative conclusion stated in the paper is that poorly performing entities or attributes coincide on average with insufficient training data (Chen et al., 26 Sep 2025).

The interpretation is deliberately limited. The framework does not claim that low training-data frequency fully explains the observed failures, nor that simply adding examples would eliminate them. The paper notes counterexamples, including badminton and sensor, which remain problematic even when $x = (e,S), \quad e \in E,\; S \subseteq A.$2. It suggests data quality issues, inherent generation difficulty, training procedure limitations, and architectural constraints as alternative explanations.

Several limitations are explicit. The pruning rule rests on a monotonicity assumption that is plausible but not universal; adding an attribute can sometimes reduce ambiguity and help generation. The current search handles one entity with multiple attributes, not multi-entity compositions or more complex scenes. The discovered landscape depends on the corpus vocabulary, the semantic-validity filtering, and the handcrafted prompt templates. Failure detection depends on automatic MLLM evaluation rather than exhaustive human annotation. The appendix reports 96% human alignment for entities and 86% for attributes, while the main discussion text mentions 84% for attributes. Success-rate estimation is based on 25 generated images per node; stability tests report that beyond 25 samples the average change is less than 0.58% for SD1.5 and 0.46% for SDXL Turbo. The cost remains high despite pruning and prioritization.

Methodologically, FailureAtlas belongs to a broader shift from single-score benchmarking to structured failure mapping. Related atlas-style systems in adjacent domains include ErrorMap and ErrorAtlas, which diagnose LLM failures by source category across 35 datasets and 83 models (Ashury-Tahan et al., 22 Jan 2026), and ATLAS for long-context LLMs, which measures length-dependent capability degradation over a fixed 8K–1M grid rather than relying on one headline score (Huang et al., 27 May 2026). Within that broader movement, FailureAtlas’s distinctive contribution is to make T2I auditing an active exploration problem over a structured semantic space, with error discovery, minimality, and training-distribution analysis unified in one framework.