ACTesting: Automated Cross-modal Testing Method of Text-to-Image Software

Abstract: Recently, creative generative artificial intelligence software has emerged as a pivotal assistant, enabling users to generate content and seek inspiration rapidly. Text-to-Image (T2I) software, one of the most widely used, synthesizes images with text input by engaging in a cross-modal process. However, despite substantial advancements in the T2I engine, T2I software still encounters errors when generating complex or non-realistic scenes, including omitting focal entities, low image realism, and mismatched text-image information. The cross-modal nature of T2I software complicates error detection for traditional testing methods, and the absence of test oracles further exacerbates the complexity of the testing process. To fill this gap, we propose ACTesting, an Automated Cross-modal Testing Method of Text-to-Image Software, the first testing method explicitly designed for T2I software. ACTesting utilizes the metamorphic testing principle to address the oracle problem and identifies cross-modal semantic consistency as its fundamental Metamorphic relation (MR) by employing the Entity-relationship (ER) triples. We design three kinds of mutation operators under the guidance of MR and the adaptability density constraint to construct the new input text. After generating the images based on the text, ACTesting verifies whether MR is satisfied by detecting the ER triples across two modalities to detect the errors of T2I software. In our experiments across five popular T2I software, ACTesting effectively generates error-revealing tests, resulting in a decrease in text-image consistency by up to 20% when compared to the baseline. Additionally, an ablation study demonstrates the efficacy of the proposed mutation operators. The experimental results validate that ACTesting can reliably identify errors within T2I software.

• The paper introduces ACTesting, an automated metamorphic testing method for Text-to-Image software that evaluates semantic consistency between mutated prompts and generated images without requiring a reference image oracle.
• ACTesting parses input prompts to extract entities and relationships, then applies targeted mutation operators like Entity Substitution, Attribute Modification, and Relationship Transformation to generate follow-up test cases.
• Experimental validation showed ACTesting reveals defects in T2I systems, indicated by significant drops in text-image consistency metrics (up to 20%), although effectiveness depends on the accuracy of NLP parsing and VLM-based proxy oracles.

Automated testing of Text-to-Image (T2I) software presents unique challenges due to its cross-modal nature and the inherent difficulty in defining precise test oracles for generated images. Traditional software testing techniques are often inadequate for evaluating the complex relationship between input text prompts and output image fidelity, realism, and semantic consistency. The paper "Automated Testing for Text-to-Image Software" (2312.12933) introduces ACTesting, a method designed specifically to address these challenges by leveraging metamorphic testing principles.

ACTesting Methodology

ACTesting provides a systematic framework for generating test cases and evaluating T2I software without requiring predefined image oracles. The core principle is to apply transformations (mutations) to an initial input text prompt to create a follow-up prompt. According to a defined Metamorphic Relation (MR), the images generated from the original and follow-up prompts should exhibit predictable relationships in their semantic content. Deviations from this expected relationship indicate potential defects in the T2I software.

Test Sample Construction

To manage the cross-modal complexity, ACTesting focuses on the structured semantic information within the input prompt. It parses the text to identify core semantic components, specifically entities (subjects, objects) and their relationships (predicates), often represented as triples (Subject-Predicate-Object).

1. Input Prompt Parsing: An initial text prompt is processed using NLP techniques (e.g., dependency parsing, semantic role labeling) to extract key entities and the relationships connecting them. For example, the prompt "A red cat sits on a mat" might be parsed into entities {"cat", "mat"} and the relationship {"cat", "sits on", "mat"}, with "red" as an attribute of "cat".
2. Semantic Consistency Focus: Test cases are constructed by manipulating these structured semantic elements. The fundamental principle is to maintain or predictably alter the core semantic information across the original and follow-up prompts. This structured approach aims to create targeted tests that probe the T2I model's understanding and rendering of specific objects, attributes, and interactions.

Metamorphic Relation for T2I Software

The absence of a ground truth image for a given prompt necessitates an alternative way to check correctness. ACTesting defines MRs tailored for T2I systems. A typical MR postulates that if a follow-up prompt $T'$ is derived from an original prompt $T$ through a specific semantic-preserving or systematically altering transformation, then the corresponding generated images $I = \text{T2I}(T)$ and $I' = \text{T2I}(T')$ should exhibit consistent semantic properties relative to the transformation.

For instance, if $T'$ is generated by replacing an entity in $T$ with a synonym, the MR might require that $I'$ depicts the same scene as $I$ but with the synonymous entity replacing the original one, while other elements remain consistent. Violations occur if $I'$ drastically differs from $I$ in unrelated aspects, fails to incorporate the change correctly, or exhibits significantly lower overall quality or text-image alignment compared to $I$.

Mutation Operators Guided by Adaptability Density

ACTesting employs three types of mutation operators to generate follow-up prompts ($T'$) from original prompts ($T$). The selection and application of these operators are guided by an "adaptability density" metric, intended to prioritize mutations that are syntactically valid and semantically plausible, thus more likely to yield meaningful test cases.

1. Entity Substitution (ES): Replaces identified entities (subjects or objects) with semantically related terms. This can include synonyms (e.g., "car" -> "automobile"), hypernyms (e.g., "cat" -> "animal"), or hyponyms (e.g., "animal" -> "cat"). This operator tests the model's ability to generalize across related concepts and maintain consistency when specific objects are varied.
2. Attribute Modification (AM): Alters descriptive attributes associated with entities or relationships. This involves adding, deleting, or changing adjectives or adverbs (e.g., "red cat" -> "blue cat", "running quickly" -> "running slowly", "cat" -> "fluffy cat"). This operator probes the model's handling of specific visual details and modifiers.
3. Relationship Transformation (RT): Modifies the predicate or the structure of the relationship while aiming to preserve the core semantic link or alter it predictably. Examples include changing the verb (e.g., "cat sits on mat" -> "cat lies on mat"), changing voice (active to passive), or adding prepositional phrases that modify the relationship. This tests the model's understanding of spatial, causal, or action-based interactions between entities.

The concept of "adaptability density" likely involves heuristics or learned models to estimate the probability that a given mutation results in a coherent and meaningful prompt that doesn't stray too far from the original semantic intent, thereby ensuring the validity of the metamorphic relation.

Implementation Workflow

The practical implementation of ACTesting involves several steps:

1. Prompt Input & Parsing: Receive an original text prompt $T$. Use NLP tools (e.g., spaCy, Stanford CoreNLP, or transformer-based models fine-tuned for information extraction) to extract entities, attributes, and relationship triples.
2. Mutation Application: Apply the ES, AM, and RT mutation operators based on the parsed structure and guided by adaptability density constraints. This generates one or more follow-up prompts $T'_1, T'_2, \dots, T'_n$.
3. Image Generation: Use the target T2I software (e.g., Stable Diffusion, Midjourney API) to generate image $I$ from the original prompt $T$ and images $I'_1, \dots, I'_n$ from the follow-up prompts $T'_1, \dots, T'_n$. It's often beneficial to generate multiple images per prompt (using different random seeds) to account for stochasticity.
4. Metamorphic Relation Evaluation: Compare the pair $(I, I'_i)$ for each follow-up prompt $T'_i$ against the expected outcome defined by the MR corresponding to the mutation applied. This is the most challenging step, acting as the test oracle surrogate.
   • Oracle Approximation: Since a perfect semantic comparison oracle for images doesn't exist, ACTesting relies on proxy metrics. This typically involves using pre-trained Vision-LLMs (VLMs) like CLIP or newer alternatives.
   • Consistency Metrics:
     • Calculate the text-image alignment score (e.g., CLIP score) for the original pair $(T, I)$ and the follow-up pair $(T'_i, I'_i)$. Let these be $S = \text{Score}(T, I)$ and $S'_i = \text{Score}(T'_i, I'_i)$.
     • The MR might expect $S'_i$ to be comparable to $S$. A significant drop in $S'_i$ compared to $S$ (e.g., $S - S'_i > \delta$) could indicate a defect, suggesting the model failed to handle the mutation correctly. The paper reports reductions in text-image consistency up to 20% for error-revealing tests.
     • Optionally, employ object detection or visual question answering models on $I$ and $I'_i$ to verify the presence, attributes, and relationships of key entities specifically targeted by the mutation. For example, if an entity's color was changed from red to blue, check if the object detector finds a blue object in $I'_i$ where it found a red one in $I$.
   • Violation Detection: A violation of the MR is flagged if the observed relationship between $I$ and $I'_i$ (as measured by the proxy metrics) deviates significantly from the expected relationship.

Below is a conceptual diagram of the ACTesting workflow:

graph LR A[Input Prompt T] --> B{NLP Parser}; B --> C{Extract Entities & Relations}; C --> D{Apply Mutation Operators}; D -- Guided by --> E[Adaptability Density]; D --> F[Follow-up Prompt T']; A --> G{T2I Software}; F --> G; G -- Generates --> H[Original Image I]; G -- Generates --> I[Follow-up Image I']; H --> J{Metamorphic Relation Evaluator}; I --> J; T --> J; T' --> J; J -- Uses --> K[VLM / Proxy Metrics]; J --> L[Detect Violation?]; style G fill:#f9f,stroke:#333,stroke-width:2px style J fill:#ccf,stroke:#333,stroke-width:2px style K fill:#ff9,stroke:#333,stroke-width:1px

Experimental Validation

The authors evaluated ACTesting on four unnamed but described as "widely-used" T2I software systems. The experiments aimed to demonstrate ACTesting's ability to generate error-revealing test cases compared to baseline approaches (likely random prompt generation or simpler modifications).

• Key Finding: ACTesting successfully generated tests that revealed defects, indicated by a significant reduction in text-image consistency metrics (up to 20% drop reported) for the generated follow-up images compared to the original ones, signaling a failure of the T2I model to adhere to the metamorphic relation.
• Ablation Study: An ablation study was conducted to assess the individual contribution of the three mutation operators (ES, AM, RT). The results reportedly confirmed that each operator type was effective in uncovering distinct types of failures, validating the design choices within the ACTesting framework.

Practical Considerations and Limitations

Implementing ACTesting involves several practical aspects:

• NLP Tooling: The effectiveness heavily relies on the accuracy and robustness of the NLP parser used to decompose the input prompt. Complex or ambiguous prompts can lead to parsing errors, resulting in invalid or ineffective mutations. Fine-tuning or selecting parsers adept at handling the typical structure of T2I prompts is crucial.
• Oracle Approximation Quality: The reliance on VLMs like CLIP as proxy oracles is a significant limitation. These models have known biases and limitations; they might not fully capture nuanced semantic errors or could yield noisy similarity scores, potentially leading to false positives (flagging non-errors) or false negatives (missing actual defects). The choice of VLM and the threshold for detecting violations ($\delta$) are critical tuning parameters.
• Computational Cost: Metamorphic testing inherently increases the number of inferences required. For each original prompt, ACTesting generates one or more mutated prompts and requires T2I model inference for all of them. This can be computationally expensive, especially with large test suites or slow T2I models.
• Mutation Strategy: Defining effective mutation rules and the "adaptability density" requires careful design. Poorly designed mutations might generate nonsensical prompts or changes too subtle/drastic for the MR to be meaningful. Domain knowledge about common T2I failure modes can help guide mutation design.
• Scope of Detectable Errors: ACTesting primarily targets errors related to semantic consistency, object rendering, attribute binding, and relationship understanding. It may be less effective at detecting other types of issues like low image realism (unless correlated with semantic errors), aesthetic flaws, or certain types of biases, which might require different testing approaches (e.g., human evaluation, specific bias probes).
• Scalability: The complexity of parsing and mutation generation scales with prompt length and intricacy. Evaluating image pairs also adds overhead. Applying ACTesting to very large datasets of diverse and complex prompts requires efficient implementation and potentially distributed computing resources.

Conclusion

ACTesting offers a structured and automated approach to testing T2I software, addressing the critical challenge of the missing test oracle by employing metamorphic testing. By focusing on semantic consistency through guided mutations of text prompts based on entity-relationship structures, it provides a practical method for uncovering defects related to how these models interpret and render semantic content. While reliant on imperfect NLP tools and VLM-based oracles, its systematic nature and demonstrated effectiveness in revealing errors make it a valuable technique for improving the reliability and robustness of T2I systems.