GenPilot: T2I Prompt Optimization System

• GenPilot is a plug-and-play multi-agent system for test-time prompt optimization in text-to-image generation, enhancing semantic fidelity and image coherence.
• It integrates error analysis through parallel VQA and caption-based detection, followed by clustering-driven exploration and fine-grained verification.
• The system employs a memory module for iterative refinement, enabling systematic prompt improvements without requiring backbone model fine-tuning.

GenPilot is a plug-and-play multi-agent system designed for model-agnostic, interpretable, and systematic test-time prompt optimization in text-to-image (T2I) generation. It addresses critical limitations in both user-guided and automatic prompt optimization by introducing a pipeline that integrates error analysis, clustering-driven adaptive exploration, fine-grained verification, and a memory module for iterative refinement. GenPilot’s architecture enables direct manipulation of input text, enhancing the semantic fidelity and structural coherence of generated images, particularly for complex or lengthy prompts. The system is compatible with arbitrary backbone multimodal LLMs and T2I models, requires no fine-tuning of the underlying generative models, and employs natural-language feedback at all stages (Ye et al., 8 Oct 2025).

1. System Architecture and Agent Design

GenPilot is structured around two primary stages—Error Analysis and Test-Time Prompt Optimization—using lightweight “agents”, each powered by a multimodal LLM (e.g., Qwen2.5-VL-72B):

• Error Analyzer:
  • Prompt Decomposer: Segments the original prompt $P$ into coarse meta-sentences $\{s_1, \dots, s_n\}$.
  • Parallel Detectors: Employs two agents in parallel:
  • VQA-based agent generates yes/no questions about objects, attributes, and relations, labeling responses as YES/NO with explanations $(\mathrm{type}_i, \mathrm{explanation}_i)$.
  • Caption-based agent generates a caption $C$ for the image and compares it to $P$ to produce errors $(\mathrm{type}_i, \mathrm{explanation}_i)$.
  • Error-Integration agent ($A_\mathrm{error}$): Merges errors from both branches into a unified error set $E_u$.
  • Error-Mapping agent: Maps each error to its causative sub-sentence $s_i$ in $P$ (mappings $M$).
• Clustering-Based Exploration Agent:
  • Prompt-Refinement agent: For each mapped sentence $m_i$, generates $N$ diverse revisions $\{m_i^1, \dots, m_i^N\}$ based on historical feedback.
  • Branch-Merge agent: Produces $N$ candidate prompts per sentence by integrating each $m_i^j$ into $P$.
• Fine-Grained Verifier (MLLM Scorer):
  • Evaluates $(P_{ij}, I_{ij})$, runs the VQA agent for persistency of inconsistencies, and assigns a scalar score $$S(P_{ij}) = \mathrm{avg} \lVert A_\mathrm{rate}(I_{ij}, P, A_\mathrm{vqa}(I_{ij}, P)) \rVert$$.
• Memory Module:
  • Stores clusters’ selected prompts, their images, scores, and summarizations of detected errors for future iterations.

All agent operations are underpinned by a multimodal LLM and are agnostic to the particular T2I backbone employed.

2. Algorithmic Workflow

The GenPilot pipeline is divided into an error analysis phase and an iterative prompt optimization loop:

A. Error Analysis

1. Prompt Decomposition:

$P \rightarrow \{s_1, \dots, s_n\}$

1. Detection:
   • VQA Branch: Generates question set $Q_\mathrm{vqa} = \{q_1, \ldots, q_k\}$ via MLLM over relevant objects/attributes/relations. Each $q_i$ receives a YES/NO answer, forming $e_\mathrm{vqa, i}$.
   • Caption Branch: Captions the image and compares against $P$ to produce $E_c$.
2. Integration: Errors consolidated: $E_u = A_\mathrm{error}(I, P, E_c, E_\mathrm{vqa})$.
3. Mapping: Each $e \in E_u$ is assigned to the most relevant $s_i \subset P$.

B. Iterative Prompt Optimization Loop

1. Initialization: Set $P^0$ as the original prompt.
2. For $t=1\ldots T$:
   • For each mapped sentence $m_i$ in $P^{t-1}$, generate $N$ revisions $\{m_i^j\}$.
   • Construct $N$ full prompts $P_i^j$ via the Branch-Merge agent.
   • Generate associated images $I_i^j$ and compute corresponding scores $S_i^j$ using the Fine-Grained Verifier.
   • Cluster $\{P_i^j\}$ into $k$ clusters (k-means).
   • Bayesian update:
     • Prior: $P_j = 1/k$
     • Likelihood: $L_j \propto$ average scores of cluster $j$
     • Posterior: $$P_j^\mathrm{post} = (L_j \cdot P_j) / \Sigma (L\cdot P)$$
   • Select best cluster $j^* = \operatorname{argmax} P_j^\mathrm{post}$.
   • Sample $m$ prompts $s^*$ from $j^*$, generate images, score, analyze errors, and append to memory module.
   • Set $P^t = \arg\max_{s^*} S(s^*)$.
   • Terminate if no further errors or $t=T$.

3. Memory Module and Iterative Refinement

At each iteration, tuples of sampled prompts $s^*$, generated images $I_{s^*}$, scores $S(s^*)$, and error summaries are recorded. This memory store allows the prompt refinement agent to incorporate “history feedback” at every round, promoting focused exploration of unresolved issues and preventing redundant edits. The iterative design underlies GenPilot’s capacity to address compounding and previously unresolved prompt–image inconsistencies.

4. Error Taxonomy and Refinement Patterns

GenPilot’s error analysis and refinement strategy distinguish several categories:

• Quantity: Mismatches in object count.
• Attribute: Errors in color, shape, texture, etc.
• Relation/Position: Spatial or relational misplacements.
• Background/Style: Scene or stylistic discrepancies.
• Omissions/Additions: Presence or lack of required (or forbidden) elements.

Combining VQA and captioning increases error analysis accuracy from roughly 3.9 to 4.6 out of 5 (GPT-4o rating). Thirty-five prototypical error patterns, with corresponding one-line suggested refinements, are released for common issues such as “Object Fusion Errors,” “Ambient Lighting Mismatch,” and “Temporal Ambiguity.” Frequent strategies include explicit numeric constraints (“exactly six chairs”), precision in spatial descriptors (“2 cm above”), material qualifiers (“brushed titanium”), style/lighting mandates (“soft blue moonlight”), and explicit exclusions (“no benches or lighting fixtures”).

A summary of common error categories and refinement strategies:

Error Type	Example Pattern	Typical Refinement
Quantity	“incorrect number of objects”	“exactly three apples”
Spatial	“misplaced relationships”	“directly behind”, “aligned at 45°”
Attribute	“wrong color/texture/material”	“Pantone 186C”, “brushed titanium”
Style/Atmosphere	“ambient mismatch”	“steampunk”, “melancholic mood”
Added/Omitted Obj.	“object missing/unwanted addition”	“do not include any X”

Integration of multi-source error signals and explicit refinement patterns underpins the systematic prompt improvement process.

5. Experimental Evaluation and Analysis

GenPilot was evaluated primarily on the DPG-bench (challenging, long prompts) and GenEval (short, object-focused prompts).

• DPG-bench:
  • 264 prompts, baseline average score $<$0.81.
  • Stable Diffusion v1.4: 53.16 (baseline) $\rightarrow$ 62.12 with GenPilot ($+8.96$, $+16.9\%$ relative).
  • FLUX.1 schnell: 68.16 $\rightarrow$ 73.32 ($+7.6\%$).
  • DALL-E 3: 72.04 $\rightarrow$ 74.08 ($+2.8\%$).
  • GenPilot consistently outperformed approaches like Prompt Engineering, MagicPrompt, BeautifulPrompt, and naïve Test-Time Scaling across all backbone models.
• GenEval:
  • FLUX.1 schnell: 65.82% $\rightarrow$ 69.60% ($+3.78$ absolute, $+5.7\%$ relative).
  • PixArt-α: 46.73% $\rightarrow$ 48.54%.
  • Substantial gains, especially in positional accuracy ($+12$ points), color/attribute, and counting.
  • Ablation studies demonstrated performance drops without the memory module (to 66.05%) or without clustering (to 66.27%), and with alternative MLLMs/captioners (e.g., MiniCPM-V2.0 vs Qwen: 69.82 vs 73.32).

These results demonstrate GenPilot's effectiveness on both long-complex and short-atomic prompt tasks (Ye et al., 8 Oct 2025).

6. Strengths, Limitations, and Potential Implications

Strengths

• Model-Agnosticism: No fine-tuning of underlying T2I or MLLM models is required.
• Interpretability: Errors and refinements are generated in human-readable language, facilitating auditability and debugging.
• Systematic Optimization: Combines multi-source error detection, clustering-based exploration, fine-grained scoring, and memory-based iterative refinement.
• Generalizability: Supports any multimodal LLM and diffusion/transformer-based T2I system. Handles both complex and simple prompts.

Limitations

• Inference Overhead: Each optimization round can be computationally intensive; the default protocol generates up to 20 candidate prompts per iteration across 10 iterations.
• LLM Dependency: The performance is contingent on the capability of the chosen multimodal LLM agents; weaker models yield lower error detection and scoring accuracy.

A plausible implication is that improvements in MLLM performance will directly enhance GenPilot’s overall effectiveness. Furthermore, the catalog of prototypical errors and refinements provides a foundation for research in prompt engineering and could inform future developments in both automated and user-in-the-loop text-to-image pipelines.

(Ye et al., 8 Oct 2025)