GenPilot: Multi-Agent Prompt Optimization

• GenPilot is a multi-agent system that iteratively refines user prompts to correct semantic inconsistencies in text-to-image synthesis.
• Its structured pipeline incorporates error analysis, clustering-based exploration, fine-grained verification, and a memory module to ensure systematic improvements.
• Experimental results demonstrate notable alignment score gains across models, validating the effective integration of diverse agents in prompt optimization.

GenPilot is a multi-agent, test-time prompt optimization system designed to address the challenge of semantic alignment in complex text-to-image (T2I) synthesis. It operates directly on user prompts, iteratively refining them to minimize semantic inconsistencies between input descriptions and output images. GenPilot is model-agnostic, requires no additional training, and is structured to provide interpretable, systematic improvements in image-text consistency and structural coherence without modifying or fine-tuning the underlying generative model (Ye et al., 8 Oct 2025).

1. System Architecture and Operational Workflow

GenPilot is organized as a plug-and-play pipeline comprising four specialized agents: an error analysis agent, a clustering-based adaptive exploration agent, a fine-grained verification (scoring) agent, and a stateful memory module. The high-level workflow is divided into two main stages:

Stage 1 (Error Analysis):

1. Decomposition of the initial prompt $P$ into atomic meta-sentences $\{s_1,\dots,s_n\}$.
2. Generation of comprehensive coverage questions $Q$ (on objects, attributes, relations, background).
3. Application of VQA agents and caption-based analysis to extract error sets $\mathcal{E}_{vqa}$ and $\mathcal{E}_c$.
4. Integration via an error-integration agent $A_{\rm error}$, yielding a unified error list $\mathcal{E}_u$.
5. Mapping of errors to specific prompt fragments $M = \{m_i\}$.

Stage 2 (Test-Time Prompt Optimization):

1. Candidate refinements $\{m_i^1,\dots,m_i^N\}$ are generated for each mapped fragment by a refinement agent $A_{\rm refine}$.
2. A branch-merge agent $\{s_1,\dots,s_n\}$0 merges refinements into candidate prompts $\{s_1,\dots,s_n\}$1.
3. Each prompt is rendered by the base T2I model.
4. An MLLM-based scorer $\{s_1,\dots,s_n\}$2 assigns scores $\{s_1,\dots,s_n\}$3 for prompt-image pairs.
5. Prompts are embedded and clustered via $\{s_1,\dots,s_n\}$4-means with a Bayesian update, selecting top clusters.
6. The most promising candidates are sampled, rescored, and memory $\{s_1,\dots,s_n\}$5 is updated.
7. The process is repeated to convergence or for a fixed number of iterations.

This modular design allows GenPilot to systematically isolate, revise, and validate the effects of prompt changes.

2. Error Analysis and Error Integration

The error analysis module is central to GenPilot’s error localization and correction mechanism. It comprises several steps:

• Prompt Decomposition: Each initial prompt $\{s_1,\dots,s_n\}$6 is divided into meta-sentences, each focusing on distinct objects, attributes, relations, or background constraints.
• Question Generation: An MLLM agent constructs targeted coverage questions $\{s_1,\dots,s_n\}$7, verifying existence, attribute correctness, spatial relations, and background alignment.
• VQA-Based Analysis: Visual question answering is applied; errors are represented as tuples $\{s_1,\dots,s_n\}$8 collected into $\{s_1,\dots,s_n\}$9.
• Caption-Based Analysis: A captioner produces $Q$0, and semantic mismatches are detected by comparing $Q$1 with $Q$2 to yield $Q$3.
• Error Integration and Mapping: An error-integration agent fuses $Q$4 and $Q$5 into $Q$6, mapping each error to its responsible prompt fragment $Q$7.

This structured error analysis ensures that subsequent refinement focuses only on demonstrably problematic prompt components.

3. Clustering-Based Adaptive Exploration

To maximize the efficiency and coverage of prompt refinement, GenPilot employs a clustering-based adaptive exploration strategy:

• Embedding: Each candidate prompt is embedded into $Q$8 as $Q$9 via a text encoder.
• Clustering: $\mathcal{E}_{vqa}$0-means clustering groups the $\mathcal{E}_{vqa}$1 candidate embeddings into $\mathcal{E}_{vqa}$2 clusters $\mathcal{E}_{vqa}$3 by minimizing intra-cluster distances.
• Bayesian Cluster Selection: Cluster priors $\mathcal{E}_{vqa}$4 are initialized uniformly. The likelihood $\mathcal{E}_{vqa}$5 of each cluster is computed as the average candidate score; Bayes’ theorem updates posteriors $\mathcal{E}_{vqa}$6 with the best cluster $\mathcal{E}_{vqa}$7 selected as $\mathcal{E}_{vqa}$8.
• Centroid-Based Prompt Steering: Subsequent candidate prompts are sampled by nudging the prompt embedding toward the selected centroid $\mathcal{E}_{vqa}$9:

$\mathcal{E}_c$0

The refinement agent decodes these steered embeddings back to text.

This iterative exploratory mechanism concentrates resources in semantically promising regions while preserving diversity.

4. Fine-Grained Verification and Scoring

Image-prompt pairs are extensively validated using an MLLM-based scoring framework along three axes—attribute binding, relations, and background/style.

• Scoring Function: For each pair $\mathcal{E}_c$1, VQA sub-questions provide granular feedback, with the score

$\mathcal{E}_c$2

where $\mathcal{E}_c$3 issues discrete ratings (e.g., from 1 to 5).

• Semantic Consistency: Optional cosine similarity between text/image features ($\mathcal{E}_c$4) is discussed, but the primary metric remains the MLLM-based score.

This approach enables detailed, interpretable, and targeted update signals for prompt refinement at each stage.

5. Memory Module and Iterative Optimization

GenPilot maintains a persistent memory module $\mathcal{E}_c$5 that records all explored prompt-image pairs, their scores, and associated error sets:

• After each round, $\mathcal{E}_c$6 is updated by storing sampled prompts, images, scores, and errors to track the trajectory of improvement.
• The iterative cycle follows a fixed or convergence-based schedule:
  1. Error analysis.
  2. Candidate generation.
  3. Candidate scoring.
  4. Clustering and sampling.
  5. Memory update.
  6. Prompt update.
  7. Termination if no errors remain.

• The best current prompt replaces the previous one for the next iteration, ensuring monotonic error reduction.

The memory architecture supports both efficiency (by avoiding redundant exploration) and interpretability through replay and auditability of system choices.

6. Error Taxonomy and Refinement Tactics

Through analysis of 35 frequent error types, GenPilot identifies prevalent failure modes in T2I generation and associated prompt engineering tactics:

Error Category	Common Issue	Prompt Refinement Example
Quantity (counting)	Missing/extra objects	Specify “exactly $\mathcal{E}_c$7”, repeat
Relation (spatial)	Incorrect object locations	Use precise phrases (“directly behind/in front of”)
Attribute	Mismatched color, texture, shape	Employ Pantone codes, texture-specific terms
Composition & background	Omitted elements, clutter	Enumerate required elements, exclude unwanted items
Lighting & shadow	Inconsistent or implausible lighting	Define light source, color, and intensity
Style/material	Style or material mismatch	Add style/material descriptors (“steampunk”, “brushed steel”)
Other	Opacity, reflection, seasonal cues, etc.	As required per failure mode

This taxonomy systematizes prompt editing strategies aligned with recurrent T2I error cases.

7. Experimental Results and System Performance

GenPilot’s empirical validation spans two standardized datasets—DPG-bench (264 challenging prompts) and GenEval (short, object-focused prompts in seven categories). The main findings are:

• Model Range: Improvements are consistent across DALL-E 3, Stable Diffusion (v1.4/2.1/3), FLUX.1 schnell, PixArt-α, and Sana-1.0.
• Quantitative Gains: Relative increases in average alignment scores range from +1.9% (Sana-1.0) up to +16.9% (Stable Diffusion v1.4).
• Ablation Study: Performance degrades when memory or clustering components are ablated (–3.55 and –3.33 points respectively on GenEval), substantiating their importance.
• Component Impact: The contribution of each system agent is quantified:

$\mathcal{E}_c$8

for each component $\mathcal{E}_c$9.

These results illustrate that systematic, agent-based prompt optimization at test time can close the semantic gap in T2I generation even for difficult prompts and across diverse architectures.

8. Interpretability, Limitations, and Extension Directions

GenPilot yields interpretability by design, as each refinement is directly traceable to a specific identified error and annotated prompt fragment. Its model-agnostic, test-time-only operation ensures broad compatibility with existing and future T2I architectures.

Emerging directions include:

• Integration of learned error predictors to supplement or supersede MLLM-based agents.
• Exploration of gradient-guided search in prompt embedding space for efficiency gains.
• Development of human-in-the-loop procedures to balance machine and expert intuition.
• Expansion toward multi-stage prompt editing and interactive scene specification.

This suggests that further hybridization of systematic, multi-agent prompt optimization and adaptive search could yield tractable solutions for increasingly complex, open-ended T2I scenarios (Ye et al., 8 Oct 2025).