One Rewrite to Fix Them All? Type-Aware Repair Allocation for Text-to-Image Prompt Optimization

Abstract: Text-to-image (T2I) generators often fail to follow their prompts faithfully, producing wrong counts, swapped attributes, ambiguous relations, and illegible text. Prompt optimization repairs such failures by rewriting the user prompt, requiring no generator retraining, and has yielded promising results. However, existing optimizers absorb heterogeneous failures into one uniform prompt expansion, even though each calls for different repair language. We formulate semantic prompt optimization as atomic repair allocation: each failed proposition is routed to a type-conditioned repair operator before the resulting local constraints are compiled into one executable prompt. We instantiate this formulation in the training-free Type-Aware Repair Allocation (TARA) framework, which separates diagnosis, allocation, compilation, and a semantic repair gate, an accept-or-revert controller over exactly one prescribed repair that prevents semantic regressions. Extensive experiments on DSG and TIFA across four frozen generators demonstrate that TARA achieves the best semantic accuracy in all eight benchmark-generator cells, improving over VisualPrompter by 5.6 and 2.6 points on DSG and TIFA, respectively, while maintaining image quality and running fastest in our matched local setting at 16.0 seconds versus 20.0 seconds per prompt.

• The paper introduces TARA, a training-free framework that diagnoses atomic prompt failures, assigns type-specific repairs, and accepts a rewrite only when it improves the semantic score.
• TARA outperforms five baselines across eight benchmark–generator settings, improving over VisualPrompter by 5.6 points on DSG and 2.6 points on TIFA while preserving image quality and aesthetics.
• The method requires at most two image generations, runs 4 seconds faster than VisualPrompter at a matched budget, and shows that typed repair language, target recovery, and regression gating jointly drive performance.

Motivation and problem formulation

Text-to-image (T2I) generators frequently violate fine-grained prompt constraints—dropping objects, miscounting instances, swapping attributes, scrambling spatial relations, or rendering illegible text. Prompt optimization addresses these failures by rewriting the user prompt without retraining the generator. The paper's central observation is that existing optimizers, including visual-feedback methods such as VisualPrompter, collapse heterogeneous failure modes into a single uniform prompt expansion: whatever the diagnosis, the same type-agnostic enrichment rule is applied. The authors argue that different failures demand different repair language—an explicit cardinality for counts, unambiguous spatial anchors for relations, an exact quoted string for rendered text—and that uniform expansion is merely a degenerate allocation policy with one type-invariant operator.

The paper formulates semantic prompt optimization as atomic repair allocation. A frozen generator $G$ renders the user prompt $p$; following DSG, $p$ is decomposed into atomic propositions $Q(p)$, each verified by a VLM judge to yield a semantic score $s(x)$. Failed propositions are routed by an allocator $\mathcal{A}$ to type-conditioned repair operators, and a compiler $\mathcal{C}$ fuses the resulting local constraints into one executable prompt $\hat{p}$. Uniform expansion corresponds to the special case where $\mathcal{A}$ ignores the error type $t$. Crucially, atomic feedback is used not only to decide what failed but how each failure should be repaired.

The TARA framework

TARA (Type-Aware Repair Allocation) instantiates this formulation as a training-free, four-stage pipeline requiring at most two image generations per prompt:

• Diagnose: DSG-style decomposition assigns each proposition one of three states (correct, absent, wrong) and one of eight semantic categories, mapped through a fixed category-to-type table to eight repairable error types.
• Recover repair targets: because a missing entity prunes its dependent relation/action questions in the DSG dependency graph, TARA re-adds pruned but wrong or absent relation/action propositions so a single rewrite can re-describe both the object and its layout. This affects only repair targets, not the comparable score.
• Allocate and compile: a text-only portfolio of three rewrite drafts—uniform expansion (an in-method control mirroring prior optimizers), typed expansion annotated with per-type repair phrases, and minimal append—is fused by an LLM under instructions to preserve correct semantics, fix all failures, avoid unrelated objects, and stay concise. When the initial image is near-correct ($p$0), a re-seed candidate retaining the original prompt is admitted instead. No candidate is ever rendered; the portfolio adds zero image generations.
• Adopt via a semantic repair gate: exactly one repaired image is generated and adopted iff $p$1, otherwise the original output is kept. The gate is explicitly not a best-of-$p$2 selector—it performs no image search—and its cost is included in runtime accounting.

The threshold $p$3 is fixed globally; a sweep shows all values in $p$4 yield large gains ($p$5 to $p$6 points on DSG/SD 2.1) at a comparable image budget of roughly 1.8 generations.

Experimental results

Evaluation covers DSG-1k and TIFA v1.0 across four frozen generators (SD v1.5, SD v2.1, Flux-dev, Janus-Pro), three seeds, and 200 stratified prompts per seed group, against five baselines including VisualPrompter. All methods share prompts, seeds, judge, and protocol; the VLM judge never sees optimized prompt text.

Semantic accuracy. TARA achieves the best semantic accuracy in all eight benchmark–generator cells, improving over VisualPrompter by +5.6 points on DSG (76.4 vs. 70.8) and +2.6 on TIFA (85.4 vs. 82.8), with prompt-clustered paired bootstrap 95% CIs of [4.7, 6.5] and [1.6, 3.6]. Across 4,800 paired evaluations the mean gain is +4.11 points (CI [3.45, 4.80]). Re-scoring with four held-out VLM evaluators confirms the ranking (+2.4 to +3.2 overall). Gains are largest on sources demanding explicit counts, relations, poses, and text (e.g., CountBench, PoseScript, VRD, DrawText). Notably, aesthetics-oriented optimizers such as BeautifulPrompt fall far below even the raw-prompt baseline, indicating generic enrichment can actively harm semantics.

Image quality. TARA attains the highest mean CLIP score (31.86 vs. 31.45 for VisualPrompter) while staying within 0.02 of the raw prompt in aesthetic score—a contrast with baselines like NeuroPrompts that trade semantics for aesthetics.

Per-type consistency. Aggregated over 24 cells, TARA improves over the raw prompt on all eight semantic categories and over every baseline on seven, essentially matching VisualPrompter on action. Largest gains appear on style (+4.3), attribute (+4.2), entity (+4.1), and relation (+3.7). Human evaluation on 100 cases prefers TARA over originals in 62% (SD 1.5) and 58% (Flux-dev) for semantic consistency.

Efficiency and ablations. At a matched image budget of 1.64 generations per prompt, TARA averages 16.0 s versus 20.0 s for VisualPrompter, aided by shorter compiled prompts (22.7 vs. 41.6 words when repaired; 18.2 vs. 30.3 overall). A full-scale ablation (2,400 evaluations per variant) shows removing the typed-expansion branch costs −3.8 points, the repair gate −3.6, the portfolio −2.2, and recovery −1.3, indicating gains arise jointly from typed repair language and regression-controlled adoption rather than from either alone.

Limitations and open questions

Several caveats are stated plainly. Per-type adoption rates range from 22.5% (missing text) to 29.7% (missing objects), so most attempted repairs do not improve the score; residual failures stem largely from generator-side limits—multi-object counting, rare poses, long rendered strings—that no prompt can force. Accept rates vary strongly by generator (rendered text is nearly unrepairable on SD but substantially easier on Flux/Janus), showing part of the ceiling reflects generator capacity rather than optimizer quality. Two per-source cells show slight regressions relative to VisualPrompter (DiffusionDB on Flux-dev, −1.3; PartiPrompt and PaintSkill on Janus-Pro, −0.2 and −1.0), so the "best in every cell" claim holds at the aggregate benchmark–generator level, not uniformly across source categories. The allocator uses fixed hand-written templates rather than a learned router, and the main judge shares a model family with the diagnostic endpoint, though held-out evaluator re-scoring mitigates this concern. Open questions include whether learned allocation policies could outperform the fixed template mapping, and how repair language should adapt to generators with different capacity profiles.

Conclusion

The paper reframes visual-feedback prompt optimization as atomic repair allocation, in which diagnosed failure types serve as routing variables selecting type-conditioned repairs that are compiled into one concise prompt under a single-regeneration budget, with an accept-or-revert gate guarding against regressions. Empirically, TARA leads all eight benchmark–generator cells, preserves CLIP alignment and aesthetics, and runs fastest among compared optimizers, supporting the claim that atomic feedback should determine not only what is rewritten but how each failure is repaired.