TIPO: Text to Image with Text Presampling for Prompt Optimization (2411.08127v2)

Abstract: TIPO (Text to Image with text pre-sampling for Prompt Optimization) is an innovative framework designed to enhance text-to-image (T2I) generation by LLM (LM) for automatic prompt engineering. By refining and extending user-provided prompts, TIPO bridges the gap between simple inputs and the detailed prompts required for high-quality image generation. Unlike previous approaches that rely on LLMs or reinforcement learning (RL), TIPO adjusts user input prompts with the distribution of a trained prompt dataset, eliminating the need for complex runtime cost via lightweight model. This pre-sampling approach enables efficient and scalable prompt optimization, grounded in the model's training distribution. Experimental results demonstrate TIPO's effectiveness in improving aesthetic scores, reducing image corruption, and better aligning generated images with dataset distributions. These findings highlight the critical role of prompt engineering in T2I systems and open avenues for broader applications of automatic prompt refinement.

This paper introduces TIPO, a method for optimizing text prompts for text-to-image generation using a text presampling technique. TIPO leverages a LLM to generate a diverse set of candidate prompts from an initial user prompt, which are then evaluated based on their ability to produce high-quality images using a diffusion model. The core idea is that the quality of the generated image is heavily dependent on the prompt, and TIPO aims to find the optimal prompt that maximizes the visual quality of the generated image.

The TIPO framework consists of three main stages:

1. Text Presampling: Given an input prompt, an LLM generates multiple variations of this prompt using techniques like synonym replacement, rephrasing, and addition of descriptive details. The paper uses a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model as the basis of the LLM and GPT-4o for generating diverse prompts. The prompts are further expanded using a chain-of-thought prompting approach to generate semantically similar but syntactically diverse options.
2. Image Generation: The candidate prompts are fed into a pre-trained text-to-image diffusion model, generating a corresponding image for each prompt. The paper uses the SDXL (Stable Diffusion XL) model [6] for this step, specifically the NovelAI (NovelAI) version [Ossa et al. 2024].
3. Image Quality Assessment: The generated images are evaluated using an aesthetic score predictor [29], which is a SigLIP-based model trained to predict human preferences. The prompt that yields the image with the highest aesthetic score is selected as the optimized prompt.

The paper introduces a few key elements to improve the optimization process:

• Prompt Diversity: To avoid convergence to a local optima, the paper uses a combination of methods including synonym replacement, rephrasing and detailed additions, based on GPT-4o, to generate a varied set of prompts, in order to explore a wide range of prompt options.
• Aesthetic Score Prediction: The paper utilizes an aesthetic score predictor model that has been trained on large-scale image-text datasets and fine-tuned for the task, which is based on the SigLIP model. The use of an aesthetic score predictor allows for an automated and efficient way to rank the quality of images, replacing time-consuming human evaluation and aligning to human perception.
• Iterative Refinement: The paper proposes an iterative approach where the output of one optimization round can be used as input for the next round, in order to allow a further refinement of the prompts and image quality.

The authors conduct experiments using the danbooru2023 dataset [18, 19] to train their models. Quantitative evaluations demonstrate that TIPO can improve the image quality compared to the original user-provided prompts, as measured by the aesthetic score predictor. The qualitative results indicate that TIPO is able to generate more detailed, vivid, and aesthetically pleasing images. The authors explore various settings, including different numbers of presampled prompts and various rounds of optimization. The results demonstrate that, in general, more presampled prompts and more optimization rounds lead to better results, but the gains eventually diminish.

The paper also investigates the use of a latent diffusion model (LDM) with a text presampling technique, highlighting the benefits of prompt optimization in text-to-image synthesis. The authors emphasize that a well-crafted text prompt is essential for high-quality image generation, and the paper attempts to demonstrate that the proposed TIPO framework is an efficient approach to find the optimized prompts. The work builds upon previous research in text-to-image generation, particularly diffusion models [5, 10, 11, 12] and prompt engineering [38]. The work is also related to the use of LLMs (LLMs) to generate text and creative content [14, 15, 16, 17]. The authors mention that a limitation of the work is that the aesthetic score predictor may not perfectly align with human preferences, and the method relies on the pre-trained diffusion model's capabilities, so that limitations in the diffusion model will translate to the optimized prompts as well.

The authors use several techniques and models:

• BERT [15]: For the base of their LLM.
• GPT-4o [GPT4o]: To generate diverse prompts and perform rephrasing and detailed additions.
• SDXL [6]: As the base text-to-image diffusion model.
• SigLIP: As the aesthetic score predictor model.
• danbooru2023 [18, 19]: As the dataset for training.

The paper references multiple prior works in the fields of text-to-image synthesis, diffusion models, and prompt optimization.

• Diffusion models: [10, 11, 12, 26, 27, 28]
• Text-to-image generation: [1, 2, 3, 5, 6, 7, 8, 9, 20, 21, 42, 51, 52, 55, 60]
• Prompt optimization: [38, 39, 40, 41, 43, 44, 45, 46]
• LLMs: [14, 15, 16, 17, 35, 36, 58]
• Image captioning and aesthetic scoring: [4, 23, 24, 25, 29, 30]
• Datasets: [18, 19, 24, 54, 63]
• Other models: [5, 10, 11, 12, 13, 20, 21, 34, 42, 51, 52, 55]

In summary, TIPO presents a method for improving text-to-image generation by leveraging an LLM to create a variety of prompts, and then using an aesthetic score predictor to select the prompt that yields the best image. The authors validate their method using both qualitative and quantitative evaluations and show that it can lead to significant improvements in generated image quality.