Prompt Optimization (VPO)

• Prompt Optimization (VPO) is the process of algorithmically refining prompts to bridge gaps between under-specified user queries and the detailed captions used in training generative models.
• It integrates supervised fine-tuning and direct preference learning to enforce alignment principles—harmlessness, accuracy, and helpfulness—boosting metrics like 95% text alignment and +20 pp safety improvements.
• Empirical benchmarks demonstrate VPO’s effectiveness with superior win rates and enhanced video quality compared to baseline models using memory-augmented, multimodal process feedback.

Prompt Optimization (VPO)

Prompt optimization is the process of algorithmically refining, generating, or searching for prompts that elicit optimal outputs from machine learning models, particularly LLMs and diffusion-based generative models for modalities including images and videos. In the context of text-to-video generation, Video Prompt Optimization (VPO) refers specifically to learned or automated approaches that bridge the gap between terse, ambiguous, or unsafe user queries and the highly structured, safe, and richly annotated captions that state-of-the-art video diffusion models are trained on. Recent advances in VPO integrate principles from alignment research, preference learning, multi-modal process supervision, and evolutionary search, yielding robust, generalizable, and interpretable prompt refiners in both supervised and reinforcement learning settings (Cheng et al., 26 Mar 2025).

1. Motivation: The Training–Inference Gap and Alignment Challenges

State-of-the-art text-to-video diffusion models are generally trained on datasets comprising hundreds of thousands of video–caption pairs, where the textual descriptions are meticulously detailed, safe, and structurally rich. By contrast, typical user interactions at inference time involve highly condensed, under-specified, or poorly controlled queries—e.g., "beach video"—that diverge significantly from the data distribution observed in training. This mismatch induces a manifold of generation failures, from misalignment and low visual fidelity to the propagation of unsafe or undesired content (Cheng et al., 26 Mar 2025).

Naïve prompt-refinement with large LLMs (in-context learning, few-shot LLM rewriters) is insufficient. Such approaches often drift from user intent, hallucinate harmful details, or optimize for surface-level text attributes that do not correlate with downstream video quality. Optimization of prompts for LLM fluency or coverage decoupled from video-level reward is thus ineffective.

2. Foundational Principles and Taxonomy of VPO Approaches

Modern VPO frameworks are designed around three core alignment principles (Cheng et al., 26 Mar 2025):

• Harmlessness: Refined prompts must exclude or rewrite any textual content that could induce the generative model to emit unsafe, violent, sexual or otherwise prohibited video outputs.
• Accuracy: The output prompt must be strictly faithful to all user-specified details (e.g., the requested color, object, scene, or action).
• Helpfulness: The optimized prompt should inject sufficient visual and cinematic details (camera motion, lighting, action verbs) to maximize downstream video fidelity—particularly in diffusion-based models hungry for such cues.

State-of-the-art VPO solutions use a two-stage architecture:

Stage	Description	Example Methods
SFT	Supervised fine-tuning on a curated x→prompt dataset, enforcing the three principles	(Cheng et al., 26 Mar 2025, Zhu et al., 15 May 2025)
DPO/PL	Text- and video-level preference learning, guiding selection by human/LLM/judge scores	Direct Preference Optimization

These stages yield models that map user queries to safe, aligned, and richly detailed prompts, outperforming simple few-shot LLM refiners both in safety and video quality.

3. Mathematical and Algorithmic Formulation

The VPO process is formally structured as follows:

1. Supervised Fine-Tuning (SFT): Collect a large set (∼20K) of real user queries (x). Use a strong LLM to generate initial rewritten prompts (p), then adjudicate and correct them via an LLM-as-judge to yield target prompts (s). Fine-tune a prompt-rewriting model πθ on maximum-likelihood of s given x:

$$\mathcal{L}_\mathrm{SFT}(\theta) = -\mathbb{E}_{(x, s)} \left[\sum_{i=1}^{|s|} \log P_\theta(s_i|x, s_{<i})\right]$$

1. Preference Learning: Sample K candidate prompts for each x from πθ. Gather two types of pairwise preferences:
   • Text-level: For pairs where p* is strictly safer or better aligned than p, record $(x, p \prec p^*)$.
   • Video-level: Generate videos with backbone (e.g., CogVideoX), score them with a vision reward model, and record preference pairs when one video is superior.
2. Direct Preference Optimization (DPO): Optimize θ so that preferred prompts have higher likelihood than less-preferred, regularized towards a frozen reference model:

$$\mathcal{L}_\mathrm{DPO}(\theta) = -\mathbb{E}_{(x, p_w, p_\ell)} \left[ \log \sigma \left( \beta \left[ \log \pi_\theta (p_w|x) - \log \pi_{\mathrm{ref}}(p_w|x) \right] - \beta \left[ \log \pi_\theta (p_\ell|x) - \log \pi_{\mathrm{ref}}(p_\ell|x)\right] \right) \right]$$

Experiments use an LLaMA3-8B-Instruct backbone fine-tuned with DeepSpeed ZeRO-3. Query encoding includes standardized instruction templates. For video feedback, an off-the-shelf model such as VisionReward provides a multi-dimensional score for generated samples (Cheng et al., 26 Mar 2025).

4. Multimodal and Process-Level Extension

The principle of process-level feedback is critical for the multimodal VPO setting, where token inflation and insufficient feedback per optimization step can otherwise limit effectiveness. Extensions such as UniAPO generalize VPO concepts across text, image, and video, introducing an EM-inspired framework that decouples the computation of corrective feedback from prompt refinement (Zhu et al., 25 Aug 2025). In this architecture:

• Latent feedback variables aggregate both current and historical data (memories) for more stable supervision.
• Short–long term memory mechanisms ensure that context window constraints do not lead to catastrophic forgetting of previously identified failure patterns.
• Modality-agnostic process supervision: Feedback and prompt memories are maintained and retrieved identically for text, image, and video tasks, ensuring transferability.

UniAPO converges in 4–6 iterations and achieves consistent F1 improvements (+7–15 points depending on modality/task) by leveraging memory-augmented feedback and directional guidance (Zhu et al., 25 Aug 2025).

5. Empirical Validation and Quantitative Benchmarks

VPO’s impact is demonstrated across a suite of standardized benchmarks:

Metric/Benchmark	VPO	Best Baselines (GLM-4, GPT-4o)
MonetBench (video)	4.15 Overall	3.98 / 4.03
Text Alignment (%)	95	86
Safety Level 1	+20 pp vs. LLM FS	-
Win rate (human eval)	37.5% over query	14% over GLM-4

VPO outperforms Diffusion–DPO (direct RL on the video model) by 5–10% pairwise video preference and demonstrates strong generalization (+2–3% on Open-Sora 1.2 using a prompt optimizer trained on CogVideoX-2B) (Cheng et al., 26 Mar 2025).

Ablations and comparisons confirm that VPO’s prompt-level approach and video-level preference learning are both necessary—text-only feedback is insufficient, and RLHF on the diffusion model alone does not close the quality or safety gap.

6. Generalization, Compositionality, and Orthogonality to RLHF

VPO prompt optimizers trained and validated on one video model often yield improvements on other backbones, indicating a degree of model-agnosticity. Combining VPO on the prompt side with RLHF (Diffusion–DPO) on the model side yields strictly additive benefits, empirically demonstrating the orthogonality of these two axes of alignment (Cheng et al., 26 Mar 2025).

Similar multi-modal extension strategies have been validated in frameworks such as UniAPO, demonstrating consistency of improvement across text, image, and video, with modality-agnostic process-level supervision and strategic memory management yielding up to +14 F1 for video keyword extraction benchmarks (Zhu et al., 25 Aug 2025).

7. Limitations, Open Problems, and Future Directions

Current VPO frameworks leverage automated video scorers (e.g., VisionReward) for downstream feedback, but integrating human-in-the-loop video preference remains an open avenue for further boosting alignment and safety. Extending VPO to multi-modal queries (e.g., combining text, images, and sketches), real-time interactive refinement, and dynamic, long-duration video tasks pose practical challenges not yet fully solved.

Future advances may align structural and evolutionary search (e.g., multi-branch prompt architectures), symbolic reasoning, and data-driven curriculum learning within the VPO process. The core insight—tight alignment between user intent, task-specific detail injection, and downstream multimodal reward—underpins a generalizable blueprint for cross-modal, robust, and safe generative modeling (Cheng et al., 26 Mar 2025, Zhu et al., 25 Aug 2025).