---
title: 'RAPO++: Cross-Stage Prompt Optimization'
url: https://www.emergentmind.com/topics/rapo
type: topic
---

# RAPO++: Cross-Stage Prompt Optimization

Searching arXiv for papers explicitly using the term “RAPO++” and its precursor RAPO in text-to-video generation.
RAPO++ is a **cross-stage prompt optimization framework** for **text-to-video (T2V) generation** that unifies **training-data--aligned refinement**, **test-time iterative scaling**, and **large language model (LLM) fine-tuning** to improve video generation **without modifying the underlying generative backbone**. In the formal usage established by "RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scaling" [2510.20206], the term refers specifically to a three-stage system built on top of the earlier **RAPO** framework from "The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation" [2504.11739]. The framework is motivated by the observation that user prompts are often **short, unstructured, and misaligned with training data**, whereas effective prompts for diffusion-based T2V models tend to be more aligned with the prompt distribution seen during training [2510.20206].

## 1. Terminological scope and relation to earlier RAPO variants

RAPO++ is a formal method name only in the context of T2V prompt optimization. This is an important distinction because multiple contemporaneous papers use the acronym **RAPO** for unrelated reinforcement-learning or policy-optimization methods, including **Rank-Aware Policy Optimization**, **Relative-Absolute Policy Optimization**, **Reward-Aware Policy Optimization**, **Retrieval-Augmented Policy Optimization**, **Role-Aware Policy Optimization**, **Reflection-aware Adaptive Policy Optimization**, **Reflection-anchor Policy Optimization**, **Robust Adversarial Policy Optimization**, and **Regret-Aware Policy Optimization** in other problem domains [2509.06278], [2509.21871], [2510.03865], [2603.03078], [2606.27025], [2601.01718], [2605.09614], [2604.10974], [2604.07428]. By contrast, the “++” designation belongs to the prompt-optimization line introduced for T2V generation [2510.20206].

Within that line, **RAPO** denotes **Retrieval-Augmented Prompt Optimization**, while **RAPO++** extends it by adding **Sample-Specific Prompt Optimization (SSPO)** and **LLM fine-tuning on optimized prompt pairs** [2504.11739], [2510.20206]. The relation is explicit in the paper’s staging: **Stage 1** is the original RAPO-style training-data-aligned prompt optimization, and **Stages 2 and 3** constitute the “++” extension [2510.20206].

A common misconception is to interpret RAPO++ as a modification of a reinforcement-learning objective. In the T2V literature represented here, this is incorrect. RAPO++ is a **prompt optimization framework**, not a PPO-like or GRPO-like optimizer [2510.20206].

## 2. Motivation: prompt–distribution mismatch in text-to-video generation

RAPO++ is built on the premise that diffusion-based T2V models are highly sensitive to prompt formulation. The papers argue that natural user prompts are often **short**, **underspecified**, and structurally unlike the captions used to train T2V systems, whereas training prompts are typically longer, more descriptive, and more regular in structure [2504.11739], [2510.20206]. This mismatch can limit **semantic alignment**, **compositional reasoning**, **temporal stability**, and **physical plausibility** [2510.20206].

The problem is presented as more acute in T2V than in text-to-image because video generation depends not only on static object and scene fidelity but also on **temporal consistency**, **motion realism**, **object persistence**, and **interaction structure** [2510.20206]. The earlier RAPO paper also stresses that simply making prompts longer is not enough: manually adding random descriptions or relying on unconstrained LLM rewriting can introduce inaccurate or ambiguous detail and can drift away from the training prompt distribution [2504.11739].

RAPO++ addresses this by combining three distinct forms of optimization. First, it aligns prompts with the training distribution through retrieval and refactoring. Second, it performs closed-loop refinement using generated-video feedback. Third, it distills these optimized prompt transformations into a fine-tuned rewriter LLM [2510.20206]. This suggests a layered view of prompt optimization: corpus-level alignment, sample-level correction, and reusable learned rewriting behavior.

## 3. Stage 1: Retrieval-Augmented Prompt Optimization

Stage 1 of RAPO++ is the original **RAPO** module, which is designed to solve the prompt–distribution mismatch problem by retrieving semantically relevant modifiers from a relation graph built from T2V training captions and then refactoring the enriched prompt into a form closer to the training distribution [2504.11739], [2510.20206].

The Stage 1 pipeline contains three modules: **Word Augmentation Module**, **Sentence Refactoring Module**, and **Prompt Selection Module** [2510.20206]. Its retrieval substrate is a **relation graph** \(\mathcal{G}\) constructed from **Vimeo25M**, a dataset of **25 million text-video pairs**, with about **2.1M valid sentences** filtered for graph construction [2510.20206]. Each **scene** is treated as a **core node**, and its associated **subject**, **action**, and **atmosphere** descriptions are connected as **sub-nodes** [2510.20206]. Retrieval uses a **sentence transformer pre-trained model**, specifically **all-MiniLM-L6-v2**, with **cosine similarity** [2510.20206].

Given a user prompt \(x_i\), the system retrieves top-\(k\) relevant scenes, gathers connected modifiers, and selects the **top-\(k\) relevant modifiers** \(\{p_n|_{n=0}^{k-1}\}\) [2510.20206]. These modifiers are merged iteratively into the prompt using an LLM-based merge function:
\[
x^{m+1}_{i} = f(x^{m}_{i},p^m_i),
\]
where \(m=0,1,\ldots,k-1\) [2504.11739], [2510.20206]. The iterative structure is intended to preserve the original prompt semantics while gradually introducing relevant detail.

The enriched prompt is then passed to a fine-tuned **Sentence Refactoring Module** using \(L_r\), trained on approximately **86k prompt pairs** with **LoRA on LLaMA 3.1**, **8 epochs**, **batch size 32**, **LoRA rank 64**, and **single A100** [2510.20206]. The target of refactoring is not arbitrary elaboration but alignment with the training prompt distribution in terms of format and length while maintaining “original subject descriptions, actions, and scene descriptions” and appending “additional straightforward actions” only if necessary [2510.20206].

The final component is a **Prompt Selection Module** that compares the refactored RAPO candidate \(x_r\) against a naively rewritten LLM candidate \(x_n\). A discriminator \(L_d\), also fine-tuned with **LoRA on LLaMA 3.1**, **3 epochs**, **batch size 32**, **LoRA rank 64**, and **single A100**, selects the better prompt [2510.20206]. The discriminator is trained using triples \((x_i,x_r,x_n,y_d)\), where labels are derived by generating videos with both prompts and evaluating them on task-appropriate metrics [2510.20206].

This first stage already embodies the central data-alignment hypothesis. The papers report that RAPO’s optimized prompt-length distribution is **closest to the training set**, whereas user prompts are too short and other methods often generate overly long prompts with excessive detail and complex vocabulary [2504.11739], [2510.20206].

## 4. Stage 2: Sample-Specific Prompt Optimization

The “++” extension begins with **Sample-Specific Prompt Optimization (SSPO)**, which introduces **test-time scaling via iterative prompt rewriting** [2510.20206]. Unlike Stage 1, which is generic and corpus-driven, SSPO is explicitly **sample-specific**: it uses the generated video for a particular prompt to drive prompt refinement for that same instance [2510.20206].

SSPO starts from the Stage 1 refined prompt \(\rho_i^r\), generates a video \(v_i\), and evaluates it using multiple feedback sources. The first is a semantic misalignment signal \(\mathcal{M}(\rho_i^u,v_i)\), where \(\rho_i^u\) is the original user prompt. In implementation, this uses **LLaVA-OneVision** [2510.20206]. The second is a multi-verifier score:
\[
\mathcal{S}(v_{i}) = \frac{1}{K}\sum_{k=1}^{K}s_{k},
\]
where the verifiers assess dimensions such as **spatial fidelity**, **temporal coherence**, and **semantic alignment** [2510.20206]. For physical-aware video generation, SSPO can also add an optical-flow branch \(\mathcal{O}(v_i)\) to assess **flow consistency**, **object trajectories**, **motion dynamics**, and **physical plausibility** [2510.20206].

These signals are stored in a feedback memory database:
\[
\{\mathcal{M}(\rho^u_{i}, v_{i}), \mathcal{S}(v_{i}), \mathcal{O}(v_{i})\},
\]
and the next refined prompt \(\rho_{i+1}^r\) is generated using the original prompt, current refined prompt, historical feedback, and current scores [2510.20206]. The rewriting instruction explicitly asks the LLM to minimize semantic misalignment and improve temporal and spatial coherence and perceptual fidelity [2510.20206].

Candidate prompts are compared using an **Average Ranking mechanism**: each candidate receives metric-wise ranks, and the prompt with the **lowest average rank** is selected as the best prompt \(\rho_i^b\) [2510.20206]. The paper characterizes this as a way to avoid over-optimizing a single metric and instead maintain balanced improvements across semantic, spatial, temporal, and physical criteria.

SSPO is therefore a **gradient-free**, **black-box** optimization loop over prompts, not over model weights or latents [2510.20206]. This distinguishes it from RLHF-style prompt optimization and from latent-space test-time search. It also introduces a clear compute trade-off: several SSPO iterations require additional T2V generations and video evaluations, pushing inference time to roughly **\(3\times\)** that of a single-pass baseline, with about **\(\sim 2\ \mathrm{GB}\)** additional memory for **LLaVA-OneVision** [2510.20206].

## 5. Stage 3: Fine-tuning the rewriter LLM

Stage 3 closes the loop by using optimized prompt pairs from SSPO to fine-tune the rewriter LLM, thereby internalizing the expensive iterative optimization patterns discovered at test time [2510.20206]. The training data consist of prompt pairs:
\[
\{(\rho^u_{i},\rho^b_{i})|_{i=0}^{n-1}\},
\]
where \(\rho_i^u\) is the original user prompt and \(\rho_i^b\) is the best optimized prompt selected after iterative SSPO [2510.20206].

The paper states that Stage 2 rewriting uses **Qwen2.5-7B-Instruct**, while the fine-tuning stage performs **LoRA fine-tuning using LLaMA 3.1** [2510.20206]. The reported training setup includes **batch size 32**, **LoRA rank 64**, **single A100 GPU**, and **within 5 hours per iteration** [2510.20206]. The paper also notes a prompt pool of about **12k prompts** generated by GPT-4 for analysis [2510.20206].

Conceptually, Stage 3 serves two functions. First, it improves the quality of the initial prompt rewriting before test-time optimization. Second, it reduces reliance on many SSPO rounds by distilling task-specific optimization behavior into the rewriter model itself [2510.20206]. This suggests that RAPO++ is not merely a test-time search procedure; it is also a self-improving prompt-rewriting system.

## 6. Empirical performance and ablations

RAPO++ is evaluated on five T2V models—**LaVie**, **Latte**, **HunyuanVideo**, **CogVideoX**, and **Wan2.1**—and five benchmarks: **VBench**, **T2V-CompBench**, **EvalCrafter**, **VideoPhy**, and **PhyGenBench** [2510.20206].

On **VBench**, for **LaVie**, the baseline score is **80.89%**, RAPO reaches **82.38%**, and RAPO++ reaches **82.65%** [2510.20206]. For **Latte**, the corresponding scores are **77.03%**, **79.97%**, and **80.75%** [2510.20206]. The strongest submetric gains occur in **Multiple Objects**, **Spatial Relationship**, **Object Class**, and **Human Action**. For LaVie, **Multiple Objects** improves from **37.71%** to **64.86%** with RAPO and **71.89%** with RAPO++; **Spatial Relationship** improves from **37.27%** to **59.15%** and **64.76%** [2510.20206]. Similar patterns hold for Latte [2510.20206]. This suggests that the framework is especially effective for compositional and interaction-heavy prompts.

On **EvalCrafter**, for LaVie, **Text-Video Alignment** improves from **69.60** to **74.38** to **75.62**, while **Temporal Consistency** improves from **60.87** to **61.29** to **66.80** [2510.20206]. For Latte, **Text-Video Alignment** improves from **55.49** to **60.86** to **61.92** [2510.20206]. These results indicate that the “++” stages contribute not only to static prompt compliance but also to video-level temporal quality.

On **T2V-CompBench**, RAPO already yields large gains, and RAPO++ usually improves further. For LaVie, **Consistent Attribute Binding** goes from **0.620** to **0.692** to **0.742**, and **Object Interactions** from **0.760** to **0.839** to **0.849** [2510.20206]. For Latte, **Object Interactions** goes from **0.792** to **0.847** to **0.856** [2510.20206]. One nuance is that gains are not uniform on every submetric: on LaVie, **Action Binding** is **0.635** for RAPO and **0.632** for RAPO++ [2510.20206]. This indicates that the cross-stage refinement improves overall quality but does not dominate on every individual metric.

In the physics-aware setting, across four SSPO rounds on **PhyGenBench**, quality increases steadily. For **HunyuanVideo**, **PC** improves from **0.38** to **0.57** and **SA** from **0.24** to **0.42**; for **CogVideoX-5B**, **PC** improves from **0.34** to **0.53** [2510.20206]. On **VideoPhy**, improvements are also reported across **solid-solid**, **solid-fluid**, and **fluid-fluid** conditions [2510.20206]. These results support the claim that optical-flow-informed SSPO can refine prompts for physical plausibility.

The paper also reports compute-normalized gains for SSPO. Across LaVie and Latte, the additional inference cost of roughly **\(3\times\)** yields average gains of **\(3.5\%\)** on VBench and **\(18.1\%\)** on T2V-CompBench [2510.20206].

Ablations clarify component roles. For Stage 1, VBench total score on LaVie rises from **80.37%** with word augmentation only and **79.75%** with sentence refactoring only to **82.38%** for full RAPO [2510.20206]. For Stage 2 and Stage 3 on T2V-CompBench with LaVie, the progression is:

| Variant | Consistent Attribute Binding | Dynamic Attribute Binding | Action Binding | Object Interactions |
|---|---:|---:|---:|---:|
| w/o fine-tuning \(L_o\), w/o SSPO | 0.620 | 0.232 | 0.483 | 0.760 |
| w/o fine-tuning \(L_o\), w/ SSPO | 0.629 | 0.236 | 0.542 | 0.778 |
| w/ fine-tuning \(L_o\), w/o SSPO | 0.659 | 0.253 | 0.552 | 0.835 |
| w/ fine-tuning \(L_o\), w/ SSPO | **0.742** | **0.294** | **0.632** | **0.849** |

These results show that **SSPO alone helps**, **fine-tuning alone helps more**, and **combining both is best** [2510.20206].

## 7. Interpretation, model-agnosticism, and limitations

RAPO++ is repeatedly described as **model-agnostic** or “model-unaware,” meaning it does not modify T2V model weights or rely on architectural internals and can be used with both **unet-based** and **DiT-based** systems [2510.20206]. The experiments across LaVie, Latte, HunyuanVideo, CogVideoX, and Wan2.1 support this portability claim [2510.20206]. At the same time, the framework’s underlying rationale is explicitly **training-data alignment**, so its effectiveness depends on how well the prompt-optimization machinery matches the training prompt distribution relevant to the target generator [2504.11739], [2510.20206]. This suggests that “model-agnostic” should be understood as interface-level portability, not as invariance to prompt-distribution differences.

The main explicit limitation discussed is **numeracy**. The paper presents failure cases such as “Five colorful parrots perch on a tree branch” and “Three majestic giraffes graze on the leaves of tall trees in the African savannah...” where the generated videos do not preserve the correct entity counts [2510.20206]. The authors attribute this to T2V models’ tendency to blur numerical information into broader semantics and to SSPO’s lack of fine-grained count-aware feedback [2510.20206].

Other limitations are implied by the framework itself. RAPO++ depends on retrieval quality from the relation graph, LLM rewriting quality, and verifier quality in SSPO [2510.20206]. Its gains are not perfectly uniform across all metrics, and the iterative stages incur substantial inference-time cost [2510.20206]. The paper does not present a full failure taxonomy, but these trade-offs are visible in the ablations and compute analysis.

A second potential misunderstanding is to view RAPO++ as generic prompt beautification. The papers argue against this interpretation. The core mechanism is **selective, semantically relevant, training-aligned enrichment**, followed by **feedback-driven correction**, not indiscriminate elaboration [2504.11739], [2510.20206]. This suggests that the method’s main contribution lies in controlling prompt distribution and prompt semantics jointly.

## 8. Significance within T2V prompt optimization

RAPO++ occupies a specific methodological position in T2V generation. Compared with standard prompt engineering, it replaces manual heuristics with a structured pipeline grounded in training captions, iterative feedback, and learned prompt rewriting [2510.20206]. Compared with generic retrieval-augmented generation, its retrieval operates over **scene–modifier relations** derived from T2V training captions rather than over external knowledge documents [2504.11739], [2510.20206]. Compared with RLHF-style prompt optimization, it remains **black-box with respect to the generator** and optimizes prompts rather than model weights or latent trajectories [2510.20206].

Its most characteristic contribution is the unification of three temporal scales of prompt optimization: pre-inference corpus alignment, inference-time sample-specific correction, and cross-run distillation of optimized prompt behavior [2510.20206]. This makes RAPO++ best understood not as a single rewriting model, but as a prompt-control layer around T2V systems.

In the formal literature represented here, RAPO++ therefore denotes a specific T2V framework: **Stage 1** RAPO for retrieval-augmented prompt alignment, **Stage 2** SSPO for iterative verifier-guided test-time refinement, and **Stage 3** LLM fine-tuning on optimized prompt pairs [2510.20206]. Its empirical profile is strongest on **semantic alignment**, **multiple-object composition**, **spatial relations**, **object interactions**, and **temporal consistency**, while its main practical trade-off is increased inference cost and its main open challenge is numeracy [2510.20206].

Source: https://www.emergentmind.com/topics/rapo