SCMAPR: Self-Correcting Multi-Agent Prompt Refinement for Complex-Scenario Text-to-Video Generation

Abstract: Text-to-Video (T2V) generation has benefited from recent advances in diffusion models, yet current systems still struggle under complex scenarios, which are generally exacerbated by the ambiguity and underspecification of text prompts. In this work, we formulate complex-scenario prompt refinement as a stage-wise multi-agent refinement process and propose SCMAPR, i.e., a scenario-aware and Self-Correcting Multi-Agent Prompt Refinement framework for T2V prompting. SCMAPR coordinates specialized agents to (i) route each prompt to a taxonomy-grounded scenario for strategy selection, (ii) synthesize scenario-aware rewriting policies and perform policy-conditioned refinement, and (iii) conduct structured semantic verification that triggers conditional revision when violations are detected. To clarify what constitutes complex scenarios in T2V prompting, provide representative examples, and enable rigorous evaluation under such challenging conditions, we further introduce T2V-Complexity, which is a complex-scenario T2V benchmark consisting exclusively of complex-scenario prompts. Extensive experiments on 3 existing benchmarks and our T2V-Complexity benchmark demonstrate that SCMAPR consistently improves text-video alignment and overall generation quality under complex scenarios, achieving up to 2.67% and 3.28 gains in average score on VBench and EvalCrafter, and up to 0.028 improvement on T2V-CompBench over 3 State-Of-The-Art baselines. Code is available at https://github.com/HiThink-Research/SCMAPR.

• The paper presents SCMAPR's main contribution in iteratively refining text-to-video prompts via a multi-agent, self-correcting mechanism.
• It introduces a scenario-aware taxonomy and a detailed verification protocol that improves semantic fidelity and temporal consistency.
• Empirical results across multiple benchmarks demonstrate significant performance gains over previous prompt optimization approaches.

Self-Correcting Multi-Agent Prompt Refinement for Complex-Scenario Text-to-Video Generation

Introduction

Text-to-Video (T2V) generation presents unique challenges, especially under prompts that are abstract, underspecified, or demand complex scenario reasoning. Despite the progress of video diffusion models, existing approaches struggle when tasked to maintain semantic fidelity and temporal-coherent synthesis from such complex prompts. The SCMAPR framework addresses these deficiencies via a scenario-aware, verification-driven, multi-agent architecture for prompt refinement, explicitly targeting complex-scenario T2V applications. SCMAPR reframes prompt optimization as a sequence of agent-mediated stages, each introducing structure and targeted self-correction to the generation pipeline. The framework is complemented by the T2V-Complexity benchmark, which delivers a rigorous taxonomy and a balanced prompt suite for comprehensive evaluation.

Framework Architecture

SCMAPR organizes prompt refinement into five explicit stages, each delegating functionality to specialized agents. This modular design ensures interpretability, modularity, and enforceable guarantees of semantic alignment between user input and the generative prompt.

Figure 1: The SCMAPR framework, detailing five functional stages and six interacting agents for scenario-aware, verification-driven prompt refinement.

The pipeline operates as follows:

1. Scenario Routing: A Scenario Router classifies the prompt according to a ten-class taxonomy, assigning the dominant scenario tag as a routing signal for downstream modules.
2. Policy Synthesis: The Policy Generator produces a structured, scenario-conditioned rewriting policy, specifying explicit constraints and refinement guidelines.
3. Policy-Conditioned Refinement: The Prompt Refiner generates a detailed, model-optimized prompt leveraging these policy constraints.
4. Structured Semantic Verification: Atomizer and Entailment Validator agents decompose the user prompt into atomic semantic units (atoms), chunk the refined prompt, match atoms to relevant evidence, and audit for entailment/missingness/contradiction.
5. Conditional Revision: The Content Reviser updates the refined prompt to address detected omissions or contradictions. Iteration continues until coverage and consistency are satisfied.

   Figure 2: The four-step verification protocol, from atomic extraction to entailment validation, isolates semantic deviations and enables targeted correction.

Complex-Scenario Taxonomy and T2V-Complexity Benchmark

SCMAPR utilizes a rigorously defined set of ten scenario categories, capturing diverse axes of T2V challenge:

• Abstract Descriptions
• Complex Spatial Relations
• Multi-Element Scenes
• Fine-Grained Appearance
• Temporal Consistency
• Stylistic Hybrids
• Causality & Physics
• Camera Motion
• Object Interaction
• Scene Transitions

This taxonomy enables interpretable scenario tagging for refinement and facilitates the T2V-Complexity benchmark, a 1000-prompt suite partitioned evenly across the ten classes. The design addresses severe category imbalance observed in prevalent benchmarks and supports controlled, category-wise evaluation. Each prompt is annotated for failure modes, semantic coverage, and scenario-specific video criteria.

Semantic Verification and Self-Correction

Conventional prompt optimizers for T2V are incapable of enforcing atom-level fidelity. SCMAPR's intermediate verification protocol overcomes this by:

• Extracting explicit, non-paraphrased atomic constraints from the user input (e.g., entities, actions, scenery),
• Segmenting the refined prompt and mapping atoms to evidence chunks in embedding space,
• Using an entailment validator to audit the preservation of each atom,
• Triggering minimally-invasive revision only for missing or contradictory atoms, preserving the remainder of the refined prompt.

This mechanistic loop enforces strict semantic and logical coverage, which is observable and quantitatively validated at the atom level.

Figure 3: An end-to-end SCMAPR case study, highlighting routing, policy synthesis, atomization, validation, and revision for an abstract user prompt.

Empirical Results

SCMAPR was evaluated on VBench, EvalCrafter, T2V-CompBench, and the new T2V-Complexity, using both Wan2.2 and LaVie as T2V backbones. Across all pairs and benchmarks, SCMAPR consistently outperforms direct prompting, Open-Sora, and the strongest prior prompt optimization baseline (RAPO), with key findings as follows:

Benchmark	Backbone	SCMAPR Δ Avg. Score (vs. direct)	SCMAPR Δ (vs. best prior)
VBench	LaVie	+2.67%	+1.76% (RAPO)
VBench	Wan2.2	+2.02%	+0.78% (RAPO)
EvalCrafter	LaVie	+3.06	+1.27 (RAPO)
EvalCrafter	Wan2.2	+3.28	+2.20 (RAPO)
T2V-CompBench	LaVie	+0.088	+0.016 (RAPO)
T2V-CompBench	Wan2.2	+0.069	+0.028 (RAPO)
T2V-Complexity	Wan2.2	+2.74%	N/A

SCMAPR delivers dominant improvements in semantic coverage, temporal consistency, fine-grained compositionality, and hallucination reduction. Ablations confirm substantial drops without scenario routing, policy generation, and especially without atom-level self-correction.

Figure 4: Frame comparison, showing that SCMAPR's refined prompt yields a video semantically consonant with the user's intent, as opposed to the semantically unaligned output from the raw prompt.

Figure 5: Abstract Description scenario. The refined prompt transfers the abstract intent to actionable visual elements, enabling recovery from the model's free-association failure.

Figure 6: Stylistic Hybrid scenario. SCMAPR's refined prompt enables faithful mixture of styles, correcting misalignments and implausible compositions seen with the baseline.

Figure 7: Hallucination analysis—SCMAPR eliminates unsupported visuals, attaining faithful alignment with the explicit prompt.

Case Studies and Qualitative Analysis

Figure 3 demonstrates an illustrative refinement, verification, and revision cycle for the prompt "Hope blooming in the dark." The refined and iteratively revised prompt achieves 100% atom entailment with no contradictions, as reflected in the validator audit. Subsequent video outputs manifest the precise emotional and logical structure implied by the prompt taxonomy, confirming SCMAPR's effectiveness for abstract, compositional, and hybrid stylistic scenarios.

Practical and Theoretical Implications

SCMAPR’s modular, multi-agent architecture is model-agnostic and training-free, enabling generalization across T2V backbones with minimal adaptation. The scenario taxonomy and atom-level verification protocol support interpretable, fine-grained evaluation and debugging—critical attributes for advancing T2V alignment research. SCMAPR's approach to self-correction addresses central limitations of monolithic LLM-based prompt optimizers for video, especially in the presence of abstraction, compositionality, and under-specification.

Theoretically, SCMAPR demonstrates that explicit mid-level semantic supervision, grounded in task-taxonomy, can be employed to enforce strict alignment and correct for LLM or diffusion model failures in downstream generative tasks. This insight is extensible to compositional and multi-modal generation paradigms.

Limitations and Future Directions

SCMAPR introduces inference overhead via multi-agent orchestration and relies on the base LLM's reasoning capabilities. The category tagging set, while covering principal axes of current T2V challenge, is extensible as new sources of failure emerge (e.g., multi-modal synchronization, long-context dependencies). Adapting SCMAPR to multi-modal input settings and reducing dependency on atomic field selection present open avenues for future research. Model efficiency optimizations and integration with rejection-based reinforcement mechanisms offer further opportunities.

Conclusion

SCMAPR delivers a blueprint for verification-driven, scenario-conditioned prompt refinement in complex-scenario T2V generation. Its taxonomy-guided, multi-agent protocol substantially outperforms current SOTA in text-video alignment, compositional reasoning, and hallucination robustness, as validated across standard and new benchmarks. SCMAPR substantiates the practical efficacy of structured, intermediate-representation-based refinement over black-box LLM optimization for open-ended video generation.

See (2604.05489) for full methodology, code, and experimental details.