FlowZero: Zero-Shot Text-to-Video Synthesis

• FlowZero is a zero-shot text-to-video synthesis framework that integrates GPT-4 with image diffusion models to produce temporally coherent videos from a single prompt.
• It employs Dynamic Scene Syntax to provide detailed spatio-temporal guidance, enabling precise object layout and smooth motion control across frames.
• The framework achieves high semantic and temporal coherence ratings by combining iterative self-refinement with innovative motion-guided noise shifting techniques.

FlowZero is a zero-shot text-to-video synthesis framework that integrates LLMs with image diffusion models to produce temporally coherent, semantically grounded videos from a single natural language prompt. FlowZero leverages LLMs, specifically GPT-4, to convert text descriptions into a structured spatio-temporal representation—Dynamic Scene Syntax (DSS)—enabling fine-grained control over object layouts and motion dynamics for each frame. This DSS then guides a conditioned image diffusion model to generate frame sequences exhibiting smooth object motions, visual consistency, and robust alignment to textual input, without the need for any video-specific fine-tuning or supervised video data (Lu et al., 2023).

1. Dynamic Scene Syntax: Structured Spatio-Temporal Guidance

FlowZero’s main innovation is the use of Dynamic Scene Syntax, a hierarchical, structured representation of video content extracted via prompt-driven LLM reasoning. For a given video prompt $\mathcal{T}$, GPT-4 is prompted to produce:

• Natural language descriptions $T_1,\ldots,T_N$ for $N$ frames.
• Foreground object layouts $L_i$ for each frame, where $$L_i = \{ (o_{i,k}, [x_{1ik}, y_{1ik}, x_{2ik}, y_{2ik}]) \}_{k=1}^{K_i}$$, specifying object classes/attributes and normalized bounding boxes.
• Background motion parameters $\{ d_i, s_i \}$ capturing frame-wise global translations: $d_i$ is a unit direction from a discrete 8-way set (left, right, up, etc.); $s_i \in [0,1]$ is the speed scalar.

DSS is generated in two LLM calls using prompt templates: one for frame-wise scene descriptions and another for structured layouts and motion metadata. This approach yields actionable supervisory signals for precise, multi-object scene decomposition and temporal control.

2. Mathematical Formulation of Generation Process

The joint generation process is formalized as producing frame sequence $X_1,\ldots,X_N$ conditioned on DSS: $$P(X_1,\ldots,X_N \mid \text{DSS}) = \prod_{i=1}^N P(X_i \mid T_i, L_i, d_i, s_i, X_{<i})$$ Here, each term is realized via a diffusion model denoiser conditional on:

• $T_1,\ldots,T_N$0: semantic content for frame $T_1,\ldots,T_N$1 (CLIP-encoded prompt tokens).
• $T_1,\ldots,T_N$2: foreground layout guidance (GLIGEN-like layout control).
• $T_1,\ldots,T_N$3: background motion, implemented via noise phase shifts.
• $T_1,\ldots,T_N$4: cross-frame context to enable temporal consistency.

This structure enables the model to maintain local (frame-wise) semantic consistency, enforce object placement, and encode temporal dynamics.

3. Image Diffusion Model Conditioning and Motion Guidance

Video synthesis in FlowZero is driven by a U-Net diffusion backbone adapted from GLIGEN:

• Semantic Conditioning: Framewise natural language descriptions $T_1,\ldots,T_N$5 are processed with CLIP and integrated via cross-attention.
• Layout Guidance: Foreground object boxes from $T_1,\ldots,T_N$6 control object placement using gated attention, enabling explicit spatial grounding.
• Temporal Consistency: Cross-frame attention allows each frame’s denoiser to access feature maps from previous frames, supporting smooth inter-frame transitions.
• Motion-Guided Noise Shifting (MNS): Background motion coherence is achieved by initializing all frames with correlated noise. $T_1,\ldots,T_N$7 is sampled for frame 1; for $T_1,\ldots,T_N$8, $T_1,\ldots,T_N$9 is produced by applying a phase shift in the DFT domain: $N$0 where $N$1 are the 2D DFT and inverse, $N$2 are spatial frequency grids, and $N$3 encodes motion direction. This method enables smooth, frame-aligned background and camera movement as interpreted by the diffusion denoiser.

4. Iterative Self-Refinement of Scene Syntax

Due to possible errors in LLM scene decomposition (e.g., inversion of motion direction, spatial misalignment), FlowZero introduces an iterative self-refinement loop using GPT-4 as a verifier and editor:

• Initialize layouts $N$4, motions $N$5, speeds $N$6.
• At each iteration $N$7, the frame-wise DSS is validated; if misalignment with $N$8 is detected (confidence score $N$9), GPT-4 rectifies DSS using generated critiques.
• The process terminates when $L_i$0 or after $L_i$1.
• No explicit loss is minimized; validation and correction are performed via LLM-based scoring.

Empirical results indicate self-refinement improves layout verification benchmarks (multiple objects, direction, size, visibility) from [90%, 83%, 80%, 61%] to [96%, 93%, 93%, 78%].

5. Implementation Details and Hyperparameters

• Backbone: GLIGEN U-Net, 1.2B parameters, 512×512 resolution.
• LLM: GPT-4, queried via API for DSS synthesis and refinement.
• Frames per video: $L_i$2 (configurable; limited by attention compute).
• Diffusion: Standard 50-step DDIM.
• Inference hardware: NVIDIA V100; run-time $L_i$3 s per video.
• Self-refine: Threshold $L_i$4, maximum 5 iterations.

6. Quantitative and Qualitative Evaluation

FlowZero is benchmarked on CLIP score (text–frame cosine similarity), and a user study with 20 participants rating semantic accuracy, temporal coherence, and visual quality on a 1–5 scale, as well as overall ranking.

Method	Train-Free	CLIP $L_i$5	Sem. $L_i$6	Temp. $L_i$7	Qual. $L_i$8	Rank $L_i$9
AnimateDiff	–	0.244	3.15	2.75	2.97	3.42
VideoFusion	–	0.264	3.38	2.92	3.11	3.17
T2V-Z	✔	0.245	3.29	2.99	3.03	3.19
DirecT2V	✔	0.244	3.39	3.29	2.52	2.97
FlowZero	✔	0.267	4.57	4.58	4.40	2.00

Removal of any DSS component, cross-frame attention, or MNS degrades either semantic or temporal coherence. FlowZero achieves the highest ratings in both semantic and temporal dimensions among train-free baselines and exhibits robust performance in varied example prompts such as “A butterfly takes off from a flower and flies up to the right” and “A volcano gradually erupts” (Lu et al., 2023).

7. Limitations and Prospects for Extension

Current FlowZero limitations include:

• Restriction to short video clips (8–16 frames) due to cross-frame attention memory and compute.
• Background motion is confined to simple 2D translations; more complex camera motions (zoom, rotation, nonlinear trajectories) are not supported.
• Scene verification and rectification rely solely on heuristic LLM scores; integrating learned or hybrid verifiers may improve robustness.
• The DSS representation currently lacks explicit depth, occlusion, or 3D world modeling; extending to include richer priors may enable more sophisticated scene synthesis and viewpoint changes.

A plausible implication is that future research integrating more advanced spatio-temporal priors, differentiable cross-modality verifiers, or explicit 3D structure could further enhance fidelity, scalability, and controllability in zero-shot text-to-video generation.