Optical Video Generation Model

• Optical video generation models are generative systems that leverage hierarchical optical flow to create long, temporally coherent videos with realistic motion.
• The approach divides the synthesis into key frame generation, latent optical flow modeling, and frame interpolation with MotionControlNet for detailed refinement.
• Empirical evaluations show marked improvements in metrics such as FVD, FID, and motion smoothness, establishing superiority over traditional frame interpolation techniques.

An optical video generation model is a class of generative model that synthesizes video sequences by explicitly modeling and leveraging optical flow, providing fine control over temporal consistency, realistic motion, and long-sequence synthesis. These models have arisen as advanced solutions to the challenges of generating temporally coherent and visually plausible long-form videos where standard approaches—based on framewise or naive keyframe interpolation—are prone to unnatural transitions, repetition, or breakdowns in motion continuity.

1. Hierarchical Motion-Guided Video Synthesis

Optical video generation models such as LumosFlow (Chen et al., 3 Jun 2025) utilize a hierarchical synthesis pipeline that divides the task into three primary stages:

1. Key Frame Generation: Employing a large-motion text-to-video diffusion model (LMTV-DM), the framework generates a sparse sequence of key frames characterized by substantial motion diversity. This is achieved by fine-tuning on sparsified, lower-FPS training data, forcing each key frame to represent a distinct semantic or dynamic state. The formal sampling is given by $v \sim p_{\theta}(v | P)$, where $v$ is the set of key frames and $P$ the text prompt.
2. Latent Optical Flow Synthesis: Intermediate dynamics between key frames are specified by optical flows, representing pixel-wise motion fields across the video segment. The latent optical flow diffusion model (LOF-DM) operates in a compressed latent space (not direct pixels), producing forward ($F_{k\rightarrow 1}$) and backward ($F_{k\rightarrow K}$) optical flow fields for each intermediate frame $k$, conditioned on semantic features of the key frames and a linear flow prior. This approach enables the modeling of large, nonlinear, and semantically consistent motions beyond the capabilities of classic interpolative frame-by-frame methods.
3. Frame Interpolation and Refinement: Warping functions reconstruct each intermediate frame using the generated optical flows and the key frames. The explicit formulation is:

$$\hat{I}_k = \mathcal{P}\left(\mathcal{W}(I_1, \hat{F}_{k \rightarrow 1}), \mathcal{W}(I_K, \hat{F}_{k \rightarrow K})\right)$$

where $\mathcal{W}$ denotes the warping operation and $\mathcal{P}$ a learned fusion/refinement module. A dedicated diffusion-based MotionControlNet further refines these interpolated frames, reducing artifacts and ensuring temporal consistency.

2. Optical Flow Modeling and Latent Representation

Optical flow is central both for motion specification and the interpolation mechanism:

• Encoding/Decoding: All flows are encoded jointly by an optical flow VAE (OF-VAE) to a latent tensor for efficient processing ($32\times$ spatial, $4\times$ temporal compression). The VAE is trained with an $\ell_1$ reconstruction loss and a KL-regularization:

$$L_{\text{OF-VAE}} = \|F_{:\rightarrow 1} - \hat{F}_{:\rightarrow 1}\|_1 + \|F_{:\rightarrow K} - \hat{F}_{:\rightarrow K}\|_1 + \mathrm{KL}_{\text{reg}}$$

Compression in the latent domain allows the LOF-DM to capture and propagate nonlinear, large-scale motion over long temporal windows.

• Flow Prior and Conditioning: A linear flow prior is computed:

$$\hat{F}_{k \rightarrow 1}^L = k F_{K \rightarrow 1}, \quad \hat{F}_{k \rightarrow K}^L = (1 - k) F_{1 \rightarrow K}$$

Latent representations of these prior flows are concatenated with CLIP-embedded key frame features for conditioning the diffusion process, ensuring physically plausible interpolations aligned with scene semantics.

3. Advanced Interpolation and Motion Refinement

Standard video frame interpolation (VFI) techniques rely on explicit flow between consecutive frames, typically failing under high-rate ($>8\times$) interpolation or complex, non-linear motions. LumosFlow surpasses these by employing:

• High-Rate Interpolation: Achieving up to $15\times$ in-between sampling (e.g., synthesizing 16 frames between every 2 key frames), far exceeding traditional VFI while maintaining motion and appearance continuity.
• MotionControlNet: Functions analogously to ControlNet but conditions on both motion and appearance, further refining interpolated frames to enhance dynamic plausibility. MotionControlNet combines representations of both warped images and key frame semantics with the prompt, mapping to the output image:

$$y = \mathcal{F}_{\phi_1}(I_1, I_K, P) + \mathcal{Z}_{\phi_2}(I_1, I_K, P, \mathcal{W}(I_1, \hat{F}_{:\rightarrow 1}), \mathcal{W}(I_K, \hat{F}_{:\rightarrow K}))$$

4. Empirical Evaluation and Performance

On benchmarks for long video synthesis and frame interpolation, LumosFlow demonstrates:

• Long-Video Generation: Synthesis of 273 frames per video (18 key frames, 16 interpolated per segment), surpassing FreeLong, FreeNoise, and Video-Infinity in Fréchet Video Distance (FVD), Fréchet Inception Distance (FID), Motion Smoothness (M-S), and Dynamic Degree (D-D). For example, LumosFlow attains FVD = 913, FID = 479, M-S = 0.990, D-D = 0.570, with a human preference rate of 98.2% for realism and consistency.
• Interpolation Robustness: On DAVIS-7 and UCF101-7, achieves lower FVD/LPIPS and higher PSNR than RIFE, LDMVFI, VIDIM, and ablated baselines, with end-point flow error (EPE) significantly reduced due to OF-VAE/LOF-DM. Large and nonlinear motions are handled with minimal visual distortion.

Model	FVD ↓	FID ↓	M-S ↑	D-D ↑
FreeLong	1829	482	0.9752	0.218
FreeNoise	2176	481	0.9722	0.314
Video-Infinity	1789	483	0.9566	0.547
LumosFlow	913	479	0.990	0.570

5. Hierarchical and Generative Advantages over Conventional Methods

Classical video generation approaches either concatenate short clips sequentially or interpolate between sparsely generated key frames, both regimes prone to temporally repeated content or unnatural transitions. Optical video generation models with hierarchical, motion-injected architectures exhibit the following distinctions:

• Generative Motion Synthesis: Instead of fitting a unique flow per pair of frames, the flow distribution is synthesized in the latent space, conditioned on both local and global scene context, enabling large-motion, high-fidelity transitions.
• Scalability: 273-frame videos with temporally coherent structure can be synthesized in a single pipeline without cascading errors.
• Temporal Consistency: By decoupling motion synthesis, content fusion, and post-hoc refinement, artifacts from sequential generation are minimized, and long-range dependencies are preserved.

6. Mathematical Formulations and Losses

Key mathematical elements include:

• Latent Flow Prior:

$$\hat{F}_{k \rightarrow 1}^L = k F_{K \rightarrow 1}, \quad \hat{F}_{k \rightarrow K}^L = (1 - k) F_{1 \rightarrow K}$$

• OF-VAE Reconstruction Loss:

$$L_{\text{OF-VAE}} = \|F_{:\rightarrow 1} - \hat{F}_{:\rightarrow 1}\|_1 + \|F_{:\rightarrow K} - \hat{F}_{:\rightarrow K}\|_1 + \mathrm{KL}_{\text{reg}}$$

• Intermediate Frame Synthesis:

$$\hat{I}_k = \mathcal{P}\left(\mathcal{W}(I_1, \hat{F}_{k \rightarrow 1}), \mathcal{W}(I_K, \hat{F}_{k \rightarrow K})\right)$$

• MotionControlNet Output:

$$y = \mathcal{F}_{\phi_1}(I_1, I_K, P) + \mathcal{Z}_{\phi_2}(I_1, I_K, P, \mathcal{W}(I_1, \hat{F}_{:\rightarrow 1}), \mathcal{W}(I_K, \hat{F}_{:\rightarrow K}))$$

7. Limitations and Prospective Directions

While LumosFlow establishes a significant advance in long-range, temporally consistent video generation by leveraging hierarchical motion control and latent optical flow injection, several boundaries remain:

• Flow Modeling Complexity: The expressivity of LOF-DM is determined by the quality and diversity of the training set, and models may require adaptation for scenes with highly non-standard dynamics.
• Resource Considerations: Latent space processing mitigates memory and time complexity, but large-scale synthesis remains compute-intensive for extremely long outputs.
• Extension: Integration with semantic or multimodal controls (e.g., depth, user sketches) could further enhance control, and adaptation to higher spatial resolutions remains an important engineering challenge.

Conclusion

Optical video generation models such as LumosFlow reify a hierarchical paradigm for motion-guided, long video synthesis by uniting high-level content diversity, low-level motion realism, and advanced temporal interpolation using latent-space optical flow diffusion. The explicit injection of optical flow guidance drives both the high-rate generation of intermediate frames and the preservation of appearance continuity across large temporal spans, establishing the state-of-the-art for both long-form video synthesis and advanced frame interpolation.