OmniDirector: Camera Motion Cloning

• OmniDirector is a unified framework that clones complex, multi-shot camera trajectories in video diffusion systems using a visual camera grid representation.
• It employs a novel camera grid to abstract and render spatio-temporal camera motion, enabling precise and seamless video generation without cross-paired training data.
• Empirical evaluations show superior accuracy in rotation, translation, and temporal consistency, positioning OmniDirector as a robust solution for advanced cinematographic effects.

OmniDirector is a unified, director-level framework for camera-motion cloning in video diffusion models, enabling the transfer of arbitrary, multi-shot camera trajectories from reference videos without requiring cross-paired training data. At its core lies a fully visual, spatio-temporal representation of camera motion—the camera grid—which abstracts camera parameters as a rendered video sequence and interfaces seamlessly with multimodal diffusion transformers. Augmenting this physical proxy, a hierarchical prompt expansion agent translates diverse multimodal control signals into a single textual prompt, facilitating precise and coherent video generation that harmonizes camera, subject, and action (Liu et al., 11 Jun 2026).

1. Motivation and Challenges in Camera Motion Cloning

Conventional methods for video generation with camera control fall into two main categories: explicit-parametric and implicit/text-driven. Techniques that inject direct camera parameters (e.g., 6DoF extrinsics, Plücker coordinates) provide precision on elementary pans or tilts but fail for multi-shot sequences and demand labor-intensive parameter specification. Text-based or implicit conditioning offers improved usability but cannot reproduce complex, nuanced cinematographic transitions with high fidelity. Efforts to learn from cross-paired data—video pairs differing only in camera motion—foundered due to scarcity of such data in the wild and limited narrative variety in synthetic substitutes. Thus, prior to OmniDirector, no approach robustly cloned complex, multi-shot camera trajectories at a scalable or general level (Liu et al., 11 Jun 2026).

2. Camera Grid: Universal Visual Representation of Camera Motion

OmniDirector introduces the “camera grid,” a purely visual, empty-room grid video that captures camera extrinsics as spatio-temporal motion, entirely decoupled from scene content:

• Scene Construction: With extrinsics $P = \{ (R_1, t_1), \ldots, (R_T, t_T) \}$, the mean height $\bar y$ is computed and horizontal grid planes are placed at $y_\mathrm{floor} = \bar y - \Delta h$ and $y_\mathrm{ceiling} = \bar y + \Delta h$, where $\Delta h$ is proportional to the median inter-frame displacement, enforcing stability.
• Tunnel Walls: Camera translation in $X$–$Z$ is visualized by wrapping the trajectory in an annular tunnel $$W = \{(x,z)\mid r < d_\mathrm{traj}(x, z) < r + \delta\}$$; vertical lines from floor to ceiling accentuate spatial progress.
• Frame Rendering: Each world coordinate $P_w$ is projected via $P_c = R_i P_w + t_i$ and $\bar y$0, for all frames, yielding the grid video $\bar y$1.
• Special Effects: Fisheye distortions adopt the Kannala–Brandt model, while dolly zooms use a constant-size cube and a dynamically scaled picture-in-picture inset.
• Multi-Shot Extension: TransNet-V2 detects shot boundaries, introducing white transition frames; each subclip is rendered independently and concatenated, producing variable-length, multi-shot grids.

This camera grid is compatible with standard video diffusion architectures and can be procedurally generated for any reference video, scaling the construction of large training corpora (Liu et al., 11 Jun 2026).

3. Multimodal Diffusion Transformer Architecture

OmniDirector employs a Multi-Modal Diffusion Transformer (MMDiT), ingesting three coordinated inputs: a single reference image $\bar y$2 of the subject, the camera grid video $\bar y$3, and a consolidated textual prompt $\bar y$4. Key components include:

• Latent Encoding and Concatenation: Both $\bar y$5 and $\bar y$6 are separately processed by a pretrained 3D-VAE encoder $\bar y$7 yielding $\bar y$8 and $\bar y$9. These latents, concatenated with a diffusion noised target latent $y_\mathrm{floor} = \bar y - \Delta h$0, form $y_\mathrm{floor} = \bar y - \Delta h$1, which is then patchified into tokens $y_\mathrm{floor} = \bar y - \Delta h$2.
• Joint Attention Fusion: Inside each Transformer block, alternating spatial-temporal self-attention on $y_\mathrm{floor} = \bar y - \Delta h$3 is fused with cross-modal attention using $y_\mathrm{floor} = \bar y - \Delta h$4, ensuring joint alignment of image, video, and grid tokens.
• Self-Reconstruction Regularization: To ensure the camera grid is utilized, 30% of training samples use $y_\mathrm{floor} = \bar y - \Delta h$5 as both input and target, regularizing the model to reconstruct its own latent $y_\mathrm{floor} = \bar y - \Delta h$6.

Adaptive classifier-free guidance is employed: unconditional branches receive a black grid and the phrase “static camera,” with grid features injected early in diffusion to establish motion layout and image/text cues injected in later denoising stages (Liu et al., 11 Jun 2026).

4. Hierarchical Prompt Expansion Agent

Recognizing text as the shared interface for pretrained diffusion models, OmniDirector’s two-stage agent assembles diverse control cues into a unified $y_\mathrm{floor} = \bar y - \Delta h$7:

• Stage I: Camera Prompt Generation
  1. Inter-Shot Prompts ($y_\mathrm{floor} = \bar y - \Delta h$8): Using TransNet-V2, frames adjacent to shot boundaries are described via Qwen3-VL, a multimodal LLM, yielding narrations such as “cut from wide establishing shot to close-up.”
  2. Intra-Shot Prompts ($y_\mathrm{floor} = \bar y - \Delta h$9): Within subclips, relative pose $y_\mathrm{ceiling} = \bar y + \Delta h$0 is computed. Dominant translation axes, speed discretization, and Euler angle conversion yield labeled camera moves (e.g., left arc, right arc). Consecutive moves with identical labels are merged.
  3. Pose Refinement: Keyframes and prompts are refined by Qwen3-VL to resolve estimator noise through visual-linguistic cross-validation.
  4. Manual Polishing: Human review finalizes $y_\mathrm{ceiling} = \bar y + \Delta h$1 by merging inter- and refined intra-shot prompts.

• Stage II: Semantic Fusion: The text prompt $y_\mathrm{ceiling} = \bar y + \Delta h$2 user prompt $y_\mathrm{ceiling} = \bar y + \Delta h$3 and image $y_\mathrm{ceiling} = \bar y + \Delta h$4 are collectively fused by Qwen3-VL to the final text condition $y_\mathrm{ceiling} = \bar y + \Delta h$5.

The agent’s design ensures comprehensive and systematically structured prompts that harmonize camera, subject, and action (Liu et al., 11 Jun 2026).

5. Large-Scale Data Generation and Training

Leveraging the self-supervised nature of the camera grid, data is formed automatically from existing video corpora:

• 1.8 million 480p Internet videos (spanning movies, advertising, user footage) are processed.
• For each video: TransNet-V2 detects shot boundaries; DPA-V3 estimates extrinsics; the camera grid $y_\mathrm{ceiling} = \bar y + \Delta h$6 is rendered (with transition-aware white frames); a main-subject image crop $y_\mathrm{ceiling} = \bar y + \Delta h$7 is extracted.
• Training comprises 10,000 steps, batch size 64, optimizer learning rate $y_\mathrm{ceiling} = \bar y + \Delta h$8, and augmentations such as random grid coloration and pose jittering.
• The approach eliminates the need for labor-intensive cross-paired annotation, enabling scalable domain transfer (Liu et al., 11 Jun 2026).

6. Empirical Evaluation and Findings

On a rigorous set of 1,094 single- and multi-shot reference samples (from both in-domain and out-of-domain categories), OmniDirector demonstrates marked superiority over CamCloneMaster, Seedance 2.0, and LTX-LoRA:

Metric	OmniDirector	Best Prior
Relative Rotation Error	2.64°	4.11°
Rotation Precision (@4°)	83.2%	74.1%
Rel. Translation Error	16.8°	27.5°
Translation Precision (@20°)	72.7%	52.2%
Temporal Trans. Accuracy	96.5%	38.9%
Semantic Trans. Accuracy	83.8%	29.6%
Reference-Content Leakage	3.4%	56.5%

Ablation studies reveal a 20–50% degradation in camera accuracy when omitting the semantic fusion step, inter-shot prompt expansion, or adaptive guidance. Visually, the system accurately clones pans, tilts, dollies, zooms, and shot cuts, even when reference and generation domains differ drastically. During inference, OmniDirector exhibits zero-shot generalization to alternate “grid-like” representations, such as raw RGB video or Canny-edge sequences, enabling camera motion cloning without retraining (Liu et al., 11 Jun 2026).

7. Limitations and Prospects for Future Work

OmniDirector reliably handles 5–10 second sequences with 3–5 shots but relies on straightforward token concatenation, which impairs long-range temporal consistency for much longer videos. Anticipated advancements include integrating explicit temporal memory modules—such as long-context cross-attention or external memory banks—to extend capacity to feature-length sequences comprising hundreds of edits or cuts. Continued progress is expected in harmonizing complex cinematographic motion, subject preservation, and semantic control within a scalable, multimodal diffusion framework (Liu et al., 11 Jun 2026).