OmniDirector: General Multi-Shot Camera Cloning without Cross-Paired Data

Abstract: Cloning camera motion from reference videos is an important task in video generation, as videos provide intuitive and precise control. Existing methods either directly use parametric representations that fail to handle multi-shot generation or synthesize cross-paired data, which suffer from data scarcity, resulting in poor performance in complicated camera motion cloning. To address these issues, we introduce a general camera motion representation that encodes cameras as grid motion videos. This camera grid represents the camera parameters visually and supports the integration of diverse trajectories for multi-shot video generation. Building upon this, we propose OmniDirector, a unified framework trained on a million-scale camera grid-video pairs that coordinates characters, actions, and cameras to provide director-level control for multimodal diffusion transformers. Furthermore, we design a novel hierarchical prompt expansion agent that harmoniously integrates different control signals by systematically describing camera motion and visual content through understanding signal relationships. Extensive experiments demonstrate the superior performance and outstanding controllability of our framework. Project page: https://ymlinfeng.github.io/OmniDirector.github.io/

• The paper introduces a novel camera grid representation that decouples geometric control from appearance, enabling robust multi-shot video generation without paired data.
• It employs a Multi-Modal Diffusion Transformer with hierarchical prompt expansion to integrate heterogeneous control signals across inter-shot and intra-shot transitions.
• Empirical results demonstrate significant improvements in translation, rotation, and transition precision, paving the way for scalable, director-level video synthesis.

OmniDirector: A Unified Framework for General Multi-Shot Camera Cloning without Cross-Paired Data

Introduction and Motivation

The paper "OmniDirector: General Multi-Shot Camera Cloning without Cross-Paired Data" (2606.13432) presents a unified and scalable approach to video generation with fine-grained camera motion control. Conventionally, camera-controllable video generation relies on either explicit parameter injection methods, which are effective only for basic camera movements and single-shot scenarios, or cross-paired data paradigms, which are severely constrained by data scarcity and complexity. The present work introduces the camera grid: a general, visually interpretable, and scalable camera motion representation that decouples viewpoint dynamics from video content, enabling large-scale training for robust, multi-shot camera cloning.

Camera Grid: A Visual Proxy for Camera Motion

OmniDirector's central innovation is the camera grid representation. Camera motion is abstracted by extracting camera extrinsic parameters from reference videos and rendering them as trajectories through a minimalistic 3D grid scene—a room defined only by floor, ceiling, and wall lines (Figure 1). This abstracts away appearance and scene semantics, allowing the model to focus entirely on learning the geometry and spatiotemporal evolution of camera poses.

Figure 1: 3D scene grid visualizations encode the camera's trajectory as rendered from different viewpoints, serving as control signals for the generator.

Notably, the grid accommodates complex multi-shot videos by leveraging automated shot transition detectors and pose estimators to segment the video and render distinct sub-trajectories. To extend to cinematic effects, the rendering pipeline is augmented with non-linear projection (e.g., fisheye) and composite geometric cues (e.g., dolly zoom), ensuring support for diverse shooting techniques (Figure 2).

Figure 2: The camera grid flexibly encodes special effects such as fisheye lens distortion and dolly zoom by altering its rendering scheme.

This purely visual grid aligns naturally with latent representations of RGB videos, enabling seamless concatenation and fusion within transformer-based generative architectures.

Architecture and Training Paradigm

The core generative model is based on Multi-Modal Diffusion Transformers (MMDiTs). The workflow is as follows (Figure 3):

• Input Fusion: The camera grid, reference image, and target video latents are concatenated along the temporal axis after 3D-VAE encoding. These unified visual tokens are then processed with textual tokens in separate but interleaved attention streams.
• Token Interactions: Joint attention blocks mediate fine-grained cross-modal information exchange per transformer layer, allowing textual conditioning to modulate visual generation and vice versa.
• Training Strategy: The dataset is constructed at scale (>1.8M samples) by pairing internet videos with their auto-extracted camera grids, avoiding the need for curated, cross-paired training sets. A regularized self-reconstruction loss on grid inputs further enforces the model's geometric understanding by requiring the model to reproduce grid sequences in the absence of appearance signals.

  Figure 3: Architecture overview—camera grid extraction, token concatenation for MMDiT training, and hierarchical control signal fusion at inference.

Hierarchical Prompt Expansion for Multi-Signal Control

At inference, OmniDirector integrates heterogeneous control signals using a Hierarchical Prompt Expansion Agent. Initial camera prompts are decomposed hierarchically:

• Inter-shot Level: Captures scene and motion transitions across shots.
• Intra-shot Level: Encodes local camera trajectory semantics within each shot, inferring direction, speed, and type (e.g., arc, pan) from pose increments.

The prompt expansion is refined via MLLM (Qwen3-VL), which integrates visual cues (reference videos, images) and pose estimates, augmented by manual annotation for semantic accuracy. Final control signals, covering camera, objects, and user intent, are fused via a multimodal LLM into a cohesive textual prompt that guides the model generation coherently.

Empirical Results

Camera and Transition Accuracy

Extensive quantitative and qualitative evaluations demonstrate that OmniDirector achieves state-of-the-art accuracy on all major camera control metrics:

• Translation Precision (T-Pre): 72.74% (relative improvement over baseline by 39.3%)
• Rotation Precision (R-Pre): 83.18%
• Transition Temporal/semantic Precision: 96.52%/83.79%
• Reference Content Leakage: Significantly lower than all baselines (0.51% frame-level)

OmniDirector successfully transfers multi-shot camera trajectories, achieving robust geometric alignment and semantic transition matching not observed in parameter-injection methods or in prior implicit cross-paired data models. Visual results further confirm the precise and content-agnostic camera cloning across diverse domains, aspect ratios, and visual contents (Figure 4).

Figure 4: Qualitative samples showing highly accurate cloning of camera dynamics and shot transitions from reference videos.

Ablation Analysis and Emergent Properties

Ablation studies confirm the crucial role of the hierarchical prompt expansion and adaptive classifier-free guidance for mitigating signal interference and boosting performance (Figure 5). Of particular note is the model's emergent camera understanding: substituting the grid input with raw videos or Canny edge sequences at inference produces plausible camera motion effects—indicating strong cross-modal transfer and generalization without retraining (Figure 6).

Figure 5: Visualization of ablation experiments—highlighting the contributions of semantic fusion and adaptive guidance.

Figure 6: OmniDirector exhibits zero-shot camera control when conditioned on unrelated visual modalities such as raw RGB or Canny edge sequences.

Discussion and Implications

The camera grid paradigm eliminates the dependence on scarce paired datasets by decoupling geometric control from appearance, contributing a data-efficient and generalizable framework for camera-controllable video generation. The scalability of training (massive auto-labeled videos) and the flexibility to represent advanced cinematic movements position OmniDirector as a practical solution for both research and production settings.

The explicit fusion of multimodal signals using hierarchical prompt engineering aligns with trends in multimodal LLM integration, suggesting a path toward more holistic, user-driven video generative systems. The observed zero-shot generalization to unseen visual inputs highlights the potential for broader cross-modal controllability and indicates that camera motion representations may generalize robustly across related input modalities.

Limitations and Future Directions

The current token concatenation-based fusion exhibits limitations in long-term consistency for extended video sequences. Future extensions may integrate specialized temporal memory architectures (e.g., long-context cross-attention, memory banks) to address these challenges and enable persistent, coherent generation in longer or more complex video compositions.

Conclusion

OmniDirector introduces a highly generalizable, scalable, and effective paradigm for camera-controllable video generation by proposing the camera grid representation and training regime independent of cross-paired data. Its hierarchical prompt fusion mechanism and emergent cross-modal camera understanding mark significant progress toward practical, director-level video generative modeling. The demonstrated robustness across shots, scenes, and cinematic effects underscores the viability of purely visual camera signal representations in diffusion transformer frameworks.