E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness

Abstract: Stories and emotions in movies emerge through the effect of well-thought-out directing decisions, in particular camera placement and movement over time. Crafting compelling camera trajectories remains a complex iterative process, even for skilful artists. To tackle this, in this paper, we propose a dataset called the Exceptional Trajectories (E.T.) with camera trajectories along with character information and textual captions encompassing descriptions of both camera and character. To our knowledge, this is the first dataset of its kind. To show the potential applications of the E.T. dataset, we propose a diffusion-based approach, named DIRECTOR, which generates complex camera trajectories from textual captions that describe the relation and synchronisation between the camera and characters. To ensure robust and accurate evaluations, we train on the E.T. dataset CLaTr, a Contrastive Language-Trajectory embedding for evaluation metrics. We posit that our proposed dataset and method significantly advance the democratization of cinematography, making it more accessible to common users.

• The paper introduces a novel diffusion-based model (Director) that automates text-to-camera trajectory generation with integrated character movement analysis.
• The methodology leverages the E.T. dataset, featuring over 11 million frames and 120 hours of footage, to produce accurate, context-aware trajectories.
• Evaluation results show Director, especially its third variant, outperforms existing methods on key metrics like FID and precision, advancing automated cinematography.

E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness

In the paper "E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness," the authors introduce a novel dataset and method for generating camera trajectories from textual descriptions while being aware of character movements. The complexities of modern cinematography necessitate precise control over camera trajectories to convey directors’ artistic intentions effectively. However, the manual creation of such camera paths remains an intricate and iterative process, even for experienced professionals. This research seeks to democratize cinematographic capabilities by automating the generation of complex camera trajectories through text descriptions linked to character behaviors.

Dataset: Exceptional Trajectories (E.T.)

A key contribution of the paper is the E.T. dataset, which is unique in its combination of camera trajectories, character movements, and corresponding textual annotations. This dataset is extracted from real movie clips, making it highly representative of practical scenarios in cinematography. Specifically, the dataset encompasses over 11 million frames and 120 hours of footage, segmented into distinct shots with both camera and character trajectories. Each segment in the dataset also includes textual descriptions that detail the camera and character movements, further enhancing its utility for training machine learning models.

The construction of the E.T. dataset involves three core steps:

1. Extraction and pre-processing of camera and character poses, including alignment and smoothing to address noise and discontinuities in the raw data.
2. Motion tagging, which partitions trajectories into segments defined by pure camera motions along six degrees of freedom.
3. Caption generation, wherein a LLM is prompted to translate the segmented motion tags into detailed textual descriptions.

These steps ensure that the dataset is robust, varied, and enriched with detailed descriptions that accurately reflect cinematic scenes.

Methodology: Director and CLaTr

The paper introduces Director (DiffusIon tRansformEr Camera TrajectORy) as a diffusion-based model capable of generating camera trajectories from text descriptions. Director leverages the E.T. dataset to learn the correlation between camera movements and character trajectories, employing a classical diffusion framework with three distinct architectures for conditional input handling.

The three architectures explored are:

• Director A: Incorporates conditioning as in-context tokens.
• Director B: Utilizes AdaLN modulation within the transformer blocks for conditioning.
• Director C: Applies cross-attention mechanisms to leverage the full text and character trajectory sequences.

Additionally, the paper introduces CLaTr (Contrastive Language-Trajectory embedding), a robust mechanism for evaluating camera trajectory generation models. CLaTr embeds text and trajectories into a shared feature space, akin to CLIP embeddings, enabling effective computation of generative metrics such as Frechet-Inception-Distance (FID) for generated trajectories.

Experimental Results

Quantitative assessments on the E.T. dataset demonstrate that Director, particularly its third variant (Director C), significantly outperforms existing methods like CCD and MDM in terms of trajectory quality and text-camera coherence. Metrics such as FD\textsubscript{CLaTr}, CLaTr-Score, Precision, Recall, and Coverage indicate that Director's generated trajectories are more accurate and better aligned with the textual descriptions than those produced by prior methods.

Implications and Future Directions

The implications of this research are multi-faceted:

1. Practical Application: The ability to generate accurate camera trajectories from text simplifies the production process in filmmaking, potentially lowering the barriers for novice creators and small studios.
2. Theoretical Contribution: The proposed methodologies advance the field of text-to-trajectory generation, setting a new benchmark for future studies.
3. Dataset Utility: The E.T. dataset provides a rich resource for further research in cinematographic trajectory generation, laying the groundwork for more refined models.

Future developments in AI could focus on enhancing the expressiveness of trajectory captions, incorporating more nuanced information about character positions and scene context. Additionally, the exploration of multi-modal integration of audio and text for even richer trajectory descriptions could be a promising direction.

Conclusion

"E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness" provides substantial advancements in the field of automated cinematography. The introduction of the E.T. dataset and the Director model sets a new standard for the generation of camera trajectories from textual descriptions, emphasizing the crucial role of character awareness in these tasks. The robust evaluation framework offered by CLaTr further solidifies the paper's contributions, paving the way for democratizing access to sophisticated cinematographic tools.