TrajectoryMover: Generative Movement of Object Trajectories in Videos

Abstract: Generative video editing has enabled several intuitive editing operations for short video clips that would previously have been difficult to achieve, especially for non-expert editors. Existing methods focus on prescribing an object's 3D or 2D motion trajectory in a video, or on altering the appearance of an object or a scene, while preserving both the video's plausibility and identity. Yet a method to move an object's 3D motion trajectory in a video, i.e., moving an object while preserving its relative 3D motion, is currently still missing. The main challenge lies in obtaining paired video data for this scenario. Previous methods typically rely on clever data generation approaches to construct plausible paired data from unpaired videos, but this approach fails if one of the videos in a pair can not easily be constructed from the other. Instead, we introduce TrajectoryAtlas, a new data generation pipeline for large-scale synthetic paired video data and a video generator TrajectoryMover fine-tuned with this data. We show that this successfully enables generative movement of object trajectories. Project page: https://chhatrekiran.github.io/trajectorymover

• The paper introduces TrajectoryMover, a method that relocates object trajectories by integrating a synthetic data pipeline with a diffusion-based video generator.
• It employs a physics-based simulation and parameter-efficient fine-tuning to maintain foreground identity, temporal coherence, and plausible scene interactions.
• Quantitative results show state-of-the-art performance with SSIM_bg of 0.92, DINO_fg of 0.45, and IoU_traj of 0.27 on challenging synthetic video pairs.

Generative Relocation of Object Trajectories in Video: TrajectoryMover

Introduction

TrajectoryMover introduces a novel approach for generative, scene-aware relocation of an object's 3D motion trajectory within a video, controlled via simple bounding boxes in the first frame. Unlike prior approaches that require detailed, expert-specified full 2D or 3D trajectories—often an onerous burden—this framework shifts the locus of control toward intuitive editing while strictly preserving foreground identity, temporal coherence, and plausible interaction with the scene. A critical issue addressed is the inherent lack of paired training data for such direct trajectory translation, which current real and synthetic datasets fail to provide. The methodology combines a new large-scale synthetic paired video generation pipeline with deliberate fine-tuning of a diffusion-based video generator, culminating in a system that exhibits robust quantifiable advantages over state-of-the-art baselines across multiple evaluation axes.

Synthetic Data Generation Pipeline

Central to TrajectoryMover's effectiveness is the Trajectory Atlas data pipeline, which systematically constructs video pairs differing solely in the initial placement—and thus subsequent trajectory—of a single object, while holding all other scene elements constant. This is accomplished through a five-stage pipeline: asset cache preparation, preflight validation, collision-aware sampling and scaling, task-specific physics-based simulation, and canonical rendering with mask annotation. Physics simulation (via Bullet) infuses realism into object-scene interactions across a diverse taxonomy of motion types (drop, throw, roll, drag, static), while online scene modification optionally removes only non-structural obstacles in the trajectory corridor, ensuring both feasibility and diversity of object motion.

Figure 1: The data generation pipeline incorporates automated physics simulation and selective scene modification, producing instructive paired video samples for robust training.

Special attention is paid to the ambiguity of 3D placements relative to 2D bounding box constraints, particularly for air placements, by aligning target placement depths with the source via Gaussian sampling. Asset diversity (119 objects, including 98 from Objaverse and 21 geometric primitives) is critical for ensuring generalization of identity preservation. For drag motions, three parameterized planar paths (spiral, circular, S-shaped) are used to supervise motion diversity.

Figure 2: Planar drag trajectory variants visualized in 3D; these templates supervise drag motions in simulated scenes.

Model Architecture and Training Paradigm

TrajectoryMover utilizes the Wan2.1-T2V-1.3B DiT backbone. Control signals comprise source and target bounding boxes rendered as an additional input frame, fused with video token streams. The architecture concatenates target, source, and control latent streams along the temporal axis, using frame-level RoPE indexing. Fine-tuning proceeds via a multi-task regime: unconditional generation on real video data alternates with the trajectory movement task on synthetic pairs (7:3 ratio), which both preserves generative prior and imparts the new capability.

Parameter-efficient training is maximized by tuning only self-attention and projector layers, freezing the remainder of the model. The input data (video pairs and bounding boxes) are constructed to match mask-derived object locations, which are resized and temporally sampled to match backbone requirements.

Experimental Results

Quantitative Evaluation

TrajectoryMover establishes a new state-of-the-art across all major metrics relevant to trajectory-aware video editing. On a test set of 40 challenging synthetic video pairs, the system attains:

• $\text{SSIM}_\text{bg} = 0.92$, indicating near-perfect preservation of background content.
• $\text{DINO}_\text{fg} = 0.45$, the highest foreground identity consistency among all compared methods.
• $\text{IoU}_\text{traj} = 0.27$, robustly surpassing existing methods in trajectory adherence.

Competing methods, notably ATI, DaS, VACE, I2VEdit, and SFM, exhibit significant degradation in at least one critical axis, such as object identity, trajectory alignment, or scene consistency. Notably, TrajectoryMover avoids failure modes such as object fragmentation (SFM), trajectory drift (ATI, DaS), composition artifacts (VACE), and temporal fade (I2VEdit).

Figure 3: Qualitative comparison illustrates trajectory fidelity and identity preservation; pink boxes mark baseline failures, cyan boxes highlight TrajectoryMover successes.

Ablation Analysis

Systematic ablations are performed on object diversity, scene modification, and trajectory diversity components. Performance drops starkly when only primitive objects are used, confirming the necessity of a complex asset pool for identity learning. Exclusively using scene-modified or unmodified videos, or restricting to a single trajectory type (drop-only), also yields measurable performance degradation, both in detection and qualitative realism.

Figure 4: Ablation study visually demonstrates the necessity of each pipeline element for trajectory adherence and visual plausibility.

User and VLM-Based Plausibility Studies

A blinded human user study employing pairwise Bradley–Terry analysis ranks TrajectoryMover first in motion plausibility (score: 1.25 vs 0.10 for SFM, all others negative). Consistent ranking is observed with InternVL vision-LLM, corroborating the system’s superiority in object motion realism.

Figure 5: Interface for user evaluation; standard criteria applied for measuring motion plausibility and coherence.

Baseline Repurposing and Methodological Considerations

Due to a lack of directly comparable baselines for the trajectory movement task, significant engineering was invested to adapt both 2D and 3D trajectory-conditioned and video-to-video editing methods for fair comparison. Common 3D object trajectories are extracted via Video-Depth-Anything, re-anchored to target locations, and passed to the various generative pipelines (ATI, DaS, VACE, etc.) in their native control formats. The results consistently indicate that without explicit modeling of novel scene interactions, such methods fail to maintain both trajectory fidelity and physical plausibility.

Figure 6: Shared pipeline for converting source-target pairs to baseline-specific control signals, including 3D trajectory estimation and geometric re-anchoring.

Practical and Theoretical Implications

TrajectoryMover fundamentally shifts the feasibility boundary for video editors, allowing non-expert users to enact complex, physically plausible object motion edits using only bounding boxes. This removes the reliance on manual trajectory specification, significantly lowering the barrier for temporal and spatial object manipulation tasks. The synthetic data paradigm, physics-driven scene interaction modeling, and multitask fine-tuning prescribe a template for future generative video editing systems with higher-level spatial control.

Theoretically, the results raise important questions regarding the upper bounds of trajectory adherence in generative models, especially under the dual constraint of realism and identity consistency. While TrajectoryMover decisively outperforms all known baselines, the achieved peak trajectory overlap of 0.27 (IoU) suggests that tight coupling of generative priors with explicit physics simulation remains an open problem.

Future Directions

Immediate avenues for extension include adaptation to real-world videos and unpaired domains, architectural upgrades to improve strict trajectory tracking, and generalization to more nuanced object-scene manipulation tasks (e.g., dynamic scene changes, variable lighting, or interactions among multiple moving objects). Progress in large-scale synthetic dataset generation aligned with fine-grained spatio-temporal controls is also expected to drive future state-of-the-art results in generative video editing and understanding.

Conclusion

TrajectoryMover provides a robust, generalizable solution for the generative movement of object trajectories within videos, leveraging a scalable synthetic data pipeline and careful model fine-tuning. The framework achieves high scores in trajectory adherence, object identity, and plausibility, setting a new standard for the field while delineating key challenges and opportunities for future research in controllable video generation (2603.29092).