Track-Conditioned Video-to-Video Generation

• Track-conditioned video-to-video generation is defined as synthesizing videos guided by external motion tracks, ensuring physical consistency and control.
• Key methods leverage diffusion models, attention mechanisms, and variational inference to fuse structured tracking data with video representations.
• Empirical results highlight improved fidelity and trajectory adherence, yet challenges remain in scaling to complex, multi-object and 3D scenes.

Track-conditioned video-to-video generation refers to the class of generative models that synthesize videos whose content is causally and precisely guided by externally specified motion tracks, such as object trajectories, mask sequences, or 3D point correspondences. These methods address the need for controllable, physically consistent, and interaction-aware video synthesis in domains ranging from advanced scene simulation and vision data augmentation to creative content editing and robotics. Over the past two years, diffusion models and variational inference frameworks have made fine-grained trajectory conditioning feasible and robust for multi-instance and multi-modal video generation.

1. Precise Definitions and Conditioning Modalities

Track conditioning is operationalized by specifying object or region motion through structured data. Depending on the method, these data can include:

• Tracklets: Sequences of bounding boxes with class labels per object and frame (Li et al., 2023).
• Dense/Instance Masks: Per-frame, per-object binary or soft masks, uniquely identifying each tracked region (Jin et al., 8 Oct 2025).
• 3D Point Trajectories: Sparse or dense 3D world-space points (optionally paired source/target) that encode rich motion context, including depth cues (Lee et al., 1 Dec 2025).
• Coarse Trajectory Descriptors: E.g., bounding box centers, curve parametrizations, or physically plausible path tokens (Yang et al., 1 Oct 2025).
• Simulation or Flow Fields: Rendered depth, optical flow, or simulation-derived motion fields corresponding to the track data (Duan et al., 9 Oct 2025).

The conditioning signal is injected into the generative model via mechanisms tailored to the track format. These include cross-attention (tracklets or masks as tokens), explicit embedding and fusion with video tokens, or spatial masking of attention layers.

2. Key Architectures and Algorithmic Designs

Track-conditioned V2V systems fall into several architectural classes:

• Latent Diffusion Models with Instance Conditioning: TrackDiffusion (Li et al., 2023) leverages U-Net-based latent diffusion to jointly denoise latent sequences, incorporating per-object, per-frame tokens: Fourier-encoded bounding boxes, class embeddings, learnable instance IDs. Temporal instance enhancers aggregate ROI features across frames and feed them via gated cross-attention, enforcing object continuity and reducing appearance or scale drift.
• Attention-based Diffusion Transformers with Mask/Track-layer Supervision: MATRIX (Jin et al., 8 Oct 2025) extends the CogVideoX DiT backbone to concatenate first-frame mask ID maps (palette-indexed instance IDs) with the latent input. It applies mask alignment regularization on attention maps in specific "interaction-dominant" layers, explicitly linking instance trajectories with text/video attentions for semantic grounding and propagation.
• 3D Track-Conditioned Video Editors: Edit-by-Track (Lee et al., 1 Dec 2025) injects paired 3D point trajectories (source and target) into a diffusion transformer, using a 3D Track Conditioner. This module performs cross-attention from positional-encoded 3D tracks onto grid tokens, transferring context and depth cues into both source and target video representations before DiT processing.
• Variational Inference Assembly via Product of Experts: The method in "Controllable Video Synthesis via Variational Inference" (Duan et al., 9 Oct 2025) frames track and camera-conditioning as constraints imposed by separate frozen diffusion or flow models. Generation proceeds by minimizing step-wise KL divergences over an annealed sequence of distributions, using Stein Variational Gradient Descent (SVGD) to steer samples according to masked region scores (e.g., hard follow in simulated track regions, soft context elsewhere).
• Vision-LLMs for Physics-Aware Trajectory Prediction: TrajVLM-Gen (Yang et al., 1 Oct 2025) factorizes the problem into a VLM that predicts object tracks from a prompt and first frame, and a video diffusion model that conditions on these tracks using serialized track tokens and trajectory-aware attention masks.

3. Conditioning Mechanisms and Loss Formulations

The major mechanisms for injecting track information include:

• Tracklet Embedding and Self/Cross-Attention: As in TrackDiffusion, where Fourier-projected boxes and class embeddings are fused by an MLP, augmented by instance tokens, then injected throughout U-Net layers by gated self-attention (Li et al., 2023).
• Mask Alignment Regularization: MATRIX applies a composite pixel-wise loss (BCE, dice, L2) between upsampled attention maps from "interaction-dominant" layers and the ground-truth per-instance masks. Both semantic grounding (video-to-text, SGA) and propagation (video-to-video, SPA) are explicitly optimized (Jin et al., 8 Oct 2025).
• 3D-Track Cross-Attention: Edit-by-Track's conditioner module samples context from source video grids via track queries and splats it onto target video grids, with depth signals added (Lee et al., 1 Dec 2025).
• Trajectory Masked Attention: TrajVLM-Gen rasterizes trajectories onto the attention map grid, modifying attention energies and adding regularization to the diffusion objective so the object remains on track throughout the video (Yang et al., 1 Oct 2025).
• Product-of-Experts KL Minimization: In (Duan et al., 9 Oct 2025), each backbone imposes its own constraint, realized via score weighted masks in spatial/temporal regions and regularized by a cached "context prior" in under-specified areas.

The standard loss across these systems is the denoising objective of diffusion models; some additionally supervise attention maps, instance features, or mask region reconstructions.

4. Training Protocols, Datasets, and Evaluation Metrics

Training commonly proceeds in two stages:

• Stage 1: Pretrain or warm up on synthetic or easier data, building core video generation or layout control (e.g., single-image, text-driven) (Li et al., 2023, Lee et al., 1 Dec 2025).
• Stage 2: Fine-tune on real or compositional video, emphasizing track-conditioned objectives, often with explicit augmentation (e.g., perturbing tracks, simulating occlusions) (Lee et al., 1 Dec 2025, Jin et al., 8 Oct 2025).

Datasets encompass public video tracking corpora (e.g., YouTubeVIS2021, MOT-17, GOT-10k), synthetic renderings (Kubric/Mixamo), and tracking-specific collections (MATRIX-11K for multi-instance, interaction-aware mask sequences (Jin et al., 8 Oct 2025)). Pseudo-captions and tracked mask sequences are routinely generated with VLMs, segmenters, or 3D trackers.

Evaluation metrics fall into the following categories:

• Video Fidelity: Fréchet Video Distance (FVD) is ubiquitous.
• Trajectory Adherence: TrackAP (average precision on tracks), mask alignment scores (e.g., AAS in (Jin et al., 8 Oct 2025)), object success/NP rates.
• Semantic and Interaction Integrity: InterGenEval protocol in MATRIX computes KISA, SGI, and SPI for "who does what to whom" fidelity.
• Controllability vs Diversity: LPIPS on simulation regions, Camera-alignment scores, ablation-based studies (removing SVGD/context prior (Duan et al., 9 Oct 2025)).
• Human Judgment: 2AFC preference studies and QA-based integrity checks (Lee et al., 1 Dec 2025, Jin et al., 8 Oct 2025).

5. Empirical Performance, Limitations, and Ablation Insights

Empirical results across recent track-conditioned V2V works show:

• TrackDiffusion achieves TrackAP gains of +3.4–8.7 points over baselines and maintains comparable or lower FVD on YouTubeVIS and other datasets (Li et al., 2023).
• TrajVLM-Gen attains FVD improvements of 20–30% compared to prior generative baselines on UCF-101 and MSR-VTT, and demonstrates higher physics-aware trajectory following (Yang et al., 1 Oct 2025).
• Edit-by-Track delivers leading PSNR, SSIM, and LPIPS in DyCheck and in-the-wild evals, with human raters preferring its motion alignment >70% of the time (Lee et al., 1 Dec 2025).
• MATRIX attains top-1 scores on KISA, SGI, and IF among multi-instance methods, with ablations confirming that targeted regularization in "dominant" transformer layers and both SGA/SPA terms are essential (Jin et al., 8 Oct 2025).
• Variational PoE assembly enables exact trajectory and camera following while supporting diverse and high-quality completions in unconstrained regions (Duan et al., 9 Oct 2025).

Ablation studies underscore:

• Instance/track token injection and temporal consistency modules materially lift both visual and trajectory accuracy (Li et al., 2023).
• SVGD and context factorization are essential to maintain diversity and prevent drift over long generations (Duan et al., 9 Oct 2025).
• Regularizing all layers (rather than just dominant ones) or omitting mask supervision degrades output quality or interaction persistence (Jin et al., 8 Oct 2025).

Primary limitations include scale fidelity in complex scenes, lack of explicit appearance/consistency losses, absence of fine-grained physical modeling (fluid/cloth), and constraints to single-object or 2D trajectories in some systems. Mask-based regularization may struggle with diminutive or highly dynamic objects; 3D tracking can propagate noise under severe occlusions or rotation.

6. Directions for Advance and Open Challenges

Outstanding challenges and proposed research avenues include:

• Scaling to real-world, long-horizon, multi-object and 3D-trajectory conditioning for driving, robotics, or complex scene generation (Li et al., 2023, Lee et al., 1 Dec 2025).
• Integrating explicit consistency or appearance losses ($$\mathcal{L}_{\rm consistency}, \mathcal{L}_{\rm appearance}$$), physical simulation modules, or improved tracker robustness.
• Unified frameworks that flexibly blend hard track conditioning with soft context priors, balancing controllability and diversity (Duan et al., 9 Oct 2025).
• Extending mask/track-based supervision to fully 3D representation, instance-aware occlusion handling, and pose-guided generation (Yang et al., 1 Oct 2025, Lee et al., 1 Dec 2025).
• Developing safety and filtering modules for generated content (Li et al., 2023).

Track-conditioned video-to-video synthesis has advanced from coarse, single-object motion control to rich, physically plausible, interaction-aware multi-object editing. Modern methods represent an overview of structured conditioning, transformer-based diffusion backbones, and scalable regularization aligned with quantitative and qualitative metrics, establishing a robust foundation for further progress and real-world deployment.