TrajLoc: Trajectory-Attention Localization for Multi-Object Motion Control

Abstract: Controlling the motion of multiple objects in image-to-video (I2V) generation requires preserving object identities while enforcing adherence to distinct target trajectories. This becomes particularly challenging as the number of objects increases and their paths intersect or occlude one another. Existing approaches entangle multiple trajectories within a shared, dense conditioning signal, making object-level correspondence difficult to preserve in crowded scenes. We depart from this paradigm and enforce a strict, per object spatial constraint that isolates instances independently. Our method, TrajLoc, achieves this directly within the attention layers by substituting the cross-attention weights of each object token with a Gaussian heatmap centered on its target location at every frame. The same per object token interface carries trajectory and depth through a learned embedding and preserves identity by encoding first frame appearance in place of an object token. Evaluations across six datasets, featuring up to 20 simultaneously controlled objects and out of distribution real world scenes, demonstrate that our method consistently improves both visual fidelity and trajectory adherence. Applied to two architecturally distinct backbones (CogVideoX 5B and WaN 2.1 14B), our approach achieves average gains of +4.3 dB PSNR and a 51% reduction in trajectory end point error compared to the strongest baselines. Project page: https://sela-omer.github.io/traj-loc/

• The paper presents a novel trajectory-attention method that replaces traditional cross-attention weights with frame-wise Gaussian heatmaps to enforce spatial constraints.
• The approach achieves state-of-the-art results, with an average +4.3 dB PSNR improvement and 51% reduction in end-point error across both synthetic and real-world datasets.
• The method decouples motion and appearance control, proving scalable and robust for multi-object video synthesis even in crowded and occluded scenes.

TrajLoc: Trajectory-Attention Localization for Multi-Object Motion Control

Motivation and Problem Formulation

Multi-object motion control in image-to-video (I2V) diffusion models is a persistent challenge due to the need for precise object trajectory adherence and identity preservation, particularly as object count and motion complexity increase. Recent baseline approaches accomplish trajectory conditioning by fusing dense motion representations for all objects into a shared, high-dimensional tensor. This design results in entangled object signals, impaired scalability, and difficulty in preserving correspondence under occlusions or trajectory crossings, restricting practical effectiveness to scenarios with very few objects.

Method: Trajectory-Attention Localization

TrajLoc introduces a fundamentally different approach, predicated on the observation that sparse, per-object trajectory signals are semantically adequate for fine-grained multi-object control if integrated directly into the attention space of the diffusion model. TrajLoc replaces the cross-attention weights (normally computed by the model) for each object token with frame-wise Gaussian heatmaps, each centered at the current target location of the corresponding trajectory. This modification constitutes a hard, per-object spatial constraint and does not require dense, video-resolution conditioning volumes or additional control modules.

Figure 2: An overview of TrajLoc, illustrating structured prompt composition, embedding injection, and attention localization via Gaussian heatmaps at object token positions.

Token-based Conditioning is further leveraged by encoding additional object-specific information: dedicated trajectory tokens carry the temporal motion and relative depth information, while appearance tokens encode the first-frame visual identity, directly replacing the category embedding in the prompt. Both representations are learned through pretraining pipelines that ensure the trajectory embedding remains informative after passage through the text encoder, and that visual features extracted from the initial frame can strongly constrain instance-specific appearance throughout the generated sequence.

This architecture, notable for its absence of new trainable parameters or dense tensors in the attention replacement mechanism, is agnostic to the underlying cross-attention structure and is demonstrated on two state-of-the-art backbones: CogVideoX-5B (joint self-attention) and WaN 2.1-14B (explicit cross-attention).

Quantitative Results

TrajLoc’s performance is evaluated across six datasets, including in-distribution synthetic object scenes (MoVi-Extended, Pool, Football, MOTSynth) and out-of-distribution real-world settings (MOT17, DAVIS). The method is compared to four competitive baselines—ATI, Wan-Move, Tora, and MagicMotion—each relying on dense trajectory or motion conditioning. Four metrics benchmark outcomes: PSNR, LPIPS, FVD, and end-point error (EPE) measuring trajectory fidelity.

Across all datasets and both backbone classes, TrajLoc achieves state-of-the-art performance, consistently improving both pixel-wise reconstruction (average +4.3 dB PSNR) and trajectory adherence (average 51% EPE reduction), with the strongest differences on crowded, highly interactive scenes. It generalizes effectively from synthetic training data to real-world domains, outperforming all real-data-trained baselines even on OOD samples (MOT17 and DAVIS).

Ablation studies demonstrate that attention localization is the critical component, with performance degrading most when it is removed. Trajectory and appearance tokens offer complementary, but secondary, gains. Removal of depth information results in additional, task-dependent degradation, highlighting the importance of complete trajectory/context information in high-complexity scenes.

Qualitative Analysis

Qualitative comparisons reveal TrajLoc’s ability to:

• Precisely place and persistently track all objects along their assigned trajectories, even with up to 20 entities per scene.
• Preserve visual identity and depth layering across frames, while preventing the drift, duplication, or hallucination often exhibited by dense-tensor baselines under heavy occlusion or trajectory interaction.
• Maintain robust performance in OOD and natural scenes, despite exclusive synthetic-domain training.

Key Innovations and Claims

• Sparse per-object trajectory signals, if imposed directly in the cross-attention mechanism, fully suffice for high-precision, large-scale multi-object video control.
• No dense control tensors are required; the method is portable, lightweight, and model-agnostic.
• Scalability is achieved not merely in principle, but empirically—motion control degrades minimally as scene complexity increases, unlike all prior baselines.
• Generalization from synthetic to real-world data is effective, challenging assumptions about the need for data realism in trajectory-conditioned video diffusion training.

Theoretical and Practical Implications

The demonstrated sufficiency of direct per-object attention editing to enforce spatial motion constraints reorients the design space for controllable diffusion. TrajLoc effectively disentangles trajectory control from visual feature entanglement and removes a major scalability bottleneck—previously, maintaining object correspondence in crowded, interactive settings was not possible without specialized architectural modules and additional domain-specific supervision.

Practically, TrajLoc offers a path for fine-grained video content synthesis in applications such as simulation (e.g., for autonomous driving or robotic policy learning), professional post-production video editing, and any domain where synthetic video with precise, controllable object motion is required.

Theoretical implications include the potential for further abstraction, e.g., extending beyond 2D spatial Gaussians to higher-dimensional or adaptive attention forms, or combining spatial control with additional modalities (semantic, action, etc.) in a similarly sparse and decomposed fashion. TrajLoc’s minimal architectural dependence suggests opportunities for rapid adoption into emerging, larger text-to-video foundation models.

Limitations and Future Directions

Current limitations relate primarily to the static-camera and medium-resolution constraints of publicly available multi-object trajectory datasets, as well as residual distribution leakage when the model is forced outside its training modalities. Extending the paradigm to dynamic camera motion, higher-resolution synthesis, and larger linguistic action spaces represent clear next steps.

Promising future directions include:

• Integration of camera-control or full 3D-awareness for open-world synthetic video generation.
• Broader control, e.g., combining with style, interaction, or scene composition tokens for multi-level synthesis.
• Large-scale, real-world trajectory dataset curation to further improve generalization and realism.

Conclusion

TrajLoc establishes that object-level trajectory control in image-to-video diffusion is more efficiently and effectively achieved with sparse, direct attention localization and token-level identity embeddings. By decoupling motion and appearance control from dense tensors, TrajLoc enables scalable, precise, and generalizable video synthesis. This methodology challenges central assumptions in motion-controllable generation, opening pathways for highly scalable and interpretable control in video diffusion models (2607.00861).