DiTraj: Training-Free Trajectory Control

• DiTraj is a training-free framework for trajectory control in text-to-video generation that combines LLM-based foreground-background decomposition with positional embedding adjustments.
• The framework introduces Spatial-Temporal Decoupled 3D-RoPE (STD-RoPE) to align spatial embeddings across frames, enhancing cross-frame attention and trajectory adherence.
• Empirical evaluations on Wan2.1 and CogVideoX demonstrate improved trajectory controllability and video quality, validating its efficiency and generalizability.

DiTraj is a training-free framework for trajectory control in text-to-video generation with Diffusion Transformers (DiTs). It is tailored to DiT-based video generation models with 3D full attention and targets a specific controllable generation setting in which a user specifies an object trajectory, typically as bounding boxes across frames. The framework combines foreground-background separation guidance with Spatial-Temporal Decoupled 3D-RoPE (STD-RoPE), modifying only inference-time conditioning and position embedding behavior rather than requiring model retraining, inversion, or optimization. In reported experiments, DiTraj improves both trajectory controllability and video quality on DiT-based backbones such as Wan2.1 and CogVideoX (Lei et al., 26 Sep 2025).

1. Problem setting and design objective

Trajectory control in video generation is a user-friendly task in controllable video generation because it allows motion to be specified directly, for example through per-frame bounding boxes. DiTraj is motivated by two limitations identified for prior methods. First, training-based approaches require supervised datasets or model fine-tuning. Second, existing training-free approaches are generally designed for U-Net architectures and therefore do not exploit the stronger generative capabilities of DiT-based video models with 3D full attention (Lei et al., 26 Sep 2025).

The framework is designed for text-to-video generation under explicit motion constraints. Its objective is training-free, efficient, and precise trajectory control in DiT-based video generation, while avoiding both retraining and computationally expensive inversion or optimization. A central premise is that DiT’s 3D full attention is intrinsically better suited than separated spatial and temporal attention for handling spatial-temporal dependencies, especially when objects undergo large motion. This suggests that controllability can be improved not only by modifying text conditioning, but also by intervening directly in the positional structure that shapes inter-frame token interactions.

DiTraj belongs to a broader trajectory-oriented diffusion literature, but its task is distinct. FreeTraj studies tuning-free trajectory control in video diffusion models through noise construction and attention guidance (Qiu et al., 2024). DiffTraj addresses GPS trajectory synthesis with a diffusion probabilistic model (Zhu et al., 2023). RI-DiT addresses long-tail trajectory prediction in crash scenarios for autonomous driving (Chen et al., 18 Jan 2025). DiTraj, by contrast, focuses on controllable video generation in DiT-based text-to-video systems (Lei et al., 26 Sep 2025).

2. Framework composition and inference pipeline

DiTraj is organized around two components: Foreground-Background Separation Guidance and STD-RoPE. The pipeline is applicable to most DiT-based generative video models and introduces about 5% additional inference overhead (Lei et al., 26 Sep 2025).

Component	Mechanism	Intended effect
Foreground-Background Separation Guidance	LLM-based prompt partition plus cross-attention masks	Separates subject and context control
STD-RoPE	Foreground-only position embedding modification	Strengthens cross-frame attention for trajectory adherence
3D-aware trajectory control	Position embedding density regulation	Enables depth-aware control via box-size variation

At a high level, the user provides an original prompt $\mathcal{P}_{ori}$ and a trajectory $\mathbb{T}$. The prompt is decomposed into foreground and background descriptions, which are then encoded separately and recombined into a union condition embedding. During denoising, the model uses masked cross-attention so that foreground tokens align with foreground text while background tokens align with background text. In parallel, STD-RoPE alters the spatial position embeddings of foreground tokens across frames, while leaving their temporal coordinate unchanged. This strengthens inter-frame attention among object tokens and improves control, particularly for large movements.

The overall pipeline therefore operates by coupling semantic separation with positional alignment. The first mechanism enforces content allocation, while the second alters the geometry of attention. A plausible implication is that DiTraj treats trajectory control as a joint conditioning problem over language and token topology, rather than as a purely prompt-engineering or purely attention-biasing task.

3. Foreground-background separation guidance

Foreground-background separation guidance is the mechanism used to inject an object trajectory into generation. The user provides the original prompt $\mathcal{P}_{ori}$ and a bounding box trajectory $\mathbb{T}$. A LLM, with Qwen3 given as an example, converts the original prompt into a foreground prompt $\mathcal{P}_{fg}$ that describes only the main subject and a background prompt $\mathcal{P}_{bg}$ that describes only the background context (Lei et al., 26 Sep 2025):

$$\mathcal{P}_{fg}, \mathcal{P}_{bg} = LLM(\mathcal{P}_{ori})$$

Both prompts are encoded by the text encoder $\mathcal{E}_{text}$ and concatenated into a union condition embedding:

$$C^u = \text{Concatenate}(\mathcal{E}_{text}(\mathcal{P}_{fg}), \mathcal{E}_{text}(\mathcal{P}_{bg}))$$

A cross-attention mask $\mathbf{M}^{cross}$ is then constructed so that foreground video tokens, defined by the set $\mathbb{T}$0, attend only to the foreground prompt, while non-foreground tokens attend only to the background prompt:

$\mathbb{T}$1

This mask is used in cross-attention as

$\mathbb{T}$2

The separated guidance is applied during the first $\mathbb{T}$3 denoising steps. In the reported characterization, this mechanism enforces the object to be generated according to the trajectory, but is most effective for small movements. This limitation is important because it motivates the second major component of DiTraj. Semantic separation alone is insufficient when the target object moves far across frames, since the dominant obstacle then lies in how DiT attention couples position embeddings and inter-frame token matching.

4. Attention analysis and Spatial-Temporal Decoupled 3D-RoPE

DiTraj includes an explicit analysis of 3D full attention and 3D-RoPE. In DiT-based video generation, video is processed as a sequence of tokens spanning spatial and temporal dimensions after 3D-VAE compression and patchification. Each token’s position is represented by three coordinates, $\mathbb{T}$4, and 1D RoPE is applied independently to $\mathbb{T}$5, $\mathbb{T}$6, and $\mathbb{T}$7, with the resulting embeddings concatenated (Lei et al., 26 Sep 2025).

The key empirical observation is that attention maps between tokens of different frames are strongly focused along the diagonal: tokens attend mostly to others sharing the same spatial location across frames. The paper interprets this as evidence that attention scores are tightly correlated with the similarity of position embeddings. The practical consequence is that when a foreground object follows a large-movement trajectory, the tokens at its bounding-box locations across frames have dissimilar position embeddings, which leads to low cross-frame attention and weak trajectory adherence.

STD-RoPE is introduced to address this problem. Its core operation is to align the spatial position embeddings of foreground tokens across all frames while preserving temporal position embeddings. An anchor frame $\mathbb{T}$8 is selected at random from the $\mathbb{T}$9 trajectory frames, and for every frame $\mathcal{P}_{ori}$0, the spatial embeddings inside the bounding box $\mathcal{P}_{ori}$1 are replaced by those from the same relative position in the anchor box $\mathcal{P}_{ori}$2. Formally, the method splits each token’s 3D-RoPE into spatial and temporal parts, sets

$\mathcal{P}_{ori}$3

for all $\mathcal{P}_{ori}$4, and then reconstructs

$\mathcal{P}_{ori}$5

STD-RoPE is applied only during the early denoising steps, specifically the first $\mathcal{P}_{ori}$6 steps, after which normal position embeddings are restored. The effect is that foreground tokens become mutually close in embedding space across frames, greatly increasing their cross-attention score and improving representation propagation over the whole trajectory.

The method also addresses artifacts caused by repeated position embeddings, referred to as R-tokens. Since spatial alignment can create tokens with identical position embeddings, ambiguity can arise. DiTraj resolves this with a self-attention mask $\mathcal{P}_{ori}$7 that blocks attention between R-tokens and true foreground tokens:

$\mathcal{P}_{ori}$8

This combination of foreground-only positional alignment and selective masking is the DiT-specific core of DiTraj. It differs from methods that intervene primarily through noise or generic attention bias, because it targets the positional substrate that shapes 3D full attention.

5. 3D-aware trajectory control

DiTraj extends 2D trajectory control to 3D-aware control by regulating the density of position embeddings within variable-size bounding boxes. The intended use case is a trajectory in which bounding-box size changes across frames, such as a box growing when an object approaches the camera. The smallest bounding box is chosen as the anchor, and its spatial embeddings are upsampled by nearest-neighbor to match the size of each target frame’s box. For frame $\mathcal{P}_{ori}$9, the upsampled anchor embedding is assigned to $\mathbb{T}$0 (Lei et al., 26 Sep 2025).

The reported interpretation is that denser embeddings correspond to an object closer to the camera, so controlling bounding-box size enables 3D depth trajectory specification. This is not merely an auxiliary refinement of the 2D setting. It reuses the same position-embedding intervention principle but adapts it to scale variation, thereby coupling apparent depth motion to positional encoding density rather than to an explicit geometric module.

This suggests that DiTraj treats depth-aware motion as a continuation of the same representational issue seen in large 2D movement: the controllability bottleneck arises from how foreground tokens are positioned in the model’s embedding space. By manipulating density rather than only location, the method adds a simple mechanism for scale change without introducing new training objectives or external geometry estimators.

6. Empirical evaluation, ablations, and relation to adjacent methods

DiTraj is evaluated on Wan2.1 and CogVideoX against both training-based and training-free state-of-the-art methods. The evaluation uses video quality metrics from VBench—Subject Consistency (SC), Background Consistency (BC), Motion Smoothness (MS), Aesthetic Quality (AQ), and Imaging Quality (IQ)—together with trajectory-control metrics Coverage (Cov), mean IoU (mIoU), Center Distance (CD; lower is better), and AP50 (Lei et al., 26 Sep 2025).

In the quantitative comparison highlighted from Table 1, DiTraj with Wan2.1 achieves an mIoU of 47.3 versus 40.8 for the next best method, a CD of 0.09 versus 0.11, and an AP50 of 50.5 versus 49.1. With the CogVideoX backbone, the method still achieves state-of-the-art or close performance, which is presented as evidence of generalizability. In the user study summarized in Table 2, DiTraj receives over 60% of human preference votes in trajectory alignment, video-text alignment, and video quality, while the next best method remains below 13%.

The ablation results separate the contributions of the two main components. Using only foreground-background guidance improves control somewhat. Adding STD-RoPE further boosts trajectory adherence, including mIoU and AP50, without significant loss in video realism. Qualitative observations reported for the method state that it faithfully follows trajectories across large spatial displacements while preserving object appearance and background consistency, whereas baselines may lose control, generate artifacts, or fail to maintain appearance.

The most direct training-free comparison in the provided literature is FreeTraj, which controls trajectories by modifying noise sampling and attention mechanisms in video diffusion models and reports mIoU $\mathbb{T}$1 and CD $\mathbb{T}$2 against Peekaboo and TrailBlazer in its own evaluation setting (Qiu et al., 2024). DiTraj differs in both architectural target and control mechanism: it is tailored for DiT, emphasizes foreground-background prompt separation, and leverages inter-frame positional alignment through STD-RoPE rather than noise construction (Lei et al., 26 Sep 2025). This distinction matters because DiTraj’s central claim is not simply that training-free control is possible, but that DiT-specific full-attention structure can be exploited directly for more precise control.

A common misconception is to treat DiTraj as another generic diffusion-trajectory method. In the diffusion literature, similarly named systems address different tasks: DiffTraj generates privacy-preserving GPS trajectories (Zhu et al., 2023), while RI-DiT predicts long-tail driving trajectories in crash scenarios (Chen et al., 18 Jan 2025). DiTraj instead belongs to controllable text-to-video generation. Its technical contribution is the claim that trajectory adherence in DiT-based video synthesis can be improved by combining LLM-mediated semantic decomposition with inference-time manipulation of foreground position embeddings (Lei et al., 26 Sep 2025).