---
title: 'DiTraj: Training-Free Trajectory Control'
url: https://www.emergentmind.com/topics/ditraj
type: topic
---

# DiTraj: Training-Free Trajectory Control

DiTraj is a training-free framework for trajectory control in text-to-video generation with Diffusion Transformers (DiTs). It is tailored to DiT-based video generation models with 3D full attention and targets a specific controllable generation setting in which a user specifies an object trajectory, typically as bounding boxes across frames. The framework combines foreground-background separation guidance with Spatial-Temporal Decoupled 3D-RoPE (STD-RoPE), modifying only inference-time conditioning and position embedding behavior rather than requiring model retraining, inversion, or optimization. In reported experiments, DiTraj improves both trajectory controllability and video quality on DiT-based backbones such as Wan2.1 and CogVideoX [2509.21839].

## 1. Problem setting and design objective

Trajectory control in video generation is a user-friendly task in controllable video generation because it allows motion to be specified directly, for example through per-frame bounding boxes. DiTraj is motivated by two limitations identified for prior methods. First, training-based approaches require supervised datasets or model fine-tuning. Second, existing training-free approaches are generally designed for U-Net architectures and therefore do not exploit the stronger generative capabilities of DiT-based video models with 3D full attention [2509.21839].

The framework is designed for text-to-video generation under explicit motion constraints. Its objective is training-free, efficient, and precise trajectory control in DiT-based video generation, while avoiding both retraining and computationally expensive inversion or optimization. A central premise is that DiT’s 3D full attention is intrinsically better suited than separated spatial and temporal attention for handling spatial-temporal dependencies, especially when objects undergo large motion. This suggests that controllability can be improved not only by modifying text conditioning, but also by intervening directly in the positional structure that shapes inter-frame token interactions.

DiTraj belongs to a broader trajectory-oriented diffusion literature, but its task is distinct. FreeTraj studies tuning-free trajectory control in video diffusion models through noise construction and attention guidance [2406.16863]. DiffTraj addresses GPS trajectory synthesis with a diffusion probabilistic model [2304.11582]. RI-DiT addresses long-tail trajectory prediction in crash scenarios for autonomous driving [2501.16349]. DiTraj, by contrast, focuses on controllable video generation in DiT-based text-to-video systems [2509.21839].

## 2. Framework composition and inference pipeline

DiTraj is organized around two components: Foreground-Background Separation Guidance and STD-RoPE. The pipeline is applicable to most DiT-based generative video models and introduces about 5% additional inference overhead [2509.21839].

| Component | Mechanism | Intended effect |
|---|---|---|
| Foreground-Background Separation Guidance | LLM-based prompt partition plus cross-attention masks | Separates subject and context control |
| STD-RoPE | Foreground-only position embedding modification | Strengthens cross-frame attention for trajectory adherence |
| 3D-aware trajectory control | Position embedding density regulation | Enables depth-aware control via box-size variation |

At a high level, the user provides an original prompt $\mathcal{P}_{ori}$ and a trajectory $\mathbb{T}$. The prompt is decomposed into foreground and background descriptions, which are then encoded separately and recombined into a union condition embedding. During denoising, the model uses masked cross-attention so that foreground tokens align with foreground text while background tokens align with background text. In parallel, STD-RoPE alters the spatial position embeddings of foreground tokens across frames, while leaving their temporal coordinate unchanged. This strengthens inter-frame attention among object tokens and improves control, particularly for large movements.

The overall pipeline therefore operates by coupling semantic separation with positional alignment. The first mechanism enforces content allocation, while the second alters the geometry of attention. A plausible implication is that DiTraj treats trajectory control as a joint conditioning problem over language and token topology, rather than as a purely prompt-engineering or purely attention-biasing task.

## 3. Foreground-background separation guidance

Foreground-background separation guidance is the mechanism used to inject an object trajectory into generation. The user provides the original prompt $\mathcal{P}_{ori}$ and a bounding box trajectory $\mathbb{T}$. A Large Language Model, with Qwen3 given as an example, converts the original prompt into a foreground prompt $\mathcal{P}_{fg}$ that describes only the main subject and a background prompt $\mathcal{P}_{bg}$ that describes only the background context [2509.21839]:

$$
\mathcal{P}_{fg}, \mathcal{P}_{bg} = LLM(\mathcal{P}_{ori})
$$

Both prompts are encoded by the text encoder $\mathcal{E}_{text}$ and concatenated into a union condition embedding:

$$
C^u = \text{Concatenate}(\mathcal{E}_{text}(\mathcal{P}_{fg}), \mathcal{E}_{text}(\mathcal{P}_{bg}))
$$

A cross-attention mask $\mathbf{M}^{cross}$ is then constructed so that foreground video tokens, defined by the set $\mathbb{S}_{fg} = \{ i : x_i \in \mathbb{T} \}$, attend only to the foreground prompt, while non-foreground tokens attend only to the background prompt:

$$
\mathbf{M}^{cross}_{i,j} =
\begin{cases}
0 & i \in \mathbb{S}_{fg}, C^u_j \in \mathcal{E}_{text}(\mathcal{P}_{fg}) \\
0 & i \notin \mathbb{S}_{fg}, C^u_j \in \mathcal{E}_{text}(\mathcal{P}_{bg}) \\
-\infty & \text{otherwise}
\end{cases}
$$

This mask is used in cross-attention as

$$
\text{CrossAttention}(x, C^u, \mathbf{M}^{cross}) =
\text{softmax}\left( \frac{Q(x)K^T(C^u)}{\sqrt{D} + \mathbf{M}^{cross}} \right)\cdot V(C^u)
$$

The separated guidance is applied during the first $t_a$ denoising steps. In the reported characterization, this mechanism enforces the object to be generated according to the trajectory, but is most effective for small movements. This limitation is important because it motivates the second major component of DiTraj. Semantic separation alone is insufficient when the target object moves far across frames, since the dominant obstacle then lies in how DiT attention couples position embeddings and inter-frame token matching.

## 4. Attention analysis and Spatial-Temporal Decoupled 3D-RoPE

DiTraj includes an explicit analysis of 3D full attention and 3D-RoPE. In DiT-based video generation, video is processed as a sequence of tokens spanning spatial and temporal dimensions after 3D-VAE compression and patchification. Each token’s position is represented by three coordinates, $(x,y,t)$, and 1D RoPE is applied independently to $x$, $y$, and $t$, with the resulting embeddings concatenated [2509.21839].

The key empirical observation is that attention maps between tokens of different frames are strongly focused along the diagonal: tokens attend mostly to others sharing the same spatial location across frames. The paper interprets this as evidence that attention scores are tightly correlated with the similarity of position embeddings. The practical consequence is that when a foreground object follows a large-movement trajectory, the tokens at its bounding-box locations across frames have dissimilar position embeddings, which leads to low cross-frame attention and weak trajectory adherence.

STD-RoPE is introduced to address this problem. Its core operation is to align the spatial position embeddings of foreground tokens across all frames while preserving temporal position embeddings. An anchor frame $k$ is selected at random from the $f$ trajectory frames, and for every frame $i$, the spatial embeddings inside the bounding box $\mathcal{B}_i$ are replaced by those from the same relative position in the anchor box $\mathcal{B}_k$. Formally, the method splits each token’s 3D-RoPE into spatial and temporal parts, sets

$$
PE^{spatial}_i[\mathcal{B}_i] = PE^{spatial}_k[\mathcal{B}_k]
$$

for all $i$, and then reconstructs

$$
PE_{STD} = \text{Concatenate}(PE^{spatial}, PE^{temporal})
$$

STD-RoPE is applied only during the early denoising steps, specifically the first $t_b$ steps, after which normal position embeddings are restored. The effect is that foreground tokens become mutually close in embedding space across frames, greatly increasing their cross-attention score and improving representation propagation over the whole trajectory.

The method also addresses artifacts caused by repeated position embeddings, referred to as R-tokens. Since spatial alignment can create tokens with identical position embeddings, ambiguity can arise. DiTraj resolves this with a self-attention mask $\mathbf{M}^{self}$ that blocks attention between R-tokens and true foreground tokens:

$$
\mathbf{M}^{self}_{i,j} =
\begin{cases}
-\infty & (i \in \mathbb{S}_{fg}, j \in \mathbb{S}_R) \text{ or } (i \in \mathbb{S}_R, j \in \mathbb{S}_{fg}) \\
0 & \text{otherwise}
\end{cases}
$$

This combination of foreground-only positional alignment and selective masking is the DiT-specific core of DiTraj. It differs from methods that intervene primarily through noise or generic attention bias, because it targets the positional substrate that shapes 3D full attention.

## 5. 3D-aware trajectory control

DiTraj extends 2D trajectory control to 3D-aware control by regulating the density of position embeddings within variable-size bounding boxes. The intended use case is a trajectory in which bounding-box size changes across frames, such as a box growing when an object approaches the camera. The smallest bounding box is chosen as the anchor, and its spatial embeddings are upsampled by nearest-neighbor to match the size of each target frame’s box. For frame $i$, the upsampled anchor embedding is assigned to $PE^{spatial}_i[\mathcal{B}_i]$ [2509.21839].

The reported interpretation is that denser embeddings correspond to an object closer to the camera, so controlling bounding-box size enables 3D depth trajectory specification. This is not merely an auxiliary refinement of the 2D setting. It reuses the same position-embedding intervention principle but adapts it to scale variation, thereby coupling apparent depth motion to positional encoding density rather than to an explicit geometric module.

This suggests that DiTraj treats depth-aware motion as a continuation of the same representational issue seen in large 2D movement: the controllability bottleneck arises from how foreground tokens are positioned in the model’s embedding space. By manipulating density rather than only location, the method adds a simple mechanism for scale change without introducing new training objectives or external geometry estimators.

## 6. Empirical evaluation, ablations, and relation to adjacent methods

DiTraj is evaluated on Wan2.1 and CogVideoX against both training-based and training-free state-of-the-art methods. The evaluation uses video quality metrics from VBench—Subject Consistency (SC), Background Consistency (BC), Motion Smoothness (MS), Aesthetic Quality (AQ), and Imaging Quality (IQ)—together with trajectory-control metrics Coverage (Cov), mean IoU (mIoU), Center Distance (CD; lower is better), and AP50 [2509.21839].

In the quantitative comparison highlighted from Table 1, DiTraj with Wan2.1 achieves an mIoU of 47.3 versus 40.8 for the next best method, a CD of 0.09 versus 0.11, and an AP50 of 50.5 versus 49.1. With the CogVideoX backbone, the method still achieves state-of-the-art or close performance, which is presented as evidence of generalizability. In the user study summarized in Table 2, DiTraj receives over 60% of human preference votes in trajectory alignment, video-text alignment, and video quality, while the next best method remains below 13%.

The ablation results separate the contributions of the two main components. Using only foreground-background guidance improves control somewhat. Adding STD-RoPE further boosts trajectory adherence, including mIoU and AP50, without significant loss in video realism. Qualitative observations reported for the method state that it faithfully follows trajectories across large spatial displacements while preserving object appearance and background consistency, whereas baselines may lose control, generate artifacts, or fail to maintain appearance.

The most direct training-free comparison in the provided literature is FreeTraj, which controls trajectories by modifying noise sampling and attention mechanisms in video diffusion models and reports mIoU $0.281$ and CD $0.154$ against Peekaboo and TrailBlazer in its own evaluation setting [2406.16863]. DiTraj differs in both architectural target and control mechanism: it is tailored for DiT, emphasizes foreground-background prompt separation, and leverages inter-frame positional alignment through STD-RoPE rather than noise construction [2509.21839]. This distinction matters because DiTraj’s central claim is not simply that training-free control is possible, but that DiT-specific full-attention structure can be exploited directly for more precise control.

A common misconception is to treat DiTraj as another generic diffusion-trajectory method. In the diffusion literature, similarly named systems address different tasks: DiffTraj generates privacy-preserving GPS trajectories [2304.11582], while RI-DiT predicts long-tail driving trajectories in crash scenarios [2501.16349]. DiTraj instead belongs to controllable text-to-video generation. Its technical contribution is the claim that trajectory adherence in DiT-based video synthesis can be improved by combining LLM-mediated semantic decomposition with inference-time manipulation of foreground position embeddings [2509.21839].

Source: https://www.emergentmind.com/topics/ditraj