TrajVLM-Gen: Physics-Aware Video Generation

• The paper introduces a two-stage framework that predicts coarse motion trajectories with a Vision Language Model and refines video synthesis via diffusion, ensuring physical plausibility.
• It treats motion forecasting as a language-generation task by outputting future bounding box sequences as interpretable control signals for video generation.
• Evaluations show that integrating a trajectory-aware attention mask improves spatial coherence and trajectory accuracy, outperforming competitive models on benchmarks.

Searching arXiv for the specified topic and directly related work to ground the article in current papers. TrajVLM-Gen arXiv search query: "TrajVLM-Gen"

TrajVLM-Gen is a two-stage framework for physics-aware image-to-video generation in which a Vision LLM predicts coarse-grained motion trajectories and those trajectories then guide diffusion-based video synthesis (Yang et al., 1 Oct 2025). The method is motivated by the claim that current video generation systems can produce visually convincing frames while still yielding physically inconsistent motion, including drift, implausible acceleration, gravity violations, and scale changes inconsistent with camera or object dynamics (Yang et al., 1 Oct 2025). In TrajVLM-Gen, the central modeling move is to treat motion forecasting as a language-generation problem over future bounding boxes and to use the resulting trajectory signal as an explicit control variable during OpenSora-based generation (Yang et al., 1 Oct 2025).

1. Problem formulation and conceptual basis

TrajVLM-Gen addresses image-plus-text conditional video generation under an explicit physical-plausibility objective (Yang et al., 1 Oct 2025). The target is not merely to synthesize motion along an externally supplied path, but to infer a path that is itself consistent with the scene and then condition the generator so that the sampled video follows that path (Yang et al., 1 Oct 2025). The paper frames this as a response to a failure mode of video diffusion models: strong local appearance modeling does not guarantee temporally coherent dynamics or adherence to real-world motion constraints (Yang et al., 1 Oct 2025).

The framework assumes that a Vision LLM can serve as a reasoning engine for coarse motion prediction because it has access to both the input image and the text prompt (Yang et al., 1 Oct 2025). This differentiates TrajVLM-Gen from control methods that use camera control, drag-based motion control, or trajectory-guided diffusion as purely external signals rather than inferring motion from scene content (Yang et al., 1 Oct 2025). It also differentiates the method from prior language-model-based trajectory prediction methods that ignore full visual context and therefore weaken physical grounding (Yang et al., 1 Oct 2025).

A key example used in the paper is that a cyclist on a mountain path should follow a trajectory shaped by terrain, slope, and gravity rather than a naive straight-line extrapolation (Yang et al., 1 Oct 2025). This suggests that the framework is intended to encode coarse physical regularities at the trajectory-planning stage before the diffusion model performs fine-grained frame synthesis.

2. Two-stage architecture and trajectory representation

The pipeline is composed of two stages: trajectory prediction with a VLM, followed by trajectory-controlled video generation (Yang et al., 1 Oct 2025). In the first stage, the model receives an image and a textual description and predicts a sequence of future object bounding boxes (Yang et al., 1 Oct 2025). In the second stage, the predicted trajectory sequence is passed to an OpenSora-based video diffusion model, where trajectory tokens are appended to the prompt and an attention-based mechanism constrains generation so that motion aligns with the predicted path (Yang et al., 1 Oct 2025).

The trajectory representation is box-based. Each box is written as

$p = (x_1, y_1, x_2, y_2),$

where $(x_1, y_1)$ and $(x_2, y_2)$ are the top-left and bottom-right coordinates, and a trajectory is represented as

$[p_1, p_2, \ldots, p_n].$

The paper emphasizes that the output is textual, which allows the trajectory predictor to be trained and decoded using standard language-model token generation (Yang et al., 1 Oct 2025).

The VLM stack for trajectory prediction uses a SigLIP2 vision encoder, a lightweight projection layer, and Qwen2.5-8B as the LLM (Yang et al., 1 Oct 2025). The image is encoded into visual features $Z_v$, projected into the word embedding space as $H_v$, and concatenated with instruction tokens $H_{\text{ins}}$ before entering the LLM (Yang et al., 1 Oct 2025). This makes the first stage architecturally similar to contemporary multimodal LLM systems, but with trajectory text rather than free-form captions as the prediction target.

3. Reasoning-oriented trajectory prediction

TrajVLM-Gen casts coarse motion prediction as structured language generation (Yang et al., 1 Oct 2025). To encourage physically informed forecasting, the model is trained to output a response of the form

$<\text{Reason}> - [p_1, p_2, \ldots, p_n].$

The reasoning component is intended to decompose the task into three steps: analyze the scene and identify the relevant physical law category, reason about motion patterns, and predict how the bounding boxes change over time (Yang et al., 1 Oct 2025).

The paper associates this design with chain-of-thought style reasoning, explicitly linking the forecasting process to categories such as gravity, elasticity, perspective effects, and direction changes (Yang et al., 1 Oct 2025). The output remains a sequence of future boxes, but the intermediate reasoning text is meant to push the model beyond shallow pattern matching (Yang et al., 1 Oct 2025).

The training objective is not given as a fully expanded bespoke loss, but the formulation implies supervised instruction tuning over both reasoning tokens and trajectory tokens. The paper states the likelihood target conceptually as

$$\max_\theta \log p_\theta(\langle \text{Reason} \rangle, [p_1,\ldots,p_n] \mid X_v, \text{prompt}),$$

or equivalently a negative log-likelihood or cross-entropy objective over the generated output sequence (Yang et al., 1 Oct 2025). No additional custom objective for trajectory prediction is specified beyond this language-model formulation (Yang et al., 1 Oct 2025).

This representation makes the coarse motion plan legible as text. A plausible implication is that the method treats trajectory prediction not as direct geometric regression but as multimodal sequence modeling with an explicitly interpretable intermediate format.

4. Trajectory-conditioned OpenSora generation

The second stage uses OpenSora as the base video generator (Yang et al., 1 Oct 2025). The predicted box sequence is converted into text and appended to the prompt so that the diffusion model receives both semantic information and motion information (Yang et al., 1 Oct 2025). The paper states that conditioning is not limited to prompt concatenation: it also introduces a trajectory-aware attention mechanism intended to align generated motion with the coarse trajectory (Yang et al., 1 Oct 2025).

During inference, the video model produces cross-attention maps $A_t$, which are compared to a trajectory mask $(x_1, y_1)$0 derived from the coarse trajectory boxes (Yang et al., 1 Oct 2025). The paper presents the constraint as an energy function,

$(x_1, y_1)$1

while also noting, in effect, that the typesetting is malformed in the manuscript excerpt (Yang et al., 1 Oct 2025). The intended semantics are described in prose: the mask term increases attention values for trajectory-related regions or tokens, the Laplacian term smooths attention spatially and discourages abrupt changes, and $(x_1, y_1)$2 is a learnable or tunable weight (Yang et al., 1 Oct 2025).

The stated purpose of this mechanism is to refine coarse trajectory guidance into fine-grained, spatially coherent attention during diffusion (Yang et al., 1 Oct 2025). The paper further reports that raw trajectory text alone is not sufficient and that the attention-mask refinement materially improves performance (Yang et al., 1 Oct 2025). This is one of the central architectural claims of the work: the trajectory serves as a high-level motion plan, but alignment between that plan and generated video depends on explicit attention control.

5. Dataset construction, supervision, and training configuration

A major contribution of the work is a trajectory prediction dataset assembled from tracking corpora (Yang et al., 1 Oct 2025). The paper lists TNL2K, LaSOT, LaSOT-ext, OTB99-Lang, and GOT-10K as the tracking sources and describes a reverse localization setup in which future bounding boxes are predicted from the first frame (Yang et al., 1 Oct 2025). Video clips are segmented into 24-frame clips so that the prediction horizon remains manageable and motion remains predictable (Yang et al., 1 Oct 2025).

The total corpus contains 1.3M image-video-trajectory pairs, divided into 650K global alignment caption samples from LLaVA1.5-665K, 600K trajectory pretraining samples from the tracking datasets, and 50K instruction-following samples (Yang et al., 1 Oct 2025). For sample curation, the paper states that Qwen2.5-VL-72B is used to filter bad descriptions, low-resolution videos, and tracking interruptions (Yang et al., 1 Oct 2025). Additional filtering includes aesthetic scoring with a CLIP+MLP scorer at threshold 6.5 and optical-flow consistency validation using UniMatch (Yang et al., 1 Oct 2025).

To construct instruction-following physics labels, Gemini2.5-pro is used to describe video content and assign physics-related labels based on motion statistics (Yang et al., 1 Oct 2025). The labels include gravity for vertical acceleration, elastic for frequent direction changes or rebounds, and perspective camera projection for scale-varying motion (Yang et al., 1 Oct 2025). This creates the physics-aware instruction dataset that supports language-guided trajectory generation (Yang et al., 1 Oct 2025).

The reported training configuration uses the Qwen2.5 framework, 384×384 input resolution, siglip2-so400m-patch14-384 as the visual encoder, batch size 128, and learning rate $(x_1, y_1)$3 for instruction fine-tuning (Yang et al., 1 Oct 2025). Instruction fine-tuning takes about 1 day, while inference uses 4 A800 GPUs and takes about 1 hour (Yang et al., 1 Oct 2025).

6. Evaluation results and ablations

TrajVLM-Gen is evaluated on both multimodal understanding and video-generation benchmarks (Yang et al., 1 Oct 2025). On general MLLM benchmarks, the paper reports results on POPE, MME, MMBench, MMMU, VQAv2, GQA, ScienceQA, and VizWiz (Yang et al., 1 Oct 2025). Reported scores include POPE 88.1 at 384 resolution and 86.8 at 224 resolution, MME$(x_1, y_1)$4 1545 at 384, MME$(x_1, y_1)$5 301 at 384, MMB 67.6 at 384 and 68.1 at 224, MMMU 36.3 at 384, VQAv2 81.2, GQA 64.3, ScienceQA 70.2, and VizWiz 57.9 (Yang et al., 1 Oct 2025). The paper interprets these results as evidence that trajectory supervision does not degrade general multimodal reasoning and may improve it (Yang et al., 1 Oct 2025).

For trajectory generation evaluation, the work adopts a detection-based scheme from LVD with categories Numeracy, Attribution, Visibility, Dynamics, Sequential, and Average (Yang et al., 1 Oct 2025). TrajVLM-Gen reports Numeracy 89%, Attribution 95%, Visibility 92%, Dynamics 95%, Sequential 86%, and Average 89.6%, compared with an average of 49.4% for LVD(GPT-4) (Yang et al., 1 Oct 2025).

For video quality, the paper evaluates on UCF-101 and MSR-VTT using FVD (Yang et al., 1 Oct 2025). The reported results are UCF-101 FVD = 545 and MSR-VTT FVD = 539 (Yang et al., 1 Oct 2025). The comparison table is described as competitive with or better than CogVideo, MagicVideo, Make-A-Video, VideoLDM, ModelScope, and LVD, and the paper specifically notes that on MSR-VTT TrajVLM-Gen outperforms ModelScope at 550 and LVD at 565 (Yang et al., 1 Oct 2025).

The main ablation concerns the attention-mask mechanism (Yang et al., 1 Oct 2025). Without the attention mask, the reported FVD is 632 on UCF-101 and 558 on MSR-VTT; with the attention mask, the corresponding values are 545 and 539 (Yang et al., 1 Oct 2025). The paper also summarizes this effect as a rise in trajectory accuracy from 67.3% to 82.1% in the relevant setting, although the exact table context is stated to be only partially visible in the excerpt (Yang et al., 1 Oct 2025). The factual takeaway given in the paper is that coarse trajectory text alone is not sufficient and that attention-based refinement is important for preserving spatial consistency (Yang et al., 1 Oct 2025).

7. Limitations, interpretive issues, and relation to adjacent trajectory-generation research

The paper indicates several limitations (Yang et al., 1 Oct 2025). The method depends on the quality of the predicted coarse trajectory, so errors in VLM reasoning propagate into the generation stage (Yang et al., 1 Oct 2025). The trajectory representation is box-based and therefore does not explicitly model fine articulated motion, deformable objects, or non-rigid dynamics (Yang et al., 1 Oct 2025). The attention formulation is described somewhat loosely, and the equation in the manuscript is not perfectly typeset (Yang et al., 1 Oct 2025). The dataset is built from tracking data and is therefore mostly object-centric, which may limit coverage of complex multi-agent physical interactions (Yang et al., 1 Oct 2025). The generation stage also inherits limitations from OpenSora and from the mismatch between 2D box trajectories and true 3D motion (Yang et al., 1 Oct 2025).

Within the broader literature, TrajVLM-Gen belongs to a wider class of systems that use language-model or vision-language-model priors to structure trajectory generation, but it does so in a distinct application regime. Traj-VLN formulates continuous-environment navigation as autoregressive pixel-space trajectory generation for Vision-and-Language Navigation in Continuous Environments, using a VLM to predict sequences of pixel coordinates rather than future video motion (Fu et al., 12 Jul 2026). Traj-LLM repurposes a pre-trained GPT-2 backbone for autonomous-driving trajectory forecasting through structured encoding of trajectories and HD maps, avoiding explicit prompt engineering and using a multi-modal Laplace decoder (Lan et al., 2024). VL-TGS decomposes outdoor mapless navigation into CVAE-based candidate trajectory generation and VLM-based semantic selection among projected path options (Song et al., 2024). TrajDLM, by contrast, targets synthetic GPS trajectory generation on road networks through topology-aware block diffusion over road-segment sequences (Wongso et al., 11 May 2026).

These comparisons suggest that TrajVLM-Gen should be understood not as a general-purpose trajectory model, but as a multimodal video-generation framework in which trajectory prediction functions as an intermediate planning representation (Yang et al., 1 Oct 2025). A plausible implication is that its significance lies in connecting scene-conditioned motion reasoning to controllable diffusion synthesis rather than in trajectory forecasting as an end task.