---
title: 'TrajVLM-Gen: Physics-Aware Video Generation'
url: https://www.emergentmind.com/topics/trajvlm-gen
type: topic
---

# TrajVLM-Gen: Physics-Aware Video Generation

Searching arXiv for the specified topic and directly related work to ground the article in current papers.
TrajVLM-Gen arXiv search query: "TrajVLM-Gen"

TrajVLM-Gen is a two-stage framework for physics-aware image-to-video generation in which a Vision Language Model predicts coarse-grained motion trajectories and those trajectories then guide diffusion-based video synthesis [2510.00806]. The method is motivated by the claim that current video generation systems can produce visually convincing frames while still yielding physically inconsistent motion, including drift, implausible acceleration, gravity violations, and scale changes inconsistent with camera or object dynamics [2510.00806]. In TrajVLM-Gen, the central modeling move is to treat motion forecasting as a language-generation problem over future bounding boxes and to use the resulting trajectory signal as an explicit control variable during OpenSora-based generation [2510.00806].

## 1. Problem formulation and conceptual basis

TrajVLM-Gen addresses image-plus-text conditional video generation under an explicit physical-plausibility objective [2510.00806]. The target is not merely to synthesize motion along an externally supplied path, but to infer a path that is itself consistent with the scene and then condition the generator so that the sampled video follows that path [2510.00806]. The paper frames this as a response to a failure mode of video diffusion models: strong local appearance modeling does not guarantee temporally coherent dynamics or adherence to real-world motion constraints [2510.00806].

The framework assumes that a Vision Language Model can serve as a reasoning engine for coarse motion prediction because it has access to both the input image and the text prompt [2510.00806]. This differentiates TrajVLM-Gen from control methods that use camera control, drag-based motion control, or trajectory-guided diffusion as purely external signals rather than inferring motion from scene content [2510.00806]. It also differentiates the method from prior language-model-based trajectory prediction methods that ignore full visual context and therefore weaken physical grounding [2510.00806].

A key example used in the paper is that a cyclist on a mountain path should follow a trajectory shaped by terrain, slope, and gravity rather than a naive straight-line extrapolation [2510.00806]. This suggests that the framework is intended to encode coarse physical regularities at the trajectory-planning stage before the diffusion model performs fine-grained frame synthesis.

## 2. Two-stage architecture and trajectory representation

The pipeline is composed of two stages: trajectory prediction with a VLM, followed by trajectory-controlled video generation [2510.00806]. In the first stage, the model receives an image and a textual description and predicts a sequence of future object bounding boxes [2510.00806]. In the second stage, the predicted trajectory sequence is passed to an OpenSora-based video diffusion model, where trajectory tokens are appended to the prompt and an attention-based mechanism constrains generation so that motion aligns with the predicted path [2510.00806].

The trajectory representation is box-based. Each box is written as
$$
p = (x_1, y_1, x_2, y_2),
$$
where $(x_1, y_1)$ and $(x_2, y_2)$ are the top-left and bottom-right coordinates, and a trajectory is represented as
$$
[p_1, p_2, \ldots, p_n].
$$
The paper emphasizes that the output is textual, which allows the trajectory predictor to be trained and decoded using standard language-model token generation [2510.00806].

The VLM stack for trajectory prediction uses a SigLIP2 vision encoder, a lightweight projection layer, and Qwen2.5-8B as the language model [2510.00806]. The image is encoded into visual features $Z_v$, projected into the word embedding space as $H_v$, and concatenated with instruction tokens $H_{\text{ins}}$ before entering the LLM [2510.00806]. This makes the first stage architecturally similar to contemporary multimodal LLM systems, but with trajectory text rather than free-form captions as the prediction target.

## 3. Reasoning-oriented trajectory prediction

TrajVLM-Gen casts coarse motion prediction as structured language generation [2510.00806]. To encourage physically informed forecasting, the model is trained to output a response of the form
$$
<\text{Reason}> - [p_1, p_2, \ldots, p_n].
$$
The reasoning component is intended to decompose the task into three steps: analyze the scene and identify the relevant physical law category, reason about motion patterns, and predict how the bounding boxes change over time [2510.00806].

The paper associates this design with chain-of-thought style reasoning, explicitly linking the forecasting process to categories such as gravity, elasticity, perspective effects, and direction changes [2510.00806]. The output remains a sequence of future boxes, but the intermediate reasoning text is meant to push the model beyond shallow pattern matching [2510.00806].

The training objective is not given as a fully expanded bespoke loss, but the formulation implies supervised instruction tuning over both reasoning tokens and trajectory tokens. The paper states the likelihood target conceptually as
$$
\max_\theta \log p_\theta(\langle \text{Reason} \rangle, [p_1,\ldots,p_n] \mid X_v, \text{prompt}),
$$
or equivalently a negative log-likelihood or cross-entropy objective over the generated output sequence [2510.00806]. No additional custom objective for trajectory prediction is specified beyond this language-model formulation [2510.00806].

This representation makes the coarse motion plan legible as text. A plausible implication is that the method treats trajectory prediction not as direct geometric regression but as multimodal sequence modeling with an explicitly interpretable intermediate format.

## 4. Trajectory-conditioned OpenSora generation

The second stage uses OpenSora as the base video generator [2510.00806]. The predicted box sequence is converted into text and appended to the prompt so that the diffusion model receives both semantic information and motion information [2510.00806]. The paper states that conditioning is not limited to prompt concatenation: it also introduces a trajectory-aware attention mechanism intended to align generated motion with the coarse trajectory [2510.00806].

During inference, the video model produces cross-attention maps $A_t$, which are compared to a trajectory mask $M_{\text{traj}}$ derived from the coarse trajectory boxes [2510.00806]. The paper presents the constraint as an energy function,
$$
E\left(A_t\right)=M_{\text {traj } \odot \operatorname{sign}\left(A_t\right)-\lambda \nabla^2 A_t,
$$
while also noting, in effect, that the typesetting is malformed in the manuscript excerpt [2510.00806]. The intended semantics are described in prose: the mask term increases attention values for trajectory-related regions or tokens, the Laplacian term smooths attention spatially and discourages abrupt changes, and $\lambda$ is a learnable or tunable weight [2510.00806].

The stated purpose of this mechanism is to refine coarse trajectory guidance into fine-grained, spatially coherent attention during diffusion [2510.00806]. The paper further reports that raw trajectory text alone is not sufficient and that the attention-mask refinement materially improves performance [2510.00806]. This is one of the central architectural claims of the work: the trajectory serves as a high-level motion plan, but alignment between that plan and generated video depends on explicit attention control.

## 5. Dataset construction, supervision, and training configuration

A major contribution of the work is a trajectory prediction dataset assembled from tracking corpora [2510.00806]. The paper lists TNL2K, LaSOT, LaSOT-ext, OTB99-Lang, and GOT-10K as the tracking sources and describes a reverse localization setup in which future bounding boxes are predicted from the first frame [2510.00806]. Video clips are segmented into 24-frame clips so that the prediction horizon remains manageable and motion remains predictable [2510.00806].

The total corpus contains 1.3M image-video-trajectory pairs, divided into 650K global alignment caption samples from LLaVA1.5-665K, 600K trajectory pretraining samples from the tracking datasets, and 50K instruction-following samples [2510.00806]. For sample curation, the paper states that Qwen2.5-VL-72B is used to filter bad descriptions, low-resolution videos, and tracking interruptions [2510.00806]. Additional filtering includes aesthetic scoring with a CLIP+MLP scorer at threshold 6.5 and optical-flow consistency validation using UniMatch [2510.00806].

To construct instruction-following physics labels, Gemini2.5-pro is used to describe video content and assign physics-related labels based on motion statistics [2510.00806]. The labels include gravity for vertical acceleration, elastic for frequent direction changes or rebounds, and perspective camera projection for scale-varying motion [2510.00806]. This creates the physics-aware instruction dataset that supports language-guided trajectory generation [2510.00806].

The reported training configuration uses the Qwen2.5 framework, 384×384 input resolution, siglip2-so400m-patch14-384 as the visual encoder, batch size 128, and learning rate $2\mathrm{e}{-5}$ for instruction fine-tuning [2510.00806]. Instruction fine-tuning takes about 1 day, while inference uses 4 A800 GPUs and takes about 1 hour [2510.00806].

## 6. Evaluation results and ablations

TrajVLM-Gen is evaluated on both multimodal understanding and video-generation benchmarks [2510.00806]. On general MLLM benchmarks, the paper reports results on POPE, MME, MMBench, MMMU, VQAv2, GQA, ScienceQA, and VizWiz [2510.00806]. Reported scores include POPE 88.1 at 384 resolution and 86.8 at 224 resolution, MME$^p$ 1545 at 384, MME$^c$ 301 at 384, MMB 67.6 at 384 and 68.1 at 224, MMMU 36.3 at 384, VQAv2 81.2, GQA 64.3, ScienceQA 70.2, and VizWiz 57.9 [2510.00806]. The paper interprets these results as evidence that trajectory supervision does not degrade general multimodal reasoning and may improve it [2510.00806].

For trajectory generation evaluation, the work adopts a detection-based scheme from LVD with categories Numeracy, Attribution, Visibility, Dynamics, Sequential, and Average [2510.00806]. TrajVLM-Gen reports Numeracy 89%, Attribution 95%, Visibility 92%, Dynamics 95%, Sequential 86%, and Average 89.6%, compared with an average of 49.4% for LVD(GPT-4) [2510.00806].

For video quality, the paper evaluates on UCF-101 and MSR-VTT using FVD [2510.00806]. The reported results are UCF-101 FVD = 545 and MSR-VTT FVD = 539 [2510.00806]. The comparison table is described as competitive with or better than CogVideo, MagicVideo, Make-A-Video, VideoLDM, ModelScope, and LVD, and the paper specifically notes that on MSR-VTT TrajVLM-Gen outperforms ModelScope at 550 and LVD at 565 [2510.00806].

The main ablation concerns the attention-mask mechanism [2510.00806]. Without the attention mask, the reported FVD is 632 on UCF-101 and 558 on MSR-VTT; with the attention mask, the corresponding values are 545 and 539 [2510.00806]. The paper also summarizes this effect as a rise in trajectory accuracy from 67.3% to 82.1% in the relevant setting, although the exact table context is stated to be only partially visible in the excerpt [2510.00806]. The factual takeaway given in the paper is that coarse trajectory text alone is not sufficient and that attention-based refinement is important for preserving spatial consistency [2510.00806].

## 7. Limitations, interpretive issues, and relation to adjacent trajectory-generation research

The paper indicates several limitations [2510.00806]. The method depends on the quality of the predicted coarse trajectory, so errors in VLM reasoning propagate into the generation stage [2510.00806]. The trajectory representation is box-based and therefore does not explicitly model fine articulated motion, deformable objects, or non-rigid dynamics [2510.00806]. The attention formulation is described somewhat loosely, and the equation in the manuscript is not perfectly typeset [2510.00806]. The dataset is built from tracking data and is therefore mostly object-centric, which may limit coverage of complex multi-agent physical interactions [2510.00806]. The generation stage also inherits limitations from OpenSora and from the mismatch between 2D box trajectories and true 3D motion [2510.00806].

Within the broader literature, TrajVLM-Gen belongs to a wider class of systems that use language-model or vision-language-model priors to structure trajectory generation, but it does so in a distinct application regime. Traj-VLN formulates continuous-environment navigation as autoregressive pixel-space trajectory generation for Vision-and-Language Navigation in Continuous Environments, using a VLM to predict sequences of pixel coordinates rather than future video motion [2607.10744]. Traj-LLM repurposes a pre-trained GPT-2 backbone for autonomous-driving trajectory forecasting through structured encoding of trajectories and HD maps, avoiding explicit prompt engineering and using a multi-modal Laplace decoder [2405.04909]. VL-TGS decomposes outdoor mapless navigation into CVAE-based candidate trajectory generation and VLM-based semantic selection among projected path options [2408.02454]. TrajDLM, by contrast, targets synthetic GPS trajectory generation on road networks through topology-aware block diffusion over road-segment sequences [2605.10020].

These comparisons suggest that TrajVLM-Gen should be understood not as a general-purpose trajectory model, but as a multimodal video-generation framework in which trajectory prediction functions as an intermediate planning representation [2510.00806]. A plausible implication is that its significance lies in connecting scene-conditioned motion reasoning to controllable diffusion synthesis rather than in trajectory forecasting as an end task.

Source: https://www.emergentmind.com/topics/trajvlm-gen