TrajVLM: Vision-Language Trajectory Modeling

• TrajVLM is a family of multimodal models that utilize trajectory signals as primary objects to drive tasks like physics-aware video generation, autonomous driving planning, and spatial reasoning.
• The systems employ diverse trajectory representations—ranging from bounding boxes and BEV waypoints to tokenized segments and human attention traces—tailored for specific applications.
• Empirical results across domains show enhanced performance in video generation, driving safety, long-horizon reasoning, and multimodal spatial understanding.

TrajVLM is a trajectory-centered vision-language modeling paradigm whose exact meaning varies across recent arXiv literature. In one explicit usage, TrajVLM denotes the vision–language trajectory predictor inside TrajVLM-Gen: a Large Vision–LLM (LVLM) that takes an image and an optional text prompt, reasons about scene physics via chain-of-thought, and outputs a coarse, temporally ordered trajectory for a target object as a sequence of bounding boxes (Yang et al., 1 Oct 2025). In other usage, the term denotes a Trajectory Vision-LLM for autonomous driving that emits ego trajectories in Bird’s-Eye View (BEV) coordinates (Liu et al., 20 Dec 2025), a LLaVA-style vision-LLM whose connector is an end-to-end trajectory tokenizer (Zheng et al., 26 Feb 2026), or, more generally, trajectory-aware vision-language modeling conditioned on human attention traces (Yang et al., 23 Feb 2026). This suggests that TrajVLM is best understood as a family of architectures in which trajectories are elevated from auxiliary signals to first-class multimodal objects.

1. Terminological scope and conceptual range

The term “TrajVLM” does not denote a single canonical model. Instead, it appears as a recurring label for systems that bind trajectory structure to visual and linguistic reasoning in different ways. Some systems generate numeric trajectories directly; others tokenize trajectories as compact visual abstractions; still others use human attention trajectories as localization and explanation signals. A closely related line of work, Traj-MLLM, does not introduce a model explicitly named “TrajVLM,” but it is described as fulfilling the same role by turning raw GPS trajectories into multiview, interleaved image–text sequences for trajectory data mining (Liu et al., 25 Aug 2025).

Usage in the literature	Trajectory primitive	Main task
TrajVLM in TrajVLM-Gen	Sequence of bounding boxes	Physics-aware image-to-video generation
LLaViDA as TrajVLM	Ego-local BEV waypoints	Autonomous driving planning
TrajVLM with TrajTok	Object-trajectory tokens	Long-video reasoning
TraceVision as TrajVLM	Human attention trajectories	Spatial understanding and localization
Traj-MLLM as trajectory VLM	Raw GPS trajectories mapped to image–text sequences	Trajectory data mining

A common misconception is to treat TrajVLM as synonymous with driving planners. The literature is broader. TrajVLM-Gen uses trajectories to guide video diffusion (Yang et al., 1 Oct 2025); TrajTok uses trajectory tokens to improve long-video question answering (Zheng et al., 26 Feb 2026); TraceVision uses attention traces to ground descriptions and segmentation (Yang et al., 23 Feb 2026); and Traj-MLLM reframes trajectory mining as multimodal reasoning over map-rendered views and structured text (Liu et al., 25 Aug 2025).

2. Representations and fusion mechanisms

Across these systems, the central design decision is how a trajectory is represented and how it is fused with visual and textual streams. In TrajVLM-Gen, each frame’s box is represented as $$p_t = (x_{1,t}, y_{1,t}, x_{2,t}, y_{2,t}) \in \mathbb{R}^4$$, and a trajectory is the ordered sequence $P = [p_1, p_2, \ldots, p_T]$; the box center can also be written as $$x_t = ((x_{1,t}+x_{2,t})/2, (y_{1,t}+y_{2,t})/2) \in \mathbb{R}^2$$ (Yang et al., 1 Oct 2025). In LLaViDA, the target trajectory is a sequence of three ego-local BEV waypoints over a $3\,\text{s}$ horizon, represented as lateral-longitudinal pairs $T_{ego} = \{(w1_t^*, w2_t^*)\}_{t=1}^3$ (Liu et al., 20 Dec 2025). In TraceVision, a raw attention trace is a sequence $T = \{p_1, p_2, \ldots, p_N\}$ with $p_j = (x_j, y_j, t_j)$, later simplified into semantic keypoints and quantized into coordinate tokens formatted as <Traj>(x,y) (Yang et al., 23 Feb 2026). In TrajTok’s TrajVLM, the trajectory concept is object-centric rather than coordinate-explicit: frame-wise visual features are reorganized into a compact set of object-trajectory tokens whose number depends on semantic complexity rather than video duration (Zheng et al., 26 Feb 2026).

The fusion strategy is equally variable but structurally consistent. TrajVLM-Gen encodes an image with SigLIP2, projects the visual features into the Qwen2.5-8B token space, concatenates them with instruction tokens, and asks the LVLM to output "<Reason> – [p1, p2, …, pT]" (Yang et al., 1 Oct 2025). LLaViDA serializes ego context, critical-object states, and a schema prompt as $I=\langle S_{ego}, S_{obj}, p \rangle$, then generates a single-turn output $$O=\langle think\rangle T_{obj}, R_{sem}\langle/think\rangle, A_{meta}, T_{ego}$$ (Liu et al., 20 Dec 2025). TrajTok’s TrajVLM follows the LLaVA paradigm: projected trajectory tokens are concatenated before text tokens and consumed by the LLM through standard Transformer self-attention, with no separate cross-attention block (Zheng et al., 26 Feb 2026). TraceVision instead uses a dedicated Trajectory-aware Visual Perception module in which visual queries attend to trajectory embeddings and trajectory queries attend back to enhanced visual features, producing a bidirectional fusion loop (Yang et al., 23 Feb 2026).

This variety of representations indicates that “trajectory” is not restricted to future motion coordinates. Depending on the task, it may denote object motion, ego planning, tokenized object tubes, or continuous human attention traces.

3. Representative system families

Physics-aware generation: TrajVLM-Gen is a two-stage image-to-video framework. Stage 1 uses TrajVLM to predict coarse motion trajectories from a single image $X_v$ and an optional text prompt $P = [p_1, p_2, \ldots, p_T]$0. Stage 2 converts the predicted $P = [p_1, p_2, \ldots, p_T]$1 into a textual trajectory string, appends it to the prompt, and uses trajectory-aware attention optimization inside an OpenSora-based video diffusion model to refine motion during sampling (Yang et al., 1 Oct 2025). The defining claim of this system is that physical plausibility is introduced not by retraining the generator, but by LVLM reasoning and inference-time attention guidance.

Autonomous driving planning: LLaViDA operationalizes TrajVLM as a “VLM Trajectory Planner for Autonomous Driving.” It uses six calibrated surround-view cameras during supervised fine-tuning, lightweight serialized perception from BEVFormer, ego history and mission goals, and a fixed schema prompt to generate scene semantics, multi-agent motion forecasts, a meta-action, and numeric BEV waypoints in a single inference turn (Liu et al., 20 Dec 2025). The model’s explicit chain-of-thought is not merely explanatory; it forecasts critical-object trajectories $P = [p_1, p_2, \ldots, p_T]$2 and grounds planning in road layout, weather, signage, and ego intent.

Long-video reasoning: In TrajTok, TrajVLM denotes a LLaVA-style vision-LLM that replaces standard patch pooling with TrajTok, an end-to-end differentiable trajectory tokenizer (Zheng et al., 26 Feb 2026). A universal segmenter produces soft and hard spatiotemporal masks; a trajectory encoder aggregates and refines object-specific features; each trajectory can emit $P = [p_1, p_2, \ldots, p_T]$3 tokens; and the resulting visual tokens are projected into the Qwen3-4B embedding space. The primary purpose is not trajectory prediction in the classical forecasting sense, but compact temporal grounding for long-horizon reasoning.

Trajectory-aware spatial understanding: TraceVision uses “TrajVLM” in the broader sense of trajectory-aware vision-language modeling. It treats human attention trajectories as continuous, temporally ordered signals of intent and fuses them with visual features via the Trajectory-aware Visual Perception module (Yang et al., 23 Feb 2026). The model supports trajectory-guided captioning, text-guided trajectory prediction, referring localization, referring segmentation, and video scene understanding, extending the trajectory concept from motion planning into interpretable region-language association.

Trajectory data mining: Traj-MLLM extends the idea into geospatial analytics. Raw trajectories $P = [p_1, p_2, \ldots, p_T]$4, where each point contains $P = [p_1, p_2, \ldots, p_T]$5, are segmented, rendered on map tiles, paired with structured textual descriptors, and assembled into temporally interleaved image–text sequences $P = [p_1, p_2, \ldots, p_T]$6 for training-free multimodal reasoning (Liu et al., 25 Aug 2025). The paper states that it does not introduce a model explicitly named “TrajVLM,” but that it fulfills the same role by bridging trajectory modality to vision-language inputs.

4. Data construction and learning objectives

A notable trait of TrajVLM systems is that their datasets and losses are usually specialized to the trajectory form they manipulate. TrajVLM-Gen builds a trajectory prediction dataset of 1.3M image–video–trajectory triplets, including 650K global alignment captions from LLaVA1.5-665K, 600K trajectory pretraining clips from TNL2K, LaSOT, LaSOT-ext, OTB99-Lang, and GOT-10K, and 50K trajectory instruction-following examples enriched with physics labels (Yang et al., 1 Oct 2025). The generic trajectory supervision is coordinate regression, written as either $P = [p_1, p_2, \ldots, p_T]$7 or $P = [p_1, p_2, \ldots, p_T]$8, while kinematic statistics $P = [p_1, p_2, \ldots, p_T]$9, $$x_t = ((x_{1,t}+x_{2,t})/2, (y_{1,t}+y_{2,t})/2) \in \mathbb{R}^2$$0, and $$x_t = ((x_{1,t}+x_{2,t})/2, (y_{1,t}+y_{2,t})/2) \in \mathbb{R}^2$$1 are used for labeling “gravity,” “elastic,” and “perspective camera projection” rather than as explicit losses. During generation, trajectory guidance is expressed through an attention energy

$$x_t = ((x_{1,t}+x_{2,t})/2, (y_{1,t}+y_{2,t})/2) \in \mathbb{R}^2$$2

which biases cross-attention toward masked regions and enforces spatial smoothness (Yang et al., 1 Oct 2025).

LLaViDA uses the NuScenes-TP dataset, comprising 23,423 training samples and 6,019 test samples, each a 40-frame sequence at 2 Hz with ego and object trajectories, meta-actions, and GPT-4o synthesized reasoning traces (Liu et al., 20 Dec 2025). Its supervised fine-tuning objective is token-level cross-entropy over reasoning, meta-action, and numeric trajectory outputs,

$$x_t = ((x_{1,t}+x_{2,t})/2, (y_{1,t}+y_{2,t})/2) \in \mathbb{R}^2$$3

with $$x_t = ((x_{1,t}+x_{2,t})/2, (y_{1,t}+y_{2,t})/2) \in \mathbb{R}^2$$4, followed by Trajectory Preference Optimization. TPO samples $$x_t = ((x_{1,t}+x_{2,t})/2, (y_{1,t}+y_{2,t})/2) \in \mathbb{R}^2$$5 responses per instance with temperature $$x_t = ((x_{1,t}+x_{2,t})/2, (y_{1,t}+y_{2,t})/2) \in \mathbb{R}^2$$6, defines positive and negative trajectories by average L2 displacement over the $$x_t = ((x_{1,t}+x_{2,t})/2, (y_{1,t}+y_{2,t})/2) \in \mathbb{R}^2$$7 horizon, and optimizes a DPO-style preference loss with $$x_t = ((x_{1,t}+x_{2,t})/2, (y_{1,t}+y_{2,t})/2) \in \mathbb{R}^2$$8 (Liu et al., 20 Dec 2025). This is a distinctive attempt to inject regression-like supervision directly into a generative objective.

TrajTok’s TrajVLM is trained differently because its core problem is tokenization rather than direct coordinate prediction. The universal segmenter is supervised with pseudo panoptic trajectory masks using Dice loss plus Focal loss, and the paper explicitly states that cross-entropy is not used (Zheng et al., 26 Feb 2026). Visual patch features are gradient-detached before the Perceiver to stabilize training. TrajVLM itself is aligned by language modeling: a caption pretraining stage on PixMo for 1 epoch, followed by 10,000-step instruction tuning on approximately 5M VideoQA examples, using 8×A100 (80GB), bfloat16, batch size 32, learning rate $$x_t = ((x_{1,t}+x_{2,t})/2, (y_{1,t}+y_{2,t})/2) \in \mathbb{R}^2$$9 for the LLM, and $3\,\text{s}$0 for the ViT and connector (Zheng et al., 26 Feb 2026).

TraceVision adopts a three-stage curriculum. Stage 1 uses 0.25M Localized Narratives samples while freezing the visual encoder and LLM; Stage 1.5 jointly trains on 1.33M mixed samples across LN, LNV, PNG, and REC/RES; Stage 2 performs instruction fine-tuning on 1.3M samples, primarily RILN, reasoning segmentation, and interactive editing (Yang et al., 23 Feb 2026). The core autoregressive loss is

$3\,\text{s}$1

and segmentation supervision adds

$3\,\text{s}$2

with default $3\,\text{s}$3 and $3\,\text{s}$4 (Yang et al., 23 Feb 2026).

Traj-MLLM stands apart by being training-free. It uses prompt optimization with $3\,\text{s}$5 seed trajectories per task, fixes the role and output-format components of the system prompt, and iteratively refines task description and domain knowledge via MLLM feedback rather than gradient updates (Liu et al., 25 Aug 2025). This suggests a separate branch of TrajVLM research in which the trajectory representation is engineered for MLLM compatibility instead of learned end to end.

5. Empirical performance across domains

The empirical profile of TrajVLM is strongly task-dependent. In image-to-video generation, TrajVLM-Gen reports FVD = 545 on UCF-101 and FVD = 539 on MSR-VTT, outperforming LVD on both datasets and being competitive with VideoLDM on UCF-101 (Yang et al., 1 Oct 2025). An ablation shows that trajectory-based attention masks improve FVD on UCF-101 from 632 without masks to 545 with masks, and on MSR-VTT from 558 to 539 (Yang et al., 1 Oct 2025). The same system also reports 81.2% on VQAv2, 88.1% on POPE, 36.3% on MMMU, and 89.6% average across Numeracy, Attribution, Visibility, Dynamics, and Sequential in LVD-style detection-based evaluation (Yang et al., 1 Oct 2025).

In autonomous driving, LLaViDA reports state-of-the-art open-loop planning on NuScenes. Under the ST-P3 protocol, it achieves horizon-wise L2 errors of 0.14 / 0.28 / 0.51 m, an average of 0.31 m, and collision rates of 0.03 / 0.07 / 0.19%, averaging 0.10% (Liu et al., 20 Dec 2025). Under UniAD, it reports 0.19 / 0.54 / 1.09 m with average 0.61 m, and collision rates 0.06 / 0.09 / 0.50%, averaging 0.22% (Liu et al., 20 Dec 2025). The paper attributes a major part of this gain to TPO: ST-P3 average L2 improves from 0.39 to 0.31 m, and UniAD average L2 from 0.78 to 0.61 m when moving from SFT-only to SFT+TPO (Liu et al., 20 Dec 2025).

For long-video reasoning, TrajTok’s TrajVLM reports consistent gains over patch-pooling baselines on long-video benchmarks, including +8.8% on LongVideoBench and +5.4% on LVBench over PatchVLM with pool size $3\,\text{s}$6 (Zheng et al., 26 Feb 2026). The paper further emphasizes that increasing patch pooling to $3\,\text{s}$7 to match TrajTok’s token count and frame support does not close the gap, indicating that object-centric grouping rather than token reduction alone drives the improvement. This is an important empirical distinction because it isolates the effect of trajectory tokenization from mere compression.

TraceVision evaluates a different axis of performance: trajectory-grounded understanding and localization. It reports controlled caption generation results of BLEU-4 = 0.328, METEOR = 0.276, ROUGE-L = 0.492, CIDEr = 1.530, and SPICE = 0.328; controlled trajectory generation with LBM(k=0)=0.117 and LBM(k=1)=0.121; referring localization on RefCOCO with [email protected] = 90.4/93.1/87.8 for val/testA/testB; referring segmentation with cIoU = 83.4/86.8/82.4 on the same splits; and video results including HC-STVG METEOR = 24.1, CIDEr = 92.6, and VideoRefer-Bench-D Avg = 3.83 (Yang et al., 23 Feb 2026). These numbers place TrajVLM-like modeling outside forecasting alone and into general multimodal grounding.

Traj-MLLM contributes another performance profile: on four public datasets it reports improvements over state-of-the-art baselines by 48.05% on travel time estimation, 15.52% on mobility prediction, 51.52% on anomaly detection, and 1.83% on transportation mode identification, without training or fine-tuning the MLLM backbones (Liu et al., 25 Aug 2025). The explicit implication is that a trajectory VLM can be competitive even when the trajectory modality is first transformed into map-rendered image–text sequences rather than modeled directly.

6. Limitations, critiques, and emerging directions

The principal limitations of TrajVLM systems are task-specific but structurally similar: ambiguity in trajectory grounding, brittleness under distribution shift, and difficulty enforcing hard physical or geometric constraints. TrajVLM-Gen notes that physics consistency can break in complex multi-agent scenes, heavy occlusions, or when ground-truth motion depends on 3D factors not captured by 2D box trajectories; it also states that without retraining the generator, long-horizon dynamics and rapid topological changes remain hard, and that OpenSora’s 3D VAE encoding challenges precise spatial control even when attention masks help (Yang et al., 1 Oct 2025). LLaViDA reports that vision-only conditioning can hallucinate small or occluded objects, that front-only efficiency variants slightly increase L2 and collision, and that no explicit rule-checker is used for traffic-rule compliance (Liu et al., 20 Dec 2025). TrajTok identifies failure modes for very small or thin objects, over-merging or fragmentation under fast motion and dense scenes, and cross-chunk consistency issues at 16-frame boundaries (Zheng et al., 26 Feb 2026). TraceVision notes dependence on LN/RILN annotations, degradation under extreme compression of trajectories, and the limits of a lightweight segmentation decoder in boundary-heavy scenes (Yang et al., 23 Feb 2026).

A broader critique emerges from work-zone trajectory planning. “Work Zones challenge VLM Trajectory Planning” reports that mainstream VLMs fail in 68.0% of work-zone planning cases, with scenario-level failure rates of 75.86% for Qwen2.5-VL, 70.37% for GPT-4o, 80.00% for Gemini 2.5, 81.93% for SimLingo, 76.63% for RoboTron-Drive, and 77.23% for DriveLM (Liao et al., 3 Oct 2025). Its mitigation, REACT-Drive, replaces direct free-form planning with retrieval, constraint rules, executable code, and verification, achieving approximately $3\,\text{s}$8 reduction in average displacement error relative to VLM baselines under evaluation with Qwen2.5-VL and 0.58 s inference time (Liao et al., 3 Oct 2025). This challenge is relevant to TrajVLM because it shows that language-grounded trajectory generation alone is not sufficient in geometrically irregular or safety-critical domains.

Related trajectory-planning papers point toward likely responses. BEV-VLM shows that using a unified BEV-HD map can yield a 44.8% decrease in displacement error and 0.00% collision rate over the $3\,\text{s}$9 horizon on nuScenes, suggesting that stronger geometric abstraction can make VLM planning more reliable (Chen et al., 27 Sep 2025). VisionTrap shows that training-time language supervision derived from a VLM and refined by an LLM improves multi-agent forecasting while retaining 53 ms real-time inference, with final ADE_10 = 1.17, MR_10 = 0.32, and FDE_1 = 8.72 in its ablation setting (Moon et al., 2024). A plausible implication is that future TrajVLM systems will increasingly combine trajectory reasoning with explicit geometry, retrieval, structured outputs, or programmatic constraints rather than relying on unconstrained multimodal generation alone.

Taken together, the literature portrays TrajVLM not as a single architecture but as an emerging research category organized around one principle: trajectories can serve simultaneously as outputs, inputs, latent tokens, supervision signals, or interpretable attention traces. The most mature systems differ sharply in domain and implementation, yet they converge on the view that multimodal intelligence improves when temporal-spatial structure is represented explicitly rather than left implicit in pixels or text alone.