---
title: 'TrajVLM: Vision-Language Trajectory Modeling'
url: https://www.emergentmind.com/topics/trajvlm
type: topic
---

# TrajVLM: Vision-Language Trajectory Modeling

TrajVLM is a trajectory-centered vision-language modeling paradigm whose exact meaning varies across recent arXiv literature. In one explicit usage, TrajVLM denotes the vision–language trajectory predictor inside TrajVLM-Gen: a Large Vision–Language Model (LVLM) that takes an image and an optional text prompt, reasons about scene physics via chain-of-thought, and outputs a coarse, temporally ordered trajectory for a target object as a sequence of bounding boxes [2510.00806]. In other usage, the term denotes a Trajectory Vision-Language Model for autonomous driving that emits ego trajectories in Bird’s-Eye View (BEV) coordinates [2512.18211], a LLaVA-style vision-language model whose connector is an end-to-end trajectory tokenizer [2602.22779], or, more generally, trajectory-aware vision-language modeling conditioned on human attention traces [2602.19768]. This suggests that TrajVLM is best understood as a family of architectures in which trajectories are elevated from auxiliary signals to first-class multimodal objects.

## 1. Terminological scope and conceptual range

The term “TrajVLM” does not denote a single canonical model. Instead, it appears as a recurring label for systems that bind trajectory structure to visual and linguistic reasoning in different ways. Some systems generate numeric trajectories directly; others tokenize trajectories as compact visual abstractions; still others use human attention trajectories as localization and explanation signals. A closely related line of work, Traj-MLLM, does not introduce a model explicitly named “TrajVLM,” but it is described as fulfilling the same role by turning raw GPS trajectories into multiview, interleaved image–text sequences for trajectory data mining [2509.00053].

| Usage in the literature | Trajectory primitive | Main task |
|---|---|---|
| TrajVLM in TrajVLM-Gen | Sequence of bounding boxes | Physics-aware image-to-video generation |
| LLaViDA as TrajVLM | Ego-local BEV waypoints | Autonomous driving planning |
| TrajVLM with TrajTok | Object-trajectory tokens | Long-video reasoning |
| TraceVision as TrajVLM | Human attention trajectories | Spatial understanding and localization |
| Traj-MLLM as trajectory VLM | Raw GPS trajectories mapped to image–text sequences | Trajectory data mining |

A common misconception is to treat TrajVLM as synonymous with driving planners. The literature is broader. TrajVLM-Gen uses trajectories to guide video diffusion [2510.00806]; TrajTok uses trajectory tokens to improve long-video question answering [2602.22779]; TraceVision uses attention traces to ground descriptions and segmentation [2602.19768]; and Traj-MLLM reframes trajectory mining as multimodal reasoning over map-rendered views and structured text [2509.00053].

## 2. Representations and fusion mechanisms

Across these systems, the central design decision is how a trajectory is represented and how it is fused with visual and textual streams. In TrajVLM-Gen, each frame’s box is represented as $p_t = (x_{1,t}, y_{1,t}, x_{2,t}, y_{2,t}) \in \mathbb{R}^4$, and a trajectory is the ordered sequence $P = [p_1, p_2, \ldots, p_T]$; the box center can also be written as $x_t = ((x_{1,t}+x_{2,t})/2, (y_{1,t}+y_{2,t})/2) \in \mathbb{R}^2$ [2510.00806]. In LLaViDA, the target trajectory is a sequence of three ego-local BEV waypoints over a $3\,\text{s}$ horizon, represented as lateral-longitudinal pairs $T_{ego} = \{(w1_t^*, w2_t^*)\}_{t=1}^3$ [2512.18211]. In TraceVision, a raw attention trace is a sequence $T = \{p_1, p_2, \ldots, p_N\}$ with $p_j = (x_j, y_j, t_j)$, later simplified into semantic keypoints and quantized into coordinate tokens formatted as `<Traj>(x,y)` [2602.19768]. In TrajTok’s TrajVLM, the trajectory concept is object-centric rather than coordinate-explicit: frame-wise visual features are reorganized into a compact set of object-trajectory tokens whose number depends on semantic complexity rather than video duration [2602.22779].

The fusion strategy is equally variable but structurally consistent. TrajVLM-Gen encodes an image with SigLIP2, projects the visual features into the Qwen2.5-8B token space, concatenates them with instruction tokens, and asks the LVLM to output `"<Reason> – [p1, p2, …, pT]"` [2510.00806]. LLaViDA serializes ego context, critical-object states, and a schema prompt as $I=\langle S_{ego}, S_{obj}, p \rangle$, then generates a single-turn output $O=\langle think\rangle T_{obj}, R_{sem}\langle/think\rangle, A_{meta}, T_{ego}$ [2512.18211]. TrajTok’s TrajVLM follows the LLaVA paradigm: projected trajectory tokens are concatenated before text tokens and consumed by the language model through standard Transformer self-attention, with no separate cross-attention block [2602.22779]. TraceVision instead uses a dedicated Trajectory-aware Visual Perception module in which visual queries attend to trajectory embeddings and trajectory queries attend back to enhanced visual features, producing a bidirectional fusion loop [2602.19768].

This variety of representations indicates that “trajectory” is not restricted to future motion coordinates. Depending on the task, it may denote object motion, ego planning, tokenized object tubes, or continuous human attention traces.

## 3. Representative system families

**Physics-aware generation:** TrajVLM-Gen is a two-stage image-to-video framework. Stage 1 uses TrajVLM to predict coarse motion trajectories from a single image $X_v$ and an optional text prompt $ins$. Stage 2 converts the predicted $[p_1,\ldots,p_T]$ into a textual trajectory string, appends it to the prompt, and uses trajectory-aware attention optimization inside an OpenSora-based video diffusion model to refine motion during sampling [2510.00806]. The defining claim of this system is that physical plausibility is introduced not by retraining the generator, but by LVLM reasoning and inference-time attention guidance.

**Autonomous driving planning:** LLaViDA operationalizes TrajVLM as a “VLM Trajectory Planner for Autonomous Driving.” It uses six calibrated surround-view cameras during supervised fine-tuning, lightweight serialized perception from BEVFormer, ego history and mission goals, and a fixed schema prompt to generate scene semantics, multi-agent motion forecasts, a meta-action, and numeric BEV waypoints in a single inference turn [2512.18211]. The model’s explicit chain-of-thought is not merely explanatory; it forecasts critical-object trajectories $T_{obj}$ and grounds planning in road layout, weather, signage, and ego intent.

**Long-video reasoning:** In TrajTok, TrajVLM denotes a LLaVA-style vision-language model that replaces standard patch pooling with TrajTok, an end-to-end differentiable trajectory tokenizer [2602.22779]. A universal segmenter produces soft and hard spatiotemporal masks; a trajectory encoder aggregates and refines object-specific features; each trajectory can emit $n \in \{1,2,4\}$ tokens; and the resulting visual tokens are projected into the Qwen3-4B embedding space. The primary purpose is not trajectory prediction in the classical forecasting sense, but compact temporal grounding for long-horizon reasoning.

**Trajectory-aware spatial understanding:** TraceVision uses “TrajVLM” in the broader sense of trajectory-aware vision-language modeling. It treats human attention trajectories as continuous, temporally ordered signals of intent and fuses them with visual features via the Trajectory-aware Visual Perception module [2602.19768]. The model supports trajectory-guided captioning, text-guided trajectory prediction, referring localization, referring segmentation, and video scene understanding, extending the trajectory concept from motion planning into interpretable region-language association.

**Trajectory data mining:** Traj-MLLM extends the idea into geospatial analytics. Raw trajectories $T=\{p_1,\ldots,p_L\}$, where each point contains $(lon_l, lat_l, t_l)$, are segmented, rendered on map tiles, paired with structured textual descriptors, and assembled into temporally interleaved image–text sequences $M^T$ for training-free multimodal reasoning [2509.00053]. The paper states that it does not introduce a model explicitly named “TrajVLM,” but that it fulfills the same role by bridging trajectory modality to vision-language inputs.

## 4. Data construction and learning objectives

A notable trait of TrajVLM systems is that their datasets and losses are usually specialized to the trajectory form they manipulate. TrajVLM-Gen builds a trajectory prediction dataset of **1.3M image–video–trajectory triplets**, including **650K** global alignment captions from LLaVA1.5-665K, **600K** trajectory pretraining clips from TNL2K, LaSOT, LaSOT-ext, OTB99-Lang, and GOT-10K, and **50K** trajectory instruction-following examples enriched with physics labels [2510.00806]. The generic trajectory supervision is coordinate regression, written as either $L_{traj}=\sum_{t=1}^T \|p_t-p_{gt,t}\|_1$ or $\sum_t \|p_t-p_{gt,t}\|_2^2$, while kinematic statistics $v_t = p_t-p_{t-1}$, $a_t = v_t-v_{t-1}$, and $j_t = a_t-a_{t-1}$ are used for labeling “gravity,” “elastic,” and “perspective camera projection” rather than as explicit losses. During generation, trajectory guidance is expressed through an attention energy
$$
E(A_t)=M_{traj,t}\odot \operatorname{sign}(A_t)-\lambda \nabla^2 A_t,
$$
which biases cross-attention toward masked regions and enforces spatial smoothness [2510.00806].

LLaViDA uses the NuScenes-TP dataset, comprising **23,423 training samples** and **6,019 test samples**, each a **40-frame** sequence at **2 Hz** with ego and object trajectories, meta-actions, and GPT-4o synthesized reasoning traces [2512.18211]. Its supervised fine-tuning objective is token-level cross-entropy over reasoning, meta-action, and numeric trajectory outputs,
$$
L_{SFT}(\theta)=L_{reason}+\lambda(L_{traj}+L_{meta}),
$$
with $\lambda=1.2$, followed by Trajectory Preference Optimization. TPO samples **$K=16$** responses per instance with temperature **$\tau=1.5$**, defines positive and negative trajectories by average L2 displacement over the $3\,\text{s}$ horizon, and optimizes a DPO-style preference loss with **$\beta=0.1$** [2512.18211]. This is a distinctive attempt to inject regression-like supervision directly into a generative objective.

TrajTok’s TrajVLM is trained differently because its core problem is tokenization rather than direct coordinate prediction. The universal segmenter is supervised with pseudo panoptic trajectory masks using **Dice loss plus Focal loss**, and the paper explicitly states that cross-entropy is not used [2602.22779]. Visual patch features are gradient-detached before the Perceiver to stabilize training. TrajVLM itself is aligned by language modeling: a caption pretraining stage on PixMo for **1 epoch**, followed by **10,000-step** instruction tuning on approximately **5M** VideoQA examples, using **8×A100 (80GB)**, **bfloat16**, batch size **32**, learning rate **$1\times10^{-5}$** for the LLM, and **$5\times10^{-6}$** for the ViT and connector [2602.22779].

TraceVision adopts a three-stage curriculum. Stage 1 uses **0.25M** Localized Narratives samples while freezing the visual encoder and LLM; Stage 1.5 jointly trains on **1.33M** mixed samples across LN, LNV, PNG, and REC/RES; Stage 2 performs instruction fine-tuning on **1.3M** samples, primarily RILN, reasoning segmentation, and interactive editing [2602.19768]. The core autoregressive loss is
$$
L_{txt}=-\sum_t \log P(y_t \mid y_{1:t-1}, f_{visual}, f_{Traj}),
$$
and segmentation supervision adds
$$
L_{total}=L_{txt}+\lambda_{dice}L_{dice}+\lambda_{ref}L_{ref},
$$
with default $\lambda_{dice}=2.0$ and $\lambda_{ref}=1.0$ [2602.19768].

Traj-MLLM stands apart by being training-free. It uses prompt optimization with **$\le 10$** seed trajectories per task, fixes the role and output-format components of the system prompt, and iteratively refines task description and domain knowledge via MLLM feedback rather than gradient updates [2509.00053]. This suggests a separate branch of TrajVLM research in which the trajectory representation is engineered for MLLM compatibility instead of learned end to end.

## 5. Empirical performance across domains

The empirical profile of TrajVLM is strongly task-dependent. In image-to-video generation, TrajVLM-Gen reports **FVD = 545** on **UCF-101** and **FVD = 539** on **MSR-VTT**, outperforming LVD on both datasets and being competitive with VideoLDM on UCF-101 [2510.00806]. An ablation shows that trajectory-based attention masks improve FVD on UCF-101 from **632** without masks to **545** with masks, and on MSR-VTT from **558** to **539** [2510.00806]. The same system also reports **81.2%** on VQAv2, **88.1%** on POPE, **36.3%** on MMMU, and **89.6%** average across Numeracy, Attribution, Visibility, Dynamics, and Sequential in LVD-style detection-based evaluation [2510.00806].

In autonomous driving, LLaViDA reports state-of-the-art open-loop planning on NuScenes. Under the ST-P3 protocol, it achieves horizon-wise L2 errors of **0.14 / 0.28 / 0.51 m**, an average of **0.31 m**, and collision rates of **0.03 / 0.07 / 0.19%**, averaging **0.10%** [2512.18211]. Under UniAD, it reports **0.19 / 0.54 / 1.09 m** with average **0.61 m**, and collision rates **0.06 / 0.09 / 0.50%**, averaging **0.22%** [2512.18211]. The paper attributes a major part of this gain to TPO: ST-P3 average L2 improves from **0.39** to **0.31 m**, and UniAD average L2 from **0.78** to **0.61 m** when moving from SFT-only to SFT+TPO [2512.18211].

For long-video reasoning, TrajTok’s TrajVLM reports consistent gains over patch-pooling baselines on long-video benchmarks, including **+8.8%** on LongVideoBench and **+5.4%** on LVBench over PatchVLM with pool size $x=3$ [2602.22779]. The paper further emphasizes that increasing patch pooling to $x=9$ to match TrajTok’s token count and frame support does not close the gap, indicating that object-centric grouping rather than token reduction alone drives the improvement. This is an important empirical distinction because it isolates the effect of trajectory tokenization from mere compression.

TraceVision evaluates a different axis of performance: trajectory-grounded understanding and localization. It reports controlled caption generation results of **BLEU-4 = 0.328**, **METEOR = 0.276**, **ROUGE-L = 0.492**, **CIDEr = 1.530**, and **SPICE = 0.328**; controlled trajectory generation with **LBM(k=0)=0.117** and **LBM(k=1)=0.121**; referring localization on RefCOCO with **P@0.5 = 90.4/93.1/87.8** for val/testA/testB; referring segmentation with **cIoU = 83.4/86.8/82.4** on the same splits; and video results including **HC-STVG METEOR = 24.1**, **CIDEr = 92.6**, and **VideoRefer-Bench-D Avg = 3.83** [2602.19768]. These numbers place TrajVLM-like modeling outside forecasting alone and into general multimodal grounding.

Traj-MLLM contributes another performance profile: on four public datasets it reports improvements over state-of-the-art baselines by **48.05%** on travel time estimation, **15.52%** on mobility prediction, **51.52%** on anomaly detection, and **1.83%** on transportation mode identification, without training or fine-tuning the MLLM backbones [2509.00053]. The explicit implication is that a trajectory VLM can be competitive even when the trajectory modality is first transformed into map-rendered image–text sequences rather than modeled directly.

## 6. Limitations, critiques, and emerging directions

The principal limitations of TrajVLM systems are task-specific but structurally similar: ambiguity in trajectory grounding, brittleness under distribution shift, and difficulty enforcing hard physical or geometric constraints. TrajVLM-Gen notes that physics consistency can break in complex multi-agent scenes, heavy occlusions, or when ground-truth motion depends on 3D factors not captured by 2D box trajectories; it also states that without retraining the generator, long-horizon dynamics and rapid topological changes remain hard, and that OpenSora’s 3D VAE encoding challenges precise spatial control even when attention masks help [2510.00806]. LLaViDA reports that vision-only conditioning can hallucinate small or occluded objects, that front-only efficiency variants slightly increase L2 and collision, and that no explicit rule-checker is used for traffic-rule compliance [2512.18211]. TrajTok identifies failure modes for very small or thin objects, over-merging or fragmentation under fast motion and dense scenes, and cross-chunk consistency issues at 16-frame boundaries [2602.22779]. TraceVision notes dependence on LN/RILN annotations, degradation under extreme compression of trajectories, and the limits of a lightweight segmentation decoder in boundary-heavy scenes [2602.19768].

A broader critique emerges from work-zone trajectory planning. “Work Zones challenge VLM Trajectory Planning” reports that mainstream VLMs fail in **68.0%** of work-zone planning cases, with scenario-level failure rates of **75.86%** for Qwen2.5-VL, **70.37%** for GPT-4o, **80.00%** for Gemini 2.5, **81.93%** for SimLingo, **76.63%** for RoboTron-Drive, and **77.23%** for DriveLM [2510.02803]. Its mitigation, REACT-Drive, replaces direct free-form planning with retrieval, constraint rules, executable code, and verification, achieving approximately **$3\times$** reduction in average displacement error relative to VLM baselines under evaluation with Qwen2.5-VL and **0.58 s** inference time [2510.02803]. This challenge is relevant to TrajVLM because it shows that language-grounded trajectory generation alone is not sufficient in geometrically irregular or safety-critical domains.

Related trajectory-planning papers point toward likely responses. BEV-VLM shows that using a unified BEV-HD map can yield a **44.8%** decrease in displacement error and **0.00%** collision rate over the $3\,\text{s}$ horizon on nuScenes, suggesting that stronger geometric abstraction can make VLM planning more reliable [2509.25249]. VisionTrap shows that training-time language supervision derived from a VLM and refined by an LLM improves multi-agent forecasting while retaining **53 ms** real-time inference, with final **ADE\_10 = 1.17**, **MR\_10 = 0.32**, and **FDE\_1 = 8.72** in its ablation setting [2407.12345]. A plausible implication is that future TrajVLM systems will increasingly combine trajectory reasoning with explicit geometry, retrieval, structured outputs, or programmatic constraints rather than relying on unconstrained multimodal generation alone.

Taken together, the literature portrays TrajVLM not as a single architecture but as an emerging research category organized around one principle: trajectories can serve simultaneously as outputs, inputs, latent tokens, supervision signals, or interpretable attention traces. The most mature systems differ sharply in domain and implementation, yet they converge on the view that multimodal intelligence improves when temporal-spatial structure is represented explicitly rather than left implicit in pixels or text alone.

Source: https://www.emergentmind.com/topics/trajvlm