---
title: 'TrajTok: Trajectory Tokenization'
url: https://www.emergentmind.com/topics/trajtok
type: topic
---

# TrajTok: Trajectory Tokenization

Searching arXiv for recent papers on “TrajTok” and closely related trajectory-tokenization work.
TrajTok is a label used in recent arXiv literature for several distinct but conceptually related trajectory-centered tokenization frameworks rather than for one single canonical method. Across autonomous-driving simulation, video understanding, and raw GPS trajectory representation learning, the common premise is that trajectory structure can serve as the basic unit of computation more effectively than dense frames, raw coordinates, or uniformly discretized grids. In one line of work, TrajTok is a hybrid rule-based and data-driven trajectory tokenizer for discrete next-token-prediction behavior generation in the Waymo Open Sim Agents Challenge [2506.21618]. In another, it is an end-to-end differentiable video tokenizer that groups pixels into object or region trajectories and emits semantically adaptive video tokens [2602.22779]. In a third, it is a raw-GPS trajectory encoder built on adaptive multi-resolution H3 tokenization, factorized geometry/kinematics encoding, and masked-token pretraining [2605.20134].

## 1. Scope, terminology, and conceptual basis

In current usage, “TrajTok” denotes a family of methods whose tokenization unit is trajectory-derived rather than patch-derived or pointwise in the conventional sense. The term therefore spans at least three problem settings: behavior generation for autonomous-driving simulation, efficient video tokenization for downstream visual understanding, and trajectory representation learning from raw GPS traces. What unifies them is not a shared implementation, but a shared modeling move: convert continuous spatiotemporal structure into reusable tokens that can be consumed by transformer-style encoders or next-token predictors.

| Paper | Domain | Token unit |
|---|---|---|
| "TrajTok: Technical Report for 2025 Waymo Open Sim Agents Challenge" [2506.21618] | Driving simulation | \(c_i \in \mathbb{R}^{L \times 3}\) motion primitive |
| "TrajTok: Learning Trajectory Tokens enables better Video Understanding" [2602.22779] | Video understanding | Adaptive set \(\mathbf{Z} \in \mathbb{R}^{N \times d}\) of trajectory tokens |
| "TrajTok: Adaptive Spatial Tokenization for Trajectory Representation Learning" [2605.20134] | Raw GPS modeling | Multi-resolution H3 cell tokens with kinematic channel |

The differences are substantial. In the driving setting, a token is a short future trajectory primitive in agent-centric coordinates, used as an action in a discrete rollout policy. In the video setting, a token is a learned latent representing an object or region trajectory extracted by implicit spatiotemporal clustering. In the GPS setting, tokenization is discrete in space but not purely symbolic overall, because the model keeps a separate continuous kinematic stream for speed and heading. This suggests that “trajectory token” is best treated as a design principle rather than a fixed data type.

## 2. TrajTok for discrete behavior generation in autonomous-driving simulation

In the Waymo Open Sim Agents Challenge formulation, TrajTok is introduced for discrete next-token-prediction behavior generation models such as SMART and Trajeglish [2506.21618]. The task is to generate future multi-agent states \(\{\mathcal{S}_{1}, \ldots, \mathcal{S}_{T_f}\}\) from the HD map \(\mathcal{M}\) and past states \(\{\mathcal{S}_{-T_h}, \ldots, \mathcal{S}_{0}\}\). Instead of directly regressing future coordinates, the model predicts a token from a finite vocabulary \(\mathcal{V} = \{c_1,\dots,c_{|\mathcal V|}\}\), where each token
\[
c_i \in \mathbb{R}^{L \times 3}
\]
contains \(L\) points in agent-centric coordinates, with \((x,y,\text{yaw})\) at each point. At rollout step \(t\), the model predicts
\[
a_{t} = \mathcal{N_{\theta,V}(\mathcal{S}_{-T_h:t \times L},\mathcal{M})},
\]
with \(a_t^{(i)} \in \mathcal V\) for each agent \(i\), and then converts the token back into world coordinates through
\[
\mathcal{S}_{t\times L+1:t\times (L+1)+1} = f(a_t, \mathcal{S}_t).
\]

The tokenizer is explicitly hybrid. Rules define and partition the trajectory space, while data determines which regions are valid and helps instantiate realistic tokens. Construction proceeds in four stages: extraction of all valid trajectories of length \(L\) and normalization to the agent-centric coordinate system; flipping along the x-axis,
\[
\widetilde{D} = Concentrate( D, Flip(D)),
\]
to enforce symmetry; gridding of endpoint space; and a filtering-and-expanding step based on local occupancy. Sparse isolated occupied cells are treated as noise and filtered out, while empty cells with dense local support are expanded in. Token generation is then cellwise: if \(\hat D[i][j]\) is non-empty, the token is the mean trajectory,
\[
c_n = Mean(\hat D[i][j]),
\]
and if the cell is valid but empty, the token is synthesized by interpolation,
\[
c_n = curve\_interp(0,p[i][j],L).
\]

The main experimental setting uses \(L=5\), a re-plan frequency of \(2\) Hz, and prediction at \(10\) Hz over a \(0.5\)-second trajectory segment every \(0.5\) seconds. The challenge submission uses type-specific endpoint grids and vocabularies of **8040** for vehicles, **3001** for bicycles, and **2798** for pedestrians. The model replaces SMART’s original shared head with separate prediction heads for each agent type. Training also introduces spatial-aware label smoothing. Standard cross-entropy is written as
\[
\mathcal L = - \sum_k q_k \log p_k,
\]
but the non-ground-truth mass is redistributed according to inverse squared trajectory distance,
\[
k_i = \frac{1}{||\mathbf{c_i}-\mathbf{c_j}||^2+\varepsilon_1},
\]
with target distribution
\[
y_{i}=\left\{\begin{array}{l}
1-\varepsilon \quad \text { if } \quad i=j \\
\frac{ \varepsilon k_i}{\sum_{m=0,m \ne j}^{|\mathcal{V}|} k_m} \quad \text { if } \quad i \ne j
\end{array}\right.
\]
and \(\varepsilon = 0.1\).

Empirically, the paper reports **Realism Meta 0.7852**, **Kinematic 0.4887**, **Interactive 0.8116**, **Map-based 0.9207**, and **minADE 1.3179**, with leaderboard rank **2nd** in the 2025 Waymo Open Sim Agents Challenge submission period [2506.21618]. Compared with **SMART\_topk32** using k-disks, Realism Meta rises from **0.7814** to **0.7852**, a gain of **+0.0038**. Ablations show that TrajTok outperforms k-disks across tested vocabulary sizes, and that an intermediate tokenizer size is best: for TrajTok, **2062 / 1349 / 1290 → 0.7533**, **8040 / 3001 / 2798 → 0.7579**, and **12057 / 4183 / 3923 → 0.7554**. The report also notes several underspecified components, including the neighborhood size \(k\), the exact filtering and expansion thresholds, the interpolation function \(curve\_interp\), and the precise norm in \(||\mathbf c_i - \mathbf c_j||\). This suggests that the algorithmic structure is clear, while byte-for-byte reproducibility depends on the promised code release.

## 3. TrajTok for semantically adaptive video tokenization

In video understanding, TrajTok is an end-to-end differentiable tokenizer integrated into the visual model and optimized jointly with downstream objectives [2602.22779]. The central critique is that standard patchification generates token counts that grow with video duration and spatial resolution, while video semantics are more naturally organized around persistent entities or regions. TrajTok therefore replaces uniform space-time patches with learned trajectory tokens. Formally, it defines a mapping
\[
\mathcal{T}: \mathbf{V} \rightarrow \mathbf{Z},
\]
where
\[
\mathbf{V} \in \mathbb{R}^{T \times H \times W \times 3}, \qquad \mathbf{Z} \in \mathbb{R}^{N \times d},
\]
and \(N\) is not fixed by duration.

The tokenizer has two differentiable components: a universal segmenter and a trajectory encoder. A lightweight ConvNeXt patch encoder first produces dense features
\[
\mathbf{F} \in \mathbb{R}^{T \times h \times w \times d}, \qquad h = H/4,\; w = W/4.
\]
The segmenter uses learnable latent queries
\[
\mathbf{Q} \in \mathbb{R}^{N_q \times d},
\]
processed by Perceiver layers with 1D Rotary Positional Embeddings:
\[
\hat{\mathbf{Q}} = \text{Perceiver}(\mathbf{Q}, \text{RoPE}(\mathbf{F})).
\]
Soft segmentation is then defined by
\[
\mathbf{M}^{\text{soft}}_{k,t,i,j} = \text{softmax}_k\left(\hat{\mathbf{q}}_k \cdot \mathbf{F}_{t,i,j}\right).
\]
Initial trajectory tokens are obtained by soft aggregation,
\[
\mathbf{z}^{\text{init}}_k = \sum_{t,i,j} \mathbf{M}^{\text{soft}}_{k,t,i,j} \cdot \mathbf{F}_{t,i,j},
\]
and refinement uses masked Perceiver attention over hard masks derived by argmax over \(\mathbf{M}^{\text{soft}}\). The model can emit \(n \in \{1,2,4\}\) tokens per trajectory, with Fourier positional embeddings used to diversify the multi-token setting. In the main architecture, the tokenizer uses a ConvNeXt-tiny patch encoder, a 2-layer Perceiver in the segmenter, a 2-layer Perceiver in the trajectory encoder, 8 attention heads, and **128** learnable segmenter queries.

Training is joint. The segmenter is supervised with pseudo panoptic trajectory masks from the external TrajViT pipeline, but inference is self-contained. The default segmentation loss is Dice plus Focal:
\[
\mathcal{L}_{\text{seg}} = \mathcal{L}_{\text{dice}} + \mathcal{L}_{\text{focal}}.
\]
Downstream video representation is learned with a CLIP-style contrastive objective, and the paper states that the tokenizer and transformer are trained jointly under CLIP with added segmentation supervision. The main instantiated model, TrajViT2, uses TrajTok as its primary tokenizer; TrajAdapter uses it as a probing head for frozen pretrained backbones; and TrajVLM uses it as a connector between a vision encoder and a language model.

The empirical profile is broad [2602.22779]. On attentive probing, TrajViT2 reports **59.1** on K400 and **48.7** on SSv2, versus **54.2** and **46.3** for ViT3D. The abstract highlights **+4.8% on Kinetics-400** and **+4.1% on SSv2** over a standard video ViT. Retrieval improvements are similarly consistent: **42.2** versus **38.1** on ActivityNet vid2txt, **65.0** versus **61.1** on VATEX vid2txt, **65.3** versus **61.2** on MSR-VTT vid2txt, **75.6** versus **69.0** on COCO img2txt, and **90.4** versus **85.1** on Flickr30K img2txt. As an adapter, TrajTok improves probing on VideoMAE-v2 from **80.2 / 59.7** to **82.5 / 60.9** and on V-JEPA2 from **85.1 / 74.2** to **88.0 / 75.1** when using 4 tokens per trajectory. As a VLM connector, TrajVLM reports **+8.8% on LongVideoBench** and **+5.4% on LVBench** over PatchVLM with default pool size \(x=3\). The main caveats are also explicit: training still depends on pseudo masks, the lightweight segmenter may miss tiny objects and over-merge backgrounds, simple images such as ImageNet can induce too few segments, and long-video processing is still chunked into 16-frame clips.

A key historical antecedent is "One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectory" [2505.23617], which introduced grounded video tokenization with panoptic sub-object trajectories and implemented it through TrajViT. That earlier system used DirectSAM for key-frame panoptic segmentation, SAM2 for clipwise tracking, boundary merging with IoU \(> 0.8\), and one token per full panoptic sub-object trajectory, achieving about **10× token reduction** and an average **6%** top-5 recall improvement in video-text retrieval over ViT3D. TrajTok’s distinctive contribution relative to that line is end-to-end differentiability and co-training with the downstream objective.

## 4. TrajTok for raw GPS trajectory representation learning

In raw GPS modeling, TrajTok is a trajectory encoder designed to learn transferable representations directly from GPS traces without map matching or road-graph priors [2605.20134]. A trajectory is written as
\[
\mathcal{T} = \{p_1, p_2, \ldots, p_n\}, \quad p_i = (\phi_i, \lambda_i, t_i),
\]
and the main claim is that strong representations require jointly addressing spatial discretization and modality-aware encoding. The tokenizer therefore builds a multi-resolution H3 vocabulary by recursive density-based splitting. Starting from base resolution \(r_{\min}\), each cell with count \(> C\) is split until its count falls below threshold or maximum resolution \(r_{\max}\) is reached. On Porto, the main configuration uses \(r_{\min} = 6\), \(r_{\max} = 9\), \(C = 1000\), and yields vocabulary size \(|\mathcal{V}| = 1494\). Each GPS point is mapped to the highest-resolution vocabulary cell that contains it.

The encoder is factorized into geometric and kinematic streams. The geometric input is a learned embedding lookup
\[
\mathbf{g}_j = \mathbf{e}_{c_j},
\]
while the kinematic input is
\[
\mathbf{x}^{\text{kin}}_j = \left[ \frac{v_j}{v_{\max}},\ \sin\theta_j,\ \cos\theta_j \right],
\qquad
\mathbf{k}_j = \mathrm{MLP}(\mathbf{x}^{\text{kin}}_j),
\]
where speed is computed from haversine distance over elapsed time and heading is encoded by sine and cosine. The model uses a factorized transformer with **16 total layers**, **4 cross-attention fusion layers**, \(d_{\text{model}} = 512\), **8 attention heads**, \(d_h = 64\), and about **144M parameters**. Early layers apply modality-specific self-attention; later fusion layers use cross-attention,
\[
G^{(\ell+1)} = \mathrm{CrossAttn}\!\left(\tilde{G}^{(\ell)},\, K^{(\ell)};\, \mathbf{P}\right), \qquad
K^{(\ell+1)} = \mathrm{CrossAttn}\!\left(\tilde{K}^{(\ell)},\, G^{(\ell)};\, \mathbf{P}\right).
\]
All attention blocks are pre-norm residual blocks with 2-layer GeGLU MLPs.

Positional encoding is handled by spatiotemporal rotary embeddings. In the geometric stream, the positional tuple is
\[
\mathbf{p}_j = (\phi_j, \lambda_j, t_j),
\]
with \((d_\phi, d_\lambda, d_t) = (20, 20, 24)\) per attention head and latitude and longitude multiplied by \(10^4\) before forming rotary angles. In the kinematic stream, RoPE is temporal only, \(\mathbf{p}_j = t_j\). Queries and keys are rotated through the operator \(\mathcal{R}(\cdot;\mathbf{p})\) before attention. This design is intended to encode actual spatiotemporal relations rather than sequence index alone.

Pretraining is co-masked. The same positions are masked in both streams, and the model jointly predicts masked cells and reconstructs masked motion attributes. The geometric loss is
\[
\mathcal{L}_{\text{geom}} = - \frac{1}{|\mathcal{M}|} \sum_{j \in \mathcal{M}} \log p_\theta(c_j \mid \mathbf{G}),
\]
while the kinematic loss is
\[
\mathcal{L}_{\text{kin}} = \beta_{\text{speed}} \, \mathrm{MSE}(\hat{v}_j, v_j / v_{\max}) + \frac{1}{2}\beta_{\text{heading}} \, \mathrm{MSE}(\widehat{\sin \theta}_j, \sin \theta_j) + \frac{1}{2}\beta_{\text{heading}} \, \mathrm{MSE}(\widehat{\cos \theta}_j, \cos \theta_j),
\]
with \(\beta_{\text{speed}} = 1.0\) and \(\beta_{\text{heading}} = 1.0\), and total objective
\[
\mathcal{J} = \mathcal{L}_{\text{geom}} + \lambda \mathcal{L}_{\text{kin}}.
\]
A notable recipe detail is run-aware span masking, introduced because naive masking inside repeated-cell runs can make recovery trivial. The main masking settings are \(\rho = 0.30\) and average span length \(\ell = 6\). Training uses AdamW, learning rate \(10^{-4}\), weight decay \(0.01\), cosine decay, **20%** warmup, bf16, effective batch size **256**, **60k steps**, and a single **NVIDIA RTX 6000 Ada, 48 GiB** GPU, with roughly **1.5–2 hours per run**.

On the Porto dataset, the frozen encoder achieves **0.435 HR@1** on trajectory similarity search, **0.773 macro-F1** on classification, **42.27 s MAE** on ETA, and **38.41 s MAE** on full travel-time regression [2605.20134]. More detailed similarity results are **R5@20 = 0.983**, **MRR = 0.588**, and **NDCG@50 = 0.661** with the retrieval head, while the zero-shot encoder alone reaches **HR@1 = 0.351** and **MRR = 0.503**. The main architectural ablation is especially informative: geometric-only gives **0.286 HR@1**, **0.773 macro-F1**, **44.44** ETA MAE, and **87.52** TTE MAE; kinematic-only gives **0.028**, **0.632**, **43.71**, and **91.94**; both streams without fusion give **0.251**, **0.729**, **46.33**, and **53.09**; and full TrajTok gives **0.297**, **0.783**, **40.21**, and **47.10**. The limitations are explicit: all experiments are on Porto, there is no cross-city transfer study, and lower pretraining loss does not monotonically imply better transfer.

## 5. Shared architectural motifs and major divergences

Despite their domain differences, the three TrajTok lines share several recurrent design ideas. First, each treats tokenization as an inductive bias rather than a purely preprocessing step. The driving tokenizer defines the simulator’s discrete action space; the video tokenizer learns semantically adaptive trajectory groupings; and the GPS tokenizer shapes the symbolic space on which masked modeling operates [2506.21618; 2602.22779; 2605.20134]. Second, each uses trajectory structure to reduce ambiguity in a dense original modality. In driving, this means replacing open-ended continuous action regression with a finite vocabulary of short motion primitives. In video, it means replacing uniform patchification with entity- or region-centric trajectories. In raw GPS modeling, it means replacing uniform spatial grids with adaptive H3 cells while retaining continuous motion information through a second channel.

The token semantics, however, are not interchangeable. In the Waymo challenge setting, a token is an executable \(0.5\)-second future motion primitive in agent-centric coordinates. In end-to-end video tokenization, a token is a latent pooled representation over a spatiotemporal mask that may correspond to an object, an object part, or a background region. In raw GPS modeling, the discrete token is only the geometric half of the representation; motion attributes remain continuous. A plausible implication is that “trajectory tokenization” is less a single recipe than a spectrum of choices about what should be discretized, what should remain continuous, and where task semantics enter.

Self-supervision also differs systematically. The Waymo tokenizer improves token classification with spatial-aware label smoothing but does not pretrain a standalone tokenizer. The video tokenizer couples segmentation supervision with CLIP-style visual-language learning. The GPS encoder uses masked-token modeling that jointly reconstructs symbolic geometry and continuous kinematics. This suggests that trajectory tokenization is closely coupled to the supervision structure of the downstream domain: rollout realism in simulation, semantic grouping in video, and transferable structure in mobility analytics.

## 6. Relation to adjacent trajectory-token research and field-wide limitations

Several adjacent papers sharpen the meaning of TrajTok by showing what it is not. In action recognition, "TrAction: Action Recognition with Sparse Trajectories" [2606.03490] uses sparse **2.5D trajectories** extracted by CoTracker3 and Video Depth Anything-L, with one full trajectory \(\mathbf{p} \in \mathbb{R}^{T \times 4}\) converted into one transformer token through sinusoidal encoding, flattening, and a 2-layer MLP. The model uses **32** time steps, extracts **2500** trajectories per video, trains on random **512**-trajectory subsets, and employs masked-trajectory pretraining at default mask ratio **0.9**. It reaches **45.2** top-1 on Something-Something V2, **54.1** on EPIC-Kitchens-100 verbs, and **33.1** on Kinetics-400, while late fusion with DINOv2-B improves SSv2 from **53.6** to **62.3** and with V-JEPA 2-L from **66.7** to **68.3**. TrAction is not named TrajTok, but it exemplifies a closely related “one full trajectory, one token” design for motion-specialized video understanding.

In open-vocabulary tracking, "Attention to Trajectory: Trajectory-Aware Open-Vocabulary Tracking" [2503.08145] does not tokenize trajectories into a vocabulary, but treats each trajectory as a persistent memory object with a smoothed feature state, a bank of historical embeddings, and a category bank. Its TraCLIP module aggregates per-frame CLIP features with self-attention and average pooling into a trajectory embedding, then matches that embedding against vanilla and attribute-enriched text prompts. On OV-TAO, the full TRACT system improves a baseline from **37.5 TETA / 16.9 ClsA** to **38.6 TETA / 20.3 ClsA**, with Trajectory Feature Aggregation providing the largest classification gain. This line reinforces a broader interpretation of trajectory tokens as stateful semantic objects rather than merely discrete symbols.

Across the literature, several limitations recur. Tokenizer design is often heuristic or dependent on external infrastructure. The Waymo tokenizer leaves neighborhood thresholds and interpolation details unspecified. The video tokenizer still depends on pseudo trajectory masks during training, and its lightweight segmenter may miss small objects. Grounded video tokenization in the TrajViT line depends on segmentation and tracking quality, and its inference bottleneck is the tracking pipeline [2505.23617]. The GPS encoder currently lacks cross-city evaluation. More broadly, the name “TrajTok” itself is domain-specific and non-standardized. This suggests that the term is best understood as an emerging research motif—trajectory as token—rather than as a settled architecture with fixed semantics.

Source: https://www.emergentmind.com/topics/trajtok