TrajTok: Trajectory Tokenization

• TrajTok is a family of methods that tokenize trajectories instead of raw coordinates or patches, providing tailored computation for distinct domains.
• In autonomous driving, it discretizes motion into tokens for next-token prediction, achieving measurable gains in realism and simulation performance.
• For video and GPS applications, TrajTok enables semantic grouping and adaptive spatial tokenization, leading to improved representation efficiency and accuracy.

Searching arXiv for papers on “TrajTok” and closely related trajectory-tokenization work. TrajTok is a label used in recent arXiv literature for several distinct but conceptually related trajectory-centered tokenization frameworks rather than for one single canonical method. Across autonomous-driving simulation, video understanding, and raw GPS trajectory representation learning, the common premise is that trajectory structure can serve as the basic unit of computation more effectively than dense frames, raw coordinates, or uniformly discretized grids. In one line of work, TrajTok is a hybrid rule-based and data-driven trajectory tokenizer for discrete next-token-prediction behavior generation in the Waymo Open Sim Agents Challenge (Zhang et al., 23 Jun 2025). In another, it is an end-to-end differentiable video tokenizer that groups pixels into object or region trajectories and emits semantically adaptive video tokens (Zheng et al., 26 Feb 2026). In a third, it is a raw-GPS trajectory encoder built on adaptive multi-resolution H3 tokenization, factorized geometry/kinematics encoding, and masked-token pretraining (Xiong et al., 19 May 2026).

1. Scope, terminology, and conceptual basis

In current usage, “TrajTok” denotes a family of methods whose tokenization unit is trajectory-derived rather than patch-derived or pointwise in the conventional sense. The term therefore spans at least three problem settings: behavior generation for autonomous-driving simulation, efficient video tokenization for downstream visual understanding, and trajectory representation learning from raw GPS traces. What unifies them is not a shared implementation, but a shared modeling move: convert continuous spatiotemporal structure into reusable tokens that can be consumed by transformer-style encoders or next-token predictors.

Paper	Domain	Token unit
"TrajTok: Technical Report for 2025 Waymo Open Sim Agents Challenge" (Zhang et al., 23 Jun 2025)	Driving simulation	$c_i \in \mathbb{R}^{L \times 3}$ motion primitive
"TrajTok: Learning Trajectory Tokens enables better Video Understanding" (Zheng et al., 26 Feb 2026)	Video understanding	Adaptive set $\mathbf{Z} \in \mathbb{R}^{N \times d}$ of trajectory tokens
"TrajTok: Adaptive Spatial Tokenization for Trajectory Representation Learning" (Xiong et al., 19 May 2026)	Raw GPS modeling	Multi-resolution H3 cell tokens with kinematic channel

The differences are substantial. In the driving setting, a token is a short future trajectory primitive in agent-centric coordinates, used as an action in a discrete rollout policy. In the video setting, a token is a learned latent representing an object or region trajectory extracted by implicit spatiotemporal clustering. In the GPS setting, tokenization is discrete in space but not purely symbolic overall, because the model keeps a separate continuous kinematic stream for speed and heading. This suggests that “trajectory token” is best treated as a design principle rather than a fixed data type.

2. TrajTok for discrete behavior generation in autonomous-driving simulation

In the Waymo Open Sim Agents Challenge formulation, TrajTok is introduced for discrete next-token-prediction behavior generation models such as SMART and Trajeglish (Zhang et al., 23 Jun 2025). The task is to generate future multi-agent states $\{\mathcal{S}_{1}, \ldots, \mathcal{S}_{T_f}\}$ from the HD map $\mathcal{M}$ and past states $\{\mathcal{S}_{-T_h}, \ldots, \mathcal{S}_{0}\}$. Instead of directly regressing future coordinates, the model predicts a token from a finite vocabulary $\mathcal{V} = \{c_1,\dots,c_{|\mathcal V|}\}$, where each token

$c_i \in \mathbb{R}^{L \times 3}$

contains $L$ points in agent-centric coordinates, with $(x,y,\text{yaw})$ at each point. At rollout step $t$, the model predicts

$\mathbf{Z} \in \mathbb{R}^{N \times d}$0

with $\mathbf{Z} \in \mathbb{R}^{N \times d}$1 for each agent $\mathbf{Z} \in \mathbb{R}^{N \times d}$2, and then converts the token back into world coordinates through

$\mathbf{Z} \in \mathbb{R}^{N \times d}$3

The tokenizer is explicitly hybrid. Rules define and partition the trajectory space, while data determines which regions are valid and helps instantiate realistic tokens. Construction proceeds in four stages: extraction of all valid trajectories of length $\mathbf{Z} \in \mathbb{R}^{N \times d}$4 and normalization to the agent-centric coordinate system; flipping along the x-axis,

$\mathbf{Z} \in \mathbb{R}^{N \times d}$5

to enforce symmetry; gridding of endpoint space; and a filtering-and-expanding step based on local occupancy. Sparse isolated occupied cells are treated as noise and filtered out, while empty cells with dense local support are expanded in. Token generation is then cellwise: if $\mathbf{Z} \in \mathbb{R}^{N \times d}$6 is non-empty, the token is the mean trajectory,

$\mathbf{Z} \in \mathbb{R}^{N \times d}$7

and if the cell is valid but empty, the token is synthesized by interpolation,

$\mathbf{Z} \in \mathbb{R}^{N \times d}$8

The main experimental setting uses $\mathbf{Z} \in \mathbb{R}^{N \times d}$9, a re-plan frequency of $\{\mathcal{S}_{1}, \ldots, \mathcal{S}_{T_f}\}$0 Hz, and prediction at $\{\mathcal{S}_{1}, \ldots, \mathcal{S}_{T_f}\}$1 Hz over a $\{\mathcal{S}_{1}, \ldots, \mathcal{S}_{T_f}\}$2-second trajectory segment every $\{\mathcal{S}_{1}, \ldots, \mathcal{S}_{T_f}\}$3 seconds. The challenge submission uses type-specific endpoint grids and vocabularies of 8040 for vehicles, 3001 for bicycles, and 2798 for pedestrians. The model replaces SMART’s original shared head with separate prediction heads for each agent type. Training also introduces spatial-aware label smoothing. Standard cross-entropy is written as

$\{\mathcal{S}_{1}, \ldots, \mathcal{S}_{T_f}\}$4

but the non-ground-truth mass is redistributed according to inverse squared trajectory distance,

$\{\mathcal{S}_{1}, \ldots, \mathcal{S}_{T_f}\}$5

with target distribution

$\{\mathcal{S}_{1}, \ldots, \mathcal{S}_{T_f}\}$6

and $\{\mathcal{S}_{1}, \ldots, \mathcal{S}_{T_f}\}$7.

Empirically, the paper reports Realism Meta 0.7852, Kinematic 0.4887, Interactive 0.8116, Map-based 0.9207, and minADE 1.3179, with leaderboard rank 2nd in the 2025 Waymo Open Sim Agents Challenge submission period (Zhang et al., 23 Jun 2025). Compared with SMART_topk32 using k-disks, Realism Meta rises from 0.7814 to 0.7852, a gain of +0.0038. Ablations show that TrajTok outperforms k-disks across tested vocabulary sizes, and that an intermediate tokenizer size is best: for TrajTok, 2062 / 1349 / 1290 → 0.7533, 8040 / 3001 / 2798 → 0.7579, and 12057 / 4183 / 3923 → 0.7554. The report also notes several underspecified components, including the neighborhood size $\{\mathcal{S}_{1}, \ldots, \mathcal{S}_{T_f}\}$8, the exact filtering and expansion thresholds, the interpolation function $\{\mathcal{S}_{1}, \ldots, \mathcal{S}_{T_f}\}$9, and the precise norm in $\mathcal{M}$0. This suggests that the algorithmic structure is clear, while byte-for-byte reproducibility depends on the promised code release.

3. TrajTok for semantically adaptive video tokenization

In video understanding, TrajTok is an end-to-end differentiable tokenizer integrated into the visual model and optimized jointly with downstream objectives (Zheng et al., 26 Feb 2026). The central critique is that standard patchification generates token counts that grow with video duration and spatial resolution, while video semantics are more naturally organized around persistent entities or regions. TrajTok therefore replaces uniform space-time patches with learned trajectory tokens. Formally, it defines a mapping

$\mathcal{M}$1

where

$\mathcal{M}$2

and $\mathcal{M}$3 is not fixed by duration.

The tokenizer has two differentiable components: a universal segmenter and a trajectory encoder. A lightweight ConvNeXt patch encoder first produces dense features

$\mathcal{M}$4

The segmenter uses learnable latent queries

$\mathcal{M}$5

processed by Perceiver layers with 1D Rotary Positional Embeddings: $\mathcal{M}$6 Soft segmentation is then defined by

$\mathcal{M}$7

Initial trajectory tokens are obtained by soft aggregation,

$\mathcal{M}$8

and refinement uses masked Perceiver attention over hard masks derived by argmax over $\mathcal{M}$9. The model can emit $\{\mathcal{S}_{-T_h}, \ldots, \mathcal{S}_{0}\}$0 tokens per trajectory, with Fourier positional embeddings used to diversify the multi-token setting. In the main architecture, the tokenizer uses a ConvNeXt-tiny patch encoder, a 2-layer Perceiver in the segmenter, a 2-layer Perceiver in the trajectory encoder, 8 attention heads, and 128 learnable segmenter queries.

Training is joint. The segmenter is supervised with pseudo panoptic trajectory masks from the external TrajViT pipeline, but inference is self-contained. The default segmentation loss is Dice plus Focal: $\{\mathcal{S}_{-T_h}, \ldots, \mathcal{S}_{0}\}$1 Downstream video representation is learned with a CLIP-style contrastive objective, and the paper states that the tokenizer and transformer are trained jointly under CLIP with added segmentation supervision. The main instantiated model, TrajViT2, uses TrajTok as its primary tokenizer; TrajAdapter uses it as a probing head for frozen pretrained backbones; and TrajVLM uses it as a connector between a vision encoder and a LLM.

The empirical profile is broad (Zheng et al., 26 Feb 2026). On attentive probing, TrajViT2 reports 59.1 on K400 and 48.7 on SSv2, versus 54.2 and 46.3 for ViT3D. The abstract highlights +4.8% on Kinetics-400 and +4.1% on SSv2 over a standard video ViT. Retrieval improvements are similarly consistent: 42.2 versus 38.1 on ActivityNet vid2txt, 65.0 versus 61.1 on VATEX vid2txt, 65.3 versus 61.2 on MSR-VTT vid2txt, 75.6 versus 69.0 on COCO img2txt, and 90.4 versus 85.1 on Flickr30K img2txt. As an adapter, TrajTok improves probing on VideoMAE-v2 from 80.2 / 59.7 to 82.5 / 60.9 and on V-JEPA2 from 85.1 / 74.2 to 88.0 / 75.1 when using 4 tokens per trajectory. As a VLM connector, TrajVLM reports +8.8% on LongVideoBench and +5.4% on LVBench over PatchVLM with default pool size $\{\mathcal{S}_{-T_h}, \ldots, \mathcal{S}_{0}\}$2. The main caveats are also explicit: training still depends on pseudo masks, the lightweight segmenter may miss tiny objects and over-merge backgrounds, simple images such as ImageNet can induce too few segments, and long-video processing is still chunked into 16-frame clips.

A key historical antecedent is "One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectory" (Zheng et al., 29 May 2025), which introduced grounded video tokenization with panoptic sub-object trajectories and implemented it through TrajViT. That earlier system used DirectSAM for key-frame panoptic segmentation, SAM2 for clipwise tracking, boundary merging with IoU $\{\mathcal{S}_{-T_h}, \ldots, \mathcal{S}_{0}\}$3, and one token per full panoptic sub-object trajectory, achieving about 10× token reduction and an average 6% top-5 recall improvement in video-text retrieval over ViT3D. TrajTok’s distinctive contribution relative to that line is end-to-end differentiability and co-training with the downstream objective.

4. TrajTok for raw GPS trajectory representation learning

In raw GPS modeling, TrajTok is a trajectory encoder designed to learn transferable representations directly from GPS traces without map matching or road-graph priors (Xiong et al., 19 May 2026). A trajectory is written as

$\{\mathcal{S}_{-T_h}, \ldots, \mathcal{S}_{0}\}$4

and the main claim is that strong representations require jointly addressing spatial discretization and modality-aware encoding. The tokenizer therefore builds a multi-resolution H3 vocabulary by recursive density-based splitting. Starting from base resolution $\{\mathcal{S}_{-T_h}, \ldots, \mathcal{S}_{0}\}$5, each cell with count $\{\mathcal{S}_{-T_h}, \ldots, \mathcal{S}_{0}\}$6 is split until its count falls below threshold or maximum resolution $\{\mathcal{S}_{-T_h}, \ldots, \mathcal{S}_{0}\}$7 is reached. On Porto, the main configuration uses $\{\mathcal{S}_{-T_h}, \ldots, \mathcal{S}_{0}\}$8, $\{\mathcal{S}_{-T_h}, \ldots, \mathcal{S}_{0}\}$9, $\mathcal{V} = \{c_1,\dots,c_{|\mathcal V|}\}$0, and yields vocabulary size $\mathcal{V} = \{c_1,\dots,c_{|\mathcal V|}\}$1. Each GPS point is mapped to the highest-resolution vocabulary cell that contains it.

The encoder is factorized into geometric and kinematic streams. The geometric input is a learned embedding lookup

$\mathcal{V} = \{c_1,\dots,c_{|\mathcal V|}\}$2

while the kinematic input is

$\mathcal{V} = \{c_1,\dots,c_{|\mathcal V|}\}$3

where speed is computed from haversine distance over elapsed time and heading is encoded by sine and cosine. The model uses a factorized transformer with 16 total layers, 4 cross-attention fusion layers, $\mathcal{V} = \{c_1,\dots,c_{|\mathcal V|}\}$4, 8 attention heads, $\mathcal{V} = \{c_1,\dots,c_{|\mathcal V|}\}$5, and about 144M parameters. Early layers apply modality-specific self-attention; later fusion layers use cross-attention,

$\mathcal{V} = \{c_1,\dots,c_{|\mathcal V|}\}$6

All attention blocks are pre-norm residual blocks with 2-layer GeGLU MLPs.

Positional encoding is handled by spatiotemporal rotary embeddings. In the geometric stream, the positional tuple is

$\mathcal{V} = \{c_1,\dots,c_{|\mathcal V|}\}$7

with $\mathcal{V} = \{c_1,\dots,c_{|\mathcal V|}\}$8 per attention head and latitude and longitude multiplied by $\mathcal{V} = \{c_1,\dots,c_{|\mathcal V|}\}$9 before forming rotary angles. In the kinematic stream, RoPE is temporal only, $c_i \in \mathbb{R}^{L \times 3}$0. Queries and keys are rotated through the operator $c_i \in \mathbb{R}^{L \times 3}$1 before attention. This design is intended to encode actual spatiotemporal relations rather than sequence index alone.

Pretraining is co-masked. The same positions are masked in both streams, and the model jointly predicts masked cells and reconstructs masked motion attributes. The geometric loss is

$c_i \in \mathbb{R}^{L \times 3}$2

while the kinematic loss is

$c_i \in \mathbb{R}^{L \times 3}$3

with $c_i \in \mathbb{R}^{L \times 3}$4 and $c_i \in \mathbb{R}^{L \times 3}$5, and total objective

$c_i \in \mathbb{R}^{L \times 3}$6

A notable recipe detail is run-aware span masking, introduced because naive masking inside repeated-cell runs can make recovery trivial. The main masking settings are $c_i \in \mathbb{R}^{L \times 3}$7 and average span length $c_i \in \mathbb{R}^{L \times 3}$8. Training uses AdamW, learning rate $c_i \in \mathbb{R}^{L \times 3}$9, weight decay $L$0, cosine decay, 20% warmup, bf16, effective batch size 256, 60k steps, and a single NVIDIA RTX 6000 Ada, 48 GiB GPU, with roughly 1.5–2 hours per run.

On the Porto dataset, the frozen encoder achieves 0.435 HR@1 on trajectory similarity search, 0.773 macro-F1 on classification, 42.27 s MAE on ETA, and 38.41 s MAE on full travel-time regression (Xiong et al., 19 May 2026). More detailed similarity results are R5@20 = 0.983, MRR = 0.588, and NDCG@50 = 0.661 with the retrieval head, while the zero-shot encoder alone reaches HR@1 = 0.351 and MRR = 0.503. The main architectural ablation is especially informative: geometric-only gives 0.286 HR@1, 0.773 macro-F1, 44.44 ETA MAE, and 87.52 TTE MAE; kinematic-only gives 0.028, 0.632, 43.71, and 91.94; both streams without fusion give 0.251, 0.729, 46.33, and 53.09; and full TrajTok gives 0.297, 0.783, 40.21, and 47.10. The limitations are explicit: all experiments are on Porto, there is no cross-city transfer study, and lower pretraining loss does not monotonically imply better transfer.

5. Shared architectural motifs and major divergences

Despite their domain differences, the three TrajTok lines share several recurrent design ideas. First, each treats tokenization as an inductive bias rather than a purely preprocessing step. The driving tokenizer defines the simulator’s discrete action space; the video tokenizer learns semantically adaptive trajectory groupings; and the GPS tokenizer shapes the symbolic space on which masked modeling operates (Zhang et al., 23 Jun 2025, Zheng et al., 26 Feb 2026, Xiong et al., 19 May 2026). Second, each uses trajectory structure to reduce ambiguity in a dense original modality. In driving, this means replacing open-ended continuous action regression with a finite vocabulary of short motion primitives. In video, it means replacing uniform patchification with entity- or region-centric trajectories. In raw GPS modeling, it means replacing uniform spatial grids with adaptive H3 cells while retaining continuous motion information through a second channel.

The token semantics, however, are not interchangeable. In the Waymo challenge setting, a token is an executable $L$1-second future motion primitive in agent-centric coordinates. In end-to-end video tokenization, a token is a latent pooled representation over a spatiotemporal mask that may correspond to an object, an object part, or a background region. In raw GPS modeling, the discrete token is only the geometric half of the representation; motion attributes remain continuous. A plausible implication is that “trajectory tokenization” is less a single recipe than a spectrum of choices about what should be discretized, what should remain continuous, and where task semantics enter.

Self-supervision also differs systematically. The Waymo tokenizer improves token classification with spatial-aware label smoothing but does not pretrain a standalone tokenizer. The video tokenizer couples segmentation supervision with CLIP-style visual-language learning. The GPS encoder uses masked-token modeling that jointly reconstructs symbolic geometry and continuous kinematics. This suggests that trajectory tokenization is closely coupled to the supervision structure of the downstream domain: rollout realism in simulation, semantic grouping in video, and transferable structure in mobility analytics.

6. Relation to adjacent trajectory-token research and field-wide limitations

Several adjacent papers sharpen the meaning of TrajTok by showing what it is not. In action recognition, "TrAction: Action Recognition with Sparse Trajectories" (Meier et al., 2 Jun 2026) uses sparse 2.5D trajectories extracted by CoTracker3 and Video Depth Anything-L, with one full trajectory $L$2 converted into one transformer token through sinusoidal encoding, flattening, and a 2-layer MLP. The model uses 32 time steps, extracts 2500 trajectories per video, trains on random 512-trajectory subsets, and employs masked-trajectory pretraining at default mask ratio 0.9. It reaches 45.2 top-1 on Something-Something V2, 54.1 on EPIC-Kitchens-100 verbs, and 33.1 on Kinetics-400, while late fusion with DINOv2-B improves SSv2 from 53.6 to 62.3 and with V-JEPA 2-L from 66.7 to 68.3. TrAction is not named TrajTok, but it exemplifies a closely related “one full trajectory, one token” design for motion-specialized video understanding.

In open-vocabulary tracking, "Attention to Trajectory: Trajectory-Aware Open-Vocabulary Tracking" (Li et al., 11 Mar 2025) does not tokenize trajectories into a vocabulary, but treats each trajectory as a persistent memory object with a smoothed feature state, a bank of historical embeddings, and a category bank. Its TraCLIP module aggregates per-frame CLIP features with self-attention and average pooling into a trajectory embedding, then matches that embedding against vanilla and attribute-enriched text prompts. On OV-TAO, the full TRACT system improves a baseline from 37.5 TETA / 16.9 ClsA to 38.6 TETA / 20.3 ClsA, with Trajectory Feature Aggregation providing the largest classification gain. This line reinforces a broader interpretation of trajectory tokens as stateful semantic objects rather than merely discrete symbols.

Across the literature, several limitations recur. Tokenizer design is often heuristic or dependent on external infrastructure. The Waymo tokenizer leaves neighborhood thresholds and interpolation details unspecified. The video tokenizer still depends on pseudo trajectory masks during training, and its lightweight segmenter may miss small objects. Grounded video tokenization in the TrajViT line depends on segmentation and tracking quality, and its inference bottleneck is the tracking pipeline (Zheng et al., 29 May 2025). The GPS encoder currently lacks cross-city evaluation. More broadly, the name “TrajTok” itself is domain-specific and non-standardized. This suggests that the term is best understood as an emerging research motif—trajectory as token—rather than as a settled architecture with fixed semantics.