TrajViT2: Differentiable Trajectory Video Transformer

• The paper introduces a differentiable trajectory-token model that replaces fixed patch-based tokenization for improved video understanding.
• It presents TrajTok—a jointly learned tokenizer combining segmentation and trajectory encoding that adapts token granularity to video complexity.
• Empirical results demonstrate enhanced retrieval and classification performance, achieving efficiency and scalability across varied video benchmarks.

Searching arXiv for TrajViT and related trajectory-token video understanding papers. TrajViT2 is a video CLIP model trained from scratch using TrajTok as its tokenizer, introduced in “TrajTok: Learning Trajectory Tokens enables better Video Understanding” (Zheng et al., 26 Feb 2026). It preserves the “one trajectory, one token” spirit of the earlier TrajViT idea, but replaces the external, slow, task-agnostic segmentation-and-tracking pipeline with a jointly learned tokenizer. In this formulation, tokenization is optimized for downstream understanding rather than for pixel-perfect segmentation, yielding fewer, more semantically meaningful tokens and adapting token granularity to the actual complexity of the video (Zheng et al., 26 Feb 2026). The model is positioned as an alternative to standard patchification in video Vision Transformers, where token count grows with both resolution and duration and many tokens are near-duplicates across time.

1. Conceptual basis

TrajViT2 is motivated by a critique of ordinary video ViTs that split every frame into fixed space-time patches. In that setting, token count grows with both resolution and duration, and many tokens are redundant across time. The paper argues that this is inefficient and scales poorly (Zheng et al., 26 Feb 2026). A trajectory-based representation instead treats an object trajectory as the primary video unit, on the grounds that objects persist across frames and many frames can be summarized as evolving instances rather than as independent patch grids.

Within this framework, trajectory tokens decouple token count from video length and align better with human perception and object-centric reasoning (Zheng et al., 26 Feb 2026). Prior trajectory tokenizers had already shown that trajectory grouping can outperform raw patch tokens, but they depended on external pipelines and were not end-to-end trainable. TrajViT2 adopts the trajectory-token premise while making the tokenizer differentiable, lightweight, and jointly optimized with the downstream objective.

This suggests that TrajViT2 should be understood less as a minor tokenizer replacement than as a reformulation of how a video transformer allocates representational capacity. The paper’s central claim is that downstream adaptability, rather than pixel-perfect segmentation fidelity, is the decisive design criterion for effective video tokenization (Zheng et al., 26 Feb 2026).

2. TrajTok as the tokenizer for TrajViT2

The tokenizer used by TrajViT2, TrajTok, has two main parts: a universal segmenter and a trajectory encoder (Zheng et al., 26 Feb 2026). The input video is denoted as $$\mathbf{V} \in \mathbb{R}^{T \times H \times W \times 3}$$, and the goal is to map it to a variable number of trajectory tokens $\mathbf{Z} \in \mathbb{R}^{N \times d}$, where $N$ is not fixed.

First, a lightweight ConvNeXt-based patch encoder extracts dense features frame-wise. Multi-scale feature maps are resized to the highest resolution, $1/4$ of the original image size, and summed into a dense spatiotemporal tensor $$\mathbf{F} \in \mathbb{R}^{T \times h \times w \times d}$$ with $h,w = H/4, W/4$ (Zheng et al., 26 Feb 2026). The design explicitly avoids a heavy mask-decoder pipeline; the paper states that for semantic grouping, coarse features are sufficient.

The segmenter performs what the paper calls implicit clustering over pixels in both space and time. It introduces $N_q$ learnable latent queries $\mathbf{Q} \in \mathbb{R}^{N_q \times d}$, which act as cluster prototypes. These queries are refined by Perceiver layers, where the queries cross-attend to $\mathbf{F}$. To encode time, the patch features receive 1D RoPE before attention (Zheng et al., 26 Feb 2026). After the Perceiver stack, the processed queries are turned into soft segmentation masks by dot-product similarity and a softmax over queries:

$$\mathbf{M}^{\text{soft}}_{k,t,i,j} = \text{softmax}_k\left(\hat{\mathbf{q}}_k \cdot \mathbf{F}_{t,i,j}\right).$$

Each feature location $\mathbf{Z} \in \mathbb{R}^{N \times d}$0 is therefore softly assigned to one of the learned query clusters. The paper emphasizes that this is done in a single forward pass, without iterative tracking, and that queries whose masks are empty are discarded so that the final number of trajectories $\mathbf{Z} \in \mathbb{R}^{N \times d}$1 varies with scene complexity (Zheng et al., 26 Feb 2026). Long videos are chunked temporally so the model can process them in parallel, further decoupling token count from duration.

The trajectory encoder then converts trajectory masks into tokens. It first computes an initial trajectory embedding by weighted aggregation:

$\mathbf{Z} \in \mathbb{R}^{N \times d}$2

This soft aggregation provides a differentiable proposal token and allows gradients from the downstream task to flow back into the segmenter (Zheng et al., 26 Feb 2026). Because a pure soft average can blur detail, the model refines the proposals with a second Perceiver module. The initial trajectory embeddings become queries again, and attention is masked by a hard segmentation map $\mathbf{Z} \in \mathbb{R}^{N \times d}$3, produced by argmax over the soft masks, so each token can attend only to the features assigned to its own cluster. The paper describes this as recovering finer motion and texture details while keeping the representation disentangled (Zheng et al., 26 Feb 2026).

3. Adaptive token granularity

A distinctive property of TrajTok within TrajViT2 is its adaptive token granularity. The paper explicitly states that a single token per trajectory can be too restrictive for long, articulated, or appearance-changing objects (Zheng et al., 26 Feb 2026). To address this, the trajectory encoder supports multiple tokens per trajectory, with $\mathbf{Z} \in \mathbb{R}^{N \times d}$4.

Under this mechanism, the same trajectory is duplicated into several sub-tokens, each with its own learnable query, and the model learns to distribute information across them. To encourage these sub-tokens to specialize rather than collapse onto the same region, their queries are initialized with Fourier positional embeddings with angular offsets (Zheng et al., 26 Feb 2026). During training, $\mathbf{Z} \in \mathbb{R}^{N \times d}$5 is randomly sampled per batch, which the paper describes as being inspired by Matryoshka representation learning.

This training strategy makes one model work at multiple granularities, and at inference the token budget can be adjusted by choosing $\mathbf{Z} \in \mathbb{R}^{N \times d}$6 (Zheng et al., 26 Feb 2026). A plausible implication is that TrajViT2 does not treat token count as a fixed architectural constant, but as a controllable operating point balancing efficiency and expressivity. In the paper’s own framing, token granularity adapts to semantic complexity rather than video duration.

4. End-to-end optimization and pretraining configuration

In the main TrajViT2 setting, the tokenizer is trained jointly with the video transformer under the CLIP objective, with additional segmentation supervision on the segmenter (Zheng et al., 26 Feb 2026). The paper states that TrajViT2 uses the same protocol as TrajViT, but now the segmenter is learned simultaneously with the transformer instead of being frozen from an external pipeline.

The full pretraining setup reported for TrajViT2 is summarized below.

Component	Specification
Backbone	ViT-Large-style transformer
Input and sampling	224-resolution inputs; 16 uniformly sampled frames at evaluation; 8 frames sampled during training
Optimization	AdamW with base LR $\mathbf{Z} \in \mathbb{R}^{N \times d}$7; weight decay $\mathbf{Z} \in \mathbb{R}^{N \times d}$8; cosine schedule with one-epoch warmup; mixed precision
Duration and scale	20 epochs; batch size of 128 videos and 1024 images on 8 A100 GPUs
Training corpus	4M video clips from Panda-70M plus 15M image-caption pairs from CC3M and CC12M
Loss	CLIP contrastive loss for video-text alignment, plus segmentation loss for the tokenizer

For the segmenter, the paper states that Dice loss and Focal loss work best, while cross-entropy is not necessary; Dice is described as especially important because it promotes complete object discovery rather than exact pixelwise classification (Zheng et al., 26 Feb 2026). The ablation tables reported in the paper further state that removing Dice hurts retrieval and segmentation the most, and that removing gradient detachment or hard masks also degrades performance substantially.

The same descriptions indicate that TrajTok is best characterized as a differentiable clustering-and-pooling module over space-time patches. The soft mask equation provides probabilistic assignment of every feature location to a trajectory cluster; the soft aggregation equation converts each cluster into an initial token; and hard masked cross-attention refines each token using only its assigned region (Zheng et al., 26 Feb 2026).

5. Empirical performance and scaling

TrajViT2 is reported to outperform prior video and token-merging baselines on both classification and retrieval benchmarks (Zheng et al., 26 Feb 2026). On retrieval, Table 1 reports 42.2 vid2txt R@5 on ActivityNet, 65.0 on VATEX, and 65.3 on MSR-VTT, above TrajViT and above ViT3D, TokenLearner, ViViT, and RLT. On classification, TrajViT2 reports 59.1 on Kinetics-400 and 48.7 on Something-Something V2, again the best among the listed methods (Zheng et al., 26 Feb 2026).

The paper characterizes these results as “best accuracy at scale” because the advantage is maintained or increased as the pretraining dataset grows from 1M to 8M video clips (Zheng et al., 26 Feb 2026). In the scaling plot described in the paper, TrajViT2 shows a stronger upward trend than TrajViT, whereas TrajViT’s advantage over a standard ViT shrinks with scale. The stated interpretation is that the tokenizer benefits more from additional data because its segmentation behavior can adapt to the task rather than being fixed by a heuristic external pipeline.

This suggests that the central empirical contribution of TrajViT2 is not merely a better small-scale tokenizer, but a tokenizer whose inductive bias remains useful under larger-scale contrastive pretraining. In that sense, the model is presented as a scaling-compatible object-centric alternative to patch-based tokenization (Zheng et al., 26 Feb 2026).

6. Efficiency profile and ablation evidence

Efficiency is a major part of the TrajViT2 claim. The paper states that the whole trajectory tokenizer has only 46M parameters, compared with the 304M-parameter ViT-Large backbone (Zheng et al., 26 Feb 2026). In FLOPs versus frame count, TrajViT2 is described as nearly as efficient as ViViT, the most efficient baseline, while standard patch-based ViT3D scales quadratically and TrajViT has a much steeper linear cost because of its external pipeline.

The stated efficiency result is therefore not only lower tokenizer overhead, but an operating point in which strong performance is maintained while matching the best token-merging baselines in inference cost (Zheng et al., 26 Feb 2026). The paper explicitly presents the retrieval and classification gains as not being purchased at the expense of large compute overhead.

The ablations clarify which components matter. Removing hierarchical features from the patch encoder hurts both segmentation quality and retrieval. Removing gradient detachment causes large drops because the queries and patch features destabilize each other. Increasing output resolution gives little retrieval gain, which the paper uses to support the claim that coarse semantic grouping is sufficient (Zheng et al., 26 Feb 2026). On the encoder side, removing the hard attention mask causes a clear performance drop, indicating that the token should remain tied to its assigned region. Fourier-based query initialization matters especially when using multiple tokens per trajectory; random initialization weakens the benefit of multiple sub-tokens. Depth beyond 2 Perceiver layers provides only small gains (Zheng et al., 26 Feb 2026).

These findings also delimit a common misconception. TrajViT2 is not presented as a system whose tokenizer becomes effective by approximating high-fidelity panoptic segmentation. The qualitative examples instead show object-level regions and trajectory masks that are semantically meaningful even if they are not pixel-perfect; very small objects may be missed and background regions may be merged, but these imperfections are treated as acceptable because the target is downstream understanding rather than segmentation quality (Zheng et al., 26 Feb 2026).

7. Extensions and broader significance

Although TrajViT2 is the paper’s CLIP-trained video transformer, TrajTok is also used in two additional systems: TrajAdapter and TrajVLM (Zheng et al., 26 Feb 2026). TrajAdapter inserts the tokenizer after a frozen video backbone and uses the resulting trajectory tokens for probing. On VideoMAE-v2 and V-JEPA2, TrajAdapter improves Kinetics-400 and SSv2 over linear probing, attentive probing, and a Perceiver-only baseline, and the gains increase as more tokens per trajectory are allowed (Zheng et al., 26 Feb 2026).

TrajVLM uses TrajTok as the connector between a vision encoder and an LLM in a LLaVA-style setup. With Qwen3-4B and SigLIP2-Huge, TrajTok beats patch-pooling connectors on long-video QA benchmarks such as LongVideoBench and LVBench (Zheng et al., 26 Feb 2026). The paper attributes this to a reduction in redundancy while preserving object-centric structure, so that the LLM receives tokens organized around temporally persistent entities rather than arbitrary patch grids.

Taken together, these extensions position TrajViT2 as the canonical instantiation of a broader tokenization principle: unified segmentation and trajectory encoding optimized end-to-end for downstream objectives. A plausible implication is that the main contribution of TrajViT2 lies not only in improved CLIP-style video pretraining, but in establishing trajectory tokenization as a reusable interface for probing and multimodal alignment. Within the paper’s own summary, TrajViT2 is a CLIP-trained video transformer that uses TrajTok instead of patchification, and its key design choice is end-to-end differentiability with downstream supervision rather than an external segmentation-and-tracking pipeline (Zheng et al., 26 Feb 2026).