TrajAdapter: Efficient Trajectory-Based Probing

Updated 4 July 2026

TrajAdapter is a probing module that reorganizes dense video features into robust trajectory tokens for improved video action classification.
The technique leverages a frozen video backbone with pretrained segmenters to generate soft and hard attention masks that group features efficiently.
Empirical results show that trajectory tokenization boosts classification accuracy over standard linear and attentive probing baselines on action recognition benchmarks.

Searching arXiv for TrajAdapter and related TrajTok work to ground the article in the cited literature. TrajAdapter is a plug-in feature-adaptation and probing module derived from TrajTok that takes dense output features from a frozen pretrained video encoder and reorganizes them into a compact set of trajectory tokens before a lightweight probing classifier predicts the downstream label (Zheng et al., 26 Feb 2026). In the TrajTok framework, it is the probing-oriented instantiation of trajectory tokenization: TrajTok is inserted after a frozen ViT backbone, the segmenter is pretrained and frozen, and only the trajectory encoder and probing head are trained. The module is positioned as a cost-effective, parameter-efficient alternative to standard probing heads, motivated by the claim that pretrained video backbones already provide rich patch-level features but that conventional probing may not fully exploit trajectory and object structure.

1. Definition and scope

TrajAdapter is defined in the downstream probing setting rather than in backbone pretraining. Its central problem is how to adapt dense pretrained video features into more informative, lower-redundancy tokens that better support downstream classification while keeping the backbone frozen (Zheng et al., 26 Feb 2026). The method therefore asks whether trajectory grouping can improve probing accuracy when a strong pretrained encoder already exists; the reported answer is affirmative.

This definition is narrow and should be distinguished from the broader TrajTok program. In the main TrajTok setting, the tokenizer is trained end-to-end, can operate directly on raw video pixels, and is used in settings such as TrajViT2 and TrajVLM. TrajAdapter, by contrast, is specifically the reuse of TrajTok as an adapter and probing head for frozen pretrained video encoders. It is therefore neither the pretrained backbone itself nor the full tokenizer in the from-scratch training sense.

A recurrent source of confusion is that “trajectory adaptation” appears in several literatures with different meanings. In the TrajTok paper, TrajAdapter denotes a probing module for video understanding. It is evaluated on action recognition rather than on retrieval or long-video reasoning; retrieval is mainly associated with TrajViT2, and long-video reasoning with TrajVLM (Zheng et al., 26 Feb 2026).

2. Placement in the video understanding pipeline

The TrajAdapter pipeline is explicitly specified as a sequence of five stages: a frozen pretrained video backbone, dense patch-level backbone features, a TrajTok segmenter that groups those features into trajectory masks, a trajectory encoder that aggregates features into trajectory tokens, and an attentive probing head that predicts class logits (Zheng et al., 26 Feb 2026). The backbone examples given are VideoMAE-v2 and V-JEPA2, both ViT-Huge.

Architecturally, TrajAdapter reuses the main TrajTok components. The segmenter produces trajectory masks over the backbone feature map. In the main architecture it uses a ConvNeXt-Tiny patch encoder, extracts dense features at $1/4$ resolution, uses $128$ learnable queries, applies Perceiver layers, uses RoPE for spatiotemporal structure, and produces soft masks by query-feature similarity. The paper gives the soft mask formulation as

$\mathbf{M}^{\text{soft}}_{k,t,i,j}=\mathrm{softmax}_k\!\left(\hat{\mathbf{q}}_k \cdot \mathbf{F}_{t,i,j}\right).$

The trajectory encoder then aggregates features using those masks. The initial trajectory token is formed by the weighted sum

$\mathbf{z}^{\text{init}}_k=\sum_{t,i,j}\mathbf{M}^{\text{soft}}_{k,t,i,j}\cdot \mathbf{F}_{t,i,j},$

after which a second Perceiver module refines the representation using hard masks for masked cross-attention (Zheng et al., 26 Feb 2026). The paper emphasizes that hard masking is part of the adaptation mechanism rather than a peripheral implementation detail.

TrajAdapter also supports the Matryoshka-inspired multi-token mechanism with $n\in\{1,2,4\}$ tokens per trajectory. Training samples $n$ randomly per batch, while inference can choose $n$ according to the compute budget. This gives the module an explicit efficiency-expressivity trade-off without changing the frozen backbone.

3. Frozen and trainable components

The frozen-versus-trainable partition is one of the defining properties of TrajAdapter. The backbone is frozen, and the segmenter is pretrained and frozen during probing. Only the trajectory encoder and probing head are trained jointly (Zheng et al., 26 Feb 2026). This is the sense in which the paper characterizes the module as a parameter-efficient plug-in rather than a full fine-tuning system.

The probing abstraction is likewise explicit. The tokenizer can accept any pretrained feature $f(\mathrm{video})$ , enabling it to operate as a plug-in feature adapter across downstream tasks. In practice, however, the reported TrajAdapter experiments are confined to video action classification. A plausible implication is that the method’s transfer claim is architectural rather than task-universal: it demonstrates that trajectory grouping is useful when dense frozen video features are already available, not that the module replaces all downstream adaptation strategies.

The design motivation is object-centric reorganization. Standard linear probing, attentive probing, and Perceiver-style probing operate over pretrained features without trajectory priors. TrajAdapter instead groups patch tokens into semantically meaningful object-centric units. The paper further states that segmentation fidelity is less important than downstream semantic grouping, which is why coarser masks, missed tiny objects, and imperfect boundaries are treated as acceptable if the resulting tokens better serve understanding (Zheng et al., 26 Feb 2026).

4. Training protocol and probing regime

TrajAdapter is trained in a supervised probing setup rather than an end-to-end self-supervised regime. The task is video action classification, the benchmarks are Kinetics-400 and Something-Something V2, the input is uniformly sampled $16$ frames, and the maximum output is $128$ trajectory tokens (Zheng et al., 26 Feb 2026).

The supplementary training details specify AdamW, learning rate $128$0, weight decay $128$1, batch size $128$2, and $128$3 training epochs. Features are layer-normalized before classification. The paper does not introduce a separate custom objective for TrajAdapter beyond the probing loss; instead, the segmenter is pretrained from pseudo trajectory masks, while the trajectory encoder and probing head are trained for classification.

The baselines are linear probing, attentive probing, and Perceiver probing with the same size and number of learnable queries as TrajAdapter but without trajectory priors. The evaluation metric is Top-1 classification accuracy. This baseline construction matters because it isolates the claimed source of improvement: the paper argues that gains over the Perceiver-only baseline indicate that the effect is not simply due to additional parameters, but to trajectory priors (Zheng et al., 26 Feb 2026).

5. Reported empirical performance

The main quantitative evidence appears in the probing table on different video backbones. TrajAdapter consistently improves over linear probing, attentive probing, and Perceiver probing on both VideoMAE-v2 and V-JEPA2, and performance increases as the number of tokens per trajectory increases (Zheng et al., 26 Feb 2026).

Method	VideoMAE-v2 (K400 / SSv2)	V-JEPA2 (K400 / SSv2)
Linear probing	79.4 / 59.1	84.5 / 73.7
Attentive probing	80.2 / 59.7	85.1 / 74.2
Perceiver probing	79.9 / 59.8	84.7 / 74.2
TrajAdapter, 1 token/traj	82.0 / 60.4	87.2 / 74.6
TrajAdapter, 2 token/traj	82.4 / 60.8	87.8 / 74.9
TrajAdapter, 4 token/traj	82.5 / 60.9	88.0 / 75.1

For VideoMAE-v2, the best setting reaches $128$4 on Kinetics-400 and $128$5 on Something-Something V2, compared with $128$6 and $128$7 for attentive probing. For V-JEPA2, the best setting reaches $128$8 and $128$9, compared with $\mathbf{M}^{\text{soft}}_{k,t,i,j}=\mathrm{softmax}_k\!\left(\hat{\mathbf{q}}_k \cdot \mathbf{F}_{t,i,j}\right).$ 0 and $\mathbf{M}^{\text{soft}}_{k,t,i,j}=\mathrm{softmax}_k\!\left(\hat{\mathbf{q}}_k \cdot \mathbf{F}_{t,i,j}\right).$ 1 for attentive probing. Even the $\mathbf{M}^{\text{soft}}_{k,t,i,j}=\mathrm{softmax}_k\!\left(\hat{\mathbf{q}}_k \cdot \mathbf{F}_{t,i,j}\right).$ 2-token-per-trajectory configuration exceeds the conventional probing baselines on both backbones (Zheng et al., 26 Feb 2026).

The paper’s concise interpretation is that TrajTok is useful not only as a tokenizer for end-to-end pretraining, but also as a better probing head for pretrained encoders. This suggests that the trajectory prior remains beneficial even when the underlying visual representation is already strong.

6. Ablations, design claims, and interpretive boundaries

The probing-related ablations focus on the trajectory encoder. Removing the hard attention mask causes a large drop, from $\mathbf{M}^{\text{soft}}_{k,t,i,j}=\mathrm{softmax}_k\!\left(\hat{\mathbf{q}}_k \cdot \mathbf{F}_{t,i,j}\right).$ 3 to $\mathbf{M}^{\text{soft}}_{k,t,i,j}=\mathrm{softmax}_k\!\left(\hat{\mathbf{q}}_k \cdot \mathbf{F}_{t,i,j}\right).$ 4 at $\mathbf{M}^{\text{soft}}_{k,t,i,j}=\mathrm{softmax}_k\!\left(\hat{\mathbf{q}}_k \cdot \mathbf{F}_{t,i,j}\right).$ 5 token per trajectory, with similarly large drops for $\mathbf{M}^{\text{soft}}_{k,t,i,j}=\mathrm{softmax}_k\!\left(\hat{\mathbf{q}}_k \cdot \mathbf{F}_{t,i,j}\right).$ 6 and $\mathbf{M}^{\text{soft}}_{k,t,i,j}=\mathrm{softmax}_k\!\left(\hat{\mathbf{q}}_k \cdot \mathbf{F}_{t,i,j}\right).$ 7 tokens per trajectory (Zheng et al., 26 Feb 2026). The paper interprets this as evidence that hard masked attention is important for keeping trajectory tokens tied to their assigned regions.

Query initialization also matters. Fourier initialization is reported as better than random, especially in multi-token settings, and random initialization reduces the benefit of increasing token count. The stated interpretation is that diversity among sub-tokens matters. Perceiver depth shows only marginal gain when increasing from $\mathbf{M}^{\text{soft}}_{k,t,i,j}=\mathrm{softmax}_k\!\left(\hat{\mathbf{q}}_k \cdot \mathbf{F}_{t,i,j}\right).$ 8 to $\mathbf{M}^{\text{soft}}_{k,t,i,j}=\mathrm{softmax}_k\!\left(\hat{\mathbf{q}}_k \cdot \mathbf{F}_{t,i,j}\right).$ 9, supporting the claim that the adapter’s main value comes from trajectory grouping rather than deep token refinement (Zheng et al., 26 Feb 2026).

These findings constrain how TrajAdapter should be understood. It is not presented as a generic “more layers help” result, nor as a claim that any token compression improves probing. The argument is narrower: trajectory grouping, hard masked attention, and token diversity are the components that materially contribute to performance. A plausible implication is that TrajAdapter’s efficacy depends on preserving a structurally meaningful assignment between grouped regions and aggregated tokens.

The term “TrajAdapter” is not used uniformly across arXiv. In the TrajTok video paper, it denotes the specific probing module described above (Zheng et al., 26 Feb 2026). By contrast, the 2021 multiple object tracking paper introduces TrajE, a trajectory estimator based on recurrent mixture density networks that acts as a generic module added to trackers such as CenterTrack and Tracktor; later commentary characterizes this as a generic “TrajAdapter” idea because it replaces conventional motion models with a learned multimodal trajectory predictor, but the module itself is named TrajE rather than TrajAdapter (Girbau et al., 2021).

A different ambiguity appears in the 2026 GPS trajectory representation paper, which explicitly states that it does not define a separate module called “TrajAdapter.” Instead, it studies a frozen pretrained TrajTok encoder with lightweight task-specific adapters and heads for similarity search, classification, estimated time of arrival, and travel-time regression (Xiong et al., 19 May 2026). The shared theme is frozen-encoder transfer, but the architectural object is not the same as TrajAdapter in video understanding.

TransferTraj provides yet another adapter-like transfer perspective. It is not presented as a TrajAdapter module; rather, it is a pretrained vehicle trajectory model intended to transfer across regions and tasks without retraining its core parameters, using RTTE and a task-transferable masking-and-recovery scheme (2505.12672). Similarly, TrueÆdapt addresses online trajectory adaptation in robot joint space via learned bounded-acceleration corrections to reference trajectories, which is conceptually adjacent only at the level of “trajectory adaptation” vocabulary, not at the level of module identity (Kiemel et al., 2020).

This terminological spread suggests that “TrajAdapter” is best treated as a paper-specific designation rather than a standardized subfield term. In the strict sense established by the TrajTok literature, it refers to a trajectory-based probing adapter inserted after a frozen pretrained video backbone, where dense patch-level features are reorganized into compact trajectory tokens for downstream classification (Zheng et al., 26 Feb 2026).