TrackRec: Transformer-based Action Recognition

• TrackRec is a Transformer-based method that augments RGB video pipelines with 2D point trajectories for egocentric action recognition.
• It fuses image features and sparse track embeddings to capture both foreground and background motion as key discriminative cues.
• Experiments on SSv2 show significant improvements over RGB-only baselines, highlighting the value of explicit motion representation.

TrackRec, the alias used here for TRec, denotes a Transformer-based action-recognition method that augments an RGB video pipeline with sparse 2D point trajectories for egocentric action understanding. Its central claim is that randomly sampled image points, tracked across frames by CoTracker3 and fused with image features, provide an explicit motion representation that substantially improves recognition accuracy without detecting hands, objects, or interaction regions. Although the motivation is egocentric action recognition, the reported experiments are conducted on Something-Something-v2 (SSv2), a third-person benchmark chosen because its relatively stationary camera simplifies analysis while preserving rich hand–object motion patterns (Holzmann et al., 7 Jan 2026).

1. Problem setting and scope

TrackRec targets fine-grained, short actions involving hand–object interactions from first-person videos. The modeling choices are explicitly egocentric: the approach is motivated by camera motion, occlusions, and interaction-centric cues, and it treats both foreground and background motion as potentially informative. At the same time, the experimental evaluation is restricted to SSv2, which contains short recordings of “someone doing something” with objects rather than true first-person footage (Holzmann et al., 7 Jan 2026).

The dataset used is SSv2 alone. It contains 174 action templates describing hand–object interactions such as pushing, pulling, and dropping. The task is single-action classification over these templates, with no verb–noun decomposition. Evaluation uses Top-1 and Top-5 accuracy on the standard SSv2 split; specific split counts are not re-stated. This experimental choice isolates motion discrimination under relatively stable camera conditions while retaining interaction patterns that resemble egocentric scenarios. This suggests that the reported study is best interpreted as a controlled analysis of motion representations rather than as a final benchmark on canonical egocentric datasets such as EPIC-KITCHENS-100, EGTEA Gaze+, or Ego4D (Holzmann et al., 7 Jan 2026).

A recurring misconception is that egocentric action recognition necessarily requires semantic localization of hands, manipulated objects, or interaction regions. TrackRec is constructed to test the opposite hypothesis: many randomly sampled 2D tracks, including background trajectories, may already encode enough kinematic structure to support action recognition. The method therefore treats explicit motion as the primary discriminative signal rather than as a secondary cue derived from appearance.

2. Input representation and end-to-end pipeline

TrackRec augments a standard RGB recognition pipeline with sparse 2D point trajectories. For each video clip $v$ with $T$ frames, it randomly samples $N=900$ image points across the entire frame. Sampling is deliberately unfiltered, so points may fall on hands, manipulated objects, or background. Frames are sampled at 30 fps to produce smooth tracks, and the inclusion of background motion is intentional because head or camera motion may provide contextual information (Holzmann et al., 7 Jan 2026).

These points are tracked jointly across the clip using CoTracker3. Each point $i$ yields a 2D trajectory

$p^i=\{(x_t^i,y_t^i)\}_{t=1..T}.$

Coordinates are normalized by image width and height following Track2Act-style normalization so that positions lie in a fixed range independent of resolution. Occlusion and visibility are handled internally by CoTracker; TrackRec does not introduce explicit occlusion logic or visibility thresholds.

Although 900 points are tracked, training typically uses only a random subset to reduce compute and increase diversity. The default configuration samples 400 tracks per video, while some experiments randomize the number of tracks between 200 and 400 per mini-batch. In parallel, each of the $T$ frames is processed by a ResNet18 backbone to produce image features. Track features, consisting of normalized coordinate sequences over time, are embedded by an MLP into the transformer hidden dimension. Image features and track embeddings are then concatenated as tokens and processed jointly by a transformer. A class token and multi-head attention pooling produce a single representation, and an MLP head outputs the action logits (Holzmann et al., 7 Jan 2026).

A distinctive variant uses only the initial frame together with the full set of point tracks extracted from the entire video. In that setting the model receives one image plus trajectory sequences rather than the full RGB clip. The reported results show that this variant still exceeds the RGB-only baseline, indicating that the track representation alone carries substantial action information.

3. Architecture and mathematical formulation

The image backbone is ResNet18, used to encode each frame $v_t \in \mathbb{R}^{H\times W\times 3}$ into frame features. The track tensor is

$P=\{p_t^i\}\in\mathbb{R}^{T\times N\times 2},$

where $N$ is the number of points used by the model and each point contributes 2D coordinates at every frame. Before transformer processing, tracks are normalized and reshaped to

$P' \in \mathbb{R}^{B\times (T\times 2)\times N}$

for batch size $T$0, then projected with an MLP:

$T$1

This maps the coordinate representation into the transformer hidden space (Holzmann et al., 7 Jan 2026).

The transformer has six self-attention layers, four attention heads per layer, and an intermediate feed-forward dimension of 2048. Fusion is achieved by joint self-attention over all image and track tokens; the paper does not describe a separate cross-attention block. Positional or temporal encodings are not explicitly specified. Instead, temporal structure is carried by frame ordering and the track sequences themselves.

The attention mechanism is standard multi-head self-attention:

$T$2

where $T$3 is the per-head key dimension. A class token, followed by multi-head attention pooling, yields the final video representation. Classification uses standard single-label cross-entropy over $T$4 classes:

$T$5

No trajectory smoothing loss, consistency loss, or explicit motion supervision is added; motion is learned implicitly through the classification objective (Holzmann et al., 7 Jan 2026).

The most consequential architectural point is not complexity but modality coupling. TrackRec does not treat tracks as a post hoc auxiliary descriptor. Instead, appearance and motion are presented to the same transformer as joint tokens, allowing attention to associate image evidence with track dynamics at training and inference time.

4. Training protocol and implementation details

The default observation horizon is $T$6 frames per clip, and image frames are resized to $T$7. Data augmentation consists of random crop, horizontal flip, Gaussian blur, and color jitter, with track coordinates augmented consistently with the corresponding image transforms. The backbone is finetuned rather than frozen (Holzmann et al., 7 Jan 2026).

Optimization uses AdamW with a learning rate of $T$8 for the transformer and $T$9 for the ResNet18 backbone. The learning-rate schedule is cosine warm restarts with 5000 warm-up iterations. Training uses batch size 32 for 20 epochs on an NVIDIA A40 GPU. CoTracker3 generates the 900-point trajectories per video at 30 fps, and fixed random seeds are used across experiments to ensure consistent track sampling.

The implementation does not employ explicit occlusion thresholds or visibility filtering beyond CoTracker’s internal handling. Complexity, runtime, and memory are not reported. The method is nevertheless described as lightweight because it relies on sparse tracking and a small backbone rather than dense motion fields or large-scale recognition architectures. Code or pretrained model release is not stated, but the configuration is reported in enough detail to make reproduction straightforward: ResNet18, a 6-layer transformer with 4 heads and FFN dimension 2048, 8-frame clips, 256×256 inputs, AdamW, the stated learning rates and schedule, 900 tracked points per video, and a default subset of 400 tracks during training (Holzmann et al., 7 Jan 2026).

5. Experimental results and ablation evidence

The headline comparison uses the same architecture and training setup with or without point tracks on SSv2. Under the 8-frame setting, TrackRec achieves Top-1 $N=900$0 and Top-5 $N=900$1, while the RGB-only baseline reaches Top-1 $N=900$2 and Top-5 $N=900$3. The gain is therefore $N=900$4 percentage points in Top-1 and $N=900$5 points in Top-5. In the single-image setting, where the model sees only the initial frame plus the full track set, TrackRec reaches Top-1 $N=900$6 and Top-5 $N=900$7, exceeding the corresponding RGB-only single-frame baseline at Top-1 $N=900$8 and Top-5 $N=900$9. Notably, the single-image TrackRec result of $i$0 Top-1 also surpasses the RGB-only 8-frame baseline of $i$1 Top-1 (Holzmann et al., 7 Jan 2026).

Configuration	Top-1	Top-5
TrackRec, 8 frames	61.10% ± 8.66	83.95% ± 6.62
RGB-only baseline, 8 frames	30.27% ± 8.05	53.24% ± 8.75
TrackRec, initial frame + tracks	44.70%	69.12%
RGB-only baseline, initial frame	37.50%	62.41%
Vanilla TrackRec + KDE	52.46	74.04
FilterTrackRec + KDE	36.34	60.57

The point-count ablation shows that performance remains relatively stable once the number of points reaches moderate scale. The reported Top-1/Top-5 values are 61.10/83.95 for 400 points, 60.89/82.89 for 200, 60.61/82.66 for 100, and 59.73/82.11 for 50. Below that regime, degradation becomes more pronounced: 57.75/80.60 for 25 points, 54.98/78.24 for 15, 45.19/69.10 for 5, and 24.87/46.53 for a single point. The stated interpretation is that performance is stable from 50–100 points upward, with diminishing returns beyond 100, whereas below roughly 25 points the reduction in motion diversity becomes damaging.

The background-motion ablation is equally important. Vanilla TrackRec yields 61.10 Top-1 and 83.95 Top-5. Removing high-density background-motion clusters with KDE lowers performance to 52.46 Top-1 and 74.04 Top-5, and training only on filtered points produces 36.34 Top-1 and 60.57 Top-5. This directly supports the claim that background motion is not incidental noise but an informative component of the representation.

6. Interpretation, qualitative behavior, and limitations

Qualitative visualizations show tracked points on hands, manipulated objects, and background regions. Because points are sampled randomly across the frame, the method frequently places trajectories on salient movers without any hand or object detector. Success cases are actions characterized by distinctive motion patterns, including pushing versus pulling, lifting versus dropping, or bringing objects closer. In such cases, coherent flows on hands and objects, together with consistent background parallax, appear to provide strong discriminative structure. Failure cases include actions that depend on final visual state, such as revealing, pretending, or uncovering, especially when only early frames are visible. Additional failure modes include textureless regions, rapid occlusions, and aggressive camera shake, each of which can degrade track quality (Holzmann et al., 7 Jan 2026).

The authors’ design intuition is that interaction-centric actions are often better defined by how things move than by what they look like. Random sampling avoids brittle dependencies on semantic detectors, while normalized trajectories provide a geometric signal that is less sensitive to texture and color variation. The KDE ablation further shows that global scene or camera motion contributes contextual information rather than merely confounding the classifier. This suggests that TrackRec is not simply recognizing object motion in isolation; it is learning a joint kinematic representation of local manipulation and global viewpoint dynamics.

Several limitations are explicit. Performance depends on CoTracker’s robustness, so severe occlusion, fast motion, and low-texture surfaces can produce noisy or lost tracks. The use of SSv2 leaves open the question of domain shift to true egocentric video, where narrower fields of view and stronger camera motion are common. Ambiguous classes defined by end states remain difficult when the model does not observe final frames. Proposed extensions include 3D tracks or depth-aware cues, integration of optical flow or event streams, cross-attention fusion modules, adaptive point sampling, longer temporal contexts, and semi-/self-supervised pretraining on large egocentric corpora. The paper does not state that code or pretrained models are publicly released, but the reported implementation details are sufficiently specific to support direct reproduction (Holzmann et al., 7 Jan 2026).