CoTracker: Joint Transformer for 2D Tracking

• CoTracker is a transformer-based joint tracking model that estimates 2D point trajectories in long video sequences while robustly handling occlusions and out-of-view points.
• It integrates convolutional feature extraction, local correlation, and iterative transformer refinement to capture both local and global dependencies effectively.
• The factorized attention design enhances computational efficiency, enabling the tracking of up to 70k points simultaneously and supporting diverse applications in robotics, reconstruction, and medical imaging.

CoTracker is a transformer-based model for tracking a large number of 2D points in long video sequences. Its defining premise is joint point tracking: differently from most existing approaches that track points independently, CoTracker tracks them jointly, accounting for their dependencies, and this joint formulation is reported to improve tracking accuracy, robustness, and the handling of occluded points and points outside of the camera view (Karaev et al., 2023). In the literature that followed, CoTracker appears both as a standalone tracker and as a reusable correspondence engine inside broader systems for registration, 3D reconstruction, robotics, action recognition, and medical motion analysis (Neubig et al., 13 Mar 2025).

1. Problem formulation and scope

In its original formulation, CoTracker takes a video $V = (I_t)_{t=1}^T$ of $T$ RGB frames $I_t \in \mathbb{R}^{3 \times H \times W}$, together with a user-specified set of $N$ points with starting locations and times $(P^i_{t^i}, t^i)_{i=1}^N$, and predicts a 2D trajectory and visibility for each queried point over time (Karaev et al., 2023). The model output is written as predicted track locations $\hat{P}_t^i$ and visibility flags $\hat{v}_t^i$ for all $t \ge t^i$ (Karaev et al., 2023).

This formulation places CoTracker in the “track any point” setting rather than in semantic keypoint detection, object detection, or dense optical flow. The target is not a predefined anatomical landmark, articulated joint, or object category, but an arbitrary image point selected by a user or an upstream system. That design choice is central to many downstream uses. Subsequent works use CoTracker to track arbitrary points in videofluoroscopy, endoscopic stereo images, egocentric video, and transesophageal echocardiography, often precisely because the required correspondences are not naturally expressed as semantic landmarks (Neubig et al., 13 Mar 2025).

The original model is described as an online algorithm that operates causally on short windows, but it is trained utilizing unrolled windows as a recurrent network, maintaining tracks for long periods of time even when points are occluded or leave the field of view (Karaev et al., 2023). This causal-windowed design distinguishes it from fully offline trackers that consume an entire sequence simultaneously, while still aiming at long-duration trajectory stability.

2. Architectural design

CoTracker combines convolutional image features, local correlation features, iterative refinement, and a transformer that couples trajectories across both time and track identity (Karaev et al., 2023). For each frame, a convolutional neural network extracts dense feature maps at multiple scales,

$$\phi(I_t) \in \mathbb{R}^{d \times H/k \times W/k},$$

with additional downscalings via average pooling for multiscale analysis (Karaev et al., 2023).

For each queried point, the model initializes an appearance vector $Q_t^i$ by sampling the feature map at the point’s initial location and then updates this appearance representation iteratively (Karaev et al., 2023). Around the current location estimate, it computes correlation features by matching $T$0 against a local neighborhood in the image features: $T$1 for all scales $T$2 and offsets $T$3 in a window (Karaev et al., 2023). This correlation mechanism links CoTracker to the broader RAFT-style family of local matching architectures, but the tracker does not stop at per-point local matching; the key additional step is global interaction among tracks.

Each track at time $T$4 is encoded as a token

$T$5

combining relative position, visibility, appearance, correlation, and positional encodings (Karaev et al., 2023). The transformer outputs location and feature updates

$T$6

which are applied iteratively: $T$7 Visibility is predicted after the final iteration as $T$8 (Karaev et al., 2023).

A central scalability device is its factorized attention. Rather than using standard transformer attention with cost $T$9, CoTracker alternates attention along time and track dimensions, reducing the cost to $I_t \in \mathbb{R}^{3 \times H \times W}$0 (Karaev et al., 2023). To scale further, it introduces virtual track tokens, also described as token proxies, so that only $I_t \in \mathbb{R}^{3 \times H \times W}$1 virtual tracks use full self-attention while real tracks interact with virtual tracks via cross-attention (Karaev et al., 2023). The reported effect is that CoTracker can track 70k points jointly and simultaneously at inference on a single GPU (Karaev et al., 2023).

The technical significance of this design is not merely computational. The use of joint attention means that points lying on a common object, surface, or motion pattern can “help” one another under ambiguity or occlusion. The original ablations explicitly attribute substantial gains to joint tracking over independent point tracking, and also report benefits from adding support points near a target point even when only a single target trajectory is ultimately needed (Karaev et al., 2023).

3. Training procedure, inference regime, and benchmark behavior

CoTracker operates on windows of fixed length, such as 8–24 frames, and longer videos are processed through semi-overlapping windows, with predictions from one window initializing the next (Karaev et al., 2023). If tracks start after the beginning of the current window, masking is used to ignore non-initialized track tokens (Karaev et al., 2023). This makes the model causal at inference while retaining continuity across long sequences.

Long-term tracking is enforced through unrolled or recursive training. The model is trained by unrolling several windows and propagating updates recurrently, with losses applied at each transformer iteration and window: $I_t \in \mathbb{R}^{3 \times H \times W}$2

$I_t \in \mathbb{R}^{3 \times H \times W}$3

The published description states that this encourages both accurate refinement and good long-term memory (Karaev et al., 2023).

Empirically, the original paper reports evaluation on TAP-Vid-DAVIS, DynamicReplica, and PointOdyssey, using metrics such as Average Jaccard (AJ), visibility-related accuracy, and survival duration (Karaev et al., 2023). The reported outcome is that CoTracker substantially outperforms prior trackers on standard point-tracking benchmarks (Karaev et al., 2023). The ablation results are particularly important for understanding the method’s inductive bias: joint tracking with support points improves AJ by more than 15 points over independent tracking, unrolled training improves AJ by 18 points relative to per-window training, and virtual tracks allow tracking more than seven times as many points as a non-virtual-track architecture without sacrificing accuracy (Karaev et al., 2023).

These results established CoTracker as a strong general-purpose tracker, but subsequent literature also clarifies the scope of those gains. In dense optical tracking, DOT is reported to be on par with, or better than, the best point tracking algorithms like CoTracker while being at least two orders of magnitude faster, because DOT is purpose-built for dense prediction rather than sparse or quasi-dense queried tracks (Moing et al., 2023). This distinction recurs across later comparisons: CoTracker is strongest when the task truly is point trajectory estimation, but not necessarily when the target is dense flow, semantic identity preservation, or domain-specific biomechanical keypoint inference.

4. Adaptations and derived formulations

Later work frequently treats CoTracker less as a closed tracker and more as a modular point-correspondence primitive that can be adapted to new output spaces. A clear example is TAPVid-360, where a baseline adapts CoTracker v3 to predict 3D directions rather than 2D image coordinates (Hudson et al., 26 Nov 2025). In that formulation, the last decoder layer is replaced with a linear layer outputting 9 values interpreted as a $I_t \in \mathbb{R}^{3 \times H \times W}$4 matrix, projected to the nearest valid rotation matrix, and used to update a point’s initial direction according to

$I_t \in \mathbb{R}^{3 \times H \times W}$5

This baseline is reported to outperform existing TAP and TAPVid 3D methods on the proposed benchmark (Hudson et al., 26 Nov 2025).

A different adaptation appears in “TeeTracker,” a CoTracker-based component inside the autoStrain pipeline for segmental longitudinal strain estimation in transesophageal echocardiography (Taskén et al., 4 Nov 2025). There, a myocardial mesh is initialized at a reference frame and CoTracker predicts mesh-point trajectories over the cardiac cycle. In the reported synthetic evaluation, TeeTracker outperforms a RAFT-based dense optical flow alternative called TeeFlow, achieving a mean distance error in motion estimation of 0.65 mm on a synTEE test dataset (Taskén et al., 4 Nov 2025). The same work reports that clinical validation on 16 patients yielded a mean difference of 1.09% with 95% limits of agreement from $I_t \in \mathbb{R}^{3 \times H \times W}$6 to $I_t \in \mathbb{R}^{3 \times H \times W}$7 for segmental longitudinal strain estimation (Taskén et al., 4 Nov 2025).

CoTracker is also repurposed structurally rather than only as an endpoint predictor. The Tracktention Layer does not perform tracking itself; instead, it uses an external point tracker, such as CoTracker3, to extract tracks $I_t \in \mathbb{R}^{3 \times H \times W}$8, then samples features at track locations, processes them with a transformer along each track, and splats the updated track tokens back to the image grid (Lai et al., 25 Mar 2025). Its attention weights incorporate a spatial bias

$I_t \in \mathbb{R}^{3 \times H \times W}$9

so that motion-aware correspondences from CoTracker can improve temporal alignment in tasks such as video depth prediction and colorization (Lai et al., 25 Mar 2025).

This suggests that CoTracker’s long-term scientific impact may lie as much in its role as a general correspondence substrate as in its role as a benchmarked tracker. Later systems use it to provide trajectory supervision, to constrain latent dynamics, or to define motion-aware attention routes, rather than only to emit trajectories as a final product.

5. Downstream applications

The range of downstream uses is broad and technically heterogeneous.

Paper	Domain	Role of CoTracker
(Neubig et al., 13 Mar 2025)	Videofluoroscopy motion correction	Markerless tracking for registration
(Stearns et al., 2024)	Monocular dynamic scene reconstruction	Source of dense 2D tracks for tracking loss
(Holzmann et al., 7 Jan 2026)	Egocentric action recognition	Motion cue from randomly initialized point trajectories
(Reuter et al., 11 Aug 2025)	Stereo endoscopic tissue tracking	Temporal tracking and stereo matching
(Taskén et al., 4 Nov 2025)	Transesophageal echocardiography	Myocardial mesh trajectory estimation

In videofluoroscopy, a 2025 study evaluates CoTracker, PIPs++, and TAP-Net for isolating swallowing dynamics from interfering patient motion (Neubig et al., 13 Mar 2025). The abstract states that even sparse tracking points generate morphing displacement fields that outperform ANTs, LDDMM, and VoxelMorph, and that the resulting motion correction pipeline removes disruptive motion while preserving swallowing dynamics (Neubig et al., 13 Mar 2025). Here CoTracker serves as a markerless tracker whose trajectories are converted into a displacement field for registration.

In monocular dynamic scene reconstruction, Dynamic Gaussian Marbles adds a tracking loss that takes advantage of recent progress in point tracking and specifically leverages CoTracker to generate a $N$0 grid of point tracks over a temporal window of $N$1 frames with $N$2 (Stearns et al., 2024). The resulting depth-aware tracking loss aligns projected Gaussian centers with 2D trajectories, helping constrain motion in an otherwise underconstrained monocular setting (Stearns et al., 2024).

In egocentric action recognition, TRec employs CoTracker to follow a set of randomly initialized points and uses the resulting trajectories together with image frames as input to a Transformer-based recognition model (Holzmann et al., 7 Jan 2026). The paper’s abstract emphasizes that tracking randomly sampled image points across video frames can substantially improve recognition accuracy, even when only the initial frame and its associated point tracks are provided (Holzmann et al., 7 Jan 2026). This use is notable because it does not depend on hand, object, or interaction detectors.

In endoscopic stereo tracking, a 2025 paper proposes a markerless 3D tissue tracking system that combines two CoTracker models, one for temporal tracking and one for stereo matching, to estimate 3D motion from stereo endoscopic images (Reuter et al., 11 Aug 2025). On a chicken tissue phantom, the reported Euclidean distance errors are as low as 1.1 mm at a velocity of 10 mm/s (Reuter et al., 11 Aug 2025). This is a direct example of CoTracker being used as a generic 2D correspondence module inside a stereo-lifting pipeline.

Other applications in the supplied literature include in-situ ESEM measurement of CNT forest growth, where the Meta CoTracker algorithm measured CNT growth rates reducing from 50 nm/sec to full termination over 150 seconds (Surya et al., 2024), and visual servoing for orchard pruning, where a transformer-based point tracker is used to center tracked points in the camera view for robot control (Ahmed et al., 9 Apr 2025). Taken together, these works show that CoTracker is not restricted to natural videos or conventional benchmarks; it is repeatedly transferred into scientific imaging, surgical vision, and robotics.

6. Limitations, failure modes, and comparative perspective

The downstream literature also documents several regimes in which CoTracker is insufficient or requires augmentation. In long-horizon articulated tracking, QueST argues that frame-to-frame correspondence leads to silent semantic drift and compares directly against RAFT-3D, CoTracker, and TAP-Net (Anand et al., 10 May 2026). The reported results state that QueST substantially reduces terminal drift, achieving a 67.7% Absolute Point Error improvement over TAP-Net while better preserving identity over extended horizons (Anand et al., 10 May 2026). In the detailed comparison, CoTracker is described as relying on transient tracklets and having no built-in mechanism to preserve or monitor semantic identity, whereas QueST uses persistent semantic queries and lightweight 3D physical grounding (Anand et al., 10 May 2026). The implication is not that CoTracker fails at correspondence estimation per se, but that long-horizon identity maintenance may require an explicit monitoring mechanism beyond local propagation.

Egocentric video exposes a different limitation. EgoPoints introduces a benchmark with 9x more points that go out-of-view and 59x more points that require re-identification than TAP-Vid-DAVIS, and reports that fine-tuning on semi-real K-EPIC sequences improves CoTracker across all metrics, including $N$3 by 2.7 percentage points and ReID$N$4 by 2.4 points (Darkhalil et al., 2024). Even with that improvement, the paper presents re-identification after out-of-view intervals as an open challenge (Darkhalil et al., 2024).

Domain transfer can also fail in more specialized settings. In sprint biomechanics, VideoRun2D reports that the CoTracker method showed huge differences compared to the manual labeling approach, whereas angle curves were correctly estimated by the MoveNet method with errors between $N$5 and $N$6 (Garrido-Lopez et al., 2024). In skin-feature tracking, an unsupervised Deep Feature Encodings method reports mean error ranging from 0.6 to 3.3 pixels and states that it outperforms traditional methods and the latest transformers like PIPs++ and CoTracker under significant motion conditions (Chang et al., 2024). These results indicate that generic point tracking is not automatically equivalent to anatomically or biomechanically valid landmark tracking.

A common misconception is therefore that CoTracker is a universal replacement for optical flow, pose estimation, or semantic tracking. The literature supplied here does not support that claim. Instead, it supports a narrower and more precise interpretation: CoTracker is a strong general-purpose model for joint 2D point trajectory estimation, especially when arbitrary queried correspondences and occlusion robustness matter, and it is often most effective when embedded inside larger systems that add domain structure, 3D geometry, physical priors, or semantic monitoring (Karaev et al., 2023).