TAPVid-3D

• TAPVid-3D is a benchmark for Tracking Any Point in 3D (TAP-3D) from monocular video, used to follow material points through a video sequence, predict their 3D positions, and handle occlusions and out-of-view events.
• It extends 2D TAP benchmarks by adding 3D trajectories and includes 4,569 real-world clips with varying lengths and scenarios, such as indoor studio scenes, outdoor driving, and human actions.
• TAPVid-3D evaluates 3D tracking performance using metrics like 3D Average Percentage of Distances (APD_3D) and Occlusion Accuracy (OA).

TAPVid-3D is a benchmark for long-range Tracking Any Point in 3D (TAP-3D) from monocular video. Given a visible query pixel at a selected frame, a system must follow the corresponding material point through the sequence, predict its three-dimensional position and visibility at every frame, and preserve point identity through camera motion, object motion, occlusion, and out-of-view events. Introduced in “TAPVid-3D: A Benchmark for Tracking Any Point in 3D” (Koppula et al., 2024), it extends two-dimensional TAP benchmarks such as TAPVid-DAVIS from image coordinates to 3D trajectories. The benchmark contains 4,569 real-world clips from Aria Digital Twin, DriveTrack, and Panoptic Studio, with clip lengths from 25 to 300 frames. Its metrics address depth-scale ambiguity, 3D localization, visibility prediction, and the combined correctness of spatial tracking and occlusion handling.

1. Task definition and scope

A video is represented as

$V=\{I_t\}_{t=1}^{T},$

where $I_t$ denotes frame $t$. A query is

$q=(x_q,y_q,t_q),$

specifying a visible pixel at query time $t_q$. The queried pixel identifies a material point on a solid, opaque object or scene surface. For $Q$ queries, the model predicts a 3D trajectory

$$\hat{P}^i=\{\hat{P}^i_t\}_{t=1}^{T}, \qquad \hat{P}^i_t\in\mathbb{R}^3,$$

and a binary visibility sequence

$\hat{v}^i_t\in\{0,1\}.$

Ground-truth trajectories and visibility labels are denoted by $P^i_t$ and $v^i_t$. The query point is visible by construction at $I_t$0.

TAP-3D differs from ordinary 2D point tracking because the prediction must preserve both image correspondence and depth or camera-relative 3D position. The released trajectory tensor has shape $I_t$1, with coordinates in meters; the query tensor has shape $I_t$2 and contains $I_t$3; visibility annotations have shape $I_t$4.

A track is expected to remain associated with the same material point during temporary occlusion, even though the visibility flag should indicate that visual evidence is unavailable. A point that leaves the image or becomes occluded must therefore be distinguished from a point that has been lost because of tracking failure.

TAP-3D is related to scene flow but evaluates a different capability. Scene flow generally estimates dense, instantaneous motion between adjacent frames. TAP-3D evaluates sparse, long-range point identities over tens or hundreds of frames, including occlusions, camera motion, and independently moving objects. It is also distinct from category-specific 3D pose or object tracking: it requires no known object model, articulated template, or object category.

The benchmark is intended to support research in robotic manipulation, visual odometry, dynamic mapping, motion prediction, video editing and generation, object-part tracking, and dynamic scene reconstruction. These applications motivate the simultaneous evaluation of correspondence, visibility, camera-motion compensation, and depth consistency.

2. Coordinate systems and long-range correspondence

The primary evaluation representation uses 3D points in the camera coordinate system, generally expressed in meters. The camera coordinate frame may change over time when the camera moves. The data-generation pipelines additionally use world coordinates, object coordinates for rigid objects, camera intrinsics $I_t$5, and camera extrinsics represented by transformations $I_t$6 mapping coordinates from frame $I_t$7 to frame $I_t$8.

Perspective projection is written schematically as

$I_t$9

where $t$0 denotes projection with camera intrinsics. For a pixel $t$1 with depth $t$2, standard unprojection is

$t$3

Thus, a TAP-3D system must estimate an image correspondence, an appropriate depth, and camera-relative 3D motion.

The benchmark contains clips from 25 to 300 frames and evaluates predictions over the entire clip rather than only between adjacent frames. Long-range identity preservation, recovery after occlusion, camera-motion compensation, and object-motion estimation are consequently central task properties. Static-camera sequences do not eliminate the 3D challenge: moving objects can produce nontrivial trajectories even when the camera is stationary.

The benchmark does not prescribe a single physical motion model. It includes rigid objects, articulated human motion, non-rigid motion, camera movement, and independently moving objects. A static point in world coordinates can nevertheless move substantially in camera coordinates when the camera moves; conversely, a moving object can remain visually coherent while violating the assumptions of static-scene reconstruction.

Methods developed after the benchmark illustrate different choices for handling these coordinate issues. TAPIP3D lifts image features into camera- or world-coordinate feature clouds and reports improved performance when camera motion is compensated (Zhang et al., 20 Apr 2025). SpatialTrackerV2 jointly estimates depth, pose, image trajectories, 3D trajectories, visibility, and dynamic probabilities (Xiao et al., 16 Jul 2025). TrajVG explicitly couples camera-coordinate 3D trajectories with local point maps and relative camera poses (Miao et al., 4 Feb 2026). TrackCraft3R instead predicts reference-anchored trajectories from frame-anchored reconstruction pointmaps (Nam et al., 12 May 2026).

3. Dataset composition and annotation generation

TAPVid-3D combines three real-world sources:

Subset	Clips	Frames per clip	Principal characteristics
Aria Digital Twin	1,956	300	Egocentric indoor studio scenes, camera motion, hands and objects
DriveTrack	2,457	25–300	Outdoor Waymo driving videos, moving vehicles and high velocities
Panoptic Studio	156	150	Stationary multi-camera studio, human actions and non-rigid motion
Total	4,569	25–300	2,828 videos and 255 scenes

The benchmark provides 150 minival clips, consisting of 50 clips from each source. The remaining clips form the full evaluation organization. TAPVid-3D does not define a conventional training split and is intended primarily for evaluation; models are generally trained elsewhere or pretrained on synthetic and external data.

Aria Digital Twin contains egocentric footage recorded with Aria glasses in studio environments, together with 3D replicas, segmentation masks, depth maps, object poses, camera intrinsics, and camera poses. A query pixel is unprojected at its query frame, associated with an object through the segmentation mask, transformed into object coordinates, and transported through time using object poses. This procedure assumes that the queried surface point is rigidly attached to the annotated object. Visibility is determined by comparing projected reconstructed depth with observed depth and by excluding regions covered by the operator’s hands.

DriveTrack is derived from Waymo Open Dataset footage and synchronized LiDAR point clouds. A tracked vehicle is selected using a manually annotated 3D bounding box. LiDAR points belonging to the vehicle are selected at a sampling time, transported through time using the vehicle’s annotated 3D bounding-box pose, and projected into the images. Dense depth maps are produced by interpolating sparse LiDAR measurements. A point is marked invisible if its predicted depth is behind the observed depth by more than a 5% relative margin. Segmentation filtering removes many points that do not remain on the vehicle surface.

Panoptic Studio uses multiview recordings of people performing actions. Dynamic 3D reconstructions based on Dynamic 3D Gaussians are used to estimate Gaussian motion, render pseudo-ground-truth depth maps, associate query pixels with Gaussian centers, and follow those centers through time. Stationary background points are excluded because the cameras are fixed and such tracks would be trivial. The retained tracks predominantly represent foreground deforming characters.

The annotations are therefore high-quality pseudo-ground truth rather than fully independent manual metric trajectories. Potential error sources include alignment errors between synthetic annotations and footage, LiDAR noise, imperfect vehicle bounding boxes, ambiguous Gaussian splats, numerical errors, depth interpolation artifacts, and flickering visibility labels. Automated cleanup includes object-mask filtering, visibility-flicker filtering, and foreground-mask filtering for Panoptic Studio. A large sample of trajectories and videos was manually checked, but the paper does not provide an exhaustive manual-labeling protocol or a complete per-subset error rate.

The benchmark is limited to solid, opaque objects and inherits the domain biases of its three sources. It does not cover all environments, object categories, motions, camera configurations, transparent or reflective surfaces, fluids, or partially modeled deformable objects.

4. Scale alignment and evaluation metrics

Monocular depth is often determined only up to an unknown global scale. TAPVid-3D therefore evaluates predictions under several scale-alignment protocols.

Under global median rescaling, one scale factor is applied to all predicted points and frames:

$t$4

followed by

$t$5

This is the principal setting in the main evaluation and tests whether a method reconstructs the whole scene consistently up to one scale.

Under per-trajectory rescaling, each trajectory receives an independent scale:

$t$6

This emphasizes relative depth change within a trajectory without requiring scale consistency between different points.

The appendix also defines local-neighborhood rescaling. For trajectory $t$7, a ground-truth tubelet is formed using a spatial radius $t$8:

$t$9

The corresponding predictions are scaled using the query-point ratio. The appendix uses $q=(x_q,y_q,t_q),$0 m for Aria Digital Twin and Panoptic Studio and $q=(x_q,y_q,t_q),$1 m for DriveTrack. The scale hierarchy is therefore:

$q=(x_q,y_q,t_q),$2

where the leftmost protocol is the most globally restrictive.

Position accuracy

TAPVid-3D extends the 2D TAP thresholds

$q=(x_q,y_q,t_q),$3

to depth-adaptive 3D thresholds. For a ground-truth point $q=(x_q,y_q,t_q),$4,

$q=(x_q,y_q,t_q),$5

where $q=(x_q,y_q,t_q),$6 is camera-coordinate depth and $q=(x_q,y_q,t_q),$7 is focal length. Farther points receive a larger metric tolerance because the same image displacement corresponds to a larger physical displacement.

The position-correctness indicator is

$q=(x_q,y_q,t_q),$8

The 3D Average Percentage of Distances below threshold, $q=(x_q,y_q,t_q),$9, evaluates localization at ground-truth visible times:

$t_q$0

where

$t_q$1

APD$t_q$2 does not penalize incorrect visibility predictions and does not evaluate the geometric accuracy of points labeled invisible.

Occlusion accuracy

Occlusion Accuracy is

$t_q$3

It measures visibility classification independently of spatial localization. Consequently, a method can obtain high OA while producing inaccurate 3D trajectories.

3D Average Jaccard

The combined metric is

$t_q$4

Its numerator counts true positives: points that are visible, predicted visible, and spatially correct. The denominator includes visible ground-truth points, false visible predictions during occlusion, visible points predicted invisible, and visible points predicted correctly visible but spatially incorrect. AJ$t_q$5 therefore jointly evaluates geometric tracking and visibility.

The appendix also reports fixed metric thresholds

$t_q$6

These measurements are useful when a particular physical accuracy is required, but they are not directly comparable to the principal depth-adaptive results.

The benchmark does not introduce a separate multi-track temporal smoothness metric. Long-range continuity is evaluated indirectly through trajectory-level localization and visibility. Seurat later introduced temporal coherence as an additional analysis metric while evaluating relative-depth prediction on TAPVid-3D (Cho et al., 20 Apr 2025), but that metric is not part of the original benchmark’s primary definition.

5. Baselines and benchmark findings

The original baseline study combines 2D trackers with depth or reconstruction systems, evaluates a direct 3D tracker, and includes a static-motion baseline.

The 2D-plus-monocular-depth systems are TAPIR + ZoeDepth, CoTracker + ZoeDepth, and BootsTAPIR + ZoeDepth. The 2D tracker supplies pixel trajectories and visibility, while ZoeDepth supplies per-frame depth that is unprojected with camera intrinsics.

The 2D-plus-COLMAP systems are TAPIR + COLMAP, CoTracker + COLMAP, and BootsTAPIR + COLMAP. Their image tracks are exported as features and matches, reconstructed through COLMAP’s incremental mapping pipeline, filtered by track length, and transformed into camera coordinates with estimated extrinsics. COLMAP can be effective for static background structure but does not reconstruct independently moving objects reliably.

SpatialTracker is designed specifically for 3D point tracking and combines monocular-depth initialization with reprojection consistency. TAPIR-3D is a variant trained only on synthetic Kubric videos; it predicts log depth relative to the query-frame depth and refines that estimate through four passes. Because trajectories are predicted independently, it lacks strong cross-track scale consistency.

The static baseline uses ground-truth query depth and assumes

$t_q$7

It is not fully automatic but isolates the importance of modeling camera and object motion.

Under global median scaling, the full-evaluation averages were:

Method	Average AJ	Average APD	Average OA
Static	4.9	9.4	70.7
TAPIR + COLMAP	7.4	12.4	76.1
CoTracker + COLMAP	9.3	15.0	79.1
BootsTAPIR + COLMAP	9.3	14.9	81.4
TAPIR + ZoeDepth	8.3	13.8	80.0
CoTracker + ZoeDepth	8.7	14.8	83.4
BootsTAPIR + ZoeDepth	8.8	14.8	84.8
TAPIR-3D	3.1	5.9	82.8
SpatialTracker	9.0	15.5	83.7

The 2D tracking results are substantially stronger than the 3D results. BootsTAPIR obtains overall 2D-AJ of 59.1, whereas the corresponding 3D task remains much more difficult. This gap indicates that correct image-plane correspondence does not guarantee coherent depth or 3D motion.

The relative behavior of methods depends strongly on the subset. ZoeDepth-based systems perform comparatively well on Aria and Panoptic Studio. COLMAP performs substantially better on DriveTrack, where camera motion and static environmental structure provide useful geometric constraints, although moving vehicles remain difficult. COLMAP struggles on Panoptic Studio because most of the dominant scene content is moving people. The static baseline performs poorly even on the stationary-camera Panoptic subset, demonstrating that object motion alone defeats a static-camera-coordinate assumption.

The scale protocols expose different capabilities. COLMAP improves substantially under per-trajectory and local-neighborhood scaling, suggesting that it can recover local or relative geometry while failing to establish consistent global scale across moving content. TAPIR-3D has relatively high OA but poor global-scale APD and AJ, consistent with independently predicted trajectories that lack cross-track scale consistency.

Subsequent methods report improved results on the benchmark. TAPIP3D reports average AJ$t_q$8 18.8, APD$t_q$9 27.4, and OA 86.4 under its MegaSaM-based evaluation, with world-coordinate tracking outperforming its camera-coordinate variant (Zhang et al., 20 Apr 2025). SpatialTrackerV2 reports average AJ 21.2, APD$Q$0 31.0, and OA 90.6 for its primary model using learned depth and pose (Xiao et al., 16 Jul 2025). TrajVG reports ADT AJ$Q$1/APD$Q$2/OA of 23.1/32.5/93.8 and DriveTrack values of 17.8/25.2/92.2 (Miao et al., 4 Feb 2026). These later results use different model configurations and evaluation conventions where stated, so comparisons require attention to coordinate systems, scale alignment, geometry sources, and split definitions.

Other work addresses adjacent rather than identical tasks. GMRW is a self-supervised 2D point tracker evaluated on TAP-Vid rather than TAPVid-3D and provides no TAPVid-3D score (Shrivastava et al., 2024). TAPTRv3 improves long-video 2D tracking but predicts image coordinates and visibility rather than metric 3D trajectories (Qu et al., 2024). Seurat predicts temporally varying relative depth from 2D trajectories and evaluates zero-shot on TAPVid-3D, but focuses on depth changes rather than complete end-to-end 3D correspondence (Cho et al., 20 Apr 2025).

6. Subsequent extensions, applications, and limitations

TAPVid-3D has become a reference point for several directions in 3D tracking. Geometry-aware single-camera models such as TAPIP3D and SpatialTrackerV2 lift features into 3D and attempt to compensate for camera motion. TrajVG couples sparse 3D trajectories to dense local point maps and pose estimation. TrackCraft3R repurposes a video diffusion transformer to produce dense, reference-anchored pointmaps from frame-anchored geometry (Nam et al., 12 May 2026). TrackEverything represents persistent world-coordinate scene tracks, merges co-located tracks at sliding-window boundaries, and separates static and dynamic trajectory refinement (Jain et al., 24 Sep 2026). A later state-space approach studies pose-free monocular metric tracking and reports mean absolute metric-AJ of 0.256 on the minival benchmark under fixed hardware and no median scale correction (Ogawa et al., 27 Sep 2026). ARROW uses order-invariant image-set querying and reports camera-coordinate average AJ/APD of 0.248/0.341 and world-coordinate average APD of 0.385 under its stated scale-normalized protocol (Fradlin et al., 1 Oct 2026).

Several extensions change the problem rather than simply improving the original benchmark. TAPVid-360 evaluates persistent 3D viewing directions for points outside a narrow field of view, deliberately removing metric distance while emphasizing panoramic persistence and out-of-frame tracking (Hudson et al., 26 Nov 2025). LAPA constructs a multi-camera adaptation of the Panoptic subset, TAPVid-3D-MC, and reports 37.5 APD, 90.3 OA, 30.2 3D-AJ, and 40.1 2D-AJ for its default configuration (Galoaa et al., 3 Dec 2025). These results concern synchronized calibrated multi-camera tracking and should not be conflated with the original single-stream TAPVid-3D protocol.

The benchmark’s principal limitations are methodological as well as geographic. Its annotations depend on object poses, LiDAR, rendered depth, masks, Gaussian reconstructions, and calibration rather than universally exact manual 3D trajectories. Source-specific artifacts can influence both trajectory and visibility evaluation. The benchmark covers only indoor egocentric studio scenes, outdoor driving, and instrumented human-action capture. Aria and Panoptic Studio contain consenting laboratory participants, while DriveTrack consists of public-road footage from six U.S. cities; these distributions do not represent all populations, environments, motions, or driving conditions.

Scale ambiguity remains application-dependent. Global scale is important for navigation and metric reconstruction, whereas per-trajectory or local scale may be sufficient for manipulation or imitation. The benchmark therefore reports alternative alignments rather than imposing a single universally correct definition of 3D accuracy. Fixed metric thresholds provide a physical-error interpretation, but the main benchmark setting emphasizes depth-adaptive thresholds after global scale alignment.

The benchmark also does not provide a dedicated training split or a separate explicit temporal smoothness score. Methods must generally use external or synthetic training data, and long-range continuity is inferred from trajectory accuracy rather than directly measured by a smoothness objective. Later work has consequently explored temporal coherence, recurrent state-space models, world-coordinate scene memory, explicit camera-pose reasoning, and multi-camera visibility.

TAPVid-3D’s central significance is diagnostic. Its results show that current systems can often track image coordinates substantially better than they reconstruct coherent 3D trajectories. The benchmark separates several failure sources that are confounded in ordinary 2D TAP: noisy depth, scale drift, inconsistent cross-track geometry, camera-motion ambiguity, moving-object reconstruction, occlusion recovery, and long-range identity loss. It therefore occupies an intermediate position between 2D point tracking, monocular depth estimation, scene flow, pose tracking, visual odometry, and dynamic 3D reconstruction.

The benchmark’s defining requirement is simultaneous preservation of point identity, visibility reasoning, camera and object motion understanding, and depth consistency over long monocular videos. Its initial baselines establish that no single strategy is uniformly dominant: monocular depth is effective in some indoor and human-motion settings, SfM benefits from camera-motion-rich scenes with static structure, and direct 3D trackers can provide strong visibility or relative motion while still struggling with globally consistent scale. Subsequent research has treated TAPVid-3D both as an evaluation target and as a framework for developing geometry-aware, occlusion-aware, long-horizon 3D correspondence.