TAPVid360-10k: 360° Directional Tracking Benchmark

• TAPVid360-10k is a benchmark dataset for persistent directional tracking in 360° videos, uniquely using narrow-FOV inputs with full panoramic supervision.
• It replaces traditional 2D image-plane coordinates with camera-relative 3D directions, enabling effective tracking even when objects leave the visible frame.
• The benchmark comprises 10k video samples from real 360° YouTube content, providing robust evaluation of allocentric reasoning and object permanence in dynamic scenes.

Searching arXiv for the specified paper and closely related tracking benchmarks/methods to ground the article. TAPVid360-10k is the benchmark dataset introduced for TAPVid-360, a tracking task in which a model receives a narrow-field-of-view perspective video and first-frame query pixels, and must predict, for each query point and each frame, a unit 3D direction vector in the camera coordinate frame, including when the point lies far outside the visible image. The benchmark is built from real 360° YouTube videos and uses the full panorama only for supervision, while the model itself sees only narrow-FOV perspective crops. This formulation is intended to test persistent panoramic understanding, object permanence, and allocentric reasoning without requiring dynamic 4D ground-truth scene models (Hudson et al., 26 Nov 2025).

1. Task formulation and representational shift

TAPVid360-10k is motivated by a limitation in prior Track Any Point formulations. In standard TAP, for a video

$$V=(I_t)_{t=1}^T,\qquad I_t \in \mathbb{R}^{3\times H \times W},$$

and first-frame query points, the goal is to predict 2D tracks

$$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\qquad t=1,\dots,T,\ k=1,\dots,N.$$

Some methods also predict visibility or occlusion, but the representation remains fundamentally image-plane-centered. The paper argues that this fails to express persistent tracking far outside the camera’s field of view, especially for points in the back hemisphere, more than $90^\circ$ away from the view direction (Hudson et al., 26 Nov 2025).

TAPVid-360 replaces image-plane coordinates with a camera-relative directional representation. With first-frame queries

$Q^k=(i^k,j^k),\qquad k=1,\dots,N,$

the prediction target is

$$D_t^k=(x_t^k,y_t^k,z_t^k),\qquad t=1,\dots,T,\ k=1,\dots,N,$$

subject to

$\|D_t^k\|=1.$

The benchmark therefore asks a model to maintain a panoramic, camera-relative belief about where a queried scene point lies, even when the point is no longer inside the visible crop.

The representational shift is narrower than full metric 3D reconstruction but broader than 2D or 2.5D tracking. The target is the 3D direction from the current camera center to the scene point, not the full 3D position. The paper explicitly presents this as a way to capture the panoramic “where is it?” aspect of tracking while avoiding the supervision bottleneck of dynamic real-scene 4D ground truth. A plausible implication is that TAPVid360-10k occupies an intermediate regime between classical image-plane correspondence benchmarks and full scene-centric dynamic reconstruction benchmarks.

2. Benchmark composition and reported scale

TAPVid360-10k is described as a benchmark of 10k perspective videos with ground-truth directional point tracking, derived from real 360° YouTube videos (Hudson et al., 26 Nov 2025). The model input is always a narrow-FOV perspective clip; the labels are computed using the full 360° panorama, so queried points can remain valid even when they leave the visible crop.

The paper reports the following concrete dataset statistics for the benchmark and its validation summary:

Attribute	Reported value	Context
Benchmark name	TAPVid360-10k	New benchmark for TAPVid-360
Total benchmark samples	10,000	Perspective-video/object-centered samples
Videos	4,772	Validation-set table
Clips	4,772	Validation-set table
Objects	10,000	Validation-set table
Average trajectories per clip	256	Validation-set table
Data type	360	Real 360° source data
FPS range	3.75–30	Reported in dataset table
Frame length	$T=32$	Generation and evaluation
Query points used in evaluation	256	Per 32-frame clip

The paper also states that the resampling stage yields 5k training and 10k test samples, while the comparison table uses the 4,772-clip validation set statistics. It explicitly notes an inconsistency between “10k benchmark samples” and the table showing 4,772 videos/clips in the validation set. The intended picture is that a broader generated dataset exists, the benchmark TAPVid360-10k refers to 10k perspective-video/object-centered samples, and the reported comparison table specifically summarizes the validation-set portion containing 4,772 clips.

The benchmark is notable for the distribution of annotated states. It reports 36.28M points within frame and 45.64M points out of frame, so more annotated point states are out-of-frame than in-frame. This is central to its design: unlike prior TAP datasets, the target does not disappear when a point leaves the image; the label remains a direction on the viewing sphere.

3. Source corpus, filtering, and quality control

The source corpus is the 360-1M dataset of about one million YouTube links for 360° video (Hudson et al., 26 Nov 2025). Videos are stored as 2D equirectangular projections. Because many links are unusable or not genuinely suitable 360° content, the paper describes a multistage curation pipeline.

The coarse filtering stage first retains only videos with more than 15 likes, reducing the corpus to roughly 100k videos as an initial quality filter. The highest-quality version is downloaded with yt-dlp. Metadata is inspected for side-data-list, which indicates 360 formatting. If the metadata is top and bottom, the top half is cropped and scaled to a 2:1 aspect ratio.

Incorrectly labeled or unsuitable videos are then removed using averaged scores over 10 evenly spaced frames. Four tests are applied. First, perspective/poster detection uses grayscale conversion and adaptive thresholding; if the main contour occupies only a small area and the rest is black border, the video is flagged as a poster and removed. Second, scene dynamics is estimated by sampling frames at random intervals and measuring pixel variance; static videos are removed. Third, formatting / LPIPS checks use LPIPS between left and right halves to filter 180° videos, and LPIPS between top and bottom halves to filter malformed 360 videos with wrong metadata. Fourth, seam continuity computes normalized cross-correlation between thin strips from the left and right edges; low similarity indicates a non-360 video.

The fine filtering stage operates after splitting videos into 10-second clips with FFmpeg. Three additional filters are applied: average magnitude of sparse optical flow vectors between every other frame, with clips below threshold removed; PySceneDetect to remove clips with cuts or fades; and a LAION watermark detector to remove clips above a confidence threshold. The resulting corpus contains about 130k 10-second clips that are described as “correctly formatted and contain good dynamic content or camera motions.”

Quality control extends beyond automation. The appendix describes a two-stage manual verification process: first, manual validation of the point-tracker output on object-centered perspective crops before camera-motion emulation; second, verification with a 3D viewer after camera emulation to assess plausibility, accuracy, frustum alignment, and diversity. This is an important safeguard because the benchmark relies on pseudo-ground truth rather than direct 3D capture.

4. Dataset generation pipeline and annotation model

The generation pipeline is designed to produce a narrow-FOV perspective clip together with pseudo-ground-truth camera-relative direction tracks, without requiring 3D reconstruction (Hudson et al., 26 Nov 2025).

The input equirectangular clip is

$$V_{eq}=(I_t)_{t=1}^T,\qquad I_t\in\mathbb{R}^{3\times H_{eq}\times W_{eq}}.$$

The clip is temporally subsampled to $T=32$ frames, specifically “to increase the likelihood of capturing salient dynamics within the clip.”

Dynamic objects are then identified by running Lang-SAM on frame $I_1$ with a curated vocabulary of dynamic object classes including person, bird, fish, dog, cat, horse, snake, animal, car, bike, motorcycle, train, airplane, boat, ship, helicopter, submarine, rocket, bus, truck, robot, drone, conveyor belt, wind turbine, fan, clock hands, gears, ball, frisbee, pendulum, swing, yo-yo, kite, shopping cart, and more. This yields initial masks

$$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\qquad t=1,\dots,T,\ k=1,\dots,N.$$0

Masks above a confidence threshold $$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\qquad t=1,\dots,T,\ k=1,\dots,N.$$1 are retained, and the top-$$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\qquad t=1,\dots,T,\ k=1,\dots,N.$$2 are selected. These are propagated through the full video using SAM2. If SAM2 fails to keep a consistent mask for an instance through the full clip, that instance is discarded. The surviving tracked instance masks are

$$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\qquad t=1,\dots,T,\ k=1,\dots,N.$$3

where $$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\qquad t=1,\dots,T,\ k=1,\dots,N.$$4 and

$$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\qquad t=1,\dots,T,\ k=1,\dots,N.$$5

For each surviving instance $$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\qquad t=1,\dots,T,\ k=1,\dots,N.$$6, the pipeline generates an object-centered perspective sequence. At frame $$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\qquad t=1,\dots,T,\ k=1,\dots,N.$$7, it samples an extrinsic rotation

$$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\qquad t=1,\dots,T,\ k=1,\dots,N.$$8

and an intrinsic matrix

$$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\qquad t=1,\dots,T,\ k=1,\dots,N.$$9

defining a perspective transform

$90^\circ$0

This renders a perspective video

$90^\circ$1

together with projected masks

$90^\circ$2

The transform sequence is chosen so that the object stays centered in the perspective view; this stage is used only to obtain high-quality 2D object tracks.

Query points are sampled on the first perspective frame $90^\circ$3 from inside the mask $90^\circ$4:

$90^\circ$5

These become spatio-temporal CoTracker queries

$90^\circ$6

The paper does not specify the exact query-point sampling strategy inside the mask beyond stating that the points are sampled from within the mask.

Pseudo-ground-truth 2D tracks are then obtained by running CoTracker3 on the object-centered perspective clip with queries $90^\circ$7, producing tracks

$90^\circ$8

These are filtered by total image-plane path length

$90^\circ$9

and only tracks satisfying

$Q^k=(i^k,j^k),\qquad k=1,\dots,N,$0

are retained. This removes static, distant, or minimally moving points and biases the benchmark toward dynamic object motion.

The retained pixel trajectories are converted to camera-relative directions by backprojection through the perspective intrinsics. For tracked pixel $Q^k=(i^k,j^k),\qquad k=1,\dots,N,$1, homogeneous coordinates are formed as

$Q^k=(i^k,j^k),\qquad k=1,\dots,N,$2

then backprojected via

$Q^k=(i^k,j^k),\qquad k=1,\dots,N,$3

and normalized to produce the unit direction

$Q^k=(i^k,j^k),\qquad k=1,\dots,N,$4

This normalized backprojection is the core pseudo-ground-truth signal.

Finally, once these directional tracks exist, the benchmark examples are produced by rendering new perspective videos from the same equirectangular source using another sequence of projection parameters,

$Q^k=(i^k,j^k),\qquad k=1,\dots,N,$5

whose motion is no longer tied to keeping the object centered. This is the mechanism that creates supervision for persistent out-of-view tracking: labels continue to exist because they are inherited from the full panorama, not from the visible crop.

5. Camera motion, field-of-view assumptions, and evaluation

The final benchmark camera trajectories are sampled from predefined motion strategies: static (original camera motion), spin$Q^k=(i^k,j^k),\qquad k=1,\dots,N,$6, spin$Q^k=(i^k,j^k),\qquad k=1,\dots,N,$7, spin$Q^k=(i^k,j^k),\qquad k=1,\dots,N,$8, spiral, simulated human, random, and btf (back-to-front) (Hudson et al., 26 Nov 2025). Framewise camera rotations are updated by

$Q^k=(i^k,j^k),\qquad k=1,\dots,N,$9

where $$D_t^k=(x_t^k,y_t^k,z_t^k),\qquad t=1,\dots,T,\ k=1,\dots,N,$$0 are pitch, roll, and yaw deltas and $$D_t^k=(x_t^k,y_t^k,z_t^k),\qquad t=1,\dots,T,\ k=1,\dots,N,$$1 is their composition in a fixed axis order.

The appendix gives explicit samplers for these motions. For spiral motion,

$$D_t^k=(x_t^k,y_t^k,z_t^k),\qquad t=1,\dots,T,\ k=1,\dots,N,$$2

For random motion,

$$D_t^k=(x_t^k,y_t^k,z_t^k),\qquad t=1,\dots,T,\ k=1,\dots,N,$$3

For simulated human motion,

$$D_t^k=(x_t^k,y_t^k,z_t^k),\qquad t=1,\dots,T,\ k=1,\dots,N,$$4

$$D_t^k=(x_t^k,y_t^k,z_t^k),\qquad t=1,\dots,T,\ k=1,\dots,N,$$5

$$D_t^k=(x_t^k,y_t^k,z_t^k),\qquad t=1,\dots,T,\ k=1,\dots,N,$$6

with

$$D_t^k=(x_t^k,y_t^k,z_t^k),\qquad t=1,\dots,T,\ k=1,\dots,N,$$7

$$D_t^k=(x_t^k,y_t^k,z_t^k),\qquad t=1,\dots,T,\ k=1,\dots,N,$$8

$$D_t^k=(x_t^k,y_t^k,z_t^k),\qquad t=1,\dots,T,\ k=1,\dots,N,$$9

$\|D_t^k\|=1.$0

For static motion,

$\|D_t^k\|=1.$1

For spin motion,

$\|D_t^k\|=1.$2

with noise

$\|D_t^k\|=1.$3

then zero-mean corrected by

$\|D_t^k\|=1.$4

and per-frame step

$\|D_t^k\|=1.$5

applied only about the chosen axis $\|D_t^k\|=1.$6. The back-to-front motion is defined by a midpoint

$\|D_t^k\|=1.$7

a central temporal window, and a symmetric forward-reverse construction over that window.

The benchmark uses a fixed field of view, and the limitations section explicitly states that it therefore does not test sensitivity to varying FOV, zoom, or dynamically changing intrinsics. The same section notes that handling zoom properly would require a per-patch directional encoding rather than fixed positional encoding. The evaluation section further states that, based on the field of view of the dataset, a movement of a single pixel corresponds to $\|D_t^k\|=1.$8, denoted as $\|D_t^k\|=1.$9.

Evaluation uses 256 query points and 32-frame clips. The standard TAP thresholded-accuracy metric is adapted from pixels to angular thresholds. The threshold set is

$T=32$0

and the mean over these thresholds yields the summary score denoted $T=32$1. The benchmark reports this metric separately for all, in-frame (if), and out-of-frame (oof) points. It also reports mean angular deviation, denoted $T=32$2, again split into all, IF, and OOF. The paper does not print an explicit closed-form geodesic-angle equation, but the intended quantity is the angular distance between predicted and ground-truth unit vectors.

A common misconception would be to treat out-of-view and occlusion as equivalent. TAPVid360-10k does not do so. Out-of-view is not treated as unlabeled or invalid; the target remains the camera-relative direction. By contrast, the final annotations do not include an explicit occlusion label. The paper notes that some TAP methods predict a binary visibility flag, but its own baseline does not supervise CoTracker3 confidence or visibility outputs during training. The benchmark is therefore centered on persistent directional tracking, especially through OOF intervals, rather than explicit occlusion classification.

6. Baseline adaptation, quantitative results, and position relative to prior benchmarks

The baseline model is CoTracker360, an adaptation of CoTracker3 that predicts a per-point rotation for directional updates rather than a per-frame 2D image position (Hudson et al., 26 Nov 2025). The first-frame query pixel is converted into an initial unit direction using camera intrinsics. For each point and frame, the model predicts a $T=32$3 rotation matrix by replacing CoTracker3’s last decoder layer with a linear layer having 9 outputs, reshaping to $T=32$4, and projecting to the nearest valid rotation with special orthogonal Procrustes orthonormalization. The intended directional update is

$T=32$5

Training supervises the resulting direction predictions with Huber loss on the direction vectors; angular error was reportedly less stable. The model is initialized from pretrained CoTracker3 offline, finetuned on 5k additional perspective video clips distinct from TAPVid360-10k, trained for 120 epochs with Adam, learning rate $T=32$6, on a single NVIDIA A40, with batch size 8. Due to memory constraints, training uses 32 query points and 32 frames.

The main benchmark comparison shows a clear pattern. CoTracker3 offline attains the strongest reported in-frame thresholded score among the listed methods, with $T=32$7, but its out-of-frame score drops to $T=32$8, and its OOF angular distance is $T=32$9. CoTracker360 yields $$V_{eq}=(I_t)_{t=1}^T,\qquad I_t\in\mathbb{R}^{3\times H_{eq}\times W_{eq}}.$$0, $$V_{eq}=(I_t)_{t=1}^T,\qquad I_t\in\mathbb{R}^{3\times H_{eq}\times W_{eq}}.$$1, and $$V_{eq}=(I_t)_{t=1}^T,\qquad I_t\in\mathbb{R}^{3\times H_{eq}\times W_{eq}}.$$2, together with $$V_{eq}=(I_t)_{t=1}^T,\qquad I_t\in\mathbb{R}^{3\times H_{eq}\times W_{eq}}.$$3, $$V_{eq}=(I_t)_{t=1}^T,\qquad I_t\in\mathbb{R}^{3\times H_{eq}\times W_{eq}}.$$4, and $$V_{eq}=(I_t)_{t=1}^T,\qquad I_t\in\mathbb{R}^{3\times H_{eq}\times W_{eq}}.$$5. The paper highlights that CoTracker360’s OOF thresholded accuracy of 0.1160 is about 1.3× higher than the next-best method, TAPIP3D at 0.0850, and that its OOF angular error of 10.98° is more than 4× lower than TAPIP3D’s 45.80°.

Other reported baselines reinforce the same conclusion. TAPNext records $$V_{eq}=(I_t)_{t=1}^T,\qquad I_t\in\mathbb{R}^{3\times H_{eq}\times W_{eq}}.$$6, $$V_{eq}=(I_t)_{t=1}^T,\qquad I_t\in\mathbb{R}^{3\times H_{eq}\times W_{eq}}.$$7, and $$V_{eq}=(I_t)_{t=1}^T,\qquad I_t\in\mathbb{R}^{3\times H_{eq}\times W_{eq}}.$$8. TAPIR records $$V_{eq}=(I_t)_{t=1}^T,\qquad I_t\in\mathbb{R}^{3\times H_{eq}\times W_{eq}}.$$9, $T=32$0, and $T=32$1 for the same respective fields. BootsTAPIR records $T=32$2, $T=32$3, and $T=32$4. SpatialTracker records $T=32$5 and $T=32$6. The quantitative pattern is that trackers designed around image-plane targets or frustum-limited 3D formulations degrade sharply once the target leaves the crop, whereas CoTracker360 remains able to produce reasonable directional estimates.

The appendix also reports subset analyses. By camera motion type, the best-performing subset is Spiral, with $T=32$7, $T=32$8, and $T=32$9. The most difficult subset is Spin Y, with $I_1$0 and $I_1$1, which the paper attributes to roll and inversion effects. Other subsets include Random with OOF accuracy 0.0671 and OOF AD 14.40°, Simulated Human with 0.0849 and 12.20°, btf with 0.0568 and 15.13°, Spin X with 0.1255 and 7.84°, Spin Z with 0.1379 and 7.62°, and Static with 0.0207 and 24.10°.

Relative to earlier TAP and TAPVid benchmarks, the differences are structural. Prior TAP datasets supervise image-plane coordinates, and once a point leaves the image the representation becomes awkward or meaningless unless padded coordinates are used; the paper argues that this still cannot represent points in the back hemisphere. TAPVid-3D moves toward 3D point trajectories, but the paper argues that practical methods and datasets still often rely on image coordinates plus depth and do not continue tracking points far outside the frustum. TAPVid360-10k instead supervises full-sphere camera-relative directions and uses 360° video as supervision to avoid requiring full dynamic 3D or 4D scene models.

The comparison table also situates the benchmark by source and scale: TAPVid-RGB-Stacking has 50 videos and 250 clips and is simulated, 2D; RoboTAP has 265 videos, real, 2D; TAPVid-Kinetics has 1,189 videos, real, 2D; TAPVid-KUBRIC has 38,325 videos, simulated, 2D; TAPVid-3D has 2,828 videos and 4,569 clips, real, 3D; and TAPVid360-10k has 4,772 videos, 4,772 clips, 10,000 objects, 256 trajectories per clip, real, 360. The paper emphasizes that, even though that table summarizes only the validation portion, TAPVid360-10k already provides strong coverage and more out-of-frame annotation than prior TAP-style benchmarks.

The principal caveats are explicit. The benchmark uses a fixed FOV; it supervises direction only, not distance; its labels are pseudo-ground truth generated from segmentation, object-centered rendering, and CoTracker3 pseudo-tracks rather than true 3D capture; and it does not center explicit uncertainty modeling or visibility supervision. The authors further note that their baseline would likely benefit from predicting a directional distribution rather than a single direction, especially as targets remain out of frame for longer durations. These limitations clarify that TAPVid360-10k is not a full metric scene-understanding benchmark. Its specific contribution is to benchmark persistent directional tracking and panoramic localization from ordinary narrow-FOV video.