TAPVid-360: Panoramic Directional Tracking

• TAPVid-360 is a novel benchmark that reformulates TAP by tracking camera-relative unit 3D direction vectors across 360° videos, addressing image-centric limitations.
• The method converts initial 2D query pixels into 3D directions using camera intrinsics and leverages modified CoTracker architecture via rotation prediction in SO(3).
• Empirical results show CoTracker360 achieves significant stability improvements, especially for out-of-frame points, reducing angular errors compared to traditional approaches.

Searching arXiv for TAPVid-360 and closely related 360 tracking papers to ground the article in the cited literature. Tool unavailable in this environment, so grounding the article strictly in the provided arXiv records and cited ids. TAPVid-360 is a point-tracking task, dataset, and benchmark that reformulates Track Any Point (TAP) for panoramic persistence rather than image-plane localization. Instead of predicting where a queried point lands in the current image, it requires prediction of the 3D direction from the camera to that scene point at every frame, including when the point is far outside the narrow field of view of the observed video and may lie behind the camera. The benchmark is introduced as a way to foster allocentric scene representations without requiring dynamic 4D ground-truth scene models for training: supervision is obtained from 360° videos, which are resampled into narrow field-of-view perspective clips while ground-truth directions are computed by tracking points across the full panorama with a 2D pipeline (Hudson et al., 26 Nov 2025).

1. Conceptual shift from image-centric TAP to panoramic directional persistence

TAPVid-360 is defined against the limitations of standard TAP formulations. In ordinary TAP, a perspective video $V=(I_t)_{t=1}^T$ is paired with query points in the first frame, and the model predicts 2D tracks

$$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\quad t=1,\dots,T,\; k=1,\dots,N.$$

This representation is inherently image-centric. Once a point leaves the field of view, the meaning of an image-plane prediction becomes unstable, particularly for points more than $90^\circ$ away from the current viewing direction.

The benchmark positions itself relative to both TAP and TAPVid-3D. TAPVid-3D can in principle represent 3D points,

$P_t^k=(u_t^k,v_t^k,w_t^k)\in\mathbb{R}^3,$

but the paper argues that practical supervision for full dynamic 4D tracking is difficult to obtain, and many methods effectively reduce to a 2.5D formulation of image coordinates plus depth,

$P_t^k=(i_t^k,j_t^k,w_t^k)\in\mathbb{R}^3,$

which still inherits the same failure mode outside the frustum. TAPVid-360 changes the target itself: it tracks a camera-relative direction on the viewing sphere rather than a pixel location or a depth-bearing 3D position (Hudson et al., 26 Nov 2025).

A central misconception addressed by the benchmark is that “out of frame” should be treated as an absence of supervision. In TAPVid-360, out-of-view is part of the task definition. The target remains defined across the full sphere, so object permanence is not an optional re-identification behavior after re-entry into view; it is a required prediction throughout the sequence.

2. Formal task and directional representation

Given a narrow field-of-view perspective video and query pixels in frame 1,

$Q^k=(i^k,j^k), \quad k=1,\dots,N,$

the benchmark requires prediction, at every frame, of a unit 3D direction vector

$$D_t^k=(x_t^k,y_t^k,z_t^k), \quad t=1,\dots,T,\;k=1,\dots,N,$$

subject to

$\|D_t^k\|=1.$

The tracked quantity is therefore the direction from the camera center to the corresponding scene point in camera coordinates, not a point in Euclidean space and not a spherical angle pair (Hudson et al., 26 Nov 2025).

In the data-generation pipeline, a tracked 2D pixel in a perspective frame is converted to a direction using the camera intrinsics. The paper describes this as unprojecting the homogeneous pixel ray through $\mathbf{K}_{k,t}^{-1}$ and normalizing the resulting vector. The result is explicitly a unit 3D vector in camera coordinates. This target remains meaningful even when the point is outside the image bounds or in the back hemisphere.

The baseline uses a rotation-based parameterization rather than direct angular regression. The initial query pixel is converted to an initial direction vector, and the model predicts per-point rotations in $SO(3)$ that update this direction over time. The description implies the composition

$$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\quad t=1,\dots,T,\; k=1,\dots,N.$$0

with $$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\quad t=1,\dots,T,\; k=1,\dots,N.$$1. This design preserves the “relative update” philosophy of CoTracker v3 while moving the prediction space from image coordinates to directional motion.

A useful way to situate the benchmark is the following:

Formulation	Prediction target	Failure mode emphasized in the paper
TAP	2D image coordinates	Breaks once a point leaves the image
TAPVid-3D style	3D point or 2.5D image+depth	Supervision is hard; 2.5D still breaks outside the frustum
TAPVid-360	Camera-relative unit direction	Requires persistence even for out-of-frame points

3. Dataset construction and pseudo-ground-truth generation

The dataset is built from 360° video, mainly curated from the 360-1M dataset of YouTube 360° content stored in equirectangular projection. The authors first apply coarse filtering: videos must have more than 15 likes, metadata must indicate 360° formatting, top-bottom content is cropped and scaled to $$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\quad t=1,\dots,T,\; k=1,\dots,N.$$2, “poster” videos are removed by thresholding and black-border detection, static clips are filtered using inter-frame pixel variance, LPIPS is used to reject 180° videos or wrongly formatted top-bottom content, and seam continuity between the left and right edges is checked with normalized cross-correlation. These quantities are averaged over ten sampled frames per video. Fine filtering then rejects clips with insufficient motion, cuts or scene transitions detected by PySceneDetect, and overlays detected by a LAION watermark detector. The result is around 130k 10-second clips with usable dynamic content or camera motion (Hudson et al., 26 Nov 2025).

Each clip is subsampled to $$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\quad t=1,\dots,T,\; k=1,\dots,N.$$3 frames. On the first equirectangular frame, Lang-SAM is applied with a curated list of dynamic object categories such as person, dog, bicycle, and car. The top-$$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\quad t=1,\dots,T,\; k=1,\dots,N.$$4 masks above a confidence threshold are retained and propagated through the full 32-frame 360° video using SAM2. If SAM2 cannot maintain a coherent mask, the instance is discarded.

For each successfully tracked instance, the pipeline generates an object-centered perspective video by sampling time-varying camera parameters: extrinsic rotation $$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\quad t=1,\dots,T,\; k=1,\dots,N.$$5 and intrinsic matrix $$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\quad t=1,\dots,T,\; k=1,\dots,N.$$6. Query pixels are sampled inside the projected mask in the first frame and tracked with CoTracker3 on the object-centered crop. Tracks are filtered by cumulative displacement,

$$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\quad t=1,\dots,T,\; k=1,\dots,N.$$7

and only tracks with $$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\quad t=1,\dots,T,\; k=1,\dots,N.$$8 are kept. These 2D tracks are then unprojected into camera-coordinate unit directions, which become pseudo-ground-truth targets.

The crucial step is a second resampling stage. Using the recovered directional tracks, the authors generate new narrow field-of-view videos from the same 360° source, with camera trajectories sampled from motion functions including static, spin$$P_t^k=(i_t^k,j_t^k)\in\mathbb{R}^2,\quad t=1,\dots,T,\; k=1,\dots,N.$$9, spiral, simulated human, random, and back-to-front (btf). Because the final camera is no longer object-centered, tracked points may remain visible, leave the frame, or move into the back hemisphere, while supervision is still available from the underlying panorama. This is the central benchmark-generation mechanism.

The paper describes the resulting benchmark as TAPVid360-10k and reports several counts under different conventions. The abstract states “10k perspective videos with ground-truth directional point tracking.” The main text states that the resampling stage yields 5k training and 10k test samples. Table 1, which the text says reports statistics on the validation set, lists 4772 videos / clips, 10,000 objects, 256 average trajectories per clip, FPS range 3.75–30, and data type “360.” Table 2 reports 36.28M points within frame and 45.64M points out of frame, indicating that out-of-view supervision is a dominant component of the benchmark (Hudson et al., 26 Nov 2025).

The supervision is explicitly pseudo-ground truth rather than motion-capture-grade truth. The paper identifies several potential noise sources: Lang-SAM detection and prompting, SAM2 propagation failures or drift, CoTracker3 errors on object-centered crops, projection and resampling errors, and filtering heuristics that may favor easier or more dynamic points. The appendix further describes a two-stage manual verification process: first, an operator accepts or rejects point tracks on object-centered crops; second, a visualization tool checks camera-motion strategies and frustum alignment after camera emulation.

4. CoTracker360 baseline

The benchmark introduces CoTracker360, a baseline that adapts CoTracker v3 to the directional target. The architectural modification is limited but specific: the final decoder layer is replaced with a linear layer producing 9 outputs, reshaped into a $90^\circ$0 matrix, which is then projected to the nearest valid rotation matrix using special orthogonal Procrustes orthonormalization. Thus, for each point and frame, the network predicts a rotation in $90^\circ$1 rather than an image-plane displacement (Hudson et al., 26 Nov 2025).

The initial query point is converted to a direction via the camera intrinsics, and the predicted rotation is applied to obtain the direction estimate. The model therefore retains CoTracker3’s temporal modeling and offline tracking setup while changing the output space from image coordinates to directional motion. The paper does not enumerate further internal transformer changes; the primary intervention is the output representation.

Supervision uses Huber loss on the direction vectors. The authors state that they tried angular error directly but found it less stable. They do not supervise CoTracker3’s native confidence and visibility outputs during training. The baseline is initialized from pretrained CoTracker3 offline weights and fine-tuned on a separate generated training set of 5k clips.

The reported training configuration is: 120 epochs, Adam, learning rate $90^\circ$2, single NVIDIA A40, batch size 8, and—due to memory constraints—32 query points and 32 frames during training. Evaluation uses 256 query points and 32-frame clips. These choices are important because they establish that the baseline is a direct adaptation of a strong tracker rather than a wholly new panoramic architecture.

5. Evaluation protocol and empirical findings

The benchmark compares CoTracker360 with TAPIR, BootsTAPIR, CoTracker3, SpatialTracker, TAPNext, and TAPIP3D. To make these methods compatible with TAPVid-360, their outputs are converted to directional vectors using the known camera intrinsics. This conversion is straightforward only when a point remains in or near the image; it exposes the representational limitation of image-centric trackers once points move far outside the frustum (Hudson et al., 26 Nov 2025).

The principal metric adapts TAP-style threshold accuracy $90^\circ$3 from pixel error to angular threshold error. Given the benchmark’s field of view, a movement of one pixel corresponds to

$90^\circ$4

The angular thresholds are

$90^\circ$5

The final $90^\circ$6 is the average fraction of predictions within these thresholds. Results are reported for all points, in-frame (IF) points, and out-of-frame (OOF) points. The paper also reports average angular distance $90^\circ$7, again for all, IF, and OOF subsets.

The main quantitative results are:

Method	$90^\circ$8	$90^\circ$9	$P_t^k=(u_t^k,v_t^k,w_t^k)\in\mathbb{R}^3,$0	$P_t^k=(u_t^k,v_t^k,w_t^k)\in\mathbb{R}^3,$1	$P_t^k=(u_t^k,v_t^k,w_t^k)\in\mathbb{R}^3,$2	$P_t^k=(u_t^k,v_t^k,w_t^k)\in\mathbb{R}^3,$3
TAPNext	0.0082	0.0191	0.0004	51.9752	36.5923	62.4601
TAPIR	0.0106	0.0251	0.0003	49.8086	33.8824	60.5154
BootsTAPIR	0.0126	0.0293	0.0005	48.3582	33.3154	58.3841
TAPIP3D	0.2476	0.4698	0.0850	36.4412	23.3191	45.7951
SpatialTracker	0.2239	0.4893	0.0303	38.8780	22.1635	50.4350
CoTracker3 (offline)	0.2435	0.5588	0.0158	37.4287	17.6352	50.9759
Ours (CoTracker360)	0.2386	0.4060	0.1160	8.2749	3.9496	10.9829

These numbers show that CoTracker360 is not the best method on in-frame threshold accuracy, but it is substantially more stable in angular error, especially out of frame. The paper highlights two specific comparisons: its out-of-frame threshold accuracy

$P_t^k=(u_t^k,v_t^k,w_t^k)\in\mathbb{R}^3,$4

exceeds TAPIP3D’s $P_t^k=(u_t^k,v_t^k,w_t^k)\in\mathbb{R}^3,$5 by about $P_t^k=(u_t^k,v_t^k,w_t^k)\in\mathbb{R}^3,$6, and its out-of-frame angular error

$P_t^k=(u_t^k,v_t^k,w_t^k)\in\mathbb{R}^3,$7

is more than a 4× reduction relative to TAPIP3D’s $P_t^k=(u_t^k,v_t^k,w_t^k)\in\mathbb{R}^3,$8. The intended interpretation is that image-centric methods may remain competitive on easy in-frame cases while failing catastrophically under long-range out-of-view continuation.

The appendix further reports subset behavior by camera motion. CoTracker360 performs best overall on Spiral with

$P_t^k=(u_t^k,v_t^k,w_t^k)\in\mathbb{R}^3,$9

while Spin Y is the hardest condition, with

$P_t^k=(i_t^k,j_t^k,w_t^k)\in\mathbb{R}^3,$0

The paper attributes the Spiral result to objects often remaining partly visible or near frame edges, and the Spin Y failure to the difficulty of roll-like inversion effects.

6. Relation to neighboring 360 benchmarks, misconceptions, and limitations

TAPVid-360 is not equivalent to object- or mask-centric 360 video datasets. Leader360V is a large-scale real-world 360° video dataset for instance segmentation and tracking, with Leader360-S for whole-frame segmentation and Leader360-T for multi-object tracking. Its annotations are instance masks, semantic labels, tracking identities over time, and bounding boxes; its metrics are $P_t^k=(i_t^k,j_t^k,w_t^k)\in\mathbb{R}^3,$1, $P_t^k=(i_t^k,j_t^k,w_t^k)\in\mathbb{R}^3,$2, $P_t^k=(i_t^k,j_t^k,w_t^k)\in\mathbb{R}^3,$3, $P_t^k=(i_t^k,j_t^k,w_t^k)\in\mathbb{R}^3,$4, and $P_t^k=(i_t^k,j_t^k,w_t^k)\in\mathbb{R}^3,$5. The paper explicitly does not mention TAPVid-360. Relative to TAPVid-360, Leader360V is therefore object- and mask-centric rather than point-centric; the two resources are complementary rather than interchangeable (Zhang et al., 17 Jun 2025).

A second adjacent but distinct line of work is 360° dynamic reconstruction. 4DGS360 addresses 360° dynamic object reconstruction from a single monocular video and introduces AnchorTAP3D, which produces reinforced 3D point trajectories with visibility handling for reconstruction initialization. Although this work is conceptually close to TAP-style long-range correspondence under occlusion and large viewpoint change, it is not a TAPVid-360 benchmark, does not mention TAPVid-360, and evaluates reconstruction quality rather than point-tracking accuracy (Jang et al., 23 Mar 2026).

Several limitations are explicit in the TAPVid-360 paper itself. The dataset uses a fixed field of view, so it does not yet test changing FOV or zoom. The authors suggest that proper zoom handling may require replacing fixed positional encodings with a per-patch directional encoding derived from the ray through each patch center. The benchmark also evaluates single directional predictions rather than uncertainty; the paper proposes that future work should predict a directional distribution instead of a single unit vector. Finally, because the supervision is pseudo-ground truth derived from a 2D pipeline, the dataset inherits its upstream errors (Hudson et al., 26 Nov 2025).

The benchmark’s broader significance lies in its redefinition of TAP as panoramic directional persistence. It asks not merely where a point is in the current image, but in which 3D direction that point lies relative to the camera over time. This makes out-of-view continuation a first-class part of the task and turns point tracking into a probe of persistent, allocentric scene understanding rather than short-range image-plane correspondence alone (Hudson et al., 26 Nov 2025).