Zero-Shot Point Tracking

• Zero-shot point tracking is a technique for localizing and tracking specified points in videos without additional task-specific training.
• It leverages general-purpose vision models and approaches like global feature matching, prompt-based diffusion, and keypoint-heatmap regression for robust correspondence estimation.
• The method demonstrates strong generalization across diverse datasets, achieving competitive accuracy on benchmarks and enabling flexible, annotation-free tracking.

Zero-shot point tracking refers to the task of localizing and following the trajectory of one or more specified points in a video without any task-specific or target-domain training. This paradigm relies on general-purpose vision models, often trained on large-scale, heterogeneous data, and leverages either universal visual representations, conditional generation, or prompt-based mechanisms for point propagation and correspondence. Zero-shot point tracking stands in contrast to supervised approaches and classic optical flow, as it delivers correspondence on new datasets, object types, and keypoint definitions without retraining or explicit per-task annotation.

1. Foundational Approaches and Architectures

Zero-shot point tracking is realized through diverse algorithmic frameworks, each leveraging different foundations. The principal contemporary approaches include global feature matching, task-conditioned adaptation, prompt-based diffusion, keypoint-heatmap regression combined with off-the-shelf trackers, and point-centric pipelines integrated with large segmentation models.

• Global Matching with Pretrained Vision Transformers: MegaFlow (Zhang et al., 26 Mar 2026) formulates dense and long-range point tracking as a set of global feature-matching problems between a fixed query frame and all target frames, utilizing frozen DINOv2 ViT-L features. An all-pairs dot-product similarity is computed for every candidate pair of positions, followed by spatial softmax normalization and expectation to yield sub-pixel flow fields, which are then iteratively refined via local convolutional modules and temporal attention.
• Task-Adaptive Point Embedding: The TACK architecture (Vecerik et al., 2021) employs a few-shot meta-learning framework in which a small set of annotated "support" images with per-point heatmaps is processed via a ResNet-based encoder to produce a task embedding. This embedding conditions a FiLM-modulated decoder, which predicts per-pixel probability maps for the point of interest in novel query views, enabling zero-shot transfer to new instances and points.
• Prompted Diffusion-based Counterfactual Tracking: Point Prompting (Shrivastava et al., 13 Oct 2025) exploits video diffusion models trained for synthesis. A colored marker is inserted at the query point in the first frame and the entire sequence is regenerated in a counterfactual manner using SDEdit with negative-prompt guidance. The trajectory of the marker, extracted via color thresholding, is interpreted as the point track, with additional inpainting and color suppression modules for refinement.
• Semantic Keypoint Extraction with Frozen Tracking Backbones: PlantTrack (Marri et al., 2024) demonstrates zero-shot sim-to-real transfer by combining a small, synthetic-data-trained, OpenPose-style heatmap regressor on frozen DINOv2 features, with depth masking for background suppression. The extracted semantic keypoints serve as prompts for TAPIR, a frozen transformer-based tracker, which associates and refines tracks frame-to-frame.
• Point-Centric Prompt-based Segmentation Pipelines: SAM-PT (Rajič et al., 2023) integrates sparse point propagation—using pretrained point trackers—to supply prompts to the Segment Anything Model (SAM), iteratively refining object masks via positive and negative point sets. Sparse sampling (via K-medoids) and robust occlusion/confidence estimation are key design elements.

2. Mathematical Formalism and Core Algorithms

While architectures differ, zero-shot point tracking methods are unified by the absence of per-task learning, reliance on strong visual priors, and a procedural focus on correspondence.

• In global-matching approaches, the similarity $$C_{0\to t}(\mathbf{x},\mathbf{y}) = \langle F_0(\mathbf{x}), F_t(\mathbf{y}) \rangle$$ forms the basis of all-pairs matching; normalized to $M_{0\to t}(\mathbf{x},\mathbf{y})$ via spatial softmax. Expected target location and displacement are then $$\bar G_{0\to t}(\mathbf{x}) = \sum_{\mathbf{y}} M_{0\to t}(\mathbf{x},\mathbf{y}) G(\mathbf{y})$$ and $$f^{\mathrm{init}}_{0\to t}(\mathbf{x}) = \bar G_{0\to t}(\mathbf{x}) - G(\mathbf{x})$$, respectively (Zhang et al., 26 Mar 2026).
• In the diffusion-tracking paradigm, marker propagation is explicitly induced by classifier-free guidance of the denoising score, $$\tilde{\boldsymbol\epsilon}_\theta(\mathbf x_t) = (\lambda + 1)\,\boldsymbol\epsilon_\theta(\mathbf x_t,t,\phi(I_0))-\lambda\,\boldsymbol\epsilon_\theta(\mathbf x_t,t,I_0)$$, ensuring fidelity of the edited marker across the generated video (Shrivastava et al., 13 Oct 2025).
• Task-adaptive embedding models operate by encoding each support pair into a latent vector and performing FiLM-based modulation of a deep decoder, yielding $p^T_q = \mathrm{Dec}_\theta(I^T_q, c^T)$ where $c^T$ is the mean support embedding. Prediction hinges on accurate heatmap regression and soft spatial argmax (Vecerik et al., 2021).
• Heatmap regression pipelines optimize a multi-stage mean-squared error loss over the output and ground truth heatmaps. Domain robustness is obtained via depth masking and minimal network adaptation, with all foundational backbones (e.g. DINOv2, TAPIR) kept frozen (Marri et al., 2024).
• Point-propagation with segmentation employs tracker outputs for $\{p_t^{(i)}\}$ and associated occlusion confidences $\{o_t^{(i)}\}$, filtering unreliable points and providing robust, sparse inputs for iterative mask decoding (Rajič et al., 2023).

3. Experimental Protocols and Benchmarks

Zero-shot point tracking methodologies are predominantly evaluated on large-scale tracking datasets under strict zero-shot conditions:

• TAP-Vid Benchmarks: Used for assessing both tracker-centric and flow-based methods. MegaFlow achieves $73.6\%$ mean positional accuracy across DAVIS, Kinetics, and RGB-Stacking benchmarks in zero-shot mode, outperforming supervised flow methods such as RAFT and MemFlow-T (Zhang et al., 26 Mar 2026). Diffusion-based prompting yields $M_{0\to t}(\mathbf{x},\mathbf{y})$0 and Average Jaccard $M_{0\to t}(\mathbf{x},\mathbf{y})$1 on TAP-Vid DAVIS, significantly above prior zero-shot baselines (Shrivastava et al., 13 Oct 2025).
• Sim2Real Keypoint Tracking: PlantTrack, trained on only 20 synthetic images, realizes $M_{0\to t}(\mathbf{x},\mathbf{y})$2 recall on real greenhouse imagery and maintains average reprojection drift below $M_{0\to t}(\mathbf{x},\mathbf{y})$3 px over $M_{0\to t}(\mathbf{x},\mathbf{y})$4 frames (Marri et al., 2024).
• Object-Conditioned Evaluation: TACK is benchmarked on Google Scanned Objects for keypoint transfer, reporting sub-4 px RMSE for previously unseen points, outperforming dense object-embedding baselines and approaching the accuracy of specialist sparse models (Vecerik et al., 2021).
• Video Object Segmentation: SAM-PT establishes a strong zero-shot segmentation transfer by leveraging point propagation: it achieves $M_{0\to t}(\mathbf{x},\mathbf{y})$5 on DAVIS'17, surpassing prior zero-shot and mask-prompted methods (Rajič et al., 2023).

Method	Key Principle	Typical Backbone
MegaFlow (Zhang et al., 26 Mar 2026)	Global ViT matching + CNN	DINOv2 ViT-L, CNN
PlantTrack (Marri et al., 2024)	Heatmap regressor + TAPIR	DINOv2 (frozen), TAPIR
Point Prompting (Shrivastava et al., 13 Oct 2025)	Diffusion model + prompts	Wan2/2.2, CogVideoX
TACK (Vecerik et al., 2021)	Latent embedding adaptation	ResNet, U-Net/FILM
SAM-PT (Rajič et al., 2023)	Sparse point tracking + SAM	CoTracker/PIPS, SAM

4. Robustness, Ablation, and Generalization

Explicit analysis reveals the dependence of zero-shot point tracking performance on visual prior quality, feature aggregation, and the design of point propagation or embedding mechanisms.

• Vision Priors and Transfer: Ablating vision transformer pretraining in MegaFlow yields a severe drop in accuracy (Clean EPE increases from $M_{0\to t}(\mathbf{x},\mathbf{y})$6 to $M_{0\to t}(\mathbf{x},\mathbf{y})$7, KITTI FL-all $M_{0\to t}(\mathbf{x},\mathbf{y})$8 to $M_{0\to t}(\mathbf{x},\mathbf{y})$9); feature fusion and temporal attention further improve robustness (Zhang et al., 26 Mar 2026).
• Tracker-Induced Failure Modes: Point-based pipelines such as SAM-PT show vulnerability in conditions of rapid motion, pervasive occlusion, and thin structures. Robustness is increased by frequent reinitialization, occlusion filtering, and sparse negative point use (Rajič et al., 2023).
• Conditioning and Data Augmentation: In PlantTrack, depth masking and multi-stage cascade regression are essential. Removing data augmentation or the multi-stage heatmap regressor impairs real-image recall by $$\bar G_{0\to t}(\mathbf{x}) = \sum_{\mathbf{y}} M_{0\to t}(\mathbf{x},\mathbf{y}) G(\mathbf{y})$$0 or increases keypoint error by $$\bar G_{0\to t}(\mathbf{x}) = \sum_{\mathbf{y}} M_{0\to t}(\mathbf{x},\mathbf{y}) G(\mathbf{y})$$1, respectively (Marri et al., 2024).
• Generalization Across Tasks: TACK bridges the accuracy–generality spectrum, achieving strong cross-instance transfer even when canonical keypoint definitions are absent, due to separation of "point identity" from "pixel-level localization" via a small latent code (Vecerik et al., 2021).

5. Comparative Analysis and Limitations

Zero-shot point tracking frameworks exhibit clear trade-offs between computational overhead, adaptability, and accuracy:

• Computational Demands: Diffusion-based tracking via video generation is more computationally expensive (7–30 minutes per point), while approaches like MegaFlow and TAPIR run orders-of-magnitude faster due to feedforward inference (Shrivastava et al., 13 Oct 2025, Marri et al., 2024).
• Sparse vs. Dense Correspondence: Point-centric trackers (SAM-PT, TACK) focus capacity on moderately sparse points, granting higher per-point accuracy than fully dense embedding approaches. The propagation of a carefully selected set of points enables high-fidelity mask reconstruction with reduced redundancy (Rajič et al., 2023, Vecerik et al., 2021).
• Limitations: All contemporary methods struggle with complete occlusions and non-rigid large deformations. For diffusion models, the marker may be ignored on featureless or highly symmetric surfaces, and trajectory extraction fails if counterfactual propagation breaks down (Shrivastava et al., 13 Oct 2025).
• Ablation Benchmarks: Pipeline ablation in Point Prompting shows that omitting negative prompt guidance, refinement, or color rebalancing causes significant drops in AJ and positional accuracy (Shrivastava et al., 13 Oct 2025).

6. Emerging Trends and Future Directions

Current research suggests converging paradigms that exploit the transfer abilities of vision foundation models and generative video models for zero-shot tracking:

• The use of frozen, high-dimensional representations (e.g. DINOv2, ViT) and universal backbones (SAM, TAPIR) enables plug-and-play solutions with minimal task-specific adaptation (Zhang et al., 26 Mar 2026, Marri et al., 2024, Rajič et al., 2023).
• Counterfactual methods, such as video diffusion with prompt-based editing, reveal emergent motion understanding within generative models and suggest the possibility of distilling such knowledge into more efficient end-to-end trackers (Shrivastava et al., 13 Oct 2025).
• Recent results imply a trend toward multi-modal and open-world tracking: prompt-based approaches may readily extend to multiple points, arbitrary semantic queries, and beyond color markers toward richer language or vision prompts.
• A plausible implication is that integrating memory modules, confidence modeling, or joint fine-tuning of prompt-receiver modules (e.g., tracker + segmentation head) could mitigate current drifts and occlusion failures.

In summary, zero-shot point tracking is a rapidly evolving research arena leveraging powerful pretrained priors, iterative refinement, prompt engineering, and adaptive conditioning to deliver highly generalizable correspondence estimation. Robust state-of-the-art performance has been attained on several benchmarks without domain-specific retraining, establishing a central role for foundation models in low-annotation, high-flexibility vision applications.