PointSt3R: Robust 2D/3D Point Tracking

• PointSt3R is a point tracking method that extends 3D reconstruction models with dynamic matching and an explicit visibility head for occlusion handling.
• It fine-tunes a MASt3R-based encoder–decoder architecture on synthetic datasets, optimizing both static and dynamic correspondences using specialized loss functions.
• Benchmark evaluations reveal that PointSt3R achieves competitive accuracy on datasets like TAP-Vid-DAVIS and EgoPoints while managing runtime challenges inherent to N² matching.

PointSt3R is a point tracking method derived from foundational 3D reconstruction models, specifically adapting and extending architectures such as DUSt3R and MASt3R for 2D and 3D grounded correspondence in both static and dynamic scenes. The approach introduces architectural modifications and training procedures to enhance dynamic point tracking and explicit occlusion handling, conducting tracking in a strictly pairwise setting without temporal sequence context, and achieving competitive accuracy across several standard benchmarks (Guerrier et al., 30 Oct 2025).

1. Architectural Foundations and Modifications

PointSt3R employs MASt3R’s encoder–decoder backbone, inheriting its 3D-grounded capabilities. DUSt3R originally produces per-pixel 3D pointmaps and estimates camera poses for static scenes, while MASt3R adds a dense feature‐descriptor head, enabling robust static correspondence between two views.

PointSt3R introduces two critical changes for point tracking:

• Addition of a visibility head $$\mathcal{H}_{\text{vis}}(H,H') \rightarrow V \in [0,1]^{H \times W}$$, which predicts for each pixel whether it is visible in the paired frame.
• Incorporation of a dynamic‐matching loss on descriptor features $D^1, D^2$ to directly supervise dynamic correspondences.

The training procedure involves freezing the backbone encoder and fine-tuning the decoders and all heads, including the new visibility head, on data comprising both static and dynamic correspondences.

Inference remains strictly pairwise: given a query point at pixel $i$ in frame $I^0$, its descriptor $D^0(i)$ is matched by maximum cosine similarity to descriptors in frame $I^t$, with no use of temporal context or post-processing.

2. Training Objectives and Loss Function

PointSt3R’s training regime combines multiple loss functions targeting accurate 3D grounding, point matching, and visibility:

• Confidence-weighted regression loss: Follows DUSt3R/MASt3R, penalizing errors in 3D point predictions weighted by a learned confidence $C_i^v$.
• Static-point matching loss ($L_{\text{match}}^{\text{static}}$): InfoNCE loss on ground-truth static correspondences $M^{\text{static}}$ using dense descriptors.
• Dynamic-point matching loss ($L_{\text{match}}^{\text{dynamic}}$): Identical to the static-point form, but over $D^1, D^2$0, where point 3D coordinates differ between frames.
• Visibility loss ($D^1, D^2$1): Binary cross-entropy for visibility prediction using the new head, supervising point-wise occlusion handling.

The total loss is summed as:

$D^1, D^2$2

with typical weightings $D^1, D^2$3, $D^1, D^2$4.

3. Data Regime and Training Protocol

PointSt3R is fine-tuned using 10,000 synthetic image pairs, evenly sourced from:

• PointOdyssey (deformable characters, significant camera motion)
• Kubric (rigid object collisions, CoTracker3 version)
• DynamicReplica (dynamic scenes with accurate 2D–3D tracks)

For each pair, up to 4,096 static and dynamic positive correspondences are sampled, along with negatives. The ratio $D^1, D^2$5 is varied, with results best for $D^1, D^2$6, emphasizing dynamic over static matches.

Frame pairs are drawn with broad temporal strides (e.g., 10–170 frames apart) to encourage long-range correspondence learning. Critically, training and inference are restricted to the two frames containing the query point and its correspondence—no sliding windows or additional context is utilized.

4. Evaluation Metrics and Comparative Performance

The primary evaluation metrics are:

• $D^1, D^2$7: The mean fraction of visible points tracked within thresholds $D^1, D^2$8 pixels.
• Occlusion accuracy (OA): The proportion of points with correctly predicted in-frame visibility.

Performance is benchmarked on four standard datasets—TAP-Vid-DAVIS, RoboTAP, RGB-S, EgoPoints—with the following $D^1, D^2$9 and OA results:

Dataset	CoTracker2	CoTracker3	MASt3R	PointSt3R
TAP-Vid-DAVIS	75.7	76.7	38.5	73.8
Occlusion OA (%)	88.3	90.2	—	85.8
RoboTAP	70.6	78.8	71.6	78.6
RGB-S	83.3	82.8	73.2	87.0
EgoPoints	35.5	54.2	53.5	61.3
EgoPoints (dynamic)	20.2	35.8	12.3	31.4

On 3D point tracking (PStudio minival, APD metric), PointSt3R achieves 61.2% ($i$0 ZoeDepth: 62.9%) without any depth/intrinsics, compared to CoTracker3+ZoeDepth at 66.2% (w/o GT) and 80.3% (with GT).

Key performance findings include matching or approaching state-of-the-art $i$1 on 2D benchmarks and outperforming CoTracker3 by $i$211% on EgoPoints. In dynamic-only subsets, PointSt3R exceeds CoTracker2 by at least $i$311% absolute.

5. Ablation Studies

Empirical ablations highlight the importance of architectural and data design choices:

• Dynamic/static ratio ($i$4): Increasing $i$5 from 0% to $i$695% yields consistent $i$7 improvements; exceeding 95% causes loss of static 3D grounding.
• Temporal stride: Training on large stride pairs ([10,…,170] frames) raises $i$8 to 73.8%, relative to 68.5% for short strides ([1,…,9]).
• Dataset mix: PointOdyssey-only yields $i$972.7%; adding Kubric and DynamicReplica further elevates performance.
• Visibility head: Exclusion prevents point occlusion modeling (OA undefined); its inclusion provides OA$I^0$085.8% on DAVIS.
• Inference resolution: 512×384 offers best $I^0$1; bilinear descriptor sampling improves results by ~1.1% versus nearest-neighbor.
• Runtime: Pairwise $I^0$2 matching is $I^0$31.7 s for 16 frames × 5 queries—about 5× slower than temporal-window-optimized approaches (e.g., CoTracker3).

6. Significance, Limitations, and Prospective Directions

PointSt3R demonstrates the viability of leveraging a 3D-grounded reconstruction backbone for robust point tracking in dynamic and occluded scenarios by combining synthetic dynamic data, a dedicated dynamic-matching loss, explicit occlusion prediction, and extended-temporal pairwise training. The approach eschews multi-frame temporal modeling, yet maintains or exceeds state-of-the-art performance on major benchmarks with a unified pipeline for 2D and 3D tracking using only two-frame inference.

Substantial computational cost due to $I^0$4 matching and reliance on synthetic data constitute primary limitations. Closing the sim-to-real gap and possibly integrating light temporal smoothing or multi-view inference represent promising future enhancements that may further improve performance and robustness, especially in complex dynamic environments (Guerrier et al., 30 Oct 2025).