---
title: 'PointSt3R: Robust 2D/3D Point Tracking'
url: https://www.emergentmind.com/topics/pointst3r
type: topic
---

# PointSt3R: Robust 2D/3D Point Tracking

PointSt3R is a point tracking method derived from foundational 3D reconstruction models, specifically adapting and extending architectures such as DUSt3R and MASt3R for 2D and 3D grounded correspondence in both static and dynamic scenes. The approach introduces architectural modifications and training procedures to enhance dynamic point tracking and explicit occlusion handling, conducting tracking in a strictly pairwise setting without temporal sequence context, and achieving competitive accuracy across several standard benchmarks [2510.26443].

## 1. Architectural Foundations and Modifications

PointSt3R employs MASt3R’s encoder–decoder backbone, inheriting its 3D-grounded capabilities. DUSt3R originally produces per-pixel 3D pointmaps and estimates camera poses for static scenes, while MASt3R adds a dense feature‐descriptor head, enabling robust static correspondence between two views.

PointSt3R introduces two critical changes for point tracking:

- Addition of a visibility head $\mathcal{H}_{\text{vis}}(H,H') \rightarrow V \in [0,1]^{H \times W}$, which predicts for each pixel whether it is visible in the paired frame.
- Incorporation of a dynamic‐matching loss on descriptor features $D^1, D^2$ to directly supervise dynamic correspondences.

The training procedure involves freezing the backbone encoder and fine-tuning the decoders and all heads, including the new visibility head, on data comprising both static and dynamic correspondences.

Inference remains strictly pairwise: given a query point at pixel $i$ in frame $I^0$, its descriptor $D^0(i)$ is matched by maximum cosine similarity to descriptors in frame $I^t$, with no use of temporal context or post-processing.

## 2. Training Objectives and Loss Function

PointSt3R’s training regime combines multiple loss functions targeting accurate 3D grounding, point matching, and visibility:

- **Confidence-weighted regression loss**: Follows DUSt3R/MASt3R, penalizing errors in 3D point predictions weighted by a learned confidence $C_i^v$.
- **Static-point matching loss ($L_{\text{match}}^{\text{static}}$)**: InfoNCE loss on ground-truth static correspondences $M^{\text{static}}$ using dense descriptors.
- **Dynamic-point matching loss ($L_{\text{match}}^{\text{dynamic}}$)**: Identical to the static-point form, but over $M^{\text{dynamic}}$, where point 3D coordinates differ between frames.
- **Visibility loss ($L_{\text{vis}}$)**: Binary cross-entropy for visibility prediction using the new head, supervising point-wise occlusion handling.

The total loss is summed as:
$$
L_{\text{total}} = L_{\text{conf}} + \alpha L_{\text{match}}^{\text{static}} + \beta L_{\text{match}}^{\text{dynamic}} + \gamma L_{\text{vis}}
$$
with typical weightings $\alpha = \beta = 0.075$, $\gamma = 1.0$.

## 3. Data Regime and Training Protocol

PointSt3R is fine-tuned using 10,000 synthetic image pairs, evenly sourced from:

- **PointOdyssey** (deformable characters, significant camera motion)
- **Kubric** (rigid object collisions, CoTracker3 version)
- **DynamicReplica** (dynamic scenes with accurate 2D–3D tracks)

For each pair, up to 4,096 static and dynamic positive correspondences are sampled, along with negatives. The ratio $r = |M^{\text{dynamic}}| / (|M^{\text{static}}| + |M^{\text{dynamic}}|)$ is varied, with results best for $r \approx 95\%$, emphasizing dynamic over static matches.

Frame pairs are drawn with broad temporal strides (e.g., 10–170 frames apart) to encourage long-range correspondence learning. Critically, training and inference are restricted to the two frames containing the query point and its correspondence—no sliding windows or additional context is utilized.

## 4. Evaluation Metrics and Comparative Performance

The primary evaluation metrics are:

- **$\delta_{avg}$**: The mean fraction of visible points tracked within thresholds $\{1,2,4,8,16\}$ pixels.
- **Occlusion accuracy (OA)**: The proportion of points with correctly predicted in-frame visibility.

Performance is benchmarked on four standard datasets—TAP-Vid-DAVIS, RoboTAP, RGB-S, EgoPoints—with the following $\delta_{avg}$ and OA results:

| Dataset              | CoTracker2 | CoTracker3 | MASt3R | PointSt3R |
|----------------------|------------|------------|--------|-----------|
| TAP-Vid-DAVIS        | 75.7       | 76.7       | 38.5   | 73.8      |
| Occlusion OA (%)     | 88.3       | 90.2       | —      | 85.8      |
| RoboTAP              | 70.6       | 78.8       | 71.6   | 78.6      |
| RGB-S                | 83.3       | 82.8       | 73.2   | 87.0      |
| EgoPoints            | 35.5       | 54.2       | 53.5   | 61.3      |
| EgoPoints (dynamic)  | 20.2       | 35.8       | 12.3   | 31.4      |

On 3D point tracking (PStudio minival, APD metric), PointSt3R achieves 61.2% ($+$ ZoeDepth: 62.9%) without any depth/intrinsics, compared to CoTracker3+ZoeDepth at 66.2% (w/o GT) and 80.3% (with GT).

Key performance findings include matching or approaching state-of-the-art $\delta_{avg}$ on 2D benchmarks and outperforming CoTracker3 by $+$11% on EgoPoints. In dynamic-only subsets, PointSt3R exceeds CoTracker2 by at least $+$11% absolute.

## 5. Ablation Studies

Empirical ablations highlight the importance of architectural and data design choices:

- **Dynamic/static ratio ($r$)**: Increasing $r$ from 0% to $\approx$95% yields consistent $\delta_{avg}$ improvements; exceeding 95% causes loss of static 3D grounding.
- **Temporal stride**: Training on large stride pairs ([10,…,170] frames) raises $\delta_{avg}$ to 73.8%, relative to 68.5% for short strides ([1,…,9]).
- **Dataset mix**: PointOdyssey-only yields $\delta_{avg}\approx$72.7%; adding Kubric and DynamicReplica further elevates performance.
- **Visibility head**: Exclusion prevents point occlusion modeling (OA undefined); its inclusion provides OA$\approx$85.8% on DAVIS.
- **Inference resolution**: 512×384 offers best $\delta_{avg}$; bilinear descriptor sampling improves results by ~1.1% versus nearest-neighbor.
- **Runtime**: Pairwise $N^2$ matching is $\approx$1.7 s for 16 frames × 5 queries—about 5× slower than temporal-window-optimized approaches (e.g., CoTracker3).

## 6. Significance, Limitations, and Prospective Directions

PointSt3R demonstrates the viability of leveraging a 3D-grounded reconstruction backbone for robust point tracking in dynamic and occluded scenarios by combining synthetic dynamic data, a dedicated dynamic-matching loss, explicit occlusion prediction, and extended-temporal pairwise training. The approach eschews multi-frame temporal modeling, yet maintains or exceeds state-of-the-art performance on major benchmarks with a unified pipeline for 2D and 3D tracking using only two-frame inference.

Substantial computational cost due to $N^2$ matching and reliance on synthetic data constitute primary limitations. Closing the sim-to-real gap and possibly integrating light temporal smoothing or multi-view inference represent promising future enhancements that may further improve performance and robustness, especially in complex dynamic environments [2510.26443].

Source: https://www.emergentmind.com/topics/pointst3r