Self-Supervised Any-Point Tracking by Contrastive Random Walks (2409.16288v1)

Abstract: We present a simple, self-supervised approach to the Tracking Any Point (TAP) problem. We train a global matching transformer to find cycle consistent tracks through video via contrastive random walks, using the transformer's attention-based global matching to define the transition matrices for a random walk on a space-time graph. The ability to perform "all pairs" comparisons between points allows the model to obtain high spatial precision and to obtain a strong contrastive learning signal, while avoiding many of the complexities of recent approaches (such as coarse-to-fine matching). To do this, we propose a number of design decisions that allow global matching architectures to be trained through self-supervision using cycle consistency. For example, we identify that transformer-based methods are sensitive to shortcut solutions, and propose a data augmentation scheme to address them. Our method achieves strong performance on the TapVid benchmarks, outperforming previous self-supervised tracking methods, such as DIFT, and is competitive with several supervised methods.

• The paper introduces a novel self-supervised method for any-point tracking using contrastive random walks paired with global matching transformers.
• It employs cycle-consistency loss and label warping to ensure accurate space-time correspondences without relying on labeled data.
• Extensive evaluations on TapVid benchmarks demonstrate competitive positional accuracy and average Jaccard improvements over previous self-supervised methods.

Self-Supervised Any-Point Tracking by Contrastive Random Walks

The paper "Self-Supervised Any-Point Tracking by Contrastive Random Walks" by Ayush Shrivastava and Andrew Owens introduces a novel, self-supervised method for addressing the Tracking Any Point (TAP) problem in video data. The TAP problem involves identifying space-time correspondences for any physical point in a video sequence, a crucial task in various computer vision applications. This work proposes a model that leverages global matching transformers and contrastive random walks to achieve high spatial precision without requiring labeled training data.

Methodology

The core approach revolves around the use of a global matching transformer, which allows for "all pairs" attention-based comparisons between points in different frames. This method provides two key advantages: it captures fine-grained motion and leverages a richer learning signal by considering a large number of paths in the space-time graph. The transition matrices for the random walk between video frames are defined using the self-attention mechanism of the transformer, facilitating spatial precision and enabling efficient computation.

Key Components

1. Global Matching Transformer:
   • The model extracts high-dimensional feature embeddings from input frames and employs a six-layer stack of self-attention, cross-attention, and feed-forward networks. The architecture extends GMFlow by Xu et al., which was originally applied to optical flow prediction.
2. Contrastive Random Walk:
   • The training employs a cycle-consistency loss, where the model is trained to perform random walks through time, moving from frame $t$ to $t+1$ and back, ensuring that the starting and ending points match. This is captured by equating the product of the transition matrices to the identity matrix.
3. Label Warping for Cycle Consistency:
   • To mitigate shortcut solutions, the authors propose label warping. Different random augmentations are applied to forward and backward passes of the random walk, and the supervisory signal warps the cycle-consistency label rather than the feature map, ensuring the network learns meaningful correspondences rather than positional cues.
4. Training and Evaluation:
   • The model training is performed on the TapVid-Kubric dataset using 2-frame samples, and extensively evaluated on TapVid benchmarks, including Kubric, DAVIS, Kinetics, and RGB-Stacking. The evaluation metrics include positional accuracy, occlusion accuracy, and average Jaccard.

Numerical Performance

The presented approach demonstrates significant improvements over existing self-supervised methods, outperforming them on multiple benchmarks. Specifically, the model exhibits strong performance on TapVid-Kubric and TapVid-DAVIS, showing competitive results even against several supervised methods like TAP-Net:

• TapVid-Kubric: Achieved an AJ (Average Jaccard) of 54.2 and a positional accuracy of 72.4.
• TapVid-DAVIS: Attained an AJ of 41.8 and a positional accuracy of 60.9.

These results highlight the robustness and utility of contrastive random walks coupled with global matching transformers for long-term point tracking.

Implications and Future Work

The proposed method underscores the potential of self-supervised learning for data-efficient and scalable tracking systems in computer vision. The ability to train on vast amounts of unlabeled video data opens new avenues for research and application, such as in robotics and animation. The model's reliance on global matching transformers hints at future developments that could incorporate multi-scale features and temporal refinement for enhanced performance and robustness.

Moreover, handling occlusions remains a challenging aspect. While the proposed method addresses occlusions to some extent using cycle consistency, future work could explore explicit occlusion handling mechanisms and the capability to recover tracks through extended occlusions. Integrating multi-frame information and leveraging temporal consistency could further enhance the ability to maintain accurate tracks over long video sequences.

To conclude, this paper presents a sophisticated, self-supervised tracking paradigm that significantly advances the state-of-the-art in TAP, demonstrating the efficacy and scalability of contrastive random walks and global matching transformers in deriving meaningful correspondences from video data without dependency on labeled datasets. This work paves the way for future explorations into more generalized and robust self-supervised visual tracking frameworks.