Trajectory-aligned Space-time Tokens for Few-shot Action Recognition (2407.18249v1)

Abstract: We propose a simple yet effective approach for few-shot action recognition, emphasizing the disentanglement of motion and appearance representations. By harnessing recent progress in tracking, specifically point trajectories and self-supervised representation learning, we build trajectory-aligned tokens (TATs) that capture motion and appearance information. This approach significantly reduces the data requirements while retaining essential information. To process these representations, we use a Masked Space-time Transformer that effectively learns to aggregate information to facilitate few-shot action recognition. We demonstrate state-of-the-art results on few-shot action recognition across multiple datasets. Our project page is available at https://www.cs.umd.edu/~pulkit/tats

• The paper introduces trajectory-aligned tokens that disentangle motion and appearance cues for efficient few-shot action recognition.
• It employs point tracking and DINOv2-based semantic alignment with a Masked Space-time Transformer to aggregate crucial features.
• The approach achieves state-of-the-art performance on benchmarks like Kinetics and Something-Something, excelling in one-shot scenarios.

Trajectory-aligned Space-time Tokens for Few-shot Action Recognition

The paper "Trajectory-aligned Space-time Tokens for Few-shot Action Recognition" explores an innovative methodology for few-shot action recognition by emphasizing the disentanglement of motion and appearance representations. The authors introduce a novel approach that capitalizes on recent advancements in tracking methodologies and self-supervised representation learning to build trajectory-aligned tokens (TATs). These tokens efficiently capture essential motion and appearance information, thereby enhancing the few-shot learning process by reducing data requirements without sacrificing crucial information.

Methodology

The proposed framework significantly diverges from traditional large-scale action recognition paradigms that rely on massive datasets and implicitly learned representations through extensive deep networks. Instead, this approach is tailored for the few-shot regime, where delineating subtle motion and appearance cues becomes crucial due to limited training samples. The paper focuses on leveraging point trajectories, an outcome of recent tracking advancements, and aligns these trajectories with semantic descriptors derived from self-supervised learning models such as DINOv2.

Key Components:

1. Motion Representation via Point Tracking: Motion within the video frames is modeled using a point-tracking strategy. The authors use CoTracker, which jointly estimates the trajectories of multiple points, ensuring that motion details are captured across frames, even through occlusions.
2. Semantic Alignment using DINO Tokens: Semantic features are derived from DINOv2, which offers robust image features through its self-supervised learning framework. By sampling points across the frames and aligning them with DINO tokens, the approach effectively creates trajectory-aligned tokens.
3. Masked Space-time Transformer: A Masked Space-time Transformer is designed to process these trajectory-aligned tokens. It aggregates information temporally along the trajectories and spatially across different points, enhancing the model's understanding of motion and appearance cues.
4. Set Matching Metric: The approach utilizes the bidirectional Mean Hausdorff Metric (Bi-MHM) for effective video-to-video comparison in few-shot learning scenarios.

Results and Contributions

The paper showcases state-of-the-art results across several benchmark datasets, including Kinetics and Something-Something, demonstrating enhanced performance in 5-way k-shot settings. The approach consistently outperforms contemporary methods in one-shot scenarios, confirming its effectiveness in capturing relevant action-specific cues with limited data.

Contributions include:

• A novel approach that disentangles and aligns motion and appearance cues for effective few-shot action recognition.
• Utilization of point tracking and self-supervised learning to generate trajectory-aligned tokens.
• Introduction of a Masked Space-time Transformer that efficiently aggregates these tokens.
• Showcasing superior performance compared to existing methods across various datasets.

Implications and Future Directions

The introduction of trajectory-aligned tokens in few-shot learning paradigms could significantly impact action recognition tasks, providing a compact yet information-rich representation that is particularly suited for scenarios with limited data. This method opens new possibilities for deploying action recognition systems in resource-constrained environments or applications where obtaining extensive labeled datasets is impractical.

Looking forward, the paper hints at potential advancements in AI through improved point-tracking mechanisms and better integration with other self-supervised models, potentially expanding the scope of few-shot learning applications. Moreover, refining the alignment process between motion trajectories and semantic features could further enhance recognition accuracy, making it a fertile ground for future research.