Decomposition Betters Tracking Everything Everywhere (2407.06531v2)

Abstract: Recent studies on motion estimation have advocated an optimized motion representation that is globally consistent across the entire video, preferably for every pixel. This is challenging as a uniform representation may not account for the complex and diverse motion and appearance of natural videos. We address this problem and propose a new test-time optimization method, named DecoMotion, for estimating per-pixel and long-range motion. DecoMotion explicitly decomposes video content into static scenes and dynamic objects, either of which uses a quasi-3D canonical volume to represent. DecoMotion separately coordinates the transformations between local and canonical spaces, facilitating an affine transformation for the static scene that corresponds to camera motion. For the dynamic volume, DecoMotion leverages discriminative and temporally consistent features to rectify the non-rigid transformation. The two volumes are finally fused to fully represent motion and appearance. This divide-and-conquer strategy leads to more robust tracking through occlusions and deformations and meanwhile obtains decomposed appearances. We conduct evaluations on the TAP-Vid benchmark. The results demonstrate our method boosts the point-tracking accuracy by a large margin and performs on par with some state-of-the-art dedicated point-tracking solutions.

• The paper introduces a decoupled representation approach that optimizes static scenes and dynamic objects independently for enhanced motion tracking.
• It leverages separate 3D canonical volumes and specialized transformation networks to accurately model rigid and non-rigid motions.
• DecoMotion achieves a 79.0% tracking accuracy on TAP-Vid-DAVIS, significantly outperforming baseline methods like RAFT in occlusion and deformation scenarios.

Decomposition Betters Tracking Everything Everywhere

The paper "Decomposition Betters Tracking Everything Everywhere" by Rui Li and Dong Liu from the University of Science and Technology of China presents a novel test-time optimization method for motion estimation named DecoMotion, aimed at enhancing per-pixel and long-range motion tracking in video sequences. This research addresses the limitations of previous unified motion representation methods that fail to account for the complexity and diversity of natural video motion and appearance.

The central contribution of this paper is the introduction of a decoupled representation approach for video content, where video frames are explicitly decomposed into static scenes and dynamic objects. Each component is then represented using separate quasi-3D canonical volumes, which are optimized independently to account for their distinct motion and appearance characteristics. This divide-and-conquer strategy significantly improves the robustness and accuracy of tracking points through occlusions and deformations.

Methodology

Static and Dynamic Representation

DecoMotion employs two separate 3D canonical volumes for static scenes and dynamic objects:

• Static Scenes (Static Volume)
  • Represents scenes influenced mainly by the camera's rigid motion.
  • Utilizes an affine transformation model for representing static scenes.
  • Includes a network to estimate the confidence of each 3D point being static.
• Dynamic Objects (Dynamic Volume)
  • Accounts for the complex inter-frame motion and appearance changes of dynamic objects.
  • Utilizes non-linear layers to approximate non-rigid transformations.
  • Encodes discriminative and temporally consistent features for better representation.

Transformation and Volume Fusion

• Static Transformation: A simpler network employing affine transformations with some non-linear invertible layers to better handle potential misclassifications of points.
• Dynamic Transformation: A more complex transformation incorporating Real-NVP for non-rigid motion modeling, informed by discriminative features.

The static and dynamic volumes are subsequently fused through a volumetric composition method that integrates the motion and appearance characteristics of both components, providing a comprehensive global representation for motion estimation.

Optimization

DecoMotion leverages diverse loss functions for optimization:

• Motion Rendering Loss: Measures the $L_1$ distance between predicted optical flows and ground-truth flows obtained from RAFT.
• Color Rendering Loss: Enforces similarity in pixel color between frames.
• Feature Rendering Loss: Utilizes pre-trained temporal-consistent features (e.g., from DINO) to rectify dynamic transformations, providing an additional supervisory signal to handle complex deformations and occlusions.

Evaluation

The proposed DecoMotion method is evaluated on the TAP-Vid benchmark, specifically on the TAP-Vid-DAVIS subset. The evaluations demonstrate that DecoMotion substantially improves point-tracking accuracy over the baseline methods like RAFT and OmniMotion. DecoMotion achieves a tracking position accuracy of 79.0% on the benchmark, indicating a significant enhancement.

Implications and Future Work

The implications of DecoMotion are both practical and theoretical:

• Practical: DecoMotion's improved accuracy in motion estimation enables more robust and reliable tracking through video sequences, which is beneficial for applications such as video editing, object removal, and scene reconstruction.
• Theoretical: The decomposition approach highlights the importance of separating different motion dynamics within a unified framework, providing new insights and directions for future research in motion estimation and related fields.

Future developments might explore the integration of more advanced multi-frame motion estimators and the application of DecoMotion in more complex scenes involving multiple interacting objects. Additionally, the paper acknowledges that improving the dynamic transformation with better pre-trained features remains an open area for further research.

In summary, DecoMotion represents a significant advancement in the field of motion estimation by accurately decoupling and separately optimizing the motion representations of static and dynamic components within a video. This approach not only enhances tracking accuracy but also opens new avenues for addressing the inherent complexity in natural video sequences.