RoMo: Robust Motion Segmentation Improves Structure from Motion (2411.18650v1)

Abstract: There has been extensive progress in the reconstruction and generation of 4D scenes from monocular casually-captured video. While these tasks rely heavily on known camera poses, the problem of finding such poses using structure-from-motion (SfM) often depends on robustly separating static from dynamic parts of a video. The lack of a robust solution to this problem limits the performance of SfM camera-calibration pipelines. We propose a novel approach to video-based motion segmentation to identify the components of a scene that are moving w.r.t. a fixed world frame. Our simple but effective iterative method, RoMo, combines optical flow and epipolar cues with a pre-trained video segmentation model. It outperforms unsupervised baselines for motion segmentation as well as supervised baselines trained from synthetic data. More importantly, the combination of an off-the-shelf SfM pipeline with our segmentation masks establishes a new state-of-the-art on camera calibration for scenes with dynamic content, outperforming existing methods by a substantial margin.

• The paper introduces RoMo, a method that iteratively refines motion segmentation and camera pose estimation to overcome SfM challenges in dynamic scenes.
• It leverages optical flow, epipolar geometry cues, and semantic segmentation via a lightweight MLP to accurately differentiate between static and dynamic regions.
• RoMo outperforms unsupervised baselines on benchmarks like DAVIS and MPI Sintel, promising advancements for augmented reality, autonomous navigation, and action recognition.

Overview of RoMo: Robust Motion Segmentation in Structure from Motion

The paper introduces RoMo, a novel approach to improving the performance of Structure-from-Motion (SfM) systems in videos featuring dynamic objects. The research addresses a significant limitation of existing SfM methods, which typically struggle with scenes containing dynamic movements due to their reliance on a static scene rigidity assumption. The central contribution of RoMo lies in its unique motion segmentation technique that effectively distinguishes between static and dynamic components within video scenes.

The proposed technique leverages a combination of optical flow, epipolar geometry cues, and semantic features from a pre-trained video segmentation model. By iteratively refining motion segmentation masks and camera pose estimates, RoMo outperforms existing unsupervised and synthetically-supervised motion segmentation methods, setting a new standard for camera calibration tasks in dynamic scenes.

Methodology

RoMo's methodology is grounded in a two-step iterative approach:

1. Epipolar Geometry and Optical Flow Integration: The system initially uses optical flow data to establish pixel correspondences between video frames. It then employs epipolar geometry to evaluate the consistency of these pixel movements with the estimated camera poses using Sampson distance for error approximation. This step produces initial sparse masks identifying likely static and dynamic regions.
2. Semantic Segmentation Refinement: These initial masks inform the training of a lightweight MLP classifier within the feature space of a pre-trained video segmentation model. This classifier produces refined binary motion masks for each frame. These refined masks are subsequently used to improve the fundamental matrix estimations iteratively.

The process iterates, with each cycle enhancing the quality of both the motion masks and the camera pose estimates. The final refinement step utilizes the SAMv2 video segmentation model to achieve high-resolution segmentation masks.

Results

RoMo demonstrates significant improvements in motion segmentation accuracy across standard benchmarks such as DAVIS, SegTrackv2, and FBMS59. Notably, RoMo achieves superior results to unsupervised baseline methods and competes well against methods trained with synthetic data.

In the context of SfM, RoMo dramatically enhances camera pose estimation accuracy on dynamic scene datasets, including MPI Sintel, when combined with traditional SfM pipelines like COLMAP. The paper also presents a new "Casual Motion" dataset consisting of real-world scenes with ground-truth camera trajectories, on which RoMo exhibits robustness in challenging scenarios with occlusions and varied motions.

Implications and Future Directions

RoMo's ability to decouple camera-induced motion and object motions presents significant practical implications for augmented reality, autonomous navigation, and action recognition, where dynamic scenes are prevalent. Theoretically, employing semantic features combined with geometric constraints provides a scalable framework for integrating learned and engineered vision methods.

Future developments may focus on extending this approach to handle more complex real-world conditions, such as low light and extreme motion blur scenarios. Additionally, exploring ways to further integrate RoMo with emerging neural radiance fields and 3D reconstruction techniques could bring advancements in real-time applications and virtual content creation.

Overall, RoMo contributes a robust and effective mechanism for addressing one of the core challenges in visual computing—accurate dynamic scene analysis and interpretation within the SfM framework.