RAFT-MSF++: Temporal Geometry-Motion Feature Fusion for Self-Supervised Monocular Scene Flow

Abstract: Monocular scene flow estimation aims to recover dense 3D motion from image sequences, yet most existing methods are limited to two-frame inputs, restricting temporal modeling and robustness to occlusions. We propose RAFT-MSF++, a self-supervised multi-frame framework that recurrently fuses temporal features to jointly estimate depth and scene flow. Central to our approach is the Geometry-Motion Feature (GMF), which compactly encodes coupled motion and geometry cues and is iteratively updated for effective temporal reasoning. To ensure the robustness of this temporal fusion against occlusions, we incorporate relative positional attention to inject spatial priors and an occlusion regularization module to propagate reliable motion from visible regions. These components enable the GMF to effectively propagate information even in ambiguous areas. Extensive experiments show that RAFT-MSF++ achieves 24.14% SF-all on the KITTI Scene Flow benchmark, with a 30.99% improvement over the baseline and better robustness in occluded regions. The code is available at https://github.com/sunzunyi/RAFT-MSF-PlusPlus.

• The paper introduces a Geometry-Motion Feature (GMF) that uses a recurrent fusion strategy for improved temporal coherence in scene flow estimation.
• The paper incorporates a lightweight temporal fusion module and relative positional attention to robustly align features and address occlusion challenges.
• The paper demonstrates significant performance gains on the KITTI benchmark, reducing SF-all errors and maintaining efficient inference with reduced parameters.

RAFT-MSF++: Temporal Geometry-Motion Feature Fusion for Self-Supervised Monocular Scene Flow

Problem Statement and Context

Monocular scene flow estimation seeks to infer dense 3D motion and depth fields from monocular image sequences, which is pivotal for robotic perception and autonomous driving systems. Traditional solutions rely extensively on stereo, RGB-D, or LiDAR data, constrained by the need for costly sensors and annotated 3D ground truth. Although self-supervised monocular models have emerged, most operate on two-frame regimes and are fundamentally limited in temporal coherence and occlusion handling, leading to inaccuracies in dynamic, real-world scenarios.

Figure 1: Key challenges in monocular scene flow estimation: temporal fusion and occlusion robustness.

Architecture and Methodological Innovations

RAFT-MSF++ introduces a multi-frame self-supervised architecture for monocular scene flow estimation, building upon the RAFT family with critical new design elements for spatio-temporal fusion and occlusion resilience. The core contributions are:

• Geometry-Motion Feature (GMF): RAFT-MSF++ formulates a GMF—a compact representation that jointly encodes temporal depth and 3D motion cues. A Gated Recurrent Unit (GRU) iteratively refines this representation, enabling explicit temporal reasoning beyond the two-frame paradigm. The recurrent update enhances the propagation of geometric and motion features across frames for both forward and backward scene flow estimation.
• Recurrent Temporal Fusion: Rather than relying on costly bidirectional correlation volumes, temporal information is aggregated using a lightweight temporal fusion module, which concatenates and aligns bidirectional GMFs for iterative refinement.
• Relative Positional Attention: The model injects spatial priors into the fusion process via a relative positional attention mechanism. Learnable embeddings for pairwise pixel offsets capture spatial layout and guide GMF propagation, crucial for handling ambiguous and occluded regions.
• Occlusion Regularization: Leveraging object-aware segmentation (with SAM masks), the method regularizes motion in occluded regions using geometric consistency from visible neighbors. A tailored loss term ensures the propagation of reliable motion cues even in regions where photometric consistency is violated.

  Figure 2: Visualization of Attention Maps and Feature Similarity. Multi-frame models focus attention on structure-aligned regions and selectively enhance feature similarity between occluded queries and reliable neighbors, unlike two-frame models that show noisy, unstructured attention.

Empirical Results and Key Numerical Findings

RAFT-MSF++ achieves strong quantitative and qualitative performance on the KITTI Scene Flow benchmark:

• On the KITTI Testing set, RAFT-MSF++ achieves an SF-all (scene flow outlier rate) of 24.14%, corresponding to a 30.99% relative improvement over the RAFT-MSF baseline (34.98%).
• In occluded regions (SF-occ) on the KITTI Validation set, it surpasses prior unified single-network and composite multi-network methods, demonstrating 25.38% SF-occ versus the composite EMR-MSF's 28.47%, despite utilizing only 10% of the parameters.
• With supervised fine-tuning, SF-all is further reduced to 14.53%, outperforming leading fine-tuned monocular and multi-stage architectures.
• Ablation studies robustly support the model's design: introducing GMF fusion, relative positional attention, and occlusion regularization yields a 36.6% reduction in SF-all compared to the two-frame baseline.

The architecture provides a strong trade-off, with efficient inference (0.20s per stereo pair), parameter compactness (8.19M), and extensibility as a plug-and-play module for supervised pipelines (e.g., RAFT-3D baseline improvement: 5.77% → 4.75%).

Analysis and Theoretical Implications

The experimental analysis validates several foundational claims:

• Multi-frame GMF fusion is essential for temporal coherence, outperforming bidirectional correlation-based approaches and LSTM-based temporal memory, which show limited gains or even degraded geometry estimation with >3 frames due to feature misalignment.
• Relative positional attention is only effective with rich temporal context—its contribution is negative or negligible in two-frame regimes. Multi-frame information supplies the structural cues necessary to learn meaningful spatial dependencies.
• Occlusion regularization via region-based geometric consistency enhances feature propagation and robustness in non-photometric, ambiguous zones. This sets a new direction beyond the exclusion of occlusions during self-supervised training, which is standard in contemporary literature.
• The coupling of geometry and motion modeling is fundamental for monocular scene flow; methods that decouple or exclusively refine one modality (geometry or motion) are less effective in dynamic environments.

Broader Impact, Limitations, and Future Directions

RAFT-MSF++ advances self-supervised monocular scene flow towards practical deployment, with results underscoring the necessity of joint spatio-temporal modeling and robust occlusion handling for dense monocular motion understanding. Its extensibility and plug-in nature facilitate integration into more complex, multi-modal fusion or SLAM systems.

Despite these strides, two principal limitations remain:

• The requirement for stereo during training (albeit not inference), which constrains applicability in domains lacking aligned stereo data.
• Susceptibility to degradation in degenerative (low ego-motion) cases, where stereo cues for disparity vanish.

Addressing these with online self-calibration or explicit motion/geometry decoupling, possibly inspired by EMR-MSF's rigidity priors, represents a natural next step. Additionally, scaling the GMF paradigm to broader, high-resolution, or all-weather datasets can potentially further enhance generalization and robustness.

Conclusion

RAFT-MSF++ represents a significant advance in unsupervised monocular scene flow, introducing a recurrent, temporally coherent GMF mechanism and new regularization strategies for occlusion. The design leads to substantial quantitative and qualitative improvements in both standard and occluded regions over the prevailing state of the art. Its theoretical and practical innovations chart a promising direction for future monocular motion and depth perception systems, emphasizing temporally-integrated, geometry–motion coupled representations.