RAFT-MSF++: Multi-Scale Flow Architecture

• RAFT-MSF++ is a multi-scale optical and scene flow architecture that extends RAFT’s core principles with hierarchical feature pyramids and recurrent refinement.
• It employs a four-scale coarse-to-fine design alongside temporal fusion and attention mechanisms to achieve superior benchmark performance on datasets like KITTI and Sintel.
• Innovative geometry-motion feature fusion and occlusion regularization in RAFT-MSF++ address detailed flow estimation challenges and enhance cross-domain generalization.

RAFT-MSF++ is a designation that has been applied to two influential architectures in optical flow and scene flow research: the "High Resolution Multi-Scale RAFT" and follow-up multi-frame monocular scene flow models. The shared emphasis is on extending the RAFT family’s core principles—dense correlation, recurrent refinement, and compact feature sharing—by introducing hierarchical multi-scale processing and, in recent works, temporally recurrent and attention-guided multi-frame modules. These innovations have achieved state-of-the-art results in robust optical and scene flow estimation benchmarks, particularly when fine detail, temporal coherence, and occlusion robustness are critical.

1. Evolution and Nomenclature of RAFT-MSF++

The RAFT-MSF++ lineage begins with extensions of the RAFT (Recurrent All-Pairs Field Transforms) model, originally single-scale and frame-pair only. The initial "Multi-Scale RAFT" (Jahedi et al., 2022) incorporates a hierarchical framework with multi-scale feature extraction and cost volumes. An improved variant, referred to as MS-RAFT+ or RAFT-MSF++, was subsequently developed (Jahedi et al., 2022), significantly boosting cross-benchmark performance by employing a four-scale coarse-to-fine scheme and refined upsampling. More recently, the RAFT-MSF++ designation was also used for a self-supervised, temporally recurrent monocular scene flow model, introducing geometry-motion feature fusion, attention, and occlusion-specific losses (Sun et al., 21 Apr 2026).

2. Architectural Foundations: Multi-Scale Hierarchical Design

The core architectural innovation of RAFT-MSF++ is the move from classical RAFT’s single-scale, two-frame framework to a hierarchical multi-scale and, in later models, multi-frame description.

Multiple Scales and Feature Pyramids

• The original RAFT-MSF model uses a three-level feature pyramid (U-Net style), processing flow fields at 1/16, 1/8, and 1/4 resolution in succession (Jahedi et al., 2022).
• MS-RAFT+ (RAFT-MSF++; (Jahedi et al., 2022)) extends this to four scales, from 1/16 up to 1/2 of full resolution. This is realized via efficient, on-demand local cost volume computation at each scale, avoiding the prohibitive memory requirements of all-pairs correlation at higher resolutions.
• Feature and context encoders are shared across scales, and feature fusion is achieved via skip-connections reminiscent of U-Net.

Recurrent Update Operator

• At every scale, a ConvGRU (convolutional gated recurrent unit) accepts context, current flow, and sampled cost features to iteratively refine the flow estimate.
• Flow upsampling between scales, and eventual output to full resolution, is performed using a learned convex upsampler—ensuring no spurious flow magnitudes and permitting end-to-end optimization.

Temporal Fusion in Monocular Scene Flow

• The recurrent multi-frame RAFT-MSF++ (Sun et al., 21 Apr 2026) generalizes this pipeline to three or more frames, fusing information in both temporal directions and across frames before upsampling for final scene flow and depth prediction.

3. Geometry-Motion Feature Fusion and Attention Mechanisms

Geometry-Motion Feature (GMF)

• In multi-frame RAFT-MSF++ (Sun et al., 21 Apr 2026), each iteration produces compact representations ("GMFs") embedding both geometric (depth/disparity) and motion (scene flow) information.
• Separate forward and backward GRU hidden states are projected via convolutional heads, fused bidirectionally (including sign-alignment), and re-embedded to drive recurrent updates.

Relative Positional Attention

• To robustly propagate information, especially within occluded or ambiguous regions, position-enhanced attention is applied over context features. The attention weights depend on learnable relative positional encodings, biasing aggregation to spatially proximate areas and supplementing standard correspondence-based updates.

Occlusion Regularization

• Occlusion-specific modules segment each frame into regions (using segmentation tools such as SAM), construct "reliable-point masks" from non-occluded regions, and propagate rigid motions from these reliable regions into occluded areas via a differentiable rigid alignment and penalty.

4. Loss Formulations, Fine-Tuning, and Training Protocols

Supervision Strategies

• The optical flow versions of RAFT-MSF++ employ a multi-scale, multi-iteration loss, aggregating supervision at every scale and intermediate timestep, with robust (e.g., $q=0.7$) sample-wise penalties at fine-tuning (Jahedi et al., 2022, Jahedi et al., 2022).
• Mixed-dataset fine-tuning is employed—e.g., splitting batches across Sintel, VIPER, KITTI, HD1K, and FlyingThings—for improved generalization across diverse domains (Jahedi et al., 2022).

Self-Supervised Scene Flow Loss

• Total loss in monocular scene flow RAFT-MSF++ combines occlusion propagation penalties with per-iteration disparity and scene flow terms, using discounted temporal accumulation:

$$L_{total} = \lambda_{occ} L_{occ} + \sum_{i=1}^N \zeta^{N-i} (L_d^i + \lambda_{sf} L_{sf}^i)$$

where occlusion penalties are applied only at the final iteration and all photometric/geometric terms are masked by computed occlusion maps (Sun et al., 21 Apr 2026).

Data Augmentation and Optimization

• Standard photometric and geometric augmentations are adopted (gamma, brightness, color jitter, flipping, scaling, cropping).
• Optimization uses Adam or AdamW with cosine annealing, gradient clipping, and input sizes tailored to benchmarks (e.g., 256×832 for KITTI) (Jahedi et al., 2022, Sun et al., 21 Apr 2026).

5. Empirical Performance and Benchmarking

Optical Flow: MS-RAFT+ (RAFT-MSF++) (Jahedi et al., 2022)

• KITTI 2015: Fl-all 4.15% (vs. 4.88% MS-RAFT, +15.0% relative improvement)
• Sintel Clean: EPE all 1.232 (vs. 1.374, +10.3%)
• Middlebury: EPE all 0.142 (vs. 0.184, +22.8%)
• Ranks: 1st on VIPER, 2nd on KITTI, Sintel, and Middlebury; overall RVC 2022 winner.

Monocular Scene Flow: Multi-Frame RAFT-MSF++ (Sun et al., 21 Apr 2026)

• KITTI Scene Flow: SF-all = 24.14%, 30.99% relative improvement over two-frame RAFT-MSF.
• Occlusion Robustness: SF-occ = 25.38% outperforms previous best EMR-MSF (28.47%).
• Model: 8.19M parameters, 0.20 s/frame (RTX 3090); 10 recurrent iterations found optimal.
• Ablation studies show each of GMF, occlusion regularization, and relative positional attention deliver independent improvements, with optimal performance when combined.

Benchmark	Metric	MS-RAFT+ (RAFT-MSF++)	Previous Baseline	Relative Δ
KITTI 2015	Fl-all	4.15	4.88 (MS-RAFT)	+15.0%
Sintel Clean	EPE all	1.232	1.374	+10.3%
Middlebury (Train)	EPE all	0.142	0.184	+22.8%
KITTI Scene Flow	SF-all	24.14%	34.98% (RAFT-MSF)	+30.99%

6. Contextual Significance and Design Considerations

RAFT-MSF++ architectures exemplify how hierarchical and recurrent design patterns synergize in dense correspondence estimation. By fusing coarse-to-fine multi-scale updates with temporal reasoning and explicit occlusion treatment, these models address typical failure modes in flow and scene flow, including small-structure leakage, occlusion, and generalization gaps across diverse datasets. The use of learned convex upsampling, robust multi-scale losses, and explicit positional priors, all within a compact parameter envelope, provides a blueprint for state-of-the-art performance on challenging real-world benchmarks.

A plausible implication is that the evolution from pure single-scale/two-frame methods to fully multi-scale, multi-frame, recurrent models with explicit attention and occlusion reasoning will likely define future developments in dense geometric vision tasks.

7. Connections to Broader Research Directions

The RAFT-MSF++ lineage connects with several research avenues:

• Hierarchical and coarse-to-fine pipelines, which have long improved stability and efficiency in optical flow and related tasks.
• Self-supervised and semi-supervised learning, increasingly vital for scene flow where labeled data are scarce.
• Attention mechanisms and spatial priors, paralleling advances in transformer-based vision architectures.
• Occlusion handling through both loss design and region-based geometric reasoning, crucial for real-world robustness.

The architectural modularity and open-source availability of codebases for these models (Jahedi et al., 2022, Sun et al., 21 Apr 2026) have resulted in rapid adoption and further extension for related tasks, including video understanding, SLAM, and 3D reconstruction.

---

References

• High Resolution Multi-Scale RAFT (Robust Vision Challenge 2022) (Jahedi et al., 2022)
• RAFT-MSF++: Temporal Geometry-Motion Feature Fusion for Self-Supervised Monocular Scene Flow (Sun et al., 21 Apr 2026)
• Multi-Scale RAFT: Combining Hierarchical Concepts for Learning-based Optical Flow Estimation (Jahedi et al., 2022)