---
title: 'RAFT-MSF++: Multi-Scale Flow Architecture'
url: https://www.emergentmind.com/topics/raft-msf
type: topic
---

# RAFT-MSF++: Multi-Scale Flow Architecture

RAFT-MSF++ is a designation that has been applied to two influential architectures in optical flow and scene flow research: the "High Resolution Multi-Scale RAFT" and follow-up multi-frame monocular scene flow models. The shared emphasis is on extending the RAFT family’s core principles—dense correlation, recurrent refinement, and compact feature sharing—by introducing hierarchical multi-scale processing and, in recent works, temporally recurrent and attention-guided multi-frame modules. These innovations have achieved state-of-the-art results in robust optical and scene flow estimation benchmarks, particularly when fine detail, temporal coherence, and occlusion robustness are critical.

## 1. Evolution and Nomenclature of RAFT-MSF++

The RAFT-MSF++ lineage begins with extensions of the RAFT (Recurrent All-Pairs Field Transforms) model, originally single-scale and frame-pair only. The initial "Multi-Scale RAFT" [2207.12163] incorporates a hierarchical framework with multi-scale feature extraction and cost volumes. An improved variant, referred to as MS-RAFT+ or RAFT-MSF++, was subsequently developed [2210.16900], significantly boosting cross-benchmark performance by employing a four-scale coarse-to-fine scheme and refined upsampling. More recently, the RAFT-MSF++ designation was also used for a self-supervised, temporally recurrent monocular scene flow model, introducing geometry-motion feature fusion, attention, and occlusion-specific losses [2604.19349].

## 2. Architectural Foundations: Multi-Scale Hierarchical Design

The core architectural innovation of RAFT-MSF++ is the move from classical RAFT’s single-scale, two-frame framework to a hierarchical multi-scale and, in later models, multi-frame description.

**Multiple Scales and Feature Pyramids**
- The original RAFT-MSF model uses a three-level feature pyramid (U-Net style), processing flow fields at 1/16, 1/8, and 1/4 resolution in succession [2207.12163].
- MS-RAFT+ (RAFT-MSF++; [2210.16900]) extends this to four scales, from 1/16 up to 1/2 of full resolution. This is realized via efficient, on-demand local cost volume computation at each scale, avoiding the prohibitive memory requirements of all-pairs correlation at higher resolutions.
- Feature and context encoders are shared across scales, and feature fusion is achieved via skip-connections reminiscent of U-Net.

**Recurrent Update Operator**
- At every scale, a ConvGRU (convolutional gated recurrent unit) accepts context, current flow, and sampled cost features to iteratively refine the flow estimate.
- Flow upsampling between scales, and eventual output to full resolution, is performed using a learned convex upsampler—ensuring no spurious flow magnitudes and permitting end-to-end optimization.

**Temporal Fusion in Monocular Scene Flow**
- The recurrent multi-frame RAFT-MSF++ [2604.19349] generalizes this pipeline to three or more frames, fusing information in both temporal directions and across frames before upsampling for final scene flow and depth prediction.

## 3. Geometry-Motion Feature Fusion and Attention Mechanisms

**Geometry-Motion Feature (GMF)**
- In multi-frame RAFT-MSF++ [2604.19349], each iteration produces compact representations ("GMFs") embedding both geometric (depth/disparity) and motion (scene flow) information.
- Separate forward and backward GRU hidden states are projected via convolutional heads, fused bidirectionally (including sign-alignment), and re-embedded to drive recurrent updates.

**Relative Positional Attention**
- To robustly propagate information, especially within occluded or ambiguous regions, position-enhanced attention is applied over context features. The attention weights depend on learnable relative positional encodings, biasing aggregation to spatially proximate areas and supplementing standard correspondence-based updates.

**Occlusion Regularization**
- Occlusion-specific modules segment each frame into regions (using segmentation tools such as SAM), construct "reliable-point masks" from non-occluded regions, and propagate rigid motions from these reliable regions into occluded areas via a differentiable rigid alignment and penalty.

## 4. Loss Formulations, Fine-Tuning, and Training Protocols

**Supervision Strategies**
- The optical flow versions of RAFT-MSF++ employ a multi-scale, multi-iteration loss, aggregating supervision at every scale and intermediate timestep, with robust (e.g., $q=0.7$) sample-wise penalties at fine-tuning [2210.16900, 2207.12163].
- Mixed-dataset fine-tuning is employed—e.g., splitting batches across Sintel, VIPER, KITTI, HD1K, and FlyingThings—for improved generalization across diverse domains [2210.16900].

**Self-Supervised Scene Flow Loss**
- Total loss in monocular scene flow RAFT-MSF++ combines occlusion propagation penalties with per-iteration disparity and scene flow terms, using discounted temporal accumulation:
  $$
  L_{total} = \lambda_{occ} L_{occ} + \sum_{i=1}^N \zeta^{N-i} (L_d^i + \lambda_{sf} L_{sf}^i)
  $$
  where occlusion penalties are applied only at the final iteration and all photometric/geometric terms are masked by computed occlusion maps [2604.19349].

**Data Augmentation and Optimization**
- Standard photometric and geometric augmentations are adopted (gamma, brightness, color jitter, flipping, scaling, cropping).
- Optimization uses Adam or AdamW with cosine annealing, gradient clipping, and input sizes tailored to benchmarks (e.g., 256×832 for KITTI) [2210.16900, 2604.19349].

## 5. Empirical Performance and Benchmarking

**Optical Flow: MS-RAFT+ (RAFT-MSF++) [2210.16900]**
- **KITTI 2015**: Fl-all 4.15% (vs. 4.88% MS-RAFT, +15.0% relative improvement)
- **Sintel Clean**: EPE all 1.232 (vs. 1.374, +10.3%)
- **Middlebury**: EPE all 0.142 (vs. 0.184, +22.8%)
- Ranks: 1st on VIPER, 2nd on KITTI, Sintel, and Middlebury; **overall RVC 2022 winner**.

**Monocular Scene Flow: Multi-Frame RAFT-MSF++ [2604.19349]**
- **KITTI Scene Flow**: SF-all = 24.14%, 30.99% relative improvement over two-frame RAFT-MSF.
- **Occlusion Robustness**: SF-occ = 25.38% outperforms previous best EMR-MSF (28.47%).
- Model: 8.19M parameters, 0.20 s/frame (RTX 3090); 10 recurrent iterations found optimal.
- Ablation studies show each of GMF, occlusion regularization, and relative positional attention deliver independent improvements, with optimal performance when combined.

| Benchmark         | Metric   | MS-RAFT+ (RAFT-MSF++) | Previous Baseline | Relative Δ   |
|-------------------|----------|-----------------------|-------------------|--------------|
| KITTI 2015        | Fl-all   | 4.15                  | 4.88 (MS-RAFT)    | +15.0%       |
| Sintel Clean      | EPE all  | 1.232                 | 1.374             | +10.3%       |
| Middlebury (Train)| EPE all  | 0.142                 | 0.184             | +22.8%       |
| KITTI Scene Flow  | SF-all   | 24.14%                | 34.98% (RAFT-MSF) | +30.99%      |


## 6. Contextual Significance and Design Considerations

RAFT-MSF++ architectures exemplify how hierarchical and recurrent design patterns synergize in dense correspondence estimation. By fusing coarse-to-fine multi-scale updates with temporal reasoning and explicit occlusion treatment, these models address typical failure modes in flow and scene flow, including small-structure leakage, occlusion, and generalization gaps across diverse datasets. The use of learned convex upsampling, robust multi-scale losses, and explicit positional priors, all within a compact parameter envelope, provides a blueprint for state-of-the-art performance on challenging real-world benchmarks.

A plausible implication is that the evolution from pure single-scale/two-frame methods to fully multi-scale, multi-frame, recurrent models with explicit attention and occlusion reasoning will likely define future developments in dense geometric vision tasks.

## 7. Connections to Broader Research Directions

The RAFT-MSF++ lineage connects with several research avenues:
- Hierarchical and coarse-to-fine pipelines, which have long improved stability and efficiency in optical flow and related tasks.
- Self-supervised and semi-supervised learning, increasingly vital for scene flow where labeled data are scarce.
- Attention mechanisms and spatial priors, paralleling advances in transformer-based vision architectures.
- Occlusion handling through both loss design and region-based geometric reasoning, crucial for real-world robustness.

The architectural modularity and open-source availability of codebases for these models [2210.16900, 2604.19349] have resulted in rapid adoption and further extension for related tasks, including video understanding, SLAM, and 3D reconstruction.

---

**References**

- High Resolution Multi-Scale RAFT (Robust Vision Challenge 2022) [2210.16900]
- RAFT-MSF++: Temporal Geometry-Motion Feature Fusion for Self-Supervised Monocular Scene Flow [2604.19349]
- Multi-Scale RAFT: Combining Hierarchical Concepts for Learning-based Optical Flow Estimation [2207.12163]

Source: https://www.emergentmind.com/topics/raft-msf