---
title: Multi-Scale RAFT for Dense Visual Correspondence
url: https://www.emergentmind.com/topics/multi-scale-raft-ms-raft
type: topic
---

# Multi-Scale RAFT for Dense Visual Correspondence

Multi-Scale RAFT (MS-RAFT) generalizes the Recurrent All-Pairs Field Transforms (RAFT) paradigm for dense visual correspondence by integrating classical hierarchical processing concepts, explicitly incorporating multi-scale feature extraction, hierarchical cost volumes, and coarse-to-fine optimization into RAFT’s recurrent, all-pairs matching backbone. MS-RAFT and its successors—including MS-RAFT+ and MS-RAFT-3D—demonstrate that classical hierarchical refinements remain effective when carefully synthesized with modern network-based correlation and update mechanisms for both optical flow and image-based scene flow [2207.12163][2210.16900][2506.01443].

## 1. Hierarchical Multi-Scale Architecture

MS-RAFT employs a hierarchical, coarse-to-fine design where the estimation process unfolds across a pyramid of spatial scales. For optical flow, MS-RAFT typically employs three scales at 1/16, 1/8, and 1/4 input resolution, and MS-RAFT+ extends to a fourth, 1/2 resolution scale. At each scale $s$, the recurrent network predicts a flow or motion field $\mathbf{u}^s$, initialized to zero at the coarsest scale and upsampled from the previous scale at finer levels. Every scale recursively refines the estimate using a fixed number of ConvGRU-based update iterations, followed by learning-based, convex upsampling to initialize the next-finer-level estimate [2207.12163][2210.16900][2506.01443]. This process allows large displacements to be robustly captured at coarse scales and fine structures recovered at high resolution.

In scene flow (MS-RAFT-3D), the backbone is a recurrent SE(3) field estimator (as in RAFT-3D), hierarchically nested across scales. The recurrent update block (for SE(3) and rigid motion embeddings) is identical at each scale, with upsampling of both SE(3) fields and embeddings to propagate state across the hierarchy [2506.01443].

## 2. Multi-Scale Feature and Context Encoders

The feature encoders in MS-RAFT use a Siamese U-Net style backbone, generating multi-resolution features for each input frame. Successive conv–leaky-ReLU–downsampling blocks process each frame, yielding feature maps at each scale (1/16, 1/8, 1/4, [optionally 1/2]). Feature enhancement at each level fuses upsampled coarser features with scale-specific detail using residual units [2207.12163]. The context encoder (for conditioning the update block) employs either a top-down hierarchy with learned channel widths per scale or a simplified sequence of residual blocks, leading to significant model-size reductions over earlier deep backbones (e.g., ResNet50 in RAFT-3D) with improved accuracy [2506.01443]. 

Channel width choices and U-Net-based designs empirically demonstrate lower error compared to flat or untied context-encoders [2506.01443][2207.12163]. 

## 3. Hierarchical Cost Volume and Recurrent Update Block

At each scale, MS-RAFT constructs a local cost volume via inner-product (correlation) between feature vectors for small displacement windows. Unlike RAFT, which works only at a single fixed feature scale (typically 1/8), MS-RAFT generalizes the local cost window construction to all levels of the spatial pyramid [2207.12163]. In MS-RAFT-3D, the 4D cost volume is built via all-pairs feature correlation at each scale, reflecting both optical flow and stereo cues for scene flow [2506.01443].

The update block at each scale is a ConvGRU-based module with shared parameters across pyramid levels. The update module ingests current flow/motion estimate, context features, correlation features (looked up from the local cost volume), and—in scene flow—additional embeddings (e.g., rigid-motion, confidence, edge weight estimates). Corrections to the current estimate, embedding updates, and smoothing/regularization coefficients are derived from the updated hidden state in each recurrent step. 

Convex upsampling modules, trained end-to-end, output non-negative spatial weights enabling sub-pixel-accurate, learned interpolation to finer scales, outperforming bilinear upsampling in accuracy [2207.12163][2210.16900].

## 4. Training Loss and Optimization

MS-RAFT uses a fully multi-scale, multi-iteration loss that supervises each predicted flow (or scene flow) field at every iteration of every scale, generalizing the “deep supervision” found in RAFT. The typical loss at scale $s$ and iteration $i$ is, for flow,
\[
\mathcal{L}_{s,i} = \frac{1}{N_{\text{px}}}\sum_{p}\| \mathbf{f}^{s,i}_p - \mathbf{f}^{\mathrm{gt}}_p\|_\epsilon,
\]
with $\epsilon=1\times 10^{-5}$. Losses are exponentially weighted to emphasize later scales/iterations. Fine-tuning incorporates a robust, sample-wise loss parametrized by $q=0.7$ (see [2207.12163][2210.16900]). 

Scene flow extends the target to (2D flow, depth-residual), adds weighted terms for confidence and reverse flows, and supervises via a generalized multi-scale formulation [2506.01443]. 

Optimization leverages pretraining on synthetic datasets (e.g., FlyingChairs, FlyingThings3D), followed by mixed or dataset-specific fine-tuning, with no additional data augmentation beyond original RAFT.

## 5. Key Empirical Results and Ablations

Extensive quantitative evaluation of MS-RAFT verifies substantial accuracy gains over single-scale RAFT and comparable baselines:

| Benchmark             | RAFT   | MS-RAFT | MS-RAFT+ |
|-----------------------|--------|---------|----------|
| VIPER (Test) EPE      | 0.58   | 0.49    | **0.41** |
| KITTI Fl-all          | 5.10   | 4.88    | **4.15** |
| Sintel Clean EPE      | 1.43   | 1.37    | **1.23** |
| Sintel Final (small)  | 0.55   | 0.47    | **0.42** |

MS-RAFT+ achieves an overall first place among all methods in the Robust Vision Challenge 2022 [2210.16900]. Notably, the largest relative improvements occur on non-occluded regions and “small motion” subsets [2207.12163][2210.16900]. Scene flow variants (MS-RAFT-3D, MS-RAFT-3D+) yield relative accuracy gains of 8.7% on KITTI and 65.8% on Spring over previous state of the art, while reducing model size by ∼17 million parameters [2506.01443].

Ablation studies confirm the necessity of multi-scale processing, learned upsampling, and U-Net feature encoders. Disabling bi-Laplacian embedding smoothing in scene flow nearly doubles error, underscoring the importance of spatial coherence priors [2506.01443]. 

## 6. Implementation and Computational Trade-offs

A multi-scale configuration (typically S=3, with scaling factors 1/16, 1/8, 1/4 and up to 1/2 for MS-RAFT+) with partially-shared update blocks is optimal, balancing parameter efficiency and prediction accuracy [2207.12163][2210.16900]. MS-RAFT increases parameter count (13.5 M vs 5.3 M for RAFT) and inference latency (0.3 s/pair vs 0.09 s) but remains real-time on modern hardware. MS-RAFT+ introduces on-demand cost computation with custom CUDA kernels to feasibly support a 1/2 resolution scale without prohibitive memory cost [2210.16900]. 

The convex upsampler, shared across scales, is lightweight yet critical for accurate high-resolution estimation [2207.12163].

## 7. Extensions and Broader Impact

The MS-RAFT design paradigm has been successfully generalized to 3D scene flow (MS-RAFT-3D), where dense SE(3) fields are inferred over multi-scale hierarchies. The approach is shown to outperform previous scene flow methods by a large margin, demonstrating the continued utility and extensibility of hierarchical, recurrent, all-pairs estimation [2506.01443]. 

MS-RAFT further demonstrates that blending classical multi-scale/hierarchical strategies with strong modern backbones (RAFT) yields state-of-the-art accuracy and improves generalization across datasets without the need for elaborate fine-tuning schedules [2207.12163][2210.16900]. A plausible implication is that similar hybrid designs may be beneficial across other dense correspondence and motion estimation tasks.

Source: https://www.emergentmind.com/topics/multi-scale-raft-ms-raft