---
title: 'MEMFOF Optical Flow: Low-Memory Multi-Frame Estimation'
url: https://www.emergentmind.com/topics/memfof-optical-flow
type: topic
---

# MEMFOF Optical Flow: Low-Memory Multi-Frame Estimation

MEMFOF (Memory-Efficient Multi-frame Optical Flow) is a class of multi-frame optical flow architectures designed to achieve state-of-the-art accuracy while maintaining extremely low GPU memory footprints, enabling both inference and training at high (e.g., 1080p) or even ultra-high (4K) resolutions without resorting to patching or downsampling. MEMFOF extends the canonical RAFT-style iterative refinement approach to bidirectional, multi-frame estimation, integrating architectural innovations such as reduced-resolution cost volumes, high-efficiency update operators, and domain-adaptive training protocols. At time of publication, MEMFOF led public optical flow benchmarks in accuracy and efficiency, providing a competitive alternative to both two-frame and memory-intensive multi-frame approaches [2506.23151].

## 1. Architectural Foundations of MEMFOF

MEMFOF generalizes the RAFT (Recurrent All-Pairs Field Transforms) model from its original two-frame design to a three-frame setting, enabling the leveraging of additional temporal context. The architecture is structured as follows [2506.23151]:

- **Inputs:** Triplet of consecutive frames $(I_{t-1}, I_t, I_{t+1})$.
- **Bidirectional Prediction:** Simultaneous recurrent refinement of forward $(f_{t\to t+1})$ and backward $(f_{t\to t-1})$ optical flows in a single pass.
- **Feature Extraction:** Feature maps $F_{t-1}, F_t, F_{t+1} \in \mathbb{R}^{H \times W \times D_f}$ extracted via shared CNN backbones.
- **Context Network:** A context encoder consumes the image triplet, producing initial hidden state $h^0$, context features $g$, and coarse flow initialization $f^0$.
- **Correlation Volumes:** For each direction, the all-pairs dot-product correlation volume
  $$
  C_{t,\,t\pm1}(u, v) = \langle\,F_t(u)\,,\,F_{t\pm1}(v)\rangle
  $$
  is computed, but at a reduced spatial resolution (see Section 2).
- **Iterative Update Block:** At each refinement iteration $k$:
    - Local cost slices are interpolated from the correlation volumes based on the current flow.
    - Cost and flow features are encoded and fused via separate modules.
    - A GRU-style “Updater” with global motion attention updates $h^{k+1}$.
    - Residuals $\Delta f^k$ are decoded and applied to generate updated flow fields.
- **Output:** Final upsampling (e.g., convex upsample) is applied to yield full-resolution flow.

By natively supporting high resolution and multi-frame temporal context, MEMFOF substantially outperforms traditional two-frame RAFT-based and fusion-based optical flow systems in both memory efficiency and accuracy.

## 2. Memory-Efficient Design: Correlation Reduction and Update Operator

A core innovation in MEMFOF is the reduction of the otherwise prohibitive cost of the correlation volume, which in RAFT is the dominant memory bottleneck, scaling with $O(HW \times HW)$. MEMFOF reduces the volume’s spatial resolution by a factor of two in each dimension:
$$
\left(\frac{H}{16} \frac{W}{16}\right)^2 \quad\text{vs.}\quad \left(\frac{H}{8} \frac{W}{8}\right)^2
$$
Thus, memory required for the correlation volumes is cut by a factor of four. For 1080p images (1920×1080), two correlation volumes (forward and backward) at $1/16$ resolution require $0.65$ GB [2506.23151].

The iterative update mechanism follows RAFT convention but is enhanced:
- Local “LookUp” of cost slices centered on the current flow estimate for both temporal directions.
- Cost features and flow features are separately encoded then fused with context.
- The recurrent updater (a GRU) integrates cost, motion, and context, with explicit bidirectional flow coupling.

This yields a full inference GPU memory usage of $2.09$ GB per 1080p sequence (three frames, 8 iterations), compared to $\sim 8$ GB for SEA-RAFT at similar settings.

## 3. High-Resolution Training Protocol

MEMFOF was trained directly on full-resolution 1080p data without spatial cropping or tiling, made feasible by its memory-efficient design. Training methodology included the following [2506.23151]:

- Datasets: A staged pretraining and fine-tuning sequence on TartanAir, FlyingThings, T+S+K+H (an amalgamation of standard optical flow datasets), and scenario–specific datasets (Sintel, KITTI, Spring).
- Training Scale: Data was 2× upsampled (e.g., from FlyingThings), and large crop sizes (e.g., $864\times1920$) were used to address large-magnitude motions common in FullHD data.
- Loss Function: Mixture-of-Laplace (MoL) loss, with iteration-dependent weighting factor $\gamma=0.85$.
- Computational Resource: Training full-resolution Spring-finetune (1920×1080, $N=8$ iters) required $28.5$ GB per GPU on A100.

Ablation studies confirm that training with both 2× upsampling and large uncropped images provides superior generalization to large motions and real-world video [2506.23151].

## 4. Quantitative Benchmarks and Comparative Results

MEMFOF demonstrates superior or competitive performance versus both two-frame and multi-frame baselines across all major high-resolution optical flow benchmarks [2506.23151]:

| Benchmark         | Method        | Frames | Mem (GB) | 1px↓ | EPE↓ | Fl↓   | WAUC↑   |
|-------------------|--------------|--------|----------|------|------|-------|---------|
| Spring (zero-shot)| MEMFOF       | 3      | 2.09     | 3.60 | 0.432| 1.353 | 94.481  |
| Spring (finetune) | MEMFOF       | 3      | 2.09     | 3.29 | 0.355| 1.238 | 95.186  |
| Sintel (clean)    | MEMFOF       | 3      | 2.09     | —    | 0.963| —     | —       |
| KITTI-15 (Fl-all) | MEMFOF       | 3      | 2.09     | —    | —    | 2.94  | —       |

MEMFOF, when fine-tuned on Spring, outperforms previous state-of-the-art methods in the 1px outlier rate and meets or exceeds performance on Sintel and KITTI-15, with orders-of-magnitude reduced memory usage. Inference runtime is $472$ ms (3 frames, 8 iterations) on RTX 3090 (no AMP).

Ablation also demonstrates the superiority of bidirectional refinement over unidirectional, higher correlation resolution, and three-frame context. The optimal trade-off between memory, accuracy, and runtime is at three frames, 1/16-resolution correlations, and high-dimensional features [2506.23151].

## 5. Relation to Prior Multi-Frame and Memory-Efficient Flow Methods

MEMFOF advances beyond earlier multi-frame fusion approaches that combine warped previous flows and current flows via a shallow U-Net (e.g., PWC-Fusion, [1810.10066]), as well as beyond highly memory-optimized single-frame schemes such as MeFlow's Local Orthogonal Cost Volume (LOV) [2312.03790]:

- Traditional fusion (PWC-Fusion) uses explicit warping (bilinear sampling) of past flows, computes brightness-error maps, and fuses candidate flows and errors using a CNN. While pluggable and lightweight, such approaches introduce incremental compute/memory per input frame and rely on the quality of two-frame estimation and warping consistency [1810.10066].
- MeFlow (LOV+RDMS) pursues memory efficiency by decomposing the 2D correlation into two local 1D axes and applying multi-scale search with local self-attention, delivering near-linear $O(HW)$ scaling and enabling 4K flows on commodity GPUs, but with only two-frame context [2312.03790].
- MEMFOF explicitly couples bidirectional, multi-frame flow estimation within a RAFT-like iterative recurrent backbone, integrating a compact, low-resolution, dual-direction cost volume and powerful context/motion fusions. This allows both improved handling of occlusion/large motion (by borrowing temporal cues from t-1 and t+1) and extremely low memory footprints.

A plausible implication is that MEMFOF's high-resolution protocol and memory model fill the practical gap for real-world optical flow on full-frame video data, where neither two-frame nor generic multi-frame fusions are computationally viable.

## 6. Implementation Details, Ablations, and Practical Considerations

Key engineering characteristics of MEMFOF [2506.23151]:

- **GPU Usage:** Training (1080p, batch size 32, 8 iters) peaks at 28.5 GB; inference at 2.09 GB.
- **Runtime Optimizations:** Feature reuse, fast and reusable correlation volume computation, last-stage-only convex upsampling.
- **Modularity:** Downscaling correlation volume to 1/16, feature dimension tuning, and use of three frames as default delivers optimal result without further scaling gains for more frames (diminishing returns observed for 5-frame models).
- **Bidirectional Refinement:** Simultaneous forward/backward refinements decrease endpoint error by 14.8% over unidirectional on Spring.
- **Loss and Regularization:** Mixture-of-Laplace with late-iteration weighting and no explicit smoothness or temporal regularization is used.

## 7. Limitations and Open Directions

Current limitations and future potential developments [2506.23151]:

- **Temporal Horizon:** Efficiency optimal at three frames; further extension to longer sequences has not yielded substantial additional gains.
- **Spatial-Temporal Scaling:** Possibilities for scaling to 4K or real-time inference via distillation or lighter-weight updaters.
- **Adaptive Resolution:** Dynamic, spatially-varying correlation scale is not yet exploited.
- **Memory Extension:** Incorporation of explicit long-range memory (as in transformer memory banks or recurrent contexts) remains an open research question.
- **Integration with Downstream Tasks:** Joint training for tasks such as video super-resolution or frame interpolation is unaddressed.
- **Trajectory Modeling:** No explicit velocity or trajectory constraints; flow is predicted per frame triplet.

In summary, MEMFOF establishes a practical and accurate framework for high-resolution, multi-frame optical flow estimation, closing the gap between traditional accuracy-optimized but memory-intensive models and prior lightweight but less effective memory-efficient or fusion-based models [2506.23151] [1810.10066] [2312.03790].

Source: https://www.emergentmind.com/topics/memfof-optical-flow