MEMFOF Optical Flow: Low-Memory Multi-Frame Estimation

• The paper introduces MEMFOF, a multi-frame optical flow method that extends RAFT with bidirectional, three-frame refinement to cut memory usage while delivering state-of-the-art accuracy.
• It achieves a fourfold reduction in correlation volume memory, enabling full-resolution training and inference at 2.09 GB GPU usage compared to previous methods using up to 8 GB.
• The architecture integrates a GRU-based iterative update block and domain-adaptive protocols to optimally fuse temporal cues for handling occlusions and large motions.

MEMFOF (Memory-Efficient Multi-frame Optical Flow) is a class of multi-frame optical flow architectures designed to achieve state-of-the-art accuracy while maintaining extremely low GPU memory footprints, enabling both inference and training at high (e.g., 1080p) or even ultra-high (4K) resolutions without resorting to patching or downsampling. MEMFOF extends the canonical RAFT-style iterative refinement approach to bidirectional, multi-frame estimation, integrating architectural innovations such as reduced-resolution cost volumes, high-efficiency update operators, and domain-adaptive training protocols. At time of publication, MEMFOF led public optical flow benchmarks in accuracy and efficiency, providing a competitive alternative to both two-frame and memory-intensive multi-frame approaches (Bargatin et al., 29 Jun 2025).

1. Architectural Foundations of MEMFOF

MEMFOF generalizes the RAFT (Recurrent All-Pairs Field Transforms) model from its original two-frame design to a three-frame setting, enabling the leveraging of additional temporal context. The architecture is structured as follows (Bargatin et al., 29 Jun 2025):

• Inputs: Triplet of consecutive frames $(I_{t-1}, I_t, I_{t+1})$.
• Bidirectional Prediction: Simultaneous recurrent refinement of forward $(f_{t\to t+1})$ and backward $(f_{t\to t-1})$ optical flows in a single pass.
• Feature Extraction: Feature maps $$F_{t-1}, F_t, F_{t+1} \in \mathbb{R}^{H \times W \times D_f}$$ extracted via shared CNN backbones.
• Context Network: A context encoder consumes the image triplet, producing initial hidden state $h^0$, context features $g$, and coarse flow initialization $f^0$.
• Correlation Volumes: For each direction, the all-pairs dot-product correlation volume

$$C_{t,\,t\pm1}(u, v) = \langle\,F_t(u)\,,\,F_{t\pm1}(v)\rangle$$

is computed, but at a reduced spatial resolution (see Section 2).

• Iterative Update Block: At each refinement iteration $k$:
  • Local cost slices are interpolated from the correlation volumes based on the current flow.
  • Cost and flow features are encoded and fused via separate modules.
  • A GRU-style “Updater” with global motion attention updates $h^{k+1}$.
  • Residuals $(f_{t\to t+1})$0 are decoded and applied to generate updated flow fields.
• Output: Final upsampling (e.g., convex upsample) is applied to yield full-resolution flow.

By natively supporting high resolution and multi-frame temporal context, MEMFOF substantially outperforms traditional two-frame RAFT-based and fusion-based optical flow systems in both memory efficiency and accuracy.

2. Memory-Efficient Design: Correlation Reduction and Update Operator

A core innovation in MEMFOF is the reduction of the otherwise prohibitive cost of the correlation volume, which in RAFT is the dominant memory bottleneck, scaling with $(f_{t\to t+1})$1. MEMFOF reduces the volume’s spatial resolution by a factor of two in each dimension:

$(f_{t\to t+1})$2

Thus, memory required for the correlation volumes is cut by a factor of four. For 1080p images (1920×1080), two correlation volumes (forward and backward) at $(f_{t\to t+1})$3 resolution require $(f_{t\to t+1})$4 GB (Bargatin et al., 29 Jun 2025).

The iterative update mechanism follows RAFT convention but is enhanced:

• Local “LookUp” of cost slices centered on the current flow estimate for both temporal directions.
• Cost features and flow features are separately encoded then fused with context.
• The recurrent updater (a GRU) integrates cost, motion, and context, with explicit bidirectional flow coupling.

This yields a full inference GPU memory usage of $(f_{t\to t+1})$5 GB per 1080p sequence (three frames, 8 iterations), compared to $(f_{t\to t+1})$6 GB for SEA-RAFT at similar settings.

3. High-Resolution Training Protocol

MEMFOF was trained directly on full-resolution 1080p data without spatial cropping or tiling, made feasible by its memory-efficient design. Training methodology included the following (Bargatin et al., 29 Jun 2025):

• Datasets: A staged pretraining and fine-tuning sequence on TartanAir, FlyingThings, T+S+K+H (an amalgamation of standard optical flow datasets), and scenario–specific datasets (Sintel, KITTI, Spring).
• Training Scale: Data was 2× upsampled (e.g., from FlyingThings), and large crop sizes (e.g., $(f_{t\to t+1})$7) were used to address large-magnitude motions common in FullHD data.
• Loss Function: Mixture-of-Laplace (MoL) loss, with iteration-dependent weighting factor $(f_{t\to t+1})$8.
• Computational Resource: Training full-resolution Spring-finetune (1920×1080, $(f_{t\to t+1})$9 iters) required $(f_{t\to t-1})$0 GB per GPU on A100.

Ablation studies confirm that training with both 2× upsampling and large uncropped images provides superior generalization to large motions and real-world video (Bargatin et al., 29 Jun 2025).

4. Quantitative Benchmarks and Comparative Results

MEMFOF demonstrates superior or competitive performance versus both two-frame and multi-frame baselines across all major high-resolution optical flow benchmarks (Bargatin et al., 29 Jun 2025):

Benchmark	Method	Frames	Mem (GB)	1px↓	EPE↓	Fl↓	WAUC↑
Spring (zero-shot)	MEMFOF	3	2.09	3.60	0.432	1.353	94.481
Spring (finetune)	MEMFOF	3	2.09	3.29	0.355	1.238	95.186
Sintel (clean)	MEMFOF	3	2.09	—	0.963	—	—
KITTI-15 (Fl-all)	MEMFOF	3	2.09	—	—	2.94	—

MEMFOF, when fine-tuned on Spring, outperforms previous state-of-the-art methods in the 1px outlier rate and meets or exceeds performance on Sintel and KITTI-15, with orders-of-magnitude reduced memory usage. Inference runtime is $(f_{t\to t-1})$1 ms (3 frames, 8 iterations) on RTX 3090 (no AMP).

Ablation also demonstrates the superiority of bidirectional refinement over unidirectional, higher correlation resolution, and three-frame context. The optimal trade-off between memory, accuracy, and runtime is at three frames, 1/16-resolution correlations, and high-dimensional features (Bargatin et al., 29 Jun 2025).

5. Relation to Prior Multi-Frame and Memory-Efficient Flow Methods

MEMFOF advances beyond earlier multi-frame fusion approaches that combine warped previous flows and current flows via a shallow U-Net (e.g., PWC-Fusion, (Ren et al., 2018)), as well as beyond highly memory-optimized single-frame schemes such as MeFlow's Local Orthogonal Cost Volume (LOV) (Xu et al., 2023):

• Traditional fusion (PWC-Fusion) uses explicit warping (bilinear sampling) of past flows, computes brightness-error maps, and fuses candidate flows and errors using a CNN. While pluggable and lightweight, such approaches introduce incremental compute/memory per input frame and rely on the quality of two-frame estimation and warping consistency (Ren et al., 2018).
• MeFlow (LOV+RDMS) pursues memory efficiency by decomposing the 2D correlation into two local 1D axes and applying multi-scale search with local self-attention, delivering near-linear $(f_{t\to t-1})$2 scaling and enabling 4K flows on commodity GPUs, but with only two-frame context (Xu et al., 2023).
• MEMFOF explicitly couples bidirectional, multi-frame flow estimation within a RAFT-like iterative recurrent backbone, integrating a compact, low-resolution, dual-direction cost volume and powerful context/motion fusions. This allows both improved handling of occlusion/large motion (by borrowing temporal cues from t-1 and t+1) and extremely low memory footprints.

A plausible implication is that MEMFOF's high-resolution protocol and memory model fill the practical gap for real-world optical flow on full-frame video data, where neither two-frame nor generic multi-frame fusions are computationally viable.

6. Implementation Details, Ablations, and Practical Considerations

Key engineering characteristics of MEMFOF (Bargatin et al., 29 Jun 2025):

• GPU Usage: Training (1080p, batch size 32, 8 iters) peaks at 28.5 GB; inference at 2.09 GB.
• Runtime Optimizations: Feature reuse, fast and reusable correlation volume computation, last-stage-only convex upsampling.
• Modularity: Downscaling correlation volume to 1/16, feature dimension tuning, and use of three frames as default delivers optimal result without further scaling gains for more frames (diminishing returns observed for 5-frame models).
• Bidirectional Refinement: Simultaneous forward/backward refinements decrease endpoint error by 14.8% over unidirectional on Spring.
• Loss and Regularization: Mixture-of-Laplace with late-iteration weighting and no explicit smoothness or temporal regularization is used.

7. Limitations and Open Directions

Current limitations and future potential developments (Bargatin et al., 29 Jun 2025):

• Temporal Horizon: Efficiency optimal at three frames; further extension to longer sequences has not yielded substantial additional gains.
• Spatial-Temporal Scaling: Possibilities for scaling to 4K or real-time inference via distillation or lighter-weight updaters.
• Adaptive Resolution: Dynamic, spatially-varying correlation scale is not yet exploited.
• Memory Extension: Incorporation of explicit long-range memory (as in transformer memory banks or recurrent contexts) remains an open research question.
• Integration with Downstream Tasks: Joint training for tasks such as video super-resolution or frame interpolation is unaddressed.
• Trajectory Modeling: No explicit velocity or trajectory constraints; flow is predicted per frame triplet.

In summary, MEMFOF establishes a practical and accurate framework for high-resolution, multi-frame optical flow estimation, closing the gap between traditional accuracy-optimized but memory-intensive models and prior lightweight but less effective memory-efficient or fusion-based models (Bargatin et al., 29 Jun 2025, Ren et al., 2018, Xu et al., 2023).