SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow (2405.14793v1)

Abstract: We introduce SEA-RAFT, a more simple, efficient, and accurate RAFT for optical flow. Compared with RAFT, SEA-RAFT is trained with a new loss (mixture of Laplace). It directly regresses an initial flow for faster convergence in iterative refinements and introduces rigid-motion pre-training to improve generalization. SEA-RAFT achieves state-of-the-art accuracy on the Spring benchmark with a 3.69 endpoint-error (EPE) and a 0.36 1-pixel outlier rate (1px), representing 22.9% and 17.8% error reduction from best published results. In addition, SEA-RAFT obtains the best cross-dataset generalization on KITTI and Spring. With its high efficiency, SEA-RAFT operates at least 2.3x faster than existing methods while maintaining competitive performance. The code is publicly available at https://github.com/princeton-vl/SEA-RAFT.

• The paper introduces SEA-RAFT, an optimized optical flow estimator that uses a mixture of Laplace loss and direct flow regression to enhance accuracy and convergence.
• It employs rigid-flow pre-training and architectural simplifications, replacing GRUs with ConvNeXt blocks to improve efficiency and stability.
• Empirical results demonstrate state-of-the-art performance with reduced endpoint error and faster processing, ideal for real-time computer vision applications.

SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow

The paper "SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow" presents an innovative approach to optical flow estimation, introducing SEA-RAFT, an optimized variant of the Recurrent All-Pairs Field Transforms (RAFT) model. SEA-RAFT is designed to offer a more efficient and accurate solution to the task of estimating per-pixel 2D motion between video frames, significant for numerous applications in computer vision such as video in-painting, 3D reconstruction, and frame interpolation.

Core Contributions and Methodology

SEA-RAFT achieves notable improvements over traditional RAFT models through a series of methodological enhancements:

1. Mixture of Laplace Loss: The authors propose a modification to the standard $L_1$ loss used in optical flow training. By predicting parameters of a mixture of Laplace distributions, SEA-RAFT maximizes the log-likelihood of the ground truth flow, addressing issues of overfitting in ambiguous regions and enhancing generalization capabilities.
2. Direct Flow Regression: Unlike previous frameworks that initialize the flow estimation with zeros, SEA-RAFT incorporates direct regression to predict initial flows. This change is implemented using the existing context encoder, allowing for faster convergence with fewer iterative refinements.
3. Rigid-Flow Pre-Training: The model is pre-trained on the TartanAir dataset, which consists of realistic images captured with significant 3D structural diversity, albeit limited dynamic range due to being static scenes. This pre-training step boosts the generalization performance of the model across various datasets.
4. Architectural Simplifications: The paper also addresses architectural optimization, replacing custom RAFT components with more standard modules. The feature encoder and context encoders are both simplified, benefiting from the stability and performance of pre-trained modules such as ResNet. Moreover, the GRU module in RAFT is substituted with ConvNeXt blocks, enhancing training stability and efficiency.

Empirical Findings

The experimental results demonstrate SEA-RAFT's state-of-the-art performance across multiple benchmarks. On the Spring benchmark, SEA-RAFT reported a 3.69 endpoint-error (EPE) and 0.36 1-pixel outlier rate, representing significant reductions in error metrics compared to earlier methods. Notably, it achieved these results while operating at least 2.3 times faster than comparable methods. Additionally, SEA-RAFT exhibited superior cross-dataset generalization on challenging datasets such as KITTI and Sintel.

Implications and Future Directions

SEA-RAFT's advancements suggest promising improvements in real-time video processing applications where optical flow estimation is computationally intensive. The simplified architecture facilitates easier integration of future neural building blocks, aligning with trends towards modular, efficient vision models.

Future research could explore further optimizations in the initialization of flow predictions and loss functions to refine accuracy and efficiency. Building upon this work, additional pre-training strategies incorporating more diverse motion fields could enhance adaptability to unseen motion paradigms in test environments.

In conclusion, SEA-RAFT represents a significant step forward in the optical flow domain, combining efficiency with high accuracy and offering a robust model suitable for practical deployment in various computer vision tasks. The open-source availability of the codebase provides an excellent resource for continued research and development in this area.