Deep Equilibrium Optical Flow Estimation (2204.08442v1)

Abstract: Many recent state-of-the-art (SOTA) optical flow models use finite-step recurrent update operations to emulate traditional algorithms by encouraging iterative refinements toward a stable flow estimation. However, these RNNs impose large computation and memory overheads, and are not directly trained to model such stable estimation. They can converge poorly and thereby suffer from performance degradation. To combat these drawbacks, we propose deep equilibrium (DEQ) flow estimators, an approach that directly solves for the flow as the infinite-level fixed point of an implicit layer (using any black-box solver), and differentiates through this fixed point analytically (thus requiring $O(1)$ training memory). This implicit-depth approach is not predicated on any specific model, and thus can be applied to a wide range of SOTA flow estimation model designs. The use of these DEQ flow estimators allows us to compute the flow faster using, e.g., fixed-point reuse and inexact gradients, consumes $4\sim6\times$ times less training memory than the recurrent counterpart, and achieves better results with the same computation budget. In addition, we propose a novel, sparse fixed-point correction scheme to stabilize our DEQ flow estimators, which addresses a longstanding challenge for DEQ models in general. We test our approach in various realistic settings and show that it improves SOTA methods on Sintel and KITTI datasets with substantially better computational and memory efficiency.

• The paper presents a deep equilibrium model that reframes optical flow estimation as finding an infinite-level fixed point, overcoming RNN limitations.
• It leverages a constant-memory framework that cuts training memory usage by approximately 4 to 6 times while improving computational speed.
• The DEQ approach achieves superior performance on real-world datasets, reducing the F1-all score by 21% on KITTI-15 without extra computational cost.

Deep Equilibrium Optical Flow Estimation

The paper "Deep Equilibrium Optical Flow Estimation" presents a novel approach to optical flow estimation using deep equilibrium models (DEQs). This method addresses the limitations of recurrent neural network (RNN) architectures commonly used in state-of-the-art optical flow models, such as RAFT and GMA, which emulate traditional optimization algorithms through finite-step recurrent updates. The DEQ model introduced in this paper proposes an implicit framework that reframes the problem as finding an infinite-level fixed point, thus overcoming the computational and memory overheads associated with RNNs.

Optical flow estimation is a crucial task in computer vision, involving the prediction of pixel-level motion between video frames. While RNNs with unrolled updates have demonstrated success, they are hindered by high memory consumption due to backpropagation-through-time (BPTT) and poor convergence behavior. The authors propose that by using DEQ flow estimators, these issues are significantly alleviated. This is achieved by directly modeling the optical flow as a fixed point of a shallow layer and differentiating through it using a constant-memory framework.

The DEQ approach offers multiple advantages over traditional recurrent methods:

1. Efficiency: DEQ flow estimators compute with constant training memory, approximately 4 to 6 times less than their recurrent counterparts. This is due to their reliance on the fixed-point method, which allows for the decoupling of forward computation and gradient computation.
2. Speed: Using methods like fixed-point reuse from adjacent frames and inexact gradients make DEQ flows faster and more efficient. The backward pass is notably more efficient, as it does not require storing intermediate states.
3. Compatibility: It is compatible with many existing state-of-the-art model designs, such as RAFT and GMA, indicating its adaptability to various architectures without requiring structural changes.

Key in this paper is the demonstration of DEQ flows' performance on realistic datasets such as Sintel and KITTI. The DEQ-based RAFT and GMA models showcased improvements in accuracy and computation efficiency. For instance, on zero-shot generalization on the KITTI-15 dataset, a DEQ-based RAFT model reduced the F1-all score by 21.0%, showing significant performance enhancement without increased computational budget.

A noteworthy aspect of the DEQ model is its ability to employ advanced solvers such as quasi-Newton methods for fast convergence to a stable state. Additionally, the authors propose a sparse fixed-point correction scheme that stabilizes the DEQ flow estimators, addressing a crucial challenge of training DEQ models.

Looking forward, this research implies potential advancements in how optical flow estimation and other computer vision tasks could be approached using implicit neural models. The reduction in memory and computational resources while achieving better convergence and accuracy suggests a shift towards more sustainable and efficient deep learning model designs.

In conclusion, this paper adds valuable insight into optical flow estimation, making a strong case for the applicability of equilibrium models in addressing the limitations of traditional recurrent methods. The benefits of DEQ flow estimators in computational efficiency, generalization, and compatibility mark a significant stride in computer vision, with promising implications for future AI developments.