Dense Continuous-Time Optical Flow from Events and Frames (2203.13674v2)

Abstract: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. In this work, we show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized B\'ezier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use B\'ezier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the B\'ezier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To the best of our knowledge, our model is the first method that can regress dense pixel trajectories from event data. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments not only suggest that our method successfully predicts pixel trajectories in continuous time but also that it is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public.

• The paper introduces a novel event-based methodology that computes dense optical flow using continuous pixel trajectories and e curves.
• It employs a neural network with sequential correlation volumes to extract spatio-temporal features from voxel grid representations of event data.
• Experimental results show reduced endpoint error and improved accuracy when combining event data with traditional frame images.

Dense Continuous-Time Optical Flow from Event Cameras

The paper "Dense Continuous-Time Optical Flow from Event Cameras" introduces a methodology for estimating dense continuous-time optical flow using data from event cameras. The authors, Gehrig, Muglikar, and Scaramuzza, present a novel approach leveraging the unique properties of event cameras to estimate per-pixel trajectory paths in continuous time, contrasting traditional methods which only compute pixel displacements between two distinct frames.

Traditional optical flow methods face challenges due to the blind times between frames where pixel trajectories cannot be captured. Event cameras, which register brightness changes asynchronously with very fine temporal resolution, provide a solution to this limitation by offering continuous data streams informing movement dynamics in pixel space. The proposed method harnesses this continuous data, employing a neural network with multiple correlation volumes over time to predict pixel trajectories using e curves—a polynomial representation of the motion path.

The methodology combines several innovative aspects:

1. Event-based Data Processing: Events are processed into voxel grids capturing spatio-temporal changes, which form the basis for extracting context and correlation features. This segregates the method from traditional optical flow approaches by making use of the unique asynchronous nature of event data.
2. Neural Network Architecture: The network utilizes strong inductive biases tailored for event data, using multiple sequential correlation volumes to facilitate accurate and dense optical flow predictions.
3. e Curves: These are leveraged to efficiently index correlation volumes and iteratively update the pixel trajectory paths, allowing the model to predict non-linear motions that go beyond simple displacement found in frame-based methodologies.
4. Integration with Frame Data: While primarily event-based, the method can incorporate frame data to enhance performance, adapting to scenarios where frame and event data might be jointly available.

The authors' model, distinguished as the first of its kind to regress dense pixel trajectories from event data, required novel datasets for training and evaluation. MultiFlow, a synthetic dataset introduced by the authors, simulates moving objects with rich events and ground truth trajectories, serving as a critical resource for validating the proposed approach.

Results and Implications

Evaluations on both synthetic and real-world data highlight the method's efficacy. Specifically, for real-world tests on the DSEC-Flow dataset, the model demonstrated competitive performance against traditional two-view optical flow metrics, reducing the end-point error (EPE) relative to prior methodologies. Such numerical evaluations establish the proposed approach’s competency in typical optical flow tasks while providing additional capabilities for estimating continuous trajectories.

Critically, the paper outlines improvements achievable when combining event data with frame images, markedly enhancing performance metrics across experimental trials. This points toward future development paradigms in visual sensing and motion tracking, where hybrid approaches leveraging the strengths of diverse sensor inputs could become prevalent.

As event camera technology continues to evolve, the implications of this work extend both theoretically and practically. It introduces new potential applications in robotics, particularly for motion tracking and object detection in dynamic environments where frame-based systems are impeded by high latency or blur.

Conclusion

The presented paper provides foundational insights and methodology in leveraging event camera data for continuous optical flow prediction. The proposed approach not only extends the capabilities of existing optical flow estimation techniques but also opens new research pathways in both the theoretical exploration and practical deployment of event-based vision systems. As the field of event camera research expands, integrating this methodology with emerging event-based techniques could shape the next generation of vision systems in robotics and beyond.