---
title: 'EV-FlowNet: Event-based Optical Flow'
url: https://www.emergentmind.com/topics/ev-flownet
type: topic
---

# EV-FlowNet: Event-based Optical Flow

EV-FlowNet is a self-supervised deep learning pipeline for dense optical flow estimation from event-based cameras, specifically designed to address the unique data modalities and constraints associated with asynchronous event streams. Its formulation includes a compact event-tensor encoding, a U-Net-inspired network architecture, and a self-supervised training regimen based on grayscale image supervision. EV-FlowNet served as the foundational model for subsequent sequential and spiking neural extensions that enabled temporally dense (e.g., 100 Hz) flow estimation and highly efficient low-power inference.

## 1. Event Data Representation

Event cameras emit asynchronous events $e = \{x, t, p\}$, where $x = (u,v)$ is the pixel location, $t$ is the timestamp, and $p \in \{+1, -1\}$ indicates polarity (increasing or decreasing log-intensity). An event is triggered by a threshold change in log-intensity:
\[
\log I(t+\Delta t) - \log I(t) \geq \theta
\]

EV-FlowNet aggregates the raw event stream over a window $[t_0, t_1]$ into a fixed-size, image-like tensor with four channels per pixel:
- $C^+(x)$: total count of positive events at $x$
- $C^-(x)$: total count of negative events at $x$
- $T^+(x)$: normalized timestamp of the last positive event at $x$
- $T^-(x)$: normalized timestamp of the last negative event at $x$

Formally,
\[
C^p(x) = \sum_{e \in \text{Events}} 1_{e.x = x \wedge e.p = p}
\quad
T^p(x) = \max_{e: e.x = x \wedge e.p = p} \frac{e.t - t_0}{t_1 - t_0}, \quad p \in \{+1, -1\}
\]

This encoding preserves spatial locality, event frequency, and recency, allowing downstream convolutional networks to process events as pseudo-images. Both $T^+$ and $T^-$ are clipped to $[0,1]$ for range compatibility.

## 2. Network Architecture

EV-FlowNet utilizes a multi-scale encoder–decoder structure resembling U-Net to predict per-pixel 2D flow vectors $(u(x), v(x))$. The architecture includes:

- **Encoder:** Four strided 2D convolutions ($4 \times 4$, stride 2, padding 1), doubling channels at each stage: $4 \rightarrow 64 \rightarrow 128 \rightarrow 256 \rightarrow 512$.
- **Bottleneck:** Two residual blocks, both with 512 channels:
  \[
  x \mapsto \operatorname{ReLU}(\operatorname{Conv}_{3 \times 3}(x)) \mapsto \operatorname{ReLU}(\operatorname{Conv}_{3 \times 3}(\cdot)) + x
  \]
- **Decoder:** Four upconvolutional stages. Each upsamples by $2 \times$, applies a $3 \times 3$ conv, and halves channels. Skip connections from encoder stages are concatenated (“$\oplus$”).
- **Multi-scale Flow Heads:** At each decoder level, a $2$-channel “flow head” predicts flow $(u_i, v_i)$ for that scale, passed through $\tanh$. Multi-scale flows are upsampled for loss computation and concatenated into subsequent decoder operations for refinement.

The forward pass can be summarized as:
\[
\begin{align*}
&E_0 = \mathrm{InputEventImage} \in \mathbb{R}^{H \times W \times 4} \\
&E_1 = \mathrm{Conv}_{4\times4, s2}(E_0) \\
&E_2 = \mathrm{Conv}_{4\times4, s2}(E_1) \\
&E_3 = \mathrm{Conv}_{4\times4, s2}(E_2) \\
&E_4 = \mathrm{Conv}_{4\times4, s2}(E_3) \\
&B = \mathrm{ResBlock}_{512}(\mathrm{ResBlock}_{512}(E_4)) \\
&D_1 = \mathrm{UpConv}_{3\times3}(B) \oplus E_4,~ F_1 = \mathrm{FlowHead}(D_1) \\
&...
\end{align*}
\]

## 3. Self-Supervised Loss Functions

EV-FlowNet is trained without access to ground-truth flow, instead using a self-supervised approach leveraging frame-based grayscale images synchronized with the events.

- **Photometric Loss:** The network-predicted flow is used to warp frame $I_{t+1}$ to $I_t$, with a per-pixel brightness constancy loss:
  \[
  \ell_{\text{photo}} = \sum_{x,y} \rho \left( I_t(x,y) - I_{t+1}(x + u(x,y), y + v(x,y)) \right)
  \]
  The penalty $\rho$ uses the generalized Charbonnier function:
  \[
  \rho(s) = (s^2 + \epsilon^2)^\alpha, \quad \alpha = 0.45,~ \epsilon = 10^{-3}
  \]

- **Smoothness Loss:** A first-order neighborhood term penalizes local flow variation:
  \[
  \ell_{\text{smooth}} = \sum_{x,y} \sum_{(i,j) \in \mathcal{N}(x,y)} \rho \left( u(x,y) - u(i,j) \right) + \rho \left( v(x,y) - v(i,j) \right)
  \]
  where $\mathcal{N}(x,y)$ denotes 8-connected neighboring pixels.

- **Total Multi-scale Loss:** Each intermediate and final flow estimate $F_i$ is supervised by downsampled frames, with the total loss
  \[
  L_{\mathrm{total}} = \sum_{i=1}^4 \left[ \ell_{\mathrm{photo}}(F_i) + \lambda\,\ell_{\mathrm{smooth}}(F_i) \right],\quad \lambda = 0.5
  \]

## 4. Training and Evaluation Protocol

**Training Pipeline:**
- **Data:** MVSEC “outdoor_day1” and “outdoor_day2” (DAVIS sensor: events, frames, pose/depth ground truth).
- **Event window:** For $\Delta t = k$ frames, all events in $[t, t_k]$ are aggregated for input.
- **Augmentation:** Random horizontal flips, crop to $256 \times 256$.
- **Optimization:** Adam, initial $1 \times 10^{-5}$, exponential decay (0.8 per 4 epochs), Charbonnier with $\alpha=0.45$, $\epsilon=1e-3$, smoothness $\lambda=0.5$, trained for 300k iterations ($\approx$12 hours on V100 16 GB).

**Evaluation Metrics:**
- **Average Endpoint Error (AEE):**
  \[
  \mathrm{AEE} = \frac{1}{|\Omega|} \sum_{x \in \Omega} \left\| (u_{\text{pred}}, v_{\text{pred}}) - (u_{\text{gt}}, v_{\text{gt}}) \right\|_2
  \]
  ($\Omega$ is the set of active, valid pixels)
- **Outlier Rate:** $\%$ of pixels with EE $>$ 3 px and $>$5% of magnitude.

**Ground-truth Flow:** Generated from pose/depth via:
\[
\begin{pmatrix}
\dot{x} \\ \dot{y}
\end{pmatrix}
=
\begin{bmatrix}
-\frac{1}{Z} & 0 & -\frac{x}{Z} & xy & -(1+x^2) & y \\
0 & -\frac{1}{Z} & \frac{y}{Z} & 1+y^2 & -xy & -x
\end{bmatrix}
\begin{pmatrix}
v \\ \omega
\end{pmatrix}
\]

**Results (MVSEC outdoor_day1, dt=1 and dt=4):**
| Model            | AEE (dt=1) | Outliers (dt=1) | AEE (dt=4) | Outliers (dt=4) |
|------------------|------------|----------------|------------|----------------|
| UnFlow           | 0.97 px    | 1.6%           | 2.95 px    | 40.0%          |
| EV-FlowNet$_{2R}$| 0.49 px    | 0.2%           | 1.23 px    | 7.3%           |

EV-FlowNet achieves lower AEE and outlier rates than image-based UnFlow, especially at larger $\Delta t$.

## 5. Extensions: Temporally Dense and Sequential Flow Estimation

Recent developments have extended the EV-FlowNet paradigm to recurrent neural architectures for temporally dense (e.g., 100 Hz) flow estimation [2210.01244].

- **Sequential Event Slicing:** Rather than aggregating all events over a fixed frame interval, slice the stream into fine-grained (e.g., 10 ms) windows, yielding input rates up to 100 Hz.
- **Recurrent Architectures:**
  - **LSTM-FlowNet:** A U-Net structure where all convolutions are replaced by ConvLSTM layers. Flow is predicted from the decoder's finest-scale hidden state at each timestep.
  - **EfficientSpike-FlowNet:** All nonlinearities are replaced by leaky integrate-and-fire (LIF) neurons. Membrane potentials update as:
    \[
    v_t = v_{t-1} - y_{t-1} + W \ast x_t + b
    \quad
    y_t = \Theta(v_t - V_{th})
    \]
    where $\Theta$ is the Heaviside step function.

- **Loss Function:** Pure supervised $\ell_2$ loss on flow against (possibly linearly interpolated) ground truth:
  \[
  L = \sum_{\text{frames } m} \sum_{\text{pixels } i \in \text{valid}} \left\| (u_{m,i}, v_{m,i})_{\text{pred}} - (u_{m,i}, v_{m,i})_{\text{gt}} \right\|_2^2
  \]
- **Key Training Technique:** Warm-up frames + truncated BPTT to stabilize long recurrent inference.

**Comparison (DSEC dataset):**
| Model                  | Rate   | AEE  |
|------------------------|--------|------|
| EV-FlowNet (baseline)  | 10 Hz  | 0.67 |
| LSTM-FlowNet (proposed)| 100 Hz | 0.60 |
| EfficientSpike-FlowNet | 100 Hz | 2.66 |

LSTM-FlowNet demonstrates $\approx$ 13% lower AEE than baseline EV-FlowNet with $10\times$ higher temporal resolution.

**Efficiency:**
- EV-FlowNet: $16.6$M parameters; normalized compute cost $1\times$
- LSTM-FlowNet: $53.6$M parameters, uses $\sim$40$\times$ energy (or $400\times$ at $100$ Hz)
- EfficientSpike-FlowNet: $16.6$M parameters, only $1.5\%$ of LSTM energy at $100$ Hz, leveraging sparse event-driven computation.

## 6. Limitations and Prospective Directions

Identified limitations and potential enhancements include:
- **Timestamp Saturation:** Rapid event bursts can override timestamp channels, reducing spatial discriminability.
- **Event Sparsity:** If event window $\Delta t$ is too small, insufficient input signal may result.
- **Supervisory Dependency:** The reliance on frame-based photometric loss inherits brightness-constancy assumptions and occlusion sensitivity.

Suggested extensions focus on:
- Incorporating event-only loss functions, such as motion-compensated event alignment, to obviate the need for grayscale supervision.
- Broadening self-supervised paradigms to other event-based domains (e.g., depth, egomotion, semantic segmentation) using event-specific consistency or reconstruction constraints.
- Developing richer, possibly learned, event tensor embeddings and spatio-temporal surfaces to encode more nuanced event histories.

## 7. Impact and Research Context

EV-FlowNet demonstrated that effective optical flow estimation from events can be achieved via a compact 4-channel summary and a multi-scale U-Net architecture trained with self-supervised photometric and smoothness losses [1802.06898]. Its approach outperformed frame-based optical flow baselines on event camera data and inspired sequential and neuromorphic network extensions (e.g., ConvLSTM, SNNs) supporting temporally dense and energy-efficient inference [2210.01244]. The continued development of event-driven learning paradigms, recurrent representations, and event-only supervisory signals is enabling responsive and power-efficient vision pipelines for high-speed and high-dynamic-range sensing.

Source: https://www.emergentmind.com/topics/ev-flownet