EventGAN: Synthesizing Events from Frames

• EventGAN is a generative adversarial network framework that converts paired grayscale frames into simulated event volumes, bridging the gap between image datasets and event cameras.
• The architecture employs a U-Net-style generator and a PatchGAN discriminator with adversarial, flow, and reconstruction losses to produce temporally precise, sparse events.
• Downstream applications in object detection and human pose estimation demonstrate EventGAN's effective sim-to-real generalization with significant performance improvements.

EventGAN is a generative adversarial network (GAN) framework designed to synthesize event streams from conventional image frames, enabling the use of large-scale labeled image datasets for event camera research. Event cameras, which provide high temporal resolution, low latency, and high dynamic range, have been constrained by the scarcity of labeled event-based data. EventGAN addresses this by learning to map pairs of grayscale frames to event volumes, trained on paired image/event data, and producing high-fidelity simulated events suitable for downstream computer vision applications including object detection and pose estimation (Zhu et al., 2019).

1. Generator and Discriminator Architectures

The EventGAN generator employs a U-Net-style encoder-decoder identical in backbone to EV-FlowNet. The generator receives a stacked pair of grayscale input frames $I_t, I_{t+1} \in \mathbb{R}^{1 \times H \times W}$, forming a $2 \times H \times W$ input tensor. The output is an event volume $\hat{E} \in \mathbb{R}^{C \times H \times W}$, with $C = 2B$ channels ($B = 9$ temporal bins per polarity).

• Encoder: Four convolutional layers with $4 \times 4$ kernels, stride 2, increasing from 64 to 512 channels, each with LeakyReLU activation and spectral normalization.
• Decoder: Four deconvolutional layers with $4 \times 4$ kernels, stride 2, reversing filter sizes from 512 back down to $C$, each with batch normalization and ReLU. Decoder layers employ skip connections from corresponding encoder layers. The output is clipped and normalized to $[0, 1]$ per channel.

The discriminator is a four-layer PatchGAN operating on pairs of input frames and event volumes. Each layer uses $4 \times 4$ convolutions with strides of 2 except the final, and channel counts from 64 to 256, ending in a $2 \times H \times W$0-channel output score map, with LeakyReLU activations but no normalization. 10% of real/fake labels are randomly flipped to stabilize adversarial training.

2. Loss Formulations

EventGAN’s learning objective consists of adversarial and cycle consistency losses:

• Adversarial Loss: Uses a hinge-loss formulation, alternating two discriminator updates per generator update.

$2 \times H \times W$1

$2 \times H \times W$2

• Cycle Consistency Losses: Two separate frozen critics—pre-trained on real events—evaluate optical flow consistency and image-reconstruction consistency:
  • Optical Flow Consistency:

  $2 \times H \times W$3 - Image Reconstruction Consistency:

  $2 \times H \times W$4

• Full Objective:

$2 \times H \times W$5

$2 \times H \times W$6 and $2 \times H \times W$7 are empirically set for loss magnitude balancing. Discriminator update minimizes $2 \times H \times W$8 alone.

3. Training Protocol and Dataset Construction

Training utilizes ~30 minutes of paired real event/frame data, comprising the MVSEC indoor_flying and outdoor_day sequences and a custom DAVIS-346b dataset. Sampling is 80% DAVIS / 20% MVSEC with frame offsets selected uniformly from 1 to 6 frames. No additional data augmentation is performed beyond random selection of frame gaps.

Optimization uses RAdam for 100 epochs, with GAN scheduling set at two discriminator steps per generator step. Spectral normalization is applied in the generator’s encoder, batch normalization elsewhere in the generator, and no normalization in the discriminator. 10% discriminator label flips further stabilize training. Event noise is implicitly modeled by training on real, noisy events.

4. Downstream Computer Vision Applications

EventGAN’s simulated event volumes are employed to train deep networks for two primary tasks using only image-based datasets.

4.1 Object Detection

Detection of cars is performed with a YOLOv3 detector augmented with spatial pyramid pooling. The model is initialized from ImageNet-pretrained weights, except for the first convolutional layer re-initialized to ingest $2 \times H \times W$9 input channels. Training uses simulated event volumes from KITTI, with evaluation on MVSEC outdoor_day, featuring 250 manually labeled frames with easy/hard/don’t-care splits.

Performance metrics (PASCAL VOC style) for EventGAN-simulated data are:

• Precision: 0.42
• Recall (easy/hard): 0.57 / 0.34
• Combined Recall: 0.45
• AP: 0.30
• $\hat{E} \in \mathbb{R}^{C \times H \times W}$0: 0.44 For comparison, ESIM-trained detectors yield AP: 0.02, $\hat{E} \in \mathbb{R}^{C \times H \times W}$1: 0.09; frame-trained detectors yield AP: 0.29, $\hat{E} \in \mathbb{R}^{C \times H \times W}$2: 0.45.

4.2 2D Human Pose Estimation

Pose estimation uses the “Simple Baselines” network (Xiao et al.) with a ResNet-50 encoder, initialized from ImageNet, and a decoder predicting joint heatmaps. Simulated event data from MPII+Human3.6M is used for pretraining, with evaluation and fine-tuning on DHP19.

Results:

Model	Pretrain-only	+1 epoch FT	+30 epoch FT	Real-full (140 ep)
EventGAN	14.55px/45.5%	6.76px/87.7%	6.44px/90.2%	6.39px/89.9%
ESIM	19.57px/40.5%	7.58px/85.9%	6.54px/89.9%	–
Real	–	8.94px/80.6%	6.75px/87.5%	6.39px/89.9%

$\hat{E} \in \mathbb{R}^{C \times H \times W}$3MPJPE (pixels, $\hat{E} \in \mathbb{R}^{C \times H \times W}$4), PCKh@50 (\%, $\hat{E} \in \mathbb{R}^{C \times H \times W}$5)

A notable observation is that EventGAN-based pretraining followed by one epoch of fine-tuning yields models that outperform those trained fully on real event data.

5. Ablation Analyses and Observed Limitations

Ablation studies reveal the impact of individual loss components:

• $\hat{E} \in \mathbb{R}^{C \times H \times W}$6 reconstruction only leads to dense, blurry, non-sparse volumes.
• Adversarial loss alone causes instability with unclear motion representation.
• Adversarial plus reconstruction losses yield stable but motionless outputs.
• Adversarial plus flow losses extract motion direction but omit events.
• Joint adversarial, flow, and reconstruction losses best reproduce event sparsity and motion direction.

Generalization is strong, with EventGAN maintaining a small sim-to-real domain gap, although some underestimation of motion magnitude and reductions in detection recall persist. The model requires video (frame pairs) and cannot leverage single-image datasets. Lighting conditions and low-texture scenes excluded from training may degrade generated event quality.

Loss Terms	Output Character	Failure/Artifact
$\hat{E} \in \mathbb{R}^{C \times H \times W}$7 only	Blurry, dense	No sparsity
Adversarial only	Unstable	No clear motion
Adv. + Recon	Stable	No trails
Adv. + Flow	Motion direction	Missing events
Adv. + Flow + Recon	Best match	–

6. Architectural and Methodological Innovations

EventGAN advances event simulation by combining adversarial learning with cycle consistency via frozen critics: one for flow estimation and one for frame reconstruction. This distinguishes it from prior work that relied on explicit motion modeling (e.g., parametric event simulators) or less sophisticated GAN pipelines. EventGAN implicitly models sensor noise and temporal event sparsity, achieving constant GPU-time simulation and enabling transfer from image to event data modalities. The framework demonstrates strong capacity for sim-to-real generalization across diverse high-level vision tasks (Zhu et al., 2019).