ADΔER: Lightweight Asynchronous Event Codec

• ADΔER is a lightweight, asynchronous pixel-stream representation that encodes video events using a five-tuple to precisely control intensity reconstruction and rate–distortion tradeoffs.
• It supports source-modeled lossy compression with adaptive decimation and per-pixel temporal binning, significantly reducing bandwidth while maintaining accurate event recovery.
• The system ensures backward compatibility with frame-based applications and enables dynamic, region-based adaptation, making it ideal for surveillance and resource-constrained sensing.

ADΔER (Address, Decimation, Δt Event Representation) is a lightweight, asynchronous pixel-stream representation designed to unify traditional framed video and modern event-camera modalities under a single codec and API. It provides an intermediate video representation and system framework supporting source-modeled lossy compression, efficient processing, and backward compatibility with frame-based applications while exposing fine-grained control over event-rate and algorithmic workload (Freeman, 2024).

1. Formal Definition and Data Representation

An ADΔER event is formally defined as a five-tuple:

$e = \langle x, y, c, D, t \rangle$

where:

• $x, y \in \mathbb{Z}_{16}$: 16-bit pixel coordinates (row, column)
• $c \in \mathbb{Z}_8$: 8-bit color channel index (0 = Y/luma, or 0–2 = RGB)
• $D \in \mathbb{Z}_8$: 8-bit decimation exponent
• $t \in \mathbb{Z}_{32-64}$: absolute timestamp in integer clock "ticks"

Each event encodes its own spatial address—removing the need for run-length encoding or explicit full-frame scanouts commonly required in frame-based representations. The asynchrony and sparsity of ADΔER permit highly efficient storage and computation, as events are only generated when pixel-level intensity changes cross configured thresholds.

Temporal dynamics use per-pixel temporal binning:

$\Delta t(e) = t(e) - t(e^{-})$

with $t(e^{-})$ as the timestamp for the previous event at the same $(x, y, c)$. $\Delta t_{max}$ (a user-settable latency span) bounds the temporal granularity; smaller values capture more detail at the cost of more events, while larger values throttle the event-rate on static pixels.

Intensity is estimated during reconstruction by:

$I(e) \propto 2^D / \Delta t(e)$

where only $D$ and $t$ are stored, and $\Delta t$ is recomputed during playback. This use of absolute timestamps (rather than chained $\Delta t$) ensures that errors in lossy compression affect only immediate intensity recovery.

The “decimation” exponent $D$ sets the per-pixel intensity integration threshold. Each pixel maintains an accumulator $A(t)$. When $A(t) \geq 2^{D(t)}$, the pixel emits an event, subtracts $2^{D(t)}$ from $A(t)$, and may increment $D(t)$ to coarsen its rate (throttle). A global or adaptive contrast threshold $M$ enforces emission if the signal departs too far from the last value, enabling rate–distortion tradeoff and prioritization of bandwidth to selected pixels or regions.

2. System Architecture and Codec Workflow

The ADΔER system architecture consists of three conceptual layers:

Layer	Functionality	Remark
Acquisition	Source transcoding from framed, DVS, or DAVIS sensors	Modality-agnostic intake
Representation/Compression	Temporal/spatial cube organization, arithmetic coding, ADU partitioning	Efficient, source-modeled rate–distortion
Applications	Frame reconstruction, async callbacks, DVS back-conversion	Backward compatibility and native event support

Acquisition layer:

• Framed→ADΔER: Video frames are sampled (e.g., via FFmpeg), quantized to reference intervals ($\Delta t_{ref}$), and then processed per-pixel using accumulators that emit events upon reaching $2^D$ integration.
• DVS→ADΔER: Events from a DVS camera (polarity events $\langle x,y,p,t \rangle$) are accumulated using inferred absolute log-intensity, converted to integration events, and fed to the same ADΔER accumulator model.
• DAVIS→ADΔER: APS frames are deblurred to form latent intensity images (via EDI) and DVS events provide inter-frame motion, both feeding into the ADΔER model for joint representation.

Representation/compression layer uses fixed-duration Application Data Units (ADUs) spanning $\Delta t_{adu}$ (e.g., one second, or 30 frames). Events are grouped spatially (e.g., 16×16 cubes), intra-coded for the first event in each cube, and inter-coded via prediction and arithmetic coding (CABAC). This yields an efficient two-stage pipeline—temporal averaging for source modeling, followed by entropy coding—which furnishes a direct rate–distortion tradeoff.

Application layer permits multiple downstream workflows:

• Frame reconstruction: Per-pixel accumulators update reconstructed frames for legacy applications.
• Asynchronous algorithms: Event-driven algorithms (e.g., FAST feature detection) subscribe to per-pixel callbacks, allowing rapid, local processing.
• DVS-back conversion: By thresholding on log-intensity, ADΔER events can be re-emitted as DVS polarity events, maintaining compatibility with neuromorphic and pre-existing event-based pipelines.

3. Evaluation and Benchmark Results

ADΔER’s performance was evaluated across traditional framed video, DAVIS fusion, object detection/tracking, and feature detection and control. The principal findings include:

• Framed-video transcode (YT-UGC subset, 360p–1080p):
  • For contrast threshold $M = 10$, the median event-rate is 46% of the $M=0$ baseline.
  • Video quality (VMAF) drops from approximately 99 to 95; PSNR ≈ 44 dB at high quality.
  • Encoding speed exceeds 250 FPS raw and 100 FPS compressed (on 8-core CPUs).
• DAVIS fusion (DVSNOISE20):
  • Fusion modes: (i) EDI-reconstruct→framed→ADΔER, (ii) deblur+DVS→ADΔER, (iii) DVS only→ADΔER.
  • Event recovery: Modes (ii, iii) recover >80% of original DVS events; mode (i) <10%.
  • Latency: Mode (iii) achieves ≈0.4 s per packet, mode (ii) ≈2.1 s, mode (i) >20 s at high FPS.
• Object Detection & Tracking:
  • YOLOv5 on BDD100K: mAP +1.2% overall (+3.9% in “rider” class) using (intensity, D, Δt) as RGB channels.
  • MIL-tracker (NFS data): ≤1 event/pixel/frame; AUC drop 0.9% compared to raw frames.
• Feature Detection & Rate Control (VIRAT):
  • Asynchronous FAST (on ADΔER) achieves median speedup of 43.7% over OpenCV FAST at "Low" CRF (≈0.5 event/pixel/frame).
  • Source-modeled compression (CRF {0, 3, 6, 9}) yields 2.5:1 reduction over raw ADΔER with 1–1.6 dB reduction in PSNR.
  • Region-of-interest decimation provides +3–6 dB PSNR around moving objects without additional CPU cost.
• DVS→ADΔER→DVS object detection (GEN1 automotive, HMNet):
  • Lossless→Low CRF reduces DVS bitrate by 50–85%; mAP drop for large objects: 0.2% (ADΔER+CRF) vs 8.4% (random drop) for 76.8% bitrate reduction.

Scene temporal redundancy strongly governs compression ratio (low motion yields >5:1 compression) and processing throughput (sparser events enable faster asynchronous algorithms).

4. Content-Adaptive Rate and Application Control

ADΔER exposes a unique mechanism whereby application speed is directly governed by the event-rate, which itself depends on scene activity and compressive settings. Bandwidth, accuracy, and processing speed become jointly controllable via simple parameters:

• Global CRF ($\in [0..9]$) sets $M$, $M_{max}$, $M_v$, and cluster radius. Increasing CRF predictably reduces event-rate and computational cost at the expense of PSNR (≈2.5–5 dB per step).
• Region-of-interest (ROI) decimation modulates $D_{app}$ near features/tracked objects, leaving other pixels throttled towards $\Delta t_{max}$.
• These mechanisms permit dynamic, local adaptation of event generation. For example, a surveillance system can allocate higher event rates to PTZ zones upon intrusion detection while throttling bandwidth elsewhere.

A plausible implication is enablement of highly scalable machine vision pipes in edge or cloud environments, where bandwidth and computation are matched to situational requirements.

5. Broader Applications and Implications

5.1 Large-Scale Surveillance

Conventional framed solutions lead to either high storage or global bitrate cuts, which degrade both human (VMAF) and machine (e.g., detection accuracy) performance. ADΔER’s edge-transcoding enables per-region, per-time, and per-application adaptation, reducing bitrate up to 90% in static regions and enabling sub-second detection without frame buffering. This provides a unified basis for fusing emerging event sensors (e.g., Aeveon) alongside conventional cameras.

5.2 Resource-Constrained Sensing

Embedded and mobile devices are limited in their ability to transmit high-resolution or high-frame-rate video. By adapting per-pixel decimation—as fast-moving regions trigger more events and static regions are aggressively throttled—ADΔER supports always-on computer vision even in highly resource-limited (e.g., IoT, drone) settings, removing the need for sensor-specific codecs.

6. Conclusion and Future Directions

ADΔER establishes a universal, sparse, and compressible intermediate format for video from a broad spectrum of camera modalities, including frames, DVS, DAVIS, and emerging intensity-event cameras. The representation’s exposure of raw event-rate as both a compression and algorithmic workload metric enables content-adaptive pipelines spanning software and hardware. Open-source implementations are available (https://github.com/ac-freeman/adder-codec-rs), providing a unified, narrow-waist event API to accelerate research and deployment of next-generation vision systems (Freeman, 2024).