Raw2Event: Sparse Event Conversion

• Raw2Event is a paradigm that converts dense raw observations into sparse event representations to expose key dynamics and transitions.
• It employs multiple methodologies, from threshold-based DVS simulation to neural mapping, to achieve real-time processing and improved calibration.
• Its applications span sensor imaging, video captioning, social media analysis, and reinforcement learning, illustrating both performance gains and open challenges.

Searching arXiv for the cited Raw2Event-related papers and adjacent terminology. I’m retrieving the relevant arXiv records for the Raw2Event term and the specific IDs cited in the source material. Raw2Event denotes a class of formulations that convert raw observations into event representations. In recent arXiv usage, the term refers most directly to converting raw Bayer frame streams into DVS-like events in real time, but it also covers supervised RAW→Event mapping built on paired hybrid-vision sensing for event-guided ISP, one-pass generation of temporal event sequences from untrimmed video, online extraction of local events from geo-tagged tweet streams, and automated eventization of raw observations for model-based reinforcement learning (Ning et al., 8 Sep 2025). Across these settings, the common objective is to transform dense, continuous, or weakly structured inputs into sparse event variables that expose dynamics, saliency, or decision-relevant transitions.

1. Conceptual scope and event definitions

The meaning of an “event” in Raw2Event systems is domain-dependent. In event-camera imaging, an event is a tuple $(x, y, t, p)$, where $x, y$ index the event pixel, $t$ is the timestamp, and $p \in \{+1, -1\}$ is the polarity. In event-guided ISP, the same tuple is accumulated into voxel grids aligned to RAW exposures. In dense video captioning, an event is a contiguous temporal segment represented as a $T$-dimensional binary vector whose nonzero entries indicate which segments belong to the event. In geo-tagged tweet analysis, an event is a geo-topical cluster formed around a pivot tweet by authority ascent. In event-aware world models, events are modality-specific signals generated from raw observations and paired with boundary indicators that mark the start and end time of event segments (LU et al., 31 Jan 2025).

These formulations share a structural pattern: Raw2Event replaces direct processing of the raw stream with a latent or explicit eventization stage. The replacement can be physics-based, as in DVS-style threshold crossings from raw Bayer intensities; sequence-generative, as in bi-directional event decoding over video segments; graph- and kernel-based, as in local-event detection from tweets; or auxiliary and representation-shaping, as in reinforcement learning. This suggests that Raw2Event is best understood as a general eventization paradigm rather than as a single architecture.

A second common feature is the use of sparse event variables to impose inductive bias. The sparse representation can reduce redundancy, emphasize high-frequency changes, localize semantically coherent structures, or regularize latent dynamics. The relevant “raw” modality therefore varies: unprocessed sensor voltages, RAW quad-Bayer frames, RGB-video segment features, social-media text streams with timestamps and locations, or multimodal RL observations.

2. Raw Bayer frames to DVS-like events

The most literal use of the term appears in "Raw2Event: Converting Raw Frame Camera into Event Camera" (Ning et al., 8 Sep 2025). That system uses direct access to raw Bayer sensor data to bypass the ISP and generate events from the unprocessed intensity stream in real time. The hardware stack is built around a Raspberry Pi Camera Module 3 with Sony IMX708, HDR mode, and phase-detection autofocus, with the Raspberry Pi “picamera2” stack providing a raw Bayer stream before ISP and a processed RGB stream concurrently. The system records synchronized raw frames, RGB frames, and simulated event streams side-by-side, and it also supports synchronized comparison against a DAVIS346 mono event camera in a dual-camera rig mounted on a Dobot Magician robotic arm.

Its core modeling basis is DVS-Voltmeter. The internal voltage increment at each pixel is modeled as a drift-diffusion process,

$\Delta V_d = \mu\,\Delta t + \sigma\,W(\Delta t),$

with

$$\mu = \frac{k_1}{\bar{L} + k_2}\,k_{dL} + k_4 + k_5\,\bar{L}, \qquad \sigma = \frac{k_3}{\bar{L} + k_2}\,\sqrt{\bar{L} + k_6}.$$

For raw-based simulation, intensity is taken directly from the native 10-bit raw stream,

$$L_i^{\mathrm{raw}}(x,y) = \frac{r_i(x,y)}{2^{10}-1},$$

while RGB-based simulation uses grayscale-converted ISP frames. ON and OFF events are defined by first hitting times to contrast thresholds, yielding event tuples $(t,x,y,p)$ with $p \in \{+1\ \mathrm{(ON)}, -1\ \mathrm{(OFF)}\}$.

The pipeline proceeds by capturing concurrent raw Bayer frames and RGB frames with fixed exposure and enabled autofocus; computing $x, y$0, $x, y$1, and $x, y$2 from consecutive frames; deriving $x, y$3 and $x, y$4 from calibrated parameters $x, y$5; sampling first hitting times from the inverse Gaussian distribution; and emitting events whenever the sampled hitting time satisfies $x, y$6. The implementation re-implements the original DVS-Voltmeter simulator with NumPy vectorization on Raspberry Pi for real-time prototyping, while offline conversion can use GPU acceleration on desktop.

A notable design choice is the decision to disable auto-exposure and auto-gain while leaving autofocus enabled. The stated rationale is to stabilize pixel intensities without losing the autofocus advantage that standard DVS sensors typically lack. Raw2Event therefore targets a regime in which commodity frame cameras can approximate event sensing while retaining higher resolution, HDR mode, and autofocus. The paper explicitly states that it cannot achieve microsecond latency because it is frame-rate bound, but it aims at practical regimes where standard frame cameras are sufficient.

The reported empirical results emphasize fidelity advantages over RGB-derived conversion. Raw2Event preserves more spatial detail in bright regions and edges than RGB2Event because it uses 10-bit raw values and avoids ISP tone mapping. Raw2Event also matches DAVIS temporal distributions more closely than RGB2Event in inter-event interval statistics. Embedded throughput is reported as approximately $x, y$7 FPS at $x, y$8 on Raspberry Pi 4 and approximately $x, y$9 FPS at the same resolution on Raspberry Pi 5, while an RTX 4070 Ti Super reaches up to approximately $t$0 FPS for parameter sweeps.

3. Paired RAW–event sensing and event-guided ISP

"RGB-Event ISP: The Dataset and Benchmark" established a complementary Raw2Event setting centered on paired sensing and image signal processing (LU et al., 31 Jan 2025). The dataset is captured with a confidential hybrid-vision sensor based on ALPIX-Eiger that stacks an APS and an EVS on the same die. EVS pixels are physically embedded in a quad-Bayer arrangement at the lower-right corner of each color pixel block, which gives pixel-level spatial alignment to the RAW grid. APS RAW resolution is $t$1; the EVS event map is $t$2, so RAW resolution is twice that of events in each dimension. The dataset contains $t$3 RAW frames over $t$4 scenes, with $t$5 exposure modes and $t$6 lenses, spanning apertures from $t$7 to $t$8, focal lengths from $t$9 to $p \in \{+1, -1\}$0 mm, and exposure times from $p \in \{+1, -1\}$1 to $p \in \{+1, -1\}$2 ms.

The benchmark defines explicit synchronization and calibration procedures. RAW frames are linearized by black level subtraction and fixed-pattern noise correction,

$p \in \{+1, -1\}$3

where $p \in \{+1, -1\}$4 is a global black level and $p \in \{+1, -1\}$5 is a per-row fixed-pattern vector calibrated in a dark environment. Events are synchronized to RAW by accumulating them within each RAW exposure interval $p \in \{+1, -1\}$6 or a fixed window centered at the RAW timestamp. ColorChecker parameters are propagated to subsequent frames for ISP color calibration, and the conventional ISP pipeline performs demosaicing, white balancing, BM3D denoising with $p \in \{+1, -1\}$7, color correction via a $p \in \{+1, -1\}$8 color correction matrix, and sRGB gamma. The resulting RGB references achieve CIEDE00 average approximately $p \in \{+1, -1\}$9, median approximately $T$0 across $T$1 samples, with temporal stability below $T$2 variation across $T$3 patches in a $T$4-frame test.

The paper also formalizes the event model used for learning. For log intensity $T$5, an event occurs when

$T$6

For neural fusion, events are accumulated into voxel grids,

$T$7

with time discretized into $T$8 bins over the RAW exposure window.

Within this setting, Raw2Event becomes a supervised mapping problem enabled by paired RAW–event data. The benchmark explicitly proposes two model designs. A direct predictor uses CNN, UNet, or Swin on RAW to predict a voxel grid $T$9. A physics-informed model predicts a per-pixel log-intensity trajectory $\Delta V_d = \mu\,\Delta t + \sigma\,W(\Delta t),$0 and a per-pixel threshold $\Delta V_d = \mu\,\Delta t + \sigma\,W(\Delta t),$1, then renders events via the threshold model. The benchmark further proposes a simple event-guided ISP baseline, EV-UNet, in which a UNet-like encoder-decoder takes RAW as primary input and fuses an aligned event voxel-grid at the encoder. Its outdoor performance improves average PSNR from approximately $\Delta V_d = \mu\,\Delta t + \sigma\,W(\Delta t),$2 for UNet* to approximately $\Delta V_d = \mu\,\Delta t + \sigma\,W(\Delta t),$3, whereas indoors event fusion can degrade performance because flickering artificial lights destabilize the event stream relative to intensity; the reported average is approximately $\Delta V_d = \mu\,\Delta t + \sigma\,W(\Delta t),$4 for EV-UNet versus approximately $\Delta V_d = \mu\,\Delta t + \sigma\,W(\Delta t),$5 for UNet*.

The significance of this benchmark lies in its joint treatment of sensing, calibration, ISP, and learning. It is presented as the first research on event-guided ISP, and it frames Raw2Event not merely as event simulation but as a supervised, physically aligned mapping between linearized RAW measurements and event volumes attached to exposure windows.

4. Sequential event generation from video and social streams

Outside sensor imaging, Raw2Event has also been used to describe the extraction of structured event sequences from raw temporal streams. Team RUC_AIM3’s "Technical Report at Activitynet 2020 Task 2: Exploring Sequential Events Detection for Dense Video Captioning" formulates this conversion for untrimmed video (Song et al., 2020). The video is split into $\Delta V_d = \mu\,\Delta t + \sigma\,W(\Delta t),$6 non-overlapping segments of $\Delta V_d = \mu\,\Delta t + \sigma\,W(\Delta t),$7 frames each. An event $\Delta V_d = \mu\,\Delta t + \sigma\,W(\Delta t),$8 is represented as a $\Delta V_d = \mu\,\Delta t + \sigma\,W(\Delta t),$9-dimensional binary vector, and the decoder predicts per-segment inclusion probabilities

$$\mu = \frac{k_1}{\bar{L} + k_2}\,k_{dL} + k_4 + k_5\,\bar{L}, \qquad \sigma = \frac{k_3}{\bar{L} + k_2}\,\sqrt{\bar{L} + k_6}.$$0

A bidirectional GRU encodes the segment sequence using ResNet-200 appearance features, I3D motion features, and VGGish audio features, and two event decoders generate events in forward and backward order. Matched forward and backward events are fused when temporal IoU exceeds $$\mu = \frac{k_1}{\bar{L} + k_2}\,k_{dL} + k_4 + k_5\,\bar{L}, \qquad \sigma = \frac{k_3}{\bar{L} + k_2}\,\sqrt{\bar{L} + k_6}.$$1.

This proposal-free scheme treats Raw2Event as sequence generation rather than as proposal generation plus classification. The complexity is reported as $$\mu = \frac{k_1}{\bar{L} + k_2}\,k_{dL} + k_4 + k_5\,\bar{L}, \qquad \sigma = \frac{k_3}{\bar{L} + k_2}\,\sqrt{\bar{L} + k_6}.$$2 with $$\mu = \frac{k_1}{\bar{L} + k_2}\,k_{dL} + k_4 + k_5\,\bar{L}, \qquad \sigma = \frac{k_3}{\bar{L} + k_2}\,\sqrt{\bar{L} + k_6}.$$3, which is contrasted with proposal-based systems that enumerate roughly $$\mu = \frac{k_1}{\bar{L} + k_2}\,k_{dL} + k_4 + k_5\,\bar{L}, \qquad \sigma = \frac{k_3}{\bar{L} + k_2}\,\sqrt{\bar{L} + k_6}.$$4 window proposals per video. On ActivityNet Captions validation data, the bidirectional model achieves Recall @tIoU $$\mu = \frac{k_1}{\bar{L} + k_2}\,k_{dL} + k_4 + k_5\,\bar{L}, \qquad \sigma = \frac{k_3}{\bar{L} + k_2}\,\sqrt{\bar{L} + k_6}.$$5 of $$\mu = \frac{k_1}{\bar{L} + k_2}\,k_{dL} + k_4 + k_5\,\bar{L}, \qquad \sigma = \frac{k_3}{\bar{L} + k_2}\,\sqrt{\bar{L} + k_6}.$$6 and Precision @tIoU of $$\mu = \frac{k_1}{\bar{L} + k_2}\,k_{dL} + k_4 + k_5\,\bar{L}, \qquad \sigma = \frac{k_3}{\bar{L} + k_2}\,\sqrt{\bar{L} + k_6}.$$7, while generating only $$\mu = \frac{k_1}{\bar{L} + k_2}\,k_{dL} + k_4 + k_5\,\bar{L}, \qquad \sigma = \frac{k_3}{\bar{L} + k_2}\,\sqrt{\bar{L} + k_6}.$$8 events per video on average with self-tIoU $$\mu = \frac{k_1}{\bar{L} + k_2}\,k_{dL} + k_4 + k_5\,\bar{L}, \qquad \sigma = \frac{k_3}{\bar{L} + k_2}\,\sqrt{\bar{L} + k_6}.$$9. In the downstream dense-captioning system, the final challenge testing performance reaches $$L_i^{\mathrm{raw}}(x,y) = \frac{r_i(x,y)}{2^{10}-1},$$0 METEOR.

A further extension appears in Event-Radar, which turns raw geo-tagged tweet streams into local events (Zhang et al., 2017). Each tweet is modeled as $$L_i^{\mathrm{raw}}(x,y) = \frac{r_i(x,y)}{2^{10}-1},$$1 with timestamp, location, and text. GeoBurst+ computes a cross-modal authority score

$$L_i^{\mathrm{raw}}(x,y) = \frac{r_i(x,y)}{2^{10}-1},$$2

where $$L_i^{\mathrm{raw}}(x,y) = \frac{r_i(x,y)}{2^{10}-1},$$3 is a Gaussian geo-spatial kernel and $$L_i^{\mathrm{raw}}(x,y) = \frac{r_i(x,y)}{2^{10}-1},$$4 is semantic affinity derived from approximate random walk with restart on a keyword co-occurrence graph. Tweets ascend to local pivots via repeated maximization of neighboring authority, and all tweets that converge to the same pivot form a geo-topical cluster $$L_i^{\mathrm{raw}}(x,y) = \frac{r_i(x,y)}{2^{10}-1},$$5. Candidate clusters are then classified by logistic regression using unusualness, burstiness, cluster size, and pivot-authority features. The implementation is a MEAN-stack system with MongoDB and Google Maps visualization, and it supports incremental updates as the sliding query window shifts.

These works broaden Raw2Event beyond photometric sensing. In both cases, the raw stream is temporally dense and semantically ambiguous, and eventization imposes a more compact structure: contiguous temporal segments in video and geo-topical clusters in social streams.

5. Eventization inside world models for reinforcement learning

"From Observations to Events: Event-Aware World Model for Reinforcement Learning" recasts Raw2Event as an auxiliary mechanism for learning better latent dynamics from raw observations (Peng et al., 27 Jan 2026). EAWM augments a world model with modality-specific event variables $$L_i^{\mathrm{raw}}(x,y) = \frac{r_i(x,y)}{2^{10}-1},$$6 and boundary indicators $$L_i^{\mathrm{raw}}(x,y) = \frac{r_i(x,y)}{2^{10}-1},$$7. The joint factorization is summarized as

$$L_i^{\mathrm{raw}}(x,y) = \frac{r_i(x,y)}{2^{10}-1},$$8

Here $$L_i^{\mathrm{raw}}(x,y) = \frac{r_i(x,y)}{2^{10}-1},$$9 combines sequence outputs with stochastic embeddings, and events are predicted from latent transitions rather than from labels.

The automated event generator is modality-specific. For visual inputs, it uses per-pixel log-brightness changes and Adaptive Gaussian Mixture Models. A visual event is generated when the Mahalanobis surprise of $(t,x,y,p)$0 relative to the mixture exceeds a threshold or when the matched component’s weight is low. For ordinal data, an event triggers when the normalized change exceeds a threshold $(t,x,y,p)$1. For nominal data, an event triggers when the category changes between time steps. A Generic Event Segmentor then computes a boundary gate from the fraction of positions that fired events at time $(t,x,y,p)$2. This gate suppresses the event-prediction loss at boundaries and reweights the observation loss so that dynamic regions receive greater emphasis.

Event prediction becomes an explicit auxiliary objective. Ordinal modalities use cross-entropy; visual and nominal modalities use focal loss. The aggregate event loss is

$(t,x,y,p)$3

and the total training objective is

$(t,x,y,p)$4

EAWM is instantiated in two architectures: EADream, a DreamerV3-based recurrent state-space model with observation prediction from prior states, and EASimulus, a token-based Transformer/RetNet world model with parallel observation prediction.

The reported results indicate that this eventization improves generalization and sample efficiency. EAWM is said to boost strong model-based RL baselines by $(t,x,y,p)$5 across Atari 100K, Craftax 1M, DeepMind Control 500K, and DMC-GB2 500K. EASimulus achieves mean human-normalized score $(t,x,y,p)$6 and IQM $(t,x,y,p)$7 on Atari 100K, while EADream reaches mean $(t,x,y,p)$8 and median $(t,x,y,p)$9 on ten hard DeepMind Control tasks. Ablations show that removing the event predictor lowers performance and removing GES destabilizes training. In this formulation, Raw2Event is not an output modality for downstream perception but a latent-structuring device that shapes what the world model encodes.

6. Degradation, misconceptions, and open problems

Several works caution that Raw2Event is not equivalent to simply thresholding any processed signal. The raw-frame conversion literature explicitly argues that RGB-based frame-to-event converters inherit tone mapping, compression, denoising, and automatic exposure or gain distortions, whereas raw access preserves higher bit depth and linearity (Ning et al., 8 Sep 2025). The RGB-Event ISP benchmark likewise shows that event fusion can improve outdoor reconstruction yet degrade indoor performance under flickering artificial lights, because event streams become unstable relative to intensity (LU et al., 31 Jan 2025). A plausible implication is that Raw2Event quality depends as much on acquisition and synchronization policy as on the conversion model itself.

Event degradation is an additional obstacle. "Restoring Real-World Degraded Events Improves Deblurring Quality" models threshold bias, limited bandwidth, and circuit noise, then trains RDNet to restore degraded event voxel grids before event-guided deblurring (Shen et al., 2024). The paper formalizes the ideal event relation

$p \in \{+1\ \mathrm{(ON)}, -1\ \mathrm{(OFF)}\}$0

and the blur-event linkage

$p \in \{+1\ \mathrm{(ON)}, -1\ \mathrm{(OFF)}\}$1

Its event tensor uses $p \in \{+1\ \mathrm{(ON)}, -1\ \mathrm{(OFF)}\}$2 temporal channels, and the two-stage network first restores degraded events $p \in \{+1\ \mathrm{(ON)}, -1\ \mathrm{(OFF)}\}$3 and then uses $p \in \{+1\ \mathrm{(ON)}, -1\ \mathrm{(OFF)}\}$4 to guide deblurring. On GOPRO, RDNet reaches $p \in \{+1\ \mathrm{(ON)}, -1\ \mathrm{(OFF)}\}$5 dB PSNR and $p \in \{+1\ \mathrm{(ON)}, -1\ \mathrm{(OFF)}\}$6 SSIM, compared with $p \in \{+1\ \mathrm{(ON)}, -1\ \mathrm{(OFF)}\}$7 dB and $p \in \{+1\ \mathrm{(ON)}, -1\ \mathrm{(OFF)}\}$8 for EFNet; on REBlur it reaches $p \in \{+1\ \mathrm{(ON)}, -1\ \mathrm{(OFF)}\}$9 dB and $x, y$00. Although this work does not convert non-event raw inputs into events, it establishes that downstream Raw2Event pipelines are sensitive to threshold mismatch, event thinning, shot noise, leakage noise, and hot pixels.

Open problems recur across domains. In raw-sensor imaging, timestamp precision, exposure-window selection, per-pixel contrast threshold calibration, HDR exploitation, fusion granularity, and generalization across lenses and exposure modes remain unresolved (LU et al., 31 Jan 2025). In hardware-software conversion, the main limitations are frame-rate-bounded latency, the absence of explicit refractory-period and per-pixel threshold-mismatch modeling, and sensitivity to aggressive parameter settings such as large $x, y$01, $x, y$02, and $x, y$03 (Ning et al., 8 Sep 2025). In sequential video event generation, binary masks and fixed thresholding at $x, y$04 may struggle with nested or hierarchical events, and teacher forcing creates exposure bias (Song et al., 2020). In event-aware RL, deterministic boundary detection may fail when event distributions are extremely dense or when task-relevant changes are not captured by AGMM or threshold rules (Peng et al., 27 Jan 2026).

Taken together, these limitations define the contemporary research frontier. Raw2Event systems now span calibrated sensing, simulation, sequence generation, streaming analytics, and latent-dynamics learning, but they remain constrained by synchronization accuracy, noise realism, threshold calibration, domain shift, and the choice of what counts as an event in the first place.