---
title: 'EvEnhancer: Event-Driven Video SR'
url: https://www.emergentmind.com/topics/evenhancer
type: topic
---

# EvEnhancer: Event-Driven Video SR

Searching arXiv for the named systems to ground the article in the cited papers.
First, retrieving the core "EvEnhancer" continuous space-time video super-resolution paper.
Then retrieving the follow-up/extended formulation with EvEnhancerPlus.
Also checking the similarly named VEnhancer paper because the source material explicitly notes possible name confusion.
EvEnhancer is an event-driven method for continuous space-time video super-resolution (C-STVSR) that combines low-resolution RGB frames with aligned event streams to reconstruct high-resolution, high-frame-rate video at arbitrary spatial and temporal scales [2505.04657]. In its core formulation, the model addresses the joint problem of spatial upsampling and temporal interpolation through a unified continuous representation rather than separate fixed-scale modules. The method is built around two principal components: an Event-Adapted Synthesis Module (EASM), which uses event information to model long-term motion trajectories and synthesize informative spatiotemporal features, and a Local Implicit Video Transformer (LIVT), which learns a continuous video implicit neural representation (INR) with local cross-scale spatiotemporal attention [2505.04657]. A later extension, EvEnhancerPlus, adds a controllable switching mechanism and a staged training strategy to improve efficiency while maintaining robustness at out-of-distribution (OOD) scales [2510.03833].

## 1. Definition and problem setting

EvEnhancer is defined in the literature as an approach for continuous space-time video super-resolution with events. The task is to reconstruct an arbitrary number of intermediate high-resolution frames at arbitrary timestamps and arbitrary spatial magnifications from a pair of low-resolution, low-frame-rate frames and an aligned event stream [2505.04657].

The core input notation consists of low-resolution frames $V_{LR} = \{I_t^{LR}\}$, typically focusing on endpoints $I_0^{LR}$ and $I_1^{LR}$, together with an event stream $E = \{(x_i, y_i, t_i, p_i)\}$ where $p_i \in \{+1,-1\}$ denotes polarity [2505.04657]. The target is to predict high-resolution frames at arbitrary continuous spatiotemporal coordinates, expressed as $I^{HR}(x,y,\mathcal{T}; s)$ for any $\mathcal{T} \in [0,1]$ and real spatial scale $s \ge 1$ [2505.04657].

A central design choice is to learn a continuous mapping
$$
f_\theta : (x, y, \mathcal{T}, s; V_{LR}, E) \rightarrow I(x,y,\mathcal{T}; s) \in \mathbb{R}^3,
$$
with HR pixel coordinates normalized to LR coordinates as $(x_{lr}, y_{lr}) = (x_{hr}/s, y_{hr}/s)$ [2505.04657]. The decoder uses LIIF-style cell decoding, with $c(x,y) = (1/s, 1/s)$, and reconstructs pixels through
$$
\hat I_{\mathcal{T}}^{HR}(x,y) = \phi_{MLP}([\tilde Z_{\mathcal{T}}(x,y); q_{\mathcal{T}}(x,y); c(x,y)]).
$$
This formulation is intended to support arbitrary spatial scales and arbitrary frame rates by repeated continuous querying [2505.04657].

The motivation is that prevailing C-STVSR methods often generalize poorly at OOD spatial and temporal scales, while event streams provide high temporal resolution and high dynamic range that are beneficial for motion modeling and robustness under challenging conditions [2505.04657]. This suggests that EvEnhancer occupies a distinct position relative to purely frame-based C-STVSR models: it is designed not merely for interpolation or fixed-factor upsampling, but for continuous, event-guided reconstruction across space and time.

## 2. Event representation and continuous formulation

EvEnhancer relies on event cameras’ asynchronous brightness-change measurements. In the 2025 formulation, an event is represented as $e_i = (x_i, y_i, t_i, p_i)$, with polarity triggered when log-intensity changes exceed a threshold [2505.04657]. The later EvEnhancerPlus paper restates the event formalism as $r = (\tau, x, y, p)$ and gives the firing rule
$$
p =
\begin{cases}
+1, & \text{if } \Delta L(x,y) > +c,\\
-1, & \text{if } \Delta L(x,y) < -c,
\end{cases}
$$
together with the accumulated log-intensity change
$$
\Delta L_{\tau_s \to \tau_e}(x,y) = c \int_{\tau_s}^{\tau_e} p(\tau,x,y)\, d\tau.
$$
These expressions formalize the connection between event density and local brightness change [2510.03833].

For processing, EvEnhancer converts the raw event stream into a 3D spatiotemporal voxel grid $\mathcal{V} \in \mathbb{R}^{H \times W \times (M+1)}$ by discretizing time into short segments between the endpoint frames [2505.04657]. A typical aggregation is
$$
V(x,y,m) = \sum_i \mathbf{1}[(x_i,y_i)=(x,y)] \cdot \mathbf{1}[t_i \in [\tau_m,\tau_{m+1})] \cdot w(p_i),
$$
with $w(p_i)=+1$ for positive and $-1$ for negative events [2505.04657]. The 2025 extension describes a linearly weighted voxelization,
$$
V_m = \sum_r p_r \max\!\left(0, 1 - \left|(m-1)-\frac{(\tau_r-\tau_s)}{(\tau_e-\tau_s)}M\right|\right),
$$
followed by percentile clipping normalization,
$$
\bar V_m = \min(V_m,\eta)/\eta,
$$
with $\eta$ equal to the 98-th percentile among non-zero values [2510.03833].

The event voxels are aligned to the two LR frames and bicubic-downsampled to match LR resolution [2505.04657]. Their function is to provide fine-grained temporal cues between sparse RGB observations. In the source papers, this is repeatedly linked to improved handling of rapid motion, high dynamic range variation, and OOD scales [2505.04657; 2510.03833].

## 3. Event-Adapted Synthesis Module

The Event-Adapted Synthesis Module is the first major architectural component of EvEnhancer. It is designed to fuse frames and events to learn long-term motion trajectories and informative spatiotemporal features through two submodules: Event-Modulated Alignment (EMA) and Bidirectional Recurrent Compensation (BRC) [2505.04657].

### Event-Modulated Alignment

EMA operates on a 3-level pyramid at scales $\{1\times, 1/2\times, 1/4\times\}$ [2505.04657]. At each level $l$, frame features $F_0^{(l)}$ and $F_1^{(l)}$ and event features $F_{0\to1}^{E,m(l)}$ for segment $m$ are used to estimate an initial motion vector:
$$
MV_m^{(l)} = \psi^{(l)}([F_0^{(l)}, F_1^{(l)}, O_m^{(l-1)}]),
$$
where $\psi^{(l)}$ is a Conv+LReLU block and $O_m^{(l-1)}$ is the coarse-to-fine offset propagated from the previous level [2505.04657]. Events then modulate motion through an Event Modulation Block:
$$
\tilde{MV}_m^{(l)} = \mathcal{M}^{(l)}(MV_m^{(l)}, F_{0\to1}^{E,m(l)}),
$$
and the final offsets are obtained as
$$
O_m^{(l)} = \phi^{(l)}([MV_m^{(l)}, \tilde{MV}_m^{(l)}]).
$$
Deformable convolutions align features in both forward and backward directions:
$$
F_m^{LR+,(l)} = DCN(F_0^{(l)}; O_m^{(l)}), \qquad
F_m^{LR-,(l)} = DCN(F_1^{(l)}; O_m^{(l)}).
$$
The DCN sampling rule is explicitly given as
$$
y(p_0) = \sum_k w_k \cdot x(p_0 + p_k + \Delta p_k)\cdot \Delta m_k.
$$
These aligned features are aggregated coarse-to-fine to produce a discrete aligned temporal feature sequence $\mathbb{F}=\{F_\tau^{LR}\}$ across timestamps $\tau \in \{0, 1/(M+1), \dots, 1\}$ [2505.04657].

### Bidirectional Recurrent Compensation

Because each event segment corresponds to a narrow temporal window, BRC propagates information forward and backward to enlarge temporal coverage [2505.04657]. At timestamp $\tau$, it combines event feature $F_\tau^E$, aligned frame feature $F_\tau^{LR}$, and recurrent hidden states:
$$
r_\tau^b, h_\tau^b = \mathcal{F}^b(\mathcal{C}^b(F_\tau^{LR}, F_\tau^E), h_{\tau+\Delta\tau}^b),
$$
$$
r_\tau^f, h_\tau^f = \mathcal{F}^f(\mathcal{C}^f(F_\tau^{LR}, F_\tau^E), h_{\tau-\Delta\tau}^f, r_\tau^b).
$$
The final fusion is residual:
$$
\hat F_\tau = F_\tau^{LR} + r_\tau^f + r_\tau^b.
$$
The output of EASM is thus a high-temporal-resolution feature sequence $\hat F = \{\hat F_\tau\}$ [2505.04657].

The stated architectural details are specific: RGB features are extracted by a $5\times 5$ convolution with LReLU followed by 5 residual blocks; event features use a $3\times 3$ convolution and 5 residual blocks; BRC uses a $5\times 5$ convolution for event extraction, $1\times1$ convolutions for channel attention and FFN, and $3\times3$ convolutions elsewhere [2505.04657].

A plausible implication is that EASM is the locus where event information most directly reshapes motion estimation, while subsequent modules primarily exploit the improved latent trajectory representation.

## 4. Local Implicit Video Transformer and continuous decoding

The Local Implicit Video Transformer is the second principal module. It constructs a unified continuous spatiotemporal INR by selecting local temporal neighborhoods and applying cross-scale spatiotemporal attention around each query coordinate [2505.04657].

For a query timestamp $\mathcal{T}$, LIVT selects a local temporal neighborhood of size $T^G$ by
$$
S_\tau = \arg\min_{S \subset \{\text{discrete }\tau\}, |S| = T^G} \sum_{\tau_i \in S} |\tau_i - \mathcal{T}|.
$$
Spatially, it defines a local grid of size $H^G \times W^G$ around $(x_{lr}, y_{lr})$ [2505.04657].

From $\hat F$, 3D convolutions generate embeddings $K$, $Q$, and $V$ [2505.04657]. The target query embedding is obtained by 3D trilinear sampling and upsampling:
$$
q_{\mathcal{T}} \in \mathbb{R}^{(s^2HW)\times C}.
$$
For each selected $\tau_i \in S_\tau$, keys and values are sampled on the local LR grid:
$$
k_{\tau_i}, v_{\tau_i} \in \mathbb{R}^{(s^2HW)\times(H^GW^G)\times C}.
$$

The relative spatiotemporal coordinates $\delta\mathcal{C} = \{\delta\tau,\delta x,\delta y\} \in (-1,1)$ are encoded by a cosine positional encoding
$$
g(\delta\mathcal{C}) =
[\sin(2^0\delta\mathcal{C}), \cos(2^0\delta\mathcal{C}), \dots, \sin(2^{L-1}\delta\mathcal{C}), \cos(2^{L-1}\delta\mathcal{C})],
$$
with $L=10$, followed by a linear projection $W_b$ to produce positional bias $b_{\tau_i}$ [2505.04657]. Local cross-scale attention is then computed as
$$
A_{\tau_i} = \operatorname{softmax}\left(\frac{q_{\mathcal{T}} k_{\tau_i}^T}{\sqrt{C}} + b_{\tau_i}\right), \qquad
\tilde z_{\tau_i} = A_{\tau_i} v_{\tau_i}.
$$
The unified continuous video representation is formed by concatenating the attended features:
$$
\tilde Z_{\mathcal{T}} = \operatorname{Concat}(\{\tilde z_{\tau_i}\}_{\tau_i \in S_\tau}).
$$
Finally, a 5-layer MLP with widths $[256,256,256,256,3]$ and GELU reconstructs RGB values via
$$
\hat I_{\mathcal{T}}^{HR}(x,y) = \phi_{MLP}([\tilde Z_{\mathcal{T}}(x,y); q_{\mathcal{T}}(x,y); c(x,y)]).
$$
These definitions are given explicitly in the original paper [2505.04657].

The architecture uses one $3\times3\times3$ convolution to encode $\hat F$ and three $3\times3\times3$ convolutions to produce $K$, $Q$, and $V$ [2505.04657]. Recommended settings are $M=7$ event segments, channel count $C=64$, and local grid $T^G \times H^G \times W^G = 3\times3\times3$ for the main model; the light variant reduces LIVT channels to 16 and uses $M=5$ [2505.04657].

The papers explicitly contrast this unified space-time INR with decoupled spatial and temporal INRs used in models such as VideoINR and MoTIF, arguing that direct 3D local attention preserves cross-domain dependencies and improves OOD generalization [2505.04657; 2510.03833].

## 5. Training, datasets, and empirical performance

EvEnhancer is trained end-to-end with a Charbonnier reconstruction loss on the Y channel:
$$
L_{rec} = \sum_{\mathcal{T}} \sum_{x,y}
\sqrt{|\hat I_{\mathcal{T}}^{HR}(x,y) - I_{\mathcal{T}}^{GT}(x,y)|^2 + \epsilon^2},
$$
with $\epsilon^2 = 1 \times 10^{-6}$ [2505.04657]. The paper states that no perceptual, adversarial, or explicit temporal losses are used [2505.04657].

Training follows a two-stage schedule on Adobe240 [2505.04657]. Stage 1 uses fixed $s=4$ and $t=8$ for 450K iterations. Stage 2 fine-tunes with $s$ uniformly sampled in $[1,4]$ for an additional 150K iterations [2505.04657]. Optimization uses Adam with $\beta_1=0.9$, $\beta_2=0.999$, and cosine annealing from $1e^{-4}$ to $1e^{-7}$ over 150K iterations in fine-tuning [2505.04657]. Data augmentation consists of random cropping of $32\times32$ LR patches, random rotations, and horizontal flips [2505.04657].

The principal datasets and settings reported in the paper are summarized below.

| Category | Dataset | Details |
|---|---|---|
| Training/validation/test | Adobe240 | 133 sequences; 100 train / 16 val / 17 test |
| Synthetic test | GoPro, Adobe240 | events simulated by vid2e |
| Real-world test | BS-ERGB, ALPIX-VSR | event+RGB evaluation |

Performance is reported using PSNR and SSIM on the Y channel [2505.04657].

On the in-distribution setting $(t=8, s=4)$, EvEnhancer achieves the following reported results [2505.04657]:

| Dataset split | EvEnhancer | EvEnhancer-light |
|---|---:|---:|
| GoPro-Center | 33.52 dB / 0.9295 | 33.11 dB / 0.9242 |
| GoPro-Average | 33.30 dB / 0.9279 | 32.73 dB / 0.9203 |
| Adobe240-Center | 32.43 dB / 0.9129 | 31.90 dB / 0.9033 |
| Adobe240-Average | 32.18 dB / 0.9116 | 31.47 dB / 0.8988 |

The paper states that these results outperform HR-INR, MoTIF, and VideoINR while using fewer parameters, with parameter counts of 6.55M for EvEnhancer and 5.81M for EvEnhancer-light [2505.04657].

The model also reports strong OOD results on GoPro. Representative examples include $t=12, s=4$ with 32.07 / 0.9116 for EvEnhancer; $t=16, s=6$ with 28.86 / 0.8434; $t=6, s=12$ with 25.50 / 0.7323; and temporal-only upsampling at $t=6, s=1$ with 38.80 / 0.9714 [2505.04657]. On BS-ERGB with fixed $s=4$, EvEnhancer reports 25.44 / 0.7338 at $t=4$, 24.70 / 0.7215 at $t=6$, and 24.29 / 0.7138 at $t=8$ [2505.04657]. On ALPIX-VSR for 2× VSR, it reports 40.84 dB / 0.9786 without additional fine-tuning [2505.04657].

Ablation studies identify the importance of both core modules. The paper attributes robustness at arbitrary scales to the unified space-time INR, cross-scale attention, and event-guided synthesis [2505.04657]. Qualitative analyses further report clearer textures, sharper details, and smoother temporal evolution with fewer flickering artifacts than VideoINR and MoTIF, especially under fast motion, non-linear trajectories, low light, and HDR scenes [2505.04657].

## 6. EvEnhancerPlus: efficiency-oriented extension

EvEnhancerPlus extends EvEnhancer with two additional elements: a controllable switching mechanism (CSM) and a cross-derivative training strategy, termed staged cross-optimization in the detailed description [2510.03833]. The goal is to reduce computational cost while preserving or improving reconstruction quality.

### Controllable switching mechanism

The CSM is based on the observation that spatiotemporal pixels differ in reconstruction difficulty [2510.03833]. Difficulty is estimated from event statistics using the relation between events and brightness changes:
$$
|\Delta L_{\tau_s \to \tau_e}| = c \left|\int_{\tau_s}^{\tau_e} p_r\, dr\right|.
$$
After max-min normalization,
$$
\bar \Delta L_{\tau_s \to \tau_e}
=
\frac{|\Delta L_{\tau_s \to \tau_e}| - \min |\Delta L_{\tau_s \to \tau_e}|}
{\max |\Delta L_{\tau_s \to \tau_e}| - \min |\Delta L_{\tau_s \to \tau_e}|},
$$
the per-pixel difficulty map at query time $\tau$ is
$$
R_\tau = Up(\bar \Delta L_{\min S_\tau \to \max S_\tau}).
$$
Pixels are then routed to one of $N$ upsamplers using a parameter-free distributor:
$$
\hat I_\tau^{HR}(x,y) = \sum_{n=0}^{N-1} U_n(\hat F; (\tau,x,y)) \cdot D_n(\tau,x,y),
$$
where
$$
D_n(\tau,x,y)=
\begin{cases}
1, & \text{if } R_\tau(x,y)\in(\xi_n,\xi_{n+1}],\\
0, & \text{otherwise}.
\end{cases}
$$
With $N=2$, the threshold $\xi_1$ controls the trade-off between simple and complex pathways [2510.03833].

The paper emphasizes that this mechanism is non-parametric, controllable, and flexible at inference, and reports substantial compute savings: at $t=8$, $s=4$, EvEnhancerPlus uses approximately 6.13 TFLOPs compared with 7.13 for EvEnhancer, described as a 12.8% reduction; the abstract summarizes this as about 85% of EvEnhancer’s cost [2510.03833].

### Cross-derivative training strategy

The multi-pathway system is reported to be unstable under naive joint training, with the simpler path dominating gradients [2510.03833]. To address this, EvEnhancerPlus introduces a staged training procedure:

1. Stage 1: independent pathway training at fixed $t$ and fixed $s$.
2. Stage 2: fine-tuning with variable $s \sim U[1,4]$.
3. Stage 3: freezing the shared feature extractor from the most complex pathway and updating only the remaining upsamplers.

The paper states that this staged cross-initialization preserves the capacity of the complex pathway and stabilizes the multi-pathway system [2510.03833]. The reported schedule is 450K iterations for Stage 1, 150K for Stage 2, and 150K for Stage 3, with Adam, batch size 4, cosine annealing from $1e^{-4}$ to $1e^{-7}$, and the same Y-channel Charbonnier loss [2510.03833].

Empirically, EvEnhancerPlus slightly improves or matches EvEnhancer on several benchmarks. On GoPro at $t=8$, $s=4$, it reports 33.57 dB / 0.9303 SSIM (Center) and 33.39 / 0.9291 (Average), compared with 33.52 / 0.9295 and 33.30 / 0.9279 for EvEnhancer [2510.03833]. On ALPIX-VSR, EvEnhancerPlus reports 42.01 dB / 0.9802, compared with 40.84 / 0.9786 for EvEnhancer [2510.03833]. The extension also reports improved OOD performance at settings such as $t=12, s=6$ and $t=16, s=4$ [2510.03833].

## 7. Positioning, ambiguities, and limitations

Within the event-guided C-STVSR literature, EvEnhancer is positioned against both frame-based continuous models and event-based alternatives. The papers explicitly compare against VideoINR and MoTIF, which decouple spatial and temporal INRs, and against HR-INR, which uses events but still relies on decoupled INR formulations [2505.04657; 2510.03833]. EvEnhancer’s distinct claim is that unified 3D INR with local cross-scale attention and event-guided synthesis offers better OOD generalization and stronger temporal consistency [2505.04657].

The method is also contrasted with two-stage event-based pipelines that combine video frame interpolation and video super-resolution, such as TimeLens, REFID, CBMNet-L, EGVSR, and EvTexture [2505.04657]. The single-stage continuous formulation is presented as avoiding error accumulation and enabling arbitrary querying in both space and time [2505.04657].

There is, however, a nomenclatural ambiguity around the string “EvEnhancer.” The source material explicitly notes that “EvEnhancer” may refer to the system named VEnhancer in the video-generation paper “VEnhancer: Generative Space-Time Enhancement for Video Generation” [2407.07667]. That system is a diffusion-based enhancement framework for AI-generated videos, not an event-guided C-STVSR model. It upscales spatial and temporal resolution jointly while removing artifacts and flicker through a pretrained video diffusion backbone with a video ControlNet [2407.07667]. The same data block also includes an unrelated “Enhancer” module for speaker diarization in AED-EEND [2309.06672]. These usages are technically distinct.

The limitations reported for EvEnhancer itself concern event quality, scene content, and extreme scale extrapolation. Strong event noise or extremely sparse events can degrade modulation and recurrent propagation, making robustness dependent on voxelization quality [2505.04657]. In very static scenes, where event activity is scarce, gains over frame-only methods may shrink [2505.04657]. Performance decreases at very large spatial or temporal scales, although the model remains competitive [2505.04657]. The later extension similarly notes that sparse or noisy events and extreme OOD scales such as $s \gg 12$ or $t \gg 16$ remain challenging despite unified INR and event-guided synthesis [2510.03833].

Future directions named in the papers include explicit temporal consistency losses, adaptive binning, noise-robust event encodings, multi-head attention, stronger robustness to real event noise, and broader exploitation of dynamic routing for efficiency [2505.04657; 2510.03833]. This suggests that EvEnhancer should be understood not as a closed design, but as a family of event-guided continuous reconstruction methods whose main research significance lies in unifying event-aware synthesis with continuous implicit video representations.

Source: https://www.emergentmind.com/topics/evenhancer