evTransFER: Event-based Expression Recognition

• evTransFER is a transfer learning-based framework that improves event-based facial expression recognition by pretraining on facial reconstruction tasks.
• It employs the TIE event representation with explicit temporal normalization to capture fine-grained spatio-temporal dynamics from neuromorphic cameras.
• Incorporating an LSTM module for longer-term dynamics, the method achieves up to 93.6% accuracy on the e-CK+ dataset while ensuring real-time performance.

Searching arXiv for the specified paper to ground the article in the published source. evTransFER is a transfer learning-based framework and architecture for event-based facial expression recognition using event-based cameras. It addresses two constraints that are central to neuromorphic facial analysis: the need to exploit fine-grained spatio-temporal dynamics in asynchronous event streams, and the scarcity of labeled event-based expression datasets. Its core design combines a new event representation, TIE, a reconstruction-driven transfer learning strategy in which a U-Net encoder is first trained for event-based facial reconstruction and then reused for expression recognition, and an LSTM module for longer-term temporal modeling. On the event-based facial expression database e-CK+, the framework achieves a 93.6\% recognition rate and improves accuracy by 25.9\% points or more relative to state-of-the-art performance for similar problems (Verschae et al., 5 Aug 2025).

1. Problem setting and conceptual basis

Event-based cameras, also called DVS or neuromorphic cameras, do not produce image frames at a fixed frame rate. Instead, each pixel independently triggers an event whenever the log-brightness changes by more than a threshold. This yields an asynchronous event stream with microsecond timestamps, high temporal resolution, and high dynamic range. For facial expression recognition, these properties are particularly relevant because expression formation depends on onset, offset, micro-motions, and temporal transitions that may be subtle in conventional frame-based representations (Verschae et al., 5 Aug 2025).

In this setting, the sensor output is not a dense image sequence but a sparse spatio-temporal signal that can be described as a stream of events in $(x,y,t,[p])$. The event generation model is written as

$L(u_k,t) = \log I(u_k,t)$

and an event is triggered when

$L(u_k, t_k) - L(u_k, t_k - \Delta t_k) = p_k C,$

where $p_k \in \{-1,+1\}$ is the polarity and $\Delta t_k$ is the time since the last event at the same pixel. Over an interval, the camera outputs

$$\mathcal{E}(t_N) = \{ e_k \}_{k=1}^{N} = \{(u_k, t_k, p_k)\}_{k=1}^{N}.$$

evTransFER was introduced because deep models trained from scratch on event data tend to overfit or fail to capture temporal structure when only small event-based facial expression datasets are available. Its design premise is that a representation learned for event-based facial reconstruction can encode rich facial spatio-temporal dynamics and then be transferred to the downstream recognition task. A plausible implication is that the method treats reconstruction as a pretext task for learning event-native facial features under limited supervision.

2. TIE: event representation and temporal normalization

A central component of evTransFER is TIE (Temporal/Time Information of Events), a 3-channel image-like representation derived from the Event Spike Tensor (EST). EST maps events into a spatio-temporal voxel grid,

$$S_{\pm}[x_l,y_m,t_n] = \sum_{e_k \in \mathcal{E}_\pm} f_{\pm}(x_k,y_k,t_k)\, h(x_l - x_k, y_m - y_k, t_n - t_k),$$

preserving spatial, temporal, and polarity information. However, EST is a multi-channel tensor and its performance depends strongly on the time encoding and normalization strategy (Verschae et al., 5 Aug 2025).

evTransFER refines EST through explicit temporal normalization. For a sub-sequence whose first and last timestamps are $t_1$ and $t_N$, it defines

$$\tau_k = \frac{t_k}{t_N}, \qquad \hat{\tau}_k = \frac{t_k - t_1}{t_N - t_1}.$$

These normalized variables are inserted into the measurement function and kernel: $L(u_k,t) = \log I(u_k,t)$0 with corresponding kernel forms

$L(u_k,t) = \log I(u_k,t)$1

Four combinations are evaluated: $L(u_k,t) = \log I(u_k,t)$2, $L(u_k,t) = \log I(u_k,t)$3, $L(u_k,t) = \log I(u_k,t)$4, and $L(u_k,t) = \log I(u_k,t)$5. Empirically, $L(u_k,t) = \log I(u_k,t)$6 performs best in almost all settings.

TIE then compresses the EST tensor $L(u_k,t) = \log I(u_k,t)$7 into a 3-channel representation so that standard RGB CNNs can be used. The tensor is reshaped into a 4D form, summed along one dimension to obtain $L(u_k,t) = \log I(u_k,t)$8, normalized per sample using the $L(u_k,t) = \log I(u_k,t)$9 and $L(u_k, t_k) - L(u_k, t_k - \Delta t_k) = p_k C,$0 percentiles,

$L(u_k, t_k) - L(u_k, t_k - \Delta t_k) = p_k C,$1

and then linearly mapped from $L(u_k, t_k) - L(u_k, t_k - \Delta t_k) = p_k C,$2 to $L(u_k, t_k) - L(u_k, t_k - \Delta t_k) = p_k C,$3 with clipping at those percentiles. The result is an 8-bit 3-channel image. In effect, TIE preserves where events occur, encodes when they occur within the sub-window via normalized time, and projects multi-bin temporal information into a format directly consumable by existing CNN backbones. The reported ablations indicate that switching from raw EST to TIE yields substantial gains.

3. Reconstruction-based transfer learning

The defining methodological contribution of evTransFER is its transfer learning strategy. Instead of learning facial expression recognition directly from scarce event labels, it first trains a conditional GAN for facial reconstruction from events, then transfers the trained encoder into the recognition network (Verschae et al., 5 Aug 2025).

The reconstruction model takes as input a TIE image $L(u_k, t_k) - L(u_k, t_k - \Delta t_k) = p_k C,$4 and targets the corresponding frame $L(u_k, t_k) - L(u_k, t_k - \Delta t_k) = p_k C,$5 from CK+. The generator $L(u_k, t_k) - L(u_k, t_k - \Delta t_k) = p_k C,$6 is a pix2pix-style U-Net encoder-decoder. The encoder consists of downsampling convolutional blocks with LeakyReLU and batch normalization; the decoder uses upsampling, skip connections, ReLU, and dropout $L(u_k, t_k) - L(u_k, t_k - \Delta t_k) = p_k C,$7; the final activation is Tanh; and the weights are initialized from $L(u_k, t_k) - L(u_k, t_k - \Delta t_k) = p_k C,$8. The discriminator $L(u_k, t_k) - L(u_k, t_k - \Delta t_k) = p_k C,$9 is a PatchGAN classifier operating on real or generated image pairs, with LeakyReLU in hidden layers and Sigmoid in the last layer.

The adversarial objective is

$p_k \in \{-1,+1\}$0

with the usual min-max formulation

$p_k \in \{-1,+1\}$1

The paper reports Binary Cross-Entropy as the optimization loss. Reconstruction is trained for 100 epochs with Adam and learning rate $p_k \in \{-1,+1\}$2. After roughly 50 epochs, good qualitative reconstructions are reported, although PSNR and SSIM are not emphasized because reconstruction serves only as a pretext task.

Once reconstruction training is complete, the U-Net encoder is copied into the expression recognition pipeline as a feature extractor. Three strategies are compared: training from scratch; transferring the encoder and freezing it; and transferring followed by end-to-end fine-tuning. The full evTransFER method uses the third strategy, transfer + fine-tune, and this yields the strongest performance. The interpretation given in the paper is that reconstruction compels the encoder to learn facial structure and expression-relevant spatio-temporal features from event dynamics, while the recognition stage specializes these features to the seven expression classes.

4. Recognition architecture and temporal modeling

The expression recognition architecture processes event streams in a sequence of temporally localized steps. For each labeled expression sequence, the event stream between two consecutive frames is used. In CK+ videos, this interval is approximately $p_k \in \{-1,+1\}$3 ms at 30 fps. Each interval is subdivided into $p_k \in \{-1,+1\}$4 equal portions; for the LSTM experiments, the framework uses $p_k \in \{-1,+1\}$5. A TIE image is generated for each sub-stream, typically with the best-performing normalization variant $p_k \in \{-1,+1\}$6 (Verschae et al., 5 Aug 2025).

Each TIE image is passed through the transferred U-Net encoder, yielding a feature representation. In the temporal variant, these per-slice features are stacked and sent to an LSTM described in the paper as having three LSTM units. The LSTM output is then fed to fully connected layers and a final softmax over the seven CK+ expressions: anger, contempt, disgust, fear, happiness, sadness, and surprise. The classification stage uses cross-entropy loss. In compact form, if $p_k \in \{-1,+1\}$7 is the TIE image for sub-stream $p_k \in \{-1,+1\}$8, the pipeline is

$p_k \in \{-1,+1\}$9

$\Delta t_k$0

$\Delta t_k$1

with loss

$\Delta t_k$2

This architecture combines three levels of temporal handling. First, the event representation itself contains normalized temporal information. Second, the encoder is pretrained on a reconstruction task that depends on event dynamics. Third, the LSTM explicitly models longer-term temporal development across consecutive TIE slices. The ablations show that these components are complementary rather than redundant.

5. Dataset, training protocol, and empirical results

The evaluation is conducted on e-CK+, the event counterpart of the CK+ dataset. e-CK+ is generated synthetically from CK+ frames using the V2E event emulator. The original CK+ videos are $\Delta t_k$3 at 30 fps; the emulated device is DAVIS346 at 346\times260 resolution; the event threshold is 0.15 for positive and negative events with sigma threshold variation 0.03; the DVS timestamp resolution is 1 ms; exposure is 5 ms; and cutoff frequency is 30 Hz. Only the 327 labeled sequences are used, with an 80\% train / 20\% validation split per class. Faces are cropped to a $\Delta t_k$4 aspect ratio for both reconstruction and classification (Verschae et al., 5 Aug 2025).

Recognition experiments use Adam, learning rate $\Delta t_k$5, cross-entropy loss, and ExponentialLR with decay $\Delta t_k$6 of 0.1 or 0.5 depending on method. For evTransFER, the reported setting is 30 epochs with batch size 32. The reconstruction phase precedes recognition and uses all e-CK+ sequences.

The principal quantitative comparisons on e-CK+ are as follows:

Method	Setting	Top-1 accuracy
Asynet CNN (SSR + VGG-16)	Without LSTM	67.2%
Asynet SSC (SSR + VGG-16)	Without LSTM	67.5%
EST + CNN (ResNet-34)	Without LSTM	70.9%
ViT (TIE, vit-b16)	Without LSTM	79.0%
evTransFER	Without LSTM	92.4%
EST + CNN + LSTM	With LSTM	83.5%
ViT + LSTM	With LSTM	78.9%
evTransFER + LSTM	With LSTM	93.6%

The ablation results identify the best TIE normalization as $\Delta t_k$7. Under transfer + fine-tuning with the U-Net encoder, the four TIE variants yield 91.9\%, 90.5\%, 92.4\%, and 92.0\% without LSTM, and 93.0\%, 89.2\%, 93.6\%, and 93.1\% with LSTM, respectively. For the best TIE variant, the effect of transfer learning is also explicit: without LSTM, 72.1\% for ResNet-34 from scratch on TIE, 86.6\% for transferred-and-frozen U-Net encoder, and 92.4\% for transferred + fine-tuned evTransFER; with LSTM, the corresponding accuracies are 75.2\%, 88.7\%, and 93.6\%. The paper summarizes this as approximately +15–17 points from transfer even without fine-tuning, another ~6 points from fine-tuning, and ~3–4 points from adding the LSTM.

The comparison to prior methods is similarly explicit. evTransFER + LSTM improves over the best Asynet variant by 26.1 points $\Delta t_k$8, over EST without LSTM by 22.7 points $\Delta t_k$9, and over EST + CNN + LSTM by 10.1 points $$\mathcal{E}(t_N) = \{ e_k \}_{k=1}^{N} = \{(u_k, t_k, p_k)\}_{k=1}^{N}.$$0. The reported overall gain from “TIE + ResNet-34 scratch, no LSTM” to “TIE + U-Net encoder transferred + LSTM” is +21.5 absolute percentage points. Confusion matrices further show that without transfer, “contempt”, “sadness”, and “fear” are often confused, while the transfer + fine-tune + LSTM configuration yields four classes at 100\% accuracy, with “disgust” 93\% and significant improvement for “contempt” and “sadness”.

Inference time is reported on a GeForce GTX 1050. The measured latencies are 2.09 ms for EST without LSTM, 23.4 ms for Asynet, 3.71 ms for evTransFER without LSTM, and 24.69 ms for evTransFER + LSTM. Given the 33 ms time window, the paper concludes that evTransFER can run in real time, both without LSTM and with LSTM.

6. Interpretation, limitations, and research directions

The paper attributes evTransFER’s performance to three interacting factors. First, reconstruction-based pretraining forces the encoder to model face structure and expression from event dynamics, producing features that transfer effectively to recognition. Second, TIE provides a better temporal encoding than raw EST through explicit normalization choices that strongly affect downstream accuracy. Third, the LSTM captures expression evolution over tens or hundreds of milliseconds, helping disambiguate classes with similar static appearance but different temporal signatures, such as fear and surprise (Verschae et al., 5 Aug 2025).

The framework also has clearly stated limitations. e-CK+ is based on emulated events generated with V2E rather than recordings from a real event sensor, so the training and evaluation distribution may not fully reflect real sensor noise, bias, and latency. The reconstruction pretext task requires paired frames and events, which in practice implies either a DAVIS-style sensor, a carefully calibrated beam-splitter setup, or simulated data. The source dataset is relatively small and controlled, with frontal faces and laboratory lighting, so generalization to in-the-wild neuromorphic facial expression recognition remains untested. Even at 93.6\% overall accuracy, “contempt” and “sadness” remain comparatively difficult classes.

The future directions listed in the paper follow directly from these constraints. They include evaluation on real event-based facial expression datasets and on mixed simulated-plus-real settings; extension of the transfer approach to other face tasks, including detection, landmarking, pose estimation, and 2D/3D reconstruction; exploration of multi-modal configurations such as RGB plus events and larger backbones; refinement of event representations beyond TIE; and application of reconstruction-based transfer to other event-based domains in which spatio-temporal dynamics are central, including gestures, human actions, and driving. This suggests a broader methodological point: in event-based vision, reconstruction pretraining can function as a mechanism for learning transferable latent structure when supervised datasets are small.