EmoNeuroDB: Multimodal Emotion Data Resource

Updated 6 July 2026

EmoNeuroDB is an emotion-centered neurophysiological data resource integrating EEG, facial video, and peripheral signals for diverse emotion labeling.
It encompasses multiple instantiations—including dense-array and VR-based databases—to support both offline analysis and realtime affective monitoring.
Advanced methodologies like MESNP spatial filters and CNN–LSTM models highlight its role in mapping neural, behavioral, and affective responses.

EmoNeuroDB denotes an emotion-centered neurophysiological data resource and, in several papers, a broader database-backed framework for organizing EEG and related modalities around emotion labels, spatial topology, and model outputs. In one usage, it is a shorthand for a new EEG-based multimodal database coupling dense-array EEG with facial video and action units under posed and authentic conditions; in another, it is an explicit EEG dataset collected during a VR-based emotional mimicry task; and in systems-oriented work it denotes a schema for storing PLV brain networks, MESNP filters, time-stamped multimodal streams, and arousal–valence trajectories (Li et al., 2022, Freire-Obregón et al., 16 Jul 2025, Li et al., 2020, Herbuela et al., 13 Aug 2025). This suggests a family of closely related resources centered on emotion-specific neural and behavioral representations rather than a single universally standardized release.

1. Scope and concrete instantiations

One line of work states that the authors do not use the name EmoNeuroDB and instead describe their resource as “a new EEG-based multi-modal emotion database,” but the same synthesis treats it as precisely the kind of neurophysiological emotion database that the name suggests (Li et al., 2022). A later paper uses EmoNeuroDB explicitly as the dataset name for a VR-based EEG corpus used in the FG2024 “Brain Responses to Emotional Avatars” challenge (Freire-Obregón et al., 16 Jul 2025). Systems papers then extend the term toward database design, proposing that an EmoNeuroDB should store raw streams, connectivity matrices, learned filters, classifier metadata, and time-aligned affect trajectories (Li et al., 2020, Herbuela et al., 13 Aug 2025).

Instantiation	Core modalities and protocol	Source
EEG-based multimodal database	128-channel EEG at 1000 Hz, frontal face video at 24 fps, posed expressions, posed AUs, spontaneous meditation and pain, 29 participants, 2,320 trials	(Li et al., 2022)
VR-based EmoNeuroDB	DSI-24 wireless EEG, 21 dry electrodes, 300 Hz, 40 adults, six emotions, three repetitions, ~15 s trials	(Freire-Obregón et al., 16 Jul 2025)
Realtime multimodal estimation system	EEG Fp1–Fp2, ECG, BVP, GSR/EDA, facial expression, speech, BioSignalsPlux at 100 Hz, modality-specific arousal–valence outputs	(Herbuela et al., 13 Aug 2025)

These instantiations share a common organizing principle: emotion is modeled through synchronized neural, physiological, and behavioral traces, and the database is expected to preserve enough structural information to support both inference and interpretation.

2. Modalities, protocols, and label spaces

The most detailed multimodal database formulation records 29 participants, 22 male and 7 female, ages 18–38, across three sessions. Session 1 captures posed mimicry of six basic expressions—Anger, Disgust, Fear, Happiness, Sadness, and Surprise—plus neutral. Session 2 records explicitly posed facial actions and labels trials with AU1, AU2, AU4, AU5, AU9, AU12, AU15, AU17, AU25, and AU27. Session 3 records spontaneous or authentic states, contrasting meditation with cold-pressor pain, and includes a 0–10 self-report of physical pain (Li et al., 2022). This configuration makes EmoNeuroDB simultaneously an emotion database, an AU database, and a synchronized EEG–face corpus.

A complementary realtime system broadens the modality set to facial expression, speech, EEG, ECG, BVP, and GSR/EDA, and represents affect in Russell’s Circumplex Model of Emotion. For modality $m$ at time $t$ , the state is encoded as $(a_t^{(m)}, v_t^{(m)}) \in [-1,1] \times [-1,1]$ , and intensity is defined as $I_t^{(m)} = \sqrt{(a_t^{(m)})^2 + (v_t^{(m)})^2}$ (Herbuela et al., 13 Aug 2025). The same system also stores discrete mappings: 34 labels for speech and physiological signals and 38 facial labels from the Affects38 set.

A third multimodal benchmark, built on MAHNOB-HCI, uses EEG, ECG, GSR, and eye data for two separate 3-class tasks: valence as unpleasant, neutral, and pleasant, and arousal as calm, medium aroused, and excited. Its EEG encoder uses 10 channels—F3, F4, F5, F6, F7, F8, T7, T8, P7, and P8—while ECG, GSR, and eye streams are handled as separate modalities (Lopez et al., 2024). Taken together, these protocols show that EmoNeuroDB is not confined to a single label formalism: it supports discrete basic emotions, dimensional valence–arousal, AU identities, pain vs neutral contrasts, and model-derived continuous trajectories.

3. Data models and representational layers

Topology-centered designs treat each EEG segment as a brain network rather than a channelwise feature vector. In the MESNP framework, each 10 s segment is transformed into a $32 \times 32$ weighted adjacency matrix $\text{Plv}$ , with 32 EEG channels as nodes and PLV values as weighted undirected edges. For each emotion class pair $(i,j)$ , supervised spatial filters $W_{ij}$ are learned from covariance matrices $\phi_i$ and $\phi_j$ by solving a generalized eigenvalue problem, and features are extracted as

$t$ 0

The same work recommends database entities such as NetworkSegment, MESNPFilter, MESNPFeature, and ClassifierModel, alongside storage of graph metrics $t$ 1, $t$ 2, $t$ 3, and $t$ 4 (Li et al., 2020).

A different representational stack organizes multimodal streams as time-indexed records. The realtime multimodal framework suggests storing every stream as a time-stamped sequence with fields such as signal_id, session_id, participant_id, modality, device, channel, sampling_rate, units, and raw_samples, and then materializing higher-level entities including Participants, Sessions, SignalStreams, RawSamples, Features, Models, EmotionEstimates, UserModels, and Events (Herbuela et al., 13 Aug 2025). At the query layer, a TimeSeriesEmotion view can expose arousal, valence, intensity, quadrant, category_label, and model_id for each time interval.

A further extension maps CNS and PNS signals into 3D spatial-functional tensors. That approach criticizes simple base-mean subtraction because it creates trial-specific markers and instead proposes sigmoid baseline filtering in the frequency domain, with $t$ 5 and $t$ 6, then embeds EEG surface positions and inward-mapped PNS locations into a 3D cuboid series for 4D-CNN processing (Cen et al., 2022). This suggests that an EmoNeuroDB schema may need to store not only raw signals and tabular features, but also topographic images, connectivity matrices, and volumetric tensors.

4. Learning paradigms and benchmark performance

Several model families have been built around EmoNeuroDB-style representations. The MESNP hierarchy learns supervised spatial filters directly on adjacency matrices and couples them to pairwise SVMs with one-vs-one voting. On MAHNOB-HCI it reports off-line accuracy up to $t$ 7 for 3 classes in theta, and on DEAP it reports $t$ 8 for 4 classes in alpha; in simulated on-line experiments it reports 100% for 3-class MAHNOB and 99.22% for 4-class DEAP in the best bands (Li et al., 2020).

Cross-subject modeling is substantially harder. The DEPL framework uses dynamic differential entropy, 9×9 topographic mapping, and a CNN with squeeze–excitation blocks under leave-one-subject-out evaluation. Its reported accuracies are 66.23% for valence and 68.50% for arousal on DEAP, and 70.25% and 73.27% on MAHNOB-HCI, with gamma as the strongest band (Zhong et al., 2020). A separate hypercomplex architecture, H2, uses PHC encoders for intra-modal relations and PHM fusion for inter-modal relations on MAHNOB-HCI, reaching F1 0.557 and accuracy 56.91 for arousal, and F1 0.685 and accuracy 67.87 for valence (Lopez et al., 2024).

Other models extend the notion of EmoNeuroDB beyond scalp EEG classification. Emo-Net combines confident learning with deep learning for invasive amygdala recordings in behaving monkeys and shows that data-centric label cleaning can dominate architectural sophistication; on a clean independent test set, MLP with Emo-P reaches 92.02% on the human/monkey face dataset, 86.81% on the spatial-frequency dataset, and 82.68% on the monkey-emotion dataset (Wu et al., 2023). AVF-BEL models multimodal emotion generation rather than recognition, aligning a modified CORnet-Z visual module, a simplified primary auditory cortex model, an anterior STG/STS-like fusion module, and an amygdala/OFC BEL module, and reports precision 0.79, recall 0.77, F1 0.78, and average similarity 77.69% for AVF-BEL (Wang et al., 21 Feb 2025). In audio-only work, Emo-CNN reaches 90.2% categorical accuracy on Emo-DB and then projects learned 64-dimensional emotion embeddings into a three-component PCA space interpreted as dopamine, noradrenaline, and serotonin, using geometric proximity in Lövheim’s cube to infer stress without stress labels (Deshmukh et al., 2020).

5. Interpretability and neuroanatomical alignment

The most explicit EmoNeuroDB paper uses a VR-based emotional mimicry task with 40 adults, a DSI-24 wireless EEG system, 21 dry electrodes, and six avatar emotions—joy, sadness, anger, fear, disgust, and surprise—each presented three times in trials of about 15 s. A dual-stream CNN–LSTM separates left and right hemisphere spectra, concatenates the branch outputs, and reaches 22.78% validation accuracy, exceeding SVM at 17.78%, LightGBM at 19.44%, and Random Forests at 19.44 (Freire-Obregón et al., 16 Jul 2025).

Its central contribution is post-hoc interpretability. LIME is extended to structured bi-hemispheric inputs by flattening left and right hemisphere tensors, defining a custom predictor that reshapes them back into the original dual-stream format, generating 5000 perturbations, and aggregating feature weights into per-channel relevance maps. The resulting explanations show emotion-specific asymmetries: joy is dominated by right frontal and temporal electrodes F8, F4, and T4; sadness shows high influences at Fp1, O1, F3, T5, and Fz; anger peaks at T6, P4, and O2; and symmetric-pair correlations indicate relatively bilateral frontal and parietal engagement for joy and surprise but more lateralized patterns for fear and disgust (Freire-Obregón et al., 16 Jul 2025). EmoNeuroDB, in this usage, is therefore not merely a storage layer; it is the empirical substrate for hemispheric affective neuroscience.

A different interpretability strategy is built into the model itself. AVF-BEL maps visual processing to V1, V2, V4, and IT, auditory processing to a primary auditory cortex model with G-PYR, G-PV, and G-SOM populations, multimodal integration to anterior STG/STS, and emotional learning to amygdala and OFC. Sparse weight heatmaps over $t$ 9, $(a_t^{(m)}, v_t^{(m)}) \in [-1,1] \times [-1,1]$ 0, and $(a_t^{(m)}, v_t^{(m)}) \in [-1,1] \times [-1,1]$ 1 are used to trace how sensory features drive the final Emotional Positivity Parameter (Wang et al., 21 Feb 2025). A still broader extension maps 1536-dimensional text embeddings through PCA and $(a_t^{(m)}, v_t^{(m)}) \in [-1,1] \times [-1,1]$ 2-means onto 18 emotion-related brain regions, enabling region-level comparisons such as reduced left insula and raphe nuclei engagement in depressed subjects and broad human–LLM differences across amygdala, ACC, insula, OFC, hippocampus, and VTA (Vos et al., 12 Aug 2025). This suggests that EmoNeuroDB can also function as a neuroanatomically annotated representation layer spanning language and physiology.

6. Operational architectures, limitations, and future directions

Operationally, EmoNeuroDB spans both persistent databases and realtime local systems. The multimodal estimation demo deliberately uses local-only, storage-free processing, with neurophysiological streams entering through OpenSignals and LSL, FocusCalm EEG arriving by Wi-Fi/UDP, and a Python controller updating a shared 2D arousal–valence interface in realtime (Herbuela et al., 13 Aug 2025). By contrast, a related cloud–edge EEG framework, EMAP, builds a MongoDB-backed mega-database of 1000-sample EEG segments, performs cloud cross-correlation search, returns the top 100 analogous signals to the edge, and reports mean prediction accuracies of 94% for seizure, 73% for encephalopathy, and 79% for stroke (Prabakaran et al., 2020). A plausible implication is that database-backed emotion systems can adopt similar cloud–edge partitioning for low-latency affective monitoring, even though EMAP itself targets neurological anomalies rather than emotion.

Several limitations recur. MESNP’s strongest results are explicitly subject-dependent and may not transfer to cross-subject or cross-dataset settings (Li et al., 2020). The H2 benchmark uses an 80/20 stratified random split over all samples rather than a strict subject-independent split (Lopez et al., 2024). The EEG–face corpus has 29 subjects and skewed demographics, and its spontaneous conditions are limited to pain and meditation (Li et al., 2022). Disorder-oriented emotional BCI work based on DENS uses only four Muse-compatible channels and simulates disordered EEG by adding zero-mean Gaussian noise to increase the complexity index from 24.06 to 29.01, while explicitly acknowledging the lack of real patient data (Mehta, 2024). Privacy is also central: the realtime multimodal demo avoids persistent storage precisely because emotion and neuro data are particularly sensitive, especially for neurodivergent populations (Herbuela et al., 13 Aug 2025).

Future directions are correspondingly structural. Proposed extensions include cross-subject and cross-dataset MESNP, online adaptive filters, richer multimodal fusion with peripheral physiology, facial video, and fNIRS, uncertainty-aware outputs, subject-specific calibration, longitudinal profiles, and stronger visualization of learned neural signatures (Li et al., 2020, Herbuela et al., 13 Aug 2025). In database terms, EmoNeuroDB points toward a layered resource in which raw signals, engineered features, learned representations, interpretability artifacts, and consent-governed metadata are all first-class objects.