ViBE: Visual-to-M/EEG Brain Encoding via Spatio-Temporal VAE and Distribution-Aligned Projection

Abstract: Brain encoding models not only serve to decipher how visual stimuli are transformed into neural responses, but also represent a critical step toward visual prostheses that restore vision for patients with severe vision disorders. Brain encoding involves two fundamental steps: achieving faithful reconstruction of neural responses and establishing cross-modal alignment between visual stimuli and neural responses. To this end, we propose ViBE, a novel brain encoding framework for generating magnetoencephalography (MEG) and electroencephalography (EEG) signals from visual stimuli. Specifically, we first design a spatio-temporal convolutional variational autoencoder (TSC-VAE) that captures the spatio-temporal characteristics of M/EEG signals for effective neural response reconstruction. To bridge the modality gap between visual features and neural representations, we employ Q-Former to map CLIP image embeddings to the TSC-VAE latent space, producing neural proxy embeddings. For comprehensive cross-modal alignment, we combine mean squared error (MSE) loss for point-wise feature matching with sliced Wasserstein distance (SWD) for probability distribution alignment between the neural proxy embeddings and TSC-VAE latent embeddings. We conduct extensive experiments on the THINGS-EEG2 and THINGS-MEG datasets, demonstrating the effectiveness of our approach in generating high-quality M/EEG signals from visual stimuli.

• The paper introduces a dual-stage framework that uses a spatio-temporal VAE and Q-Former to bridge visual and M/EEG signal domains.
• It demonstrates significant performance improvements, achieving subject-specific Pearson correlations of 0.635 for EEG and 0.543 for MEG.
• The method offers actionable insights for individualized BCI and neuroprosthetic applications while effectively addressing neural variability.

Authoritative Summary of “ViBE: Visual-to-M/EEG Brain Encoding via Spatio-Temporal VAE and Distribution-Aligned Projection”

Introduction and Motivation

Visual brain encoding is fundamental for understanding how external visual stimuli are processed into neural activity, contributing both to systems neuroscience and the development of neural prosthetic devices. Recent progress in deep learning has facilitated new models for fMRI-based encoding, but with limited practicality for BCI, due to cost and temporal constraints. Electro- and magnetoencephalography (M/EEG), by contrast, enable rapid, noninvasive neural readout, yet high-fidelity visual-to-M/EEG encoding remains challenging due to the complexity of the spatio-temporal neural dynamics and the significant distributional gap between visual and neural representations.

Existing methods predominantly rely on deterministic mappings (Bao et al., 4 Mar 2025, Mai et al., 14 Aug 2025), which cannot capture the one-to-many mapping structure and neural variability intrinsic to M/EEG responses. Direct alignment between high-level visual embeddings (e.g., from CLIP) and neural latent spaces is further confounded by severe feature scale misalignment, resulting in suboptimal reconstructions and poor cross-modal correspondence.

Methodology

ViBE addresses these limitations with a two-stage cross-modal generative framework, emphasizing hierarchical spatio-temporal modeling and probabilistic alignment. The architecture is structured as follows.

Stage I: The TSC-VAE (Spatio-Temporal Separated Convolutional Variational Autoencoder) is designed to reconstruct M/EEG signals by combining temporal convolutions $(1,\,k_t)$ and spatial convolutions $(k_s,\,1)$ with $k_s < C$, where $C$ is the number of spatial channels. This design (TSConvPlus) preserves hierarchical spatial information, in contrast to methods that collapse spatial dimensions. The VAE objective combines standard MSE reconstruction and a KL bottleneck.

Figure 1: Overview of the two-stage ViBE training procedure, showing the TSC-VAE (Stage I) and cross-modal Q-Former alignment (Stage II).

Stage II: Q-Former receives CLIP image embeddings and uses learnable query tokens with cross-attention to extract visual-semantic features, which are projected into the TSC-VAE latent space (“neural proxy embeddings”). A combined MSE and sliced Wasserstein distance (SWD) loss aligns neural proxy and VAE latent embeddings, aligning both pointwise and distributional structure.

Inference Pipeline

At inference, a novel image is processed by the frozen CLIP encoder, Q-Former, and the TSC-VAE decoder to generate subject-specific or subject-generalized M/EEG signal predictions.

Figure 2: End-to-end ViBE inference: image $\rightarrow$ CLIP embedding $\rightarrow$ Q-Former $\rightarrow$ TSC-VAE Decoder $\rightarrow$ predicted M/EEG signal.

Experimental Results

ViBE is systematically evaluated on THINGS-EEG2 and THINGS-MEG, with extensive baselines, including MindSimulator (Bao et al., 4 Mar 2025), SynBrain (Mai et al., 14 Aug 2025), and previous CLIP-based methods (2509.00787). Three key metrics are compared: MSE (lower is better), Pearson correlation, and cosine similarity.

Strong claims in the results:

• Subject-specific ViBE achieves Pearson 0.635 (EEG2) and 0.543 (MEG), substantially outperforming previous methods (best baseline Pearson: 0.425 EEG2, 0.379 MEG).
• Cross-subject and leave-one-subject-out experiments highlight persistent degradation due to subject variability but substantial gains over the prior state of the art when training with pooled multi-subject data.
• Ablation studies demonstrate that the TSConvPlus convolution scheme is essential; performance collapses if the spatial hierarchy is eliminated (Pearson drops from 0.941 $\rightarrow$ 0.293 (Stage I, EEG2); similar for MEG).

  Figure 3: Ablation results comparing TSConv and TSConvPlus across stages, evidencing significant improvements from preserving spatial hierarchy.

Modality Gap and Cross-Modal Alignment

Statistical analysis of embedding distributions reveals a substantial input-output scale gap: CLIP image features are $\sim$100$(k_s,\,1)$0 smaller in variance than M/EEG latent embeddings. The Q-Former brings the proxy embedding scale within $(k_s,\,1)$12.5$(k_s,\,1)$2 of the neural distribution, dramatically improving alignment and thus cross-modal mapping.

Figure 4: Distributional comparison of CLIP, TSC-VAE, and neural proxy embeddings, highlighting effective scale bridging by Q-Former.

Loss ablations confirm that including the SWD term, even at the cost of slight MSE increases, robustly enhances matched structure as measured by Pearson and cosine scores.

Neuroanatomical Analysis

Brain region ablations substantiate the critical role of occipital and temporal regions (“visual pathway”) in visual signal representation. Selective occlusion of these regions yields the largest performance decrements, in line with known neuroscience principles.

Figure 5: Brain region ablation results: removing occipital/temporal locations degrades signal prediction, affirming their representational importance.

Implications and Future Work

ViBE’s subject-specific and multi-subject protocols set a new reference for high-fidelity, image-conditioned brain encoding in temporally resolved modalities. The explicit separation and bridging of modality scales enables distributional matching that is essential for robust cross-modal generative modeling. Practically, these advances support more accurate, individualized BCI and neuroprosthetics pipelines capable of leveraging high-throughput neural signals, while theoretically, the work motivates deeper alignment modeling (scale, shape, and context). Outstanding questions include scaling to larger, more diverse populations, investigating task generalization, and integrating adaptive normalization layers to further reduce scale mismatch.

Conclusion

ViBE presents a rigorously validated and technically well-motivated approach to visual-to-M/EEG encoding, featuring (i) spatio-temporally aware variational autoencoding, (ii) cross-modal scale bridging via Q-Former, and (iii) joint alignment losses. The method demonstrates strong improvements in signal reconstruction quality across challenging datasets and advances the methodological frontier for brain encoding and prosthetic applications.