Perceptogram: Reconstructing Visual Percepts from EEG (2404.01250v2)

Abstract: Visual neural decoding from EEG has improved significantly due to diffusion models that can reconstruct high-quality images from decoded latents. While recent works have focused on relatively complex architectures to achieve good reconstruction performance from EEG, less attention has been paid to the source of this information. In this work, we attempt to discover EEG features that represent perceptual and semantic visual categories, using a simple pipeline. Notably, the high temporal resolution of EEG allows us to go beyond static semantic maps as obtained from fMRI. We show (a) Training a simple linear decoder from EEG to CLIP latent space, followed by a frozen pre-trained diffusion model, is sufficient to decode images with state-of-the-art reconstruction performance. (b) Mapping the decoded latents back to EEG using a linear encoder isolates CLIP-relevant EEG spatiotemporal features. (c) By using other latent spaces representing lower-level image features, we obtain similar time-courses of texture/hue-related information. We thus use our framework, Perceptogram, to probe EEG signals at various levels of the visual information hierarchy.

• The paper presents a two-stage latent diffusion pipeline for reconstructing images from EEG signals, adapting a method previously used for fMRI.
• The study found that EEG-based reconstructions, while less sharp than fMRI, capture significant visual information, particularly for categories like animals and food.
• Future work involves refining model components and exploring reconstruction from naturalistic or video stimuli using EEG.

Image Reconstruction from EEG Using Latent Diffusion: A Technical Overview

The paper presented explores the intersection of neuroimaging and image generation by applying latent diffusion models to reconstruct images from electroencephalography (EEG) data. This approach is inspired by the previously established methodologies in functional magnetic resonance imaging (fMRI), which have demonstrated the viability of generating visually interpretable images from brain activity signals. The primary focus of this paper is to adapt and evaluate the application of diffusion-based image reconstruction in the context of EEG, presenting an intriguing baseline for subsequent research in the domain.

Methodological Approach

The authors employ a two-stage image reconstruction pipeline, initially developed for fMRI, to decode visual stimuli from EEG signals. At the core of the methodology lies the use of a Very Deep Variational Autoencoder (VDVAE) to map EEG signals onto a latent space. Subsequent integration with a Versatile Diffusion model via Contrastive Language-Image Pre-training (CLIP) embeddings enables the generation of image reconstructions that capture both low-level visual features and high-level semantics. The reconstructed images are assessed using a suite of performance metrics, including Pixel-level Correlation (PixCorr), Structural Similarity Index Metric (SSIM), and various layers within AlexNet and InceptionV3, among others. These measures provide a comprehensive evaluation framework, distinguishing between low-level and high-level visual features.

The paper utilizes the THINGS-EEG2 dataset, comprising rapid-serial visual presentation (RSVP) of images, which is advantageous for enhancing signal-to-noise ratio but limited by the brief processing intervals of each stimulus. This introduces challenges when deciphering EEG’s contributions in terms of spatial resolution, a well-known limitation in contrast to the high spatial accuracy typical of fMRI.

Results and Implications

Notably, the paper reveals that image reconstructions from EEG, while not rivaling the quality achieved through fMRI, encapsulate a notable amount of visual information. This is especially true for specific categories such as land animals and food, supporting prior assertions on EEG's sensitivity to these categories due to early and distinct visual-evoked potentials (VEPs). The investigation also proposes improvements, suggesting a possible extension of stimulus duration to tease out later-stage cognitive processing that might enrich image reconstruction fidelity.

The research further extends into ablation studies to discern the contributions of various components (e.g., AutoKL, CLIP-Vision, CLIP-Text) within the reconstruction pipeline. These findings emphasize that a complete integration of these components is necessary for optimal reconstruction performance, indicating future work could benefit from nuanced model refinement.

Future Directions

In discussing future applications, the authors consider broadening the framework to accommodate naturalistic visual stimuli, potentially integrating EEG with technologies like rapid shutter goggles to synchronize the visual input format with that required by the reconstruction model. Additionally, exploring video reconstruction from EEG represents a promising avenue given the advancing capabilities of video generative models and the temporal resolution strengths of EEG.

The paper presents a foundational exploration of EEG-based image reconstruction, offering valuable insights into the cognitive processes associated with visual perception and the possibilities of extending such methodologies to practical brain-computer interface (BCI) applications. Researchers in the field are encouraged to leverage this baseline and explore more sophisticated algorithms and data paradigms to enhance the resolution and applicability of EEG-based visual decoding.