NeuroImagen: EEG-based Visual Reconstruction

• NeuroImagen is an advanced pipeline that reconstructs visual images from noisy EEG signals by integrating pixel-level and semantic decoding with latent diffusion models.
• The system employs GAN-based saliency map generation and contrastive triplet loss to enhance structural accuracy and semantic robustness in image reconstruction.
• It has significant applications in noninvasive neural decoding, brain-computer interfaces, and cognitive research, paving the way for adaptive neural feedback systems.

NeuroImagen refers to an advanced pipeline for reconstructing perceptual images from neural signals, specifically targeting the reconstruction of visual stimuli from electroencephalography (EEG) recordings. By integrating multi-level perceptual information decoding from EEG with state-of-the-art latent diffusion models, NeuroImagen represents a cross-disciplinary approach at the intersection of neuroscience and artificial intelligence for visual perception decoding (Lan et al., 2023).

1. Pipeline Architecture and Objectives

NeuroImagen is constructed to map noisy, time-series EEG recordings elicited by visual stimuli into high-resolution images replicating the original visual inputs. The architecture consists of two coordinated semantic extraction modules: a pixel-level decoder to estimate saliency maps (capturing color, shape, and spatial details) and a sample-level decoder to extract coarse, semantic information (such as image category or text description). Both outputs are integrated into a pretrained @@@@1@@@@ that performs the core image reconstruction.

The primary aim is to overcome the inherent noise and low spatial resolution of EEG, extracting both fine-grained and global semantic features that, together with generative modeling, support accurate visual stimulus reconstruction.

2. Multi-Level Semantic Information Extraction

The methodology employs a distinct two-line extraction process from EEG data $x \in \mathbb{R}^{C \times T}$:

• Pixel-Level Semantic Decoding: A GAN-based generator $G$ receives features $f_\theta(x)$—learned from the EEG via contrastive representation learning—and a latent vector $z \sim \mathcal{N}(0, 1)$. It generates a saliency map $M_p(x)$ that encodes rough structural and positional information:

$M_p(x) = G(z, f_\theta(x))$

The adversarial loss and mode-seeking regularization stabilizes and diversifies the saliency map output.

• Sample-Level (Semantic) Decoding: Semantic representation $M_s(x)$ is extracted from EEG via a dedicated module, guided by text embeddings (obtained via CLIP) that encode image category or caption information. This ensures semantic robustness across stimulus categories.

Both pipelines are trained with a contrastive triplet loss:

$$L_{triplet} = \max(0, \beta + \| f_\theta(x^a) - f_\theta(x^p) \|_2^2 - \| f_\theta(x^a) - f_\theta(x^n) \|_2^2)$$

where $x^a$, $x^p$, and $x^n$ are anchor, positive, and negative EEG samples, respectively, and $\beta$ is a margin parameter.

3. Latent Diffusion for Visual Image Reconstruction

After extracting pixel-level ($M_p$) and sample-level ($M_s$) semantics, the final reconstruction is accomplished by a latent diffusion model $F$, which is conditioned on both the saliency map and semantic embedding:

$$\hat{y} = F(M_p(x), M_s(x)) = F(G(f_\theta(x), h_{clip}))$$

where $h_{clip}$ represents the CLIP-derived text embedding. During inference, the diffusion process denoises and polishes the initial image, bridging the gap between EEG-based semantic code and photo-realistic image output.

GAN-based training for the saliency module uses the following adversarial losses:

• Discriminator:

$$L_{adv}^D = \max(0, 1 - D(A(y), f_\theta(x))) + \max(0, 1 + D(A(M_p(x)), f_\theta(x)))$$

• Generator:

$L_{adv}^G = -D(A(M_p(x)), f_\theta(x))$

where $A$ denotes normalization and $D$ the discriminator.

An SSIM-based loss ensures that saliency maps are structurally similar to ground-truth images:

$$L_{SSIM} = 1 - \frac{(2 \mu_x \mu_{M_p(x)} + C_1)(2 \sigma_x \sigma_{M_p(x)} + C_2)}{(\mu_x^2 + \mu_{M_p(x)}^2 + C_1)(\sigma_x^2 + \sigma_{M_p(x)}^2 + C_2)}$$

4. Experimental Validation and Performance

NeuroImagen was evaluated using a publicly available EEG-image paired dataset, where EEG was recorded from six subjects viewing 50 ImageNet images from 40 categories. The dataset was divided into 80% training, 10% validation, and 10% testing, with all EEG signals of the same image kept within the same split.

Quantitative metrics reported include:

• Top-1 Classification Accuracy: 85.6%, measured using a pretrained ImageNet classifier to assess semantic alignment between reconstructed and ground-truth images.
• Inception Score (IS): 33.50, notably higher than baseline models (Brain2Image, NeuroVision).
• SSIM: 0.249, providing evidence for improved perceptual similarity when including pixel-level guidance.

Qualitative findings demonstrate that the output images preserve both coarse semantic content and finer perceptual details, and the latent diffusion model can sometimes correct deficiencies in the noisy EEG-derived intermediate representations.

5. Applications and Broader Implications

Potential and actual applications of NeuroImagen include:

• Noninvasive decoding and reconstruction of visual perception for neuroscience research into perceptual and cognitive processes.
• Brain-computer interfaces, enabling communication or feedback mechanisms based on decoded visual experience in populations with limited expressive ability.
• Cognitive science investigations into the mapping between neural states and perceptual representations.
• Possible use in neurofeedback or adaptive/augmented reality systems, where the system adapts stimulus delivery based on decoded brain representation.

The broader significance lies in demonstrating the feasibility of reconstructing highly structured, high-dimensional visual information from noisy, low-dimensional EEG activity by leveraging contemporary advances in generative modeling.

6. Methodological Innovations and Technical Challenges

NeuroImagen introduces several methodological advances tailored to overcoming the challenges of EEG-based image reconstruction:

• Contrastive triplet loss for discriminative EEG representation learning, improving category separation despite high trial-to-trial noise and inter-individual variability.
• Joint use of pixel-level and sample-level information allows the model to recover both fine structure and high-level image semantics.
• Integration with powerful pretrained diffusion models closes the representational gap between EEG and natural images, outperforming prior purely GAN-based or regression-based frameworks.

The approach addresses fundamental limitations of EEG, namely low spatial resolution, nonstationarity, and noisy temporal dynamics, by combining adversarial learning with explicit semantic conditioning and multi-level guidance.

7. Future Directions

The paper suggests that the NeuroImagen pipeline can be further extended by:

• Scaling up the training data (more subjects, categories, and images) to improve generalization.
• Tighter integration with more sophisticated image captioning and text embedding modules (e.g., advanced vision-language transformers) to enhance semantic conditioning robustness.
• Application to more complex visual scenes and naturalistic perception tasks, possibly integrating multimodal (fMRI, MEG) brain data for improved spatial fidelity.
• Enhancing real-time capabilities for closed-loop neural interfaces, given the pipeline’s modular structure and end-to-end learnability.

In summary, NeuroImagen delivers a technically robust, quantitatively validated, and modular framework for visual stimuli reconstruction from human EEG data, setting a precedent for future cross-disciplinary research at the interface of neural signal decoding and generative modeling (Lan et al., 2023).