MEIcoder: Neural Decoding of Visual Stimuli

• MEIcoder is a neural decoding framework that reconstructs visual stimuli from primary visual cortex signals using neuron-specific Most Exciting Inputs (MEIs).
• It employs a modular architecture with a per-subject readin module and a shared core decoder, leveraging SSIM-based loss and adversarial regularization for high fidelity.
• Empirical results demonstrate state-of-the-art performance, achieving superior reconstruction quality even with small datasets and a limited number of neurons.

MEIcoder is a biologically informed neural decoding framework designed to reconstruct visual stimuli from population recordings in primary visual cortex (V1), particularly under data-scarce, low-neuron-count regimes. The method leverages precomputed neuron-specific Most Exciting Inputs (MEIs), a structural similarity–based loss, and adversarial training. Empirical results establish state-of-the-art fidelity in faithful image reconstruction from neural signals, with pronounced advantages in small datasets and settings with limited neurons or training samples (Sobotka et al., 23 Oct 2025).

1. Decoding Objective and Model Architecture

The core objective is the inversion of the neural encoding process for visual scenes. Given a recorded neural response vector $\mathbf r \in \mathbb{R}^n$ to an unknown stimulus $\mathbf y \in \mathbb{R}^{c \times h \times w}$, MEIcoder learns a decoder $$D_\theta : \mathbb{R}^n \to \mathbb{R}^{c \times h \times w}$$ that outputs a reconstruction $\hat{\mathbf y} = D_\theta(\mathbf r)$.

Data Preprocessing

All images are normalized to grayscale and downsampled to dataset-appropriate resolution (e.g., $36 \times 64$ px for Brainreader, $22 \times 36$ px for SENSORIUM, $20 \times 20$ px for Synthetic Cat V1), then z-scored. Neural responses are temporally averaged over a $\pm 500$ ms window relative to stimulus, and each neuron's activity is normalized by its inverse standard deviation computed on the training set.

Modular Architecture

MEIcoder employs a two-stage modular structure:

• Per-subject Readin Module: $$R^{(s)}_\psi : \mathbb{R}^n \to \mathbb{R}^{d_c \times h \times w}$$ processes subject-specific neural data via learnable neuron embeddings and contextualization.
• Shared Core Decoder: $$C_\phi : \mathbb{R}^{d_c \times h \times w} \to \mathbb{R}^{c \times h \times w}$$ implements a six-layer convolutional neural network translating compressed features into reconstructed images.

Each neuron $\mathbf y \in \mathbb{R}^{c \times h \times w}$0 is assigned a learnable embedding $\mathbf y \in \mathbb{R}^{c \times h \times w}$1. A shallow feed-forward network $\mathbf y \in \mathbb{R}^{c \times h \times w}$2 creates a neuron-context spatial map $\mathbf y \in \mathbb{R}^{c \times h \times w}$3 from $\mathbf y \in \mathbb{R}^{c \times h \times w}$4, which modulates the neuron’s MEI ($\mathbf y \in \mathbb{R}^{c \times h \times w}$5) through pointwise multiplication. Stacking across neurons and compression by $\mathbf y \in \mathbb{R}^{c \times h \times w}$6 convolution yields the hidden representation $\mathbf y \in \mathbb{R}^{c \times h \times w}$7 provided to the core decoder.

2. Most Exciting Inputs (MEIs): Definition and Computation

For each neuron $\mathbf y \in \mathbb{R}^{c \times h \times w}$8, an encoding model $\mathbf y \in \mathbb{R}^{c \times h \times w}$9 is trained to predict neural responses to stimuli. The MEI, $$D_\theta : \mathbb{R}^n \to \mathbb{R}^{c \times h \times w}$$0, is the visual input that maximizes the predicted response of neuron $$D_\theta : \mathbb{R}^n \to \mathbb{R}^{c \times h \times w}$$1, subject to specific normalization:

$$D_\theta : \mathbb{R}^n \to \mathbb{R}^{c \times h \times w}$$2

MEIs are numerically approximated via $$D_\theta : \mathbb{R}^n \to \mathbb{R}^{c \times h \times w}$$3 steps of gradient ascent, with projection onto the admissible set—zero mean, fixed standard deviation, and bounded range—applied at each step.

These MEIs function as neuron-specific spatial templates, encoding the receptive-field selectivity structure of each recorded cell.

3. Loss Functions and Adversarial Training

MEIcoder utilizes a compound loss for training the decoder:

• Structural Similarity (SSIM) Loss: Reconstruction fidelity is encouraged through a negative log–SSIM formulation:

$$D_\theta : \mathbb{R}^n \to \mathbb{R}^{c \times h \times w}$$4

• Adversarial Loss: To enforce naturalistic texture and global statistics, a 5-layer CNN discriminator $$D_\theta : \mathbb{R}^n \to \mathbb{R}^{c \times h \times w}$$5 is trained to distinguish real from reconstructed images. The generator loss balances SSIM and adversarial loss:

$$D_\theta : \mathbb{R}^n \to \mathbb{R}^{c \times h \times w}$$6

with $$D_\theta : \mathbb{R}^n \to \mathbb{R}^{c \times h \times w}$$7, $$D_\theta : \mathbb{R}^n \to \mathbb{R}^{c \times h \times w}$$8, and $$D_\theta : \mathbb{R}^n \to \mathbb{R}^{c \times h \times w}$$9.

• Discriminator Loss: Real and fake labels are regularized via one-sided label-smoothing noise and a balancing parameter $\hat{\mathbf y} = D_\theta(\mathbf r)$0.

4. Datasets and Unified Benchmark

MEIcoder was evaluated on three principal datasets:

• Brainreader (mouse V1): $\hat{\mathbf y} = D_\theta(\mathbf r)$18,587 neurons, $\hat{\mathbf y} = D_\theta(\mathbf r)$2 px, 4,500/500/100 train/val/test samples per mouse.
• SENSORIUM 2022 (mouse V1): 8,372 neurons, $\hat{\mathbf y} = D_\theta(\mathbf r)$3 px, 4,500/500/100 partitioning.
• Synthetic Cat V1: 46,875 neurons (model-based), $\hat{\mathbf y} = D_\theta(\mathbf r)$4 px, 45,000/5,000/250 split.

A comprehensive benchmark aggregates all subjects and datasets into a unified pipeline with approximately 160,000 samples, each subject handled by its individualized readin module and a universally shared core. No explicit image augmentation beyond random ImageNet sampling was performed; neural responses are normalized independently per neuron.

5. Experimental Results and Comparative Performance

Empirical evaluation shows that MEIcoder substantially outperforms five baseline architectures (InvEnc, EGG, MonkeySee, CAE, MindEye2) across multiple criteria:

Dataset	Baseline (SSIM)	MEIcoder (SSIM)	Baseline (Alex(5))	MEIcoder (Alex(5))
Brainreader	0.321	0.400	0.896	0.990

• Qualitative reconstructions capture fine edges and natural image statistics while avoiding texture hallucinations characteristic of generative AI inversion pipelines.
• Data efficiency: MEIcoder surpasses all baselines using as few as 1,000 training samples.
• Neuron scaling: With $\hat{\mathbf y} = D_\theta(\mathbf r)$51,000 neurons, $\hat{\mathbf y} = D_\theta(\mathbf r)$695% two-way AlexNet identification is already achieved; reconstruction quality continues to improve with increasing neuron count in synthetic datasets.

6. Ablation and Component Importance

Targeted ablations demonstrate the primacy of MEIs in decoding performance:

• Substitution of neuron MEIs with uniform maps (“No MEIs”) results in the largest performance degradation (approximately 20% relative drop in SSIM).
• Removal of neuron embeddings or substitution of SSIM with MSE loss produces comparatively smaller effects.
• These observations establish MEIs as the main driver of MEIcoder’s success, validating the hypothesis that encoding receptive field selectivity as a structural prior is crucial under data-scarce conditions.

7. Significance and Impact

The introduction of MEIcoder, with its neuron-specific MEI integration, SSIM-based reconstruction objective, adversarial regularization, and parameter-efficient modular architecture, represents a key advance in neural decoding. High-fidelity reconstructions from shallow datasets or few neuron recordings become feasible, demonstrating the potential for robust brain-machine interface design and early visual processing research.

The public release of the 160,000-sample, multi-dataset benchmark provides a standardized foundation for further studies of neural decoding under heterogeneous conditions. The demonstrated data- and neuron-efficiency, in conjunction with strong baseline outperformance, positions MEIcoder as the current state of the art for reconstructing visual stimuli from neural activity in the early visual system (Sobotka et al., 23 Oct 2025).