NeuroCLIP: CLIP Models for Neural Data

• NeuroCLIP is a design family of models that adapts contrastive CLIP techniques to align neural signals such as EEG, fNIRS, and MRI with image representations.
• It employs modality-specific adapters—like SNN modules, prompt tuning, and cross-attention layers—to bridge the gap between unstructured neural data and frozen visual backbones.
• Empirical validations across event recognition, addiction analysis, and EEG-to-image retrieval highlight its innovative approach despite challenges in generalization and dataset diversity.

NeuroCLIP denotes a set of CLIP-derived models rather than a single canonical architecture. In the current literature, the exact title has been used for at least three distinct systems: a neuromorphic event-recognition framework that transfers 2D CLIP to event-camera data through event-to-frame conversion and an SNN adapter (Guo et al., 2023), a multimodal EEG-fNIRS framework for methamphetamine addiction analysis and rTMS evaluation (Wang et al., 27 Jul 2025), and an EEG-to-image prompt-tuning method for zero-shot retrieval from brain signals (Wang et al., 12 Nov 2025). Closely related work extends the same CLIP-style principle across brain MRI, fMRI decoding, neuro-oncology segmentation, multi-subject semantic decoding, brain histology, and fMRI-to-video reconstruction, so the term also functions as a broader label for neuro- and brain-oriented CLIP adaptation (Avci et al., 23 Jun 2025, Liu et al., 2023, Gong et al., 2024, Zhou et al., 2024, Ta et al., 18 Jan 2026).

1. Terminological scope and lineage

The literature uses “NeuroCLIP” in two overlapping senses. In the narrow sense, it refers to models whose formal title is “NeuroCLIP.” In the broader sense, it refers to CLIP-style methods specialized for neural signals, neuroimaging, or neuroscience data. A common misconception is that NeuroCLIP names one fixed brain-decoding architecture. The published record instead shows multiple non-equivalent systems sharing the name, plus a larger family of adjacent methods that reuse CLIP as a representational pivot in neuroscience (Guo et al., 2023, Wang et al., 27 Jul 2025, Wang et al., 12 Nov 2025).

Model	Primary data	Main objective
NeuroCLIP (Guo et al., 2023)	Event-camera streams	Zero-shot and few-shot neuromorphic recognition
NeuroCLIP (Wang et al., 27 Jul 2025)	Simultaneous EEG and fNIRS	Addiction biomarker learning and rTMS assessment
NeuroCLIP (Wang et al., 12 Nov 2025)	EEG and images	Zero-shot EEG-to-image retrieval

This terminological plurality is not merely nominal. The event-camera model adapts 2D CLIP to asynchronous spikes (Guo et al., 2023); the addiction model replaces image-text pairing with synchronized EEG-fNIRS alignment (Wang et al., 27 Jul 2025); the EEG-image model keeps a CLIP-like visual backbone but introduces neural-aware prompt tuning and a refined contrastive objective (Wang et al., 12 Nov 2025). This suggests that “NeuroCLIP” is best read as a design family defined by modality-bridging into a CLIP-like embedding space, not by one shared network blueprint.

2. NeuroCLIP for neuromorphic event understanding

The earliest paper titled “NeuroCLIP” addresses neuromorphic visual recognition from event-camera data (Guo et al., 2023). Its starting point is the event stream

$$\varepsilon = \{e_i\}_{i=1}^I,\quad e_i=(x_i,y_i,t_i,p_i),$$

where $(x_i,y_i)$ is pixel location, $t_i$ is timestamp, and $p_i\in\{-1,1\}$ is polarity. Because standard CLIP expects frame images rather than asynchronous spikes, the model introduces an event-frame module that divides the stream into $T$ time intervals and converts each interval into a grayscale frame. Pixel intensities are assigned by a ternary rule: 255 for ON events, 0 for OFF events, and 127 otherwise, after which frames are resized to $224\times224$ and passed through a frozen CLIP visual encoder (Guo et al., 2023).

Zero-shot inference averages CLIP similarities across timesteps,

$$p = {\rm SoftMax}\left(\sum \alpha_i W_t f_{vi}\right),\quad \alpha_i=\frac{1}{T},$$

while few-shot adaptation adds an inter-timestep adapter after the CLIP visual encoder. That adapter is a three-layer residual-style SNN built from LIF neurons, with membrane dynamics

$$\bm{u}^{(t+1), \text{pre}} = \tau \bm{u}^{(t)} + \bm{c}^{(t+1)}, \quad \bm{c}^{(t+1)} = \mathbf{W}\bm{x}^{(t+1)},$$

followed by thresholded spike generation and reset (Guo et al., 2023). The architectural point is not to fine-tune CLIP end-to-end, but to keep CLIP frozen and learn a small temporal module that exploits inter-frame dynamics.

Empirically, the method established feasibility rather than domain saturation. On N-MNIST, zero-shot accuracy reached 46.19% and 16-shot accuracy reached 91.51%; on CIFAR10-DVS, the corresponding best numbers were 21.40% and 44.40%; on ES-ImageNet, 8.58% and 14.72% (Guo et al., 2023). Prompt design mattered substantially: on CIFAR10-DVS with RN50, the best zero-shot prompt was “a low quality stick figure of the object [CLASS].,” improving over bare class names. The paper’s significance lies in showing that CLIP’s semantic prior transfers nontrivially to event data, but its limitations are equally explicit: the event representation is deliberately simple, zero-shot performance on ES-ImageNet remains low, and there is no detailed ablation on timestep choice, alternative adapters, or partial CLIP fine-tuning (Guo et al., 2023).

3. NeuroCLIP for multimodal neural signals

In addiction neuroscience, “NeuroCLIP” names a multimodal framework that integrates simultaneously recorded EEG and fNIRS for methamphetamine addiction analysis and rTMS treatment evaluation (Wang et al., 27 Jul 2025). The model is organized as a progressive learning pipeline with three stages: a Signal Contrastive Alignment Network, a Signal Interweave Integrator, and an ROI-Informed Feature Gating unit. EEG and fNIRS are separately encoded with lightweight ResNet-based 1D CNN backbones, L2-normalized, and aligned through a CLIP-style bidirectional cross-entropy over the similarity matrix

$$\mathbf{S} = \alpha \mathbf{X}_f \mathbf{X}_e^\top + \beta,$$

with directional losses $\mathcal{L}_F$ and $(x_i,y_i)$0 averaged into a symmetric contrastive objective (Wang et al., 27 Jul 2025). Fusion then uses cross-attention with EEG as query and fNIRS as key/value, followed by GELU and SwiGLU gating.

The method is explicitly motivated by temporal-spatial complementarity. EEG becomes informative almost immediately after cue onset, whereas fNIRS reaches a meaningful saliency threshold only around 2.8–3.0 s after stimulus onset (Wang et al., 27 Jul 2025). On the main methamphetamine-vs-control task, NeuroCLIP achieved 97.94 ± 1.08% accuracy and 97.08 ± 1.22% sensitivity under 5-fold random-shuffle validation, and 95.63% accuracy and 95.00% sensitivity under leave-one-subject-out evaluation, outperforming EEG-only and fNIRS-only models (Wang et al., 27 Jul 2025). For rTMS evaluation, patient-specific pre/post classification averaged 96.38% accuracy and 97.83% sensitivity, and craving-category decoding achieved 89.74% accuracy, 91.03% precision, 95.65% F1, and 92.31% recall (Wang et al., 27 Jul 2025). The paper frames these results as evidence for a data-driven brain biomarker, but also notes that the cohort is small, all participants are male, and many optimization details are not reported.

A different NeuroCLIP paper targets EEG-to-image alignment on THINGS-EEG2 using a frozen CLIP-ViT visual backbone plus prompt tuning (Wang et al., 12 Nov 2025). Its key innovations are a dual-stream visual embedding pipeline, instance-level visual prompt tuning through cross-attention token fusion, shared-level visual prompt tokens inserted into the Transformer input, and a refined contrastive loss that softens hard positives and negatives. Original and dynamically filtered images are patch-embedded, fused by

$(x_i,y_i)$1

then concatenated with learnable prompt tokens

$(x_i,y_i)$2

before entering the frozen ViT (Wang et al., 12 Nov 2025). The total objective combines standard CLIP loss, a soft-target KL term, and relation-aware regularization: $(x_i,y_i)$3

The main result is strong zero-shot EEG-to-image retrieval: on THINGS-EEG2, NeuroCLIP achieved Top-1 = 63.2% and Top-5 = 90.3% intra-subject, surpassing UBP by +12.3 and +10.6 points, and 17.0% / 40.3% inter-subject, surpassing UBP by +4.6 and +6.9 points (Wang et al., 12 Nov 2025). The best backbone combination was LightProjector with ViT-B/32, and prompt-token sweeps showed best performance around four or five prompt tokens. Channel and temporal analyses were consistent with visual EEG expectations: occipital channels were most informative, and early post-stimulus segments dominated performance (Wang et al., 12 Nov 2025). As in the addiction paper, the broader implication is that NeuroCLIP no longer means only “brain decoded into CLIP space”; it can also mean “CLIP visual representations adapted to noisy neural signals.”

4. NeuroCLIP-style extensions in MRI, fMRI, histology, and neuro-oncology

Beyond the exact title, a substantial adjacent literature applies CLIP-style alignment to neuroimaging and neuroscience data. “MR-CLIP” is a metadata-guided model for brain MRI that aligns 2D MR slices with DICOM-derived text prompts rather than free-form language (Avci et al., 23 Jun 2025). Its central technical move is to replace one-positive-per-sample InfoNCE with grouped supervised contrastive learning over acquisition-derived contrast labels. Starting from 21,660 raw acquisition configurations, metadata grouping collapses the supervisory space to 1,415 unique labels (Avci et al., 23 Jun 2025). On the main benchmark, MR-CLIP with ViT-B/16 achieved 66.0% image-to-text R@1, 78.7% 3D scan-to-text R@1, 90.9% text-to-image R@1, and 82.6% linear classification accuracy, outperforming BiomedCLIP baselines and an InfoNCE-trained ViT-B/16 (Avci et al., 23 Jun 2025). The paper explicitly positions itself as a neuroimaging-oriented analogue of CLIP tailored to brain MRI contrast learning.

In fMRI decoding, “BrainCLIP” maps fMRI patterns into CLIP’s joint image-text space using symmetric contrastive losses against frozen CLIP image and text embeddings (Liu et al., 2023). It supports zero-shot visual category decoding, fMRI-image retrieval, fMRI-text retrieval, and diffusion-guided image generation within one framework. On the GOD dataset, BrainCLIP improved over BraVL on zero-shot category decoding, reaching 18.40 top-1 accuracy with CoOp prompts, versus 13.99 for BraVL (Liu et al., 2023). On NSD, it substantially outperformed ridge-regression baselines in image and text retrieval and achieved 90.8% average two-way semantic identification in its best reconstruction setting (Liu et al., 2023). “NeuroClips,” in turn, extends this line from static images to video by separating a semantics reconstructor from a perception reconstructor and injecting both into a pretrained text-to-video diffusion model (Gong et al., 2024). On cc2017, NeuroClips reached SSIM 0.390 and CLIP-pcc 0.738, versus 0.171 and 0.408 for MinD-Video, corresponding to the reported 128% improvement in SSIM and 81% improvement in spatiotemporal metrics (Gong et al., 2024).

Other variants expand the scope further. “BrainMCLIP” argues that fMRI decoding should align low-/high-level visual areas with intermediate and final CLIP layers rather than only the last layer, and reports competitive reconstruction quality with 71.7% fewer parameters than top VAE-based methods by avoiding a separate VAE pathway (Xia et al., 22 Oct 2025). “CLIP-MUSED” uses CLIP-derived representational similarity matrices and two subject-specific tokens for multi-subject fMRI semantic decoding, reaching mAP .258, AUC .877, and Hamming .030 on NSD (Zhou et al., 2024). “CytoCLIP” adapts CLIP to NISSL-stained developing human brain histology, achieving F1 0.875 for whole-region classification and 0.912 for high-resolution tile classification while also showing limited generalization across gestational ages and sectioning planes (Ta et al., 18 Jan 2026). In neuro-oncology, a CLIP-guided segmentation system based on a 3D U-Net with cross-modal semantic guidance achieved overall Dice 0.8567 on BraTS 2020 and improved enhancing-tumor Dice from 0.7277 to 0.8005, illustrating yet another sense in which CLIP is being specialized for brain data without necessarily being named NeuroCLIP (Zhang, 14 Jul 2025).

5. Shared methodological patterns

Across these papers, several recurring design motifs are visible. First, CLIP is rarely used in neuroscience as an unmodified image-text classifier. Instead, each domain inserts a bridging mechanism that makes the source signal legible to a CLIP-like space: event-frame conversion plus an SNN adapter for event cameras (Guo et al., 2023), bidirectional alignment and cross-attention for EEG-fNIRS (Wang et al., 27 Jul 2025), prompt-tuned frozen ViTs for EEG-image retrieval (Wang et al., 12 Nov 2025), metadata-to-text prompting for MRI (Avci et al., 23 Jun 2025), or fMRI-to-CLIP mapping networks and diffusion priors for image and video reconstruction (Liu et al., 2023, Gong et al., 2024).

Second, supervision often departs from hard, one-positive CLIP training. MR-CLIP groups samples by shared acquisition-derived labels and uses supervised contrastive loss because strict one-to-one alignment is too brittle for MRI contrast representation (Avci et al., 23 Jun 2025). EEG-image NeuroCLIP introduces soft targets derived from intra-modal similarity structure because many EEG-image negatives are only partially negative in a physiological sense (Wang et al., 12 Nov 2025). EEG-fNIRS NeuroCLIP uses synchronized paired trials rather than language labels, so “contrastive learning” becomes alignment between two neural modalities rather than between images and text (Wang et al., 27 Jul 2025). This suggests that NeuroCLIP-style systems increasingly treat CLIP less as a fixed objective and more as a transferable alignment principle.

Third, most systems rely on frozen or lightly adapted pretrained backbones rather than full end-to-end retraining. The event-camera NeuroCLIP freezes CLIP and trains only the inter-timestep SNN adapter (Guo et al., 2023). The EEG-image NeuroCLIP freezes the CLIP visual Transformer and trains prompt tokens, fusion modules, and projection heads (Wang et al., 12 Nov 2025). BrainCLIP freezes CLIP encoders and trains the brain-to-embedding mapper (Liu et al., 2023). This pattern reflects the same practical constraint across modalities: neural datasets are far smaller and noisier than internet-scale image-text corpora.

6. Limitations, misconceptions, and outlook

A persistent misconception is that CLIP transfer automatically yields open-vocabulary neuro-semantic competence. The evidence is mixed. In CytoCLIP, base CLIP and BiomedCLIP zero-shot performance on developmental brain histology was close to zero before domain adaptation (Ta et al., 18 Jan 2026). In MR-CLIP, BiomedCLIP without task adaptation was essentially ineffective, achieving only 1.4% image-to-text R@1 and 2.5% 3D scan-to-text R@1 (Avci et al., 23 Jun 2025). In brain-tumor segmentation, the text side is conceptually important but under-specified, so semantic guidance is beneficial without yet constituting a general-purpose neuro-LLM (Zhang, 14 Jul 2025).

Generalization also remains incomplete. The addiction NeuroCLIP was validated on only 17 MUD and 17 healthy-control participants after exclusions, with all participants male (Wang et al., 27 Jul 2025). EEG-image NeuroCLIP improved inter-subject retrieval but inter-subject performance remained far below intra-subject performance (Wang et al., 12 Nov 2025). MR-CLIP degraded under OASIS distribution shift (Avci et al., 23 Jun 2025). CytoCLIP generalized poorly across sectioning planes and only modestly across gestational ages (Ta et al., 18 Jan 2026). NeuroClips still struggled with scene transitions and depended on a multi-stage pipeline built from several pretrained systems (Gong et al., 2024).

The most plausible synthesis is therefore cautious. NeuroCLIP is not a settled architecture but an evolving research program in which CLIP-like multimodal alignment is repurposed for signals that are not naturally paired with text and are often not even image-like. Some variants use CLIP almost literally; others only preserve the contrastive, shared-embedding philosophy. The direction of travel is clear: more modality-aware bridging, more flexible supervisory structure, and tighter coupling between pretrained semantic priors and the particular statistical structure of neural data. Whether that trajectory converges toward a single, reusable neurofoundation interface remains open, but the current literature already shows that “NeuroCLIP” has become a meaningful organizing label for CLIP-derived modeling across event cameras, EEG, fNIRS, MRI, fMRI, video, and brain histology (Guo et al., 2023, Wang et al., 27 Jul 2025, Wang et al., 12 Nov 2025, Avci et al., 23 Jun 2025, Liu et al., 2023, Gong et al., 2024, Ta et al., 18 Jan 2026).