Mind's Eye Paradigm: Neural Decoding & AI

• Mind’s Eye Paradigm is a framework that leverages internal visualization to decode mental imagery and simulate perceptual reasoning in both neural and artificial systems.
• It employs multimodal methodologies, including encoder-decoder models on fMRI/EEG data and simulation-augmented language models, to reconstruct and analyze internal states.
• Current challenges such as low-resolution reconstructions and subject variability drive ongoing research towards more robust, scalable applications in neuroscience and AI.

The Mind’s Eye Paradigm encompasses a diverse set of computational and neuroscientific approaches that operationalize or exploit the human capacity for internal visualization—whether for decoding mental imagery from neural data, grounding reasoning in simulated perception, or improving artificial systems’ spatial understanding via internal generative models. Research adopting this paradigm bridges high-dimensional neuroimaging, artificial neural network architectures, model-based reasoning, and LLM prompting to either reconstruct, leverage, or simulate “seeing with the mind’s eye.”

1. Definitions and Theoretical Foundations

The Mind’s Eye Paradigm refers to frameworks in which internal perceptual representations—whether arising biologically (as in mental imagery and hallucination), or in artificial systems (as imagined or rendered internal states)—are central objects of inference, grounding, or reasoning. Its cognitive basis lies in evidence that humans generate intermediate visual states to enable imagination, spatial reasoning, and problem-solving, as seen in spatial navigation (Tolman, 1948), mental rotation (Shepard & Metzler, 1971), and the workings of the working memory “visuospatial sketchpad” (Wu et al., 2024).

In computational and AI contexts, the paradigm describes architectures or procedures in which a model generates or manipulates internal “projections” of hypotheses or states and uses these as substrates for subsequent decision-making (Berntsen et al., 2016, Liu et al., 2022). In neuroscientific applications, the paradigm encompasses decoding or reconstructing such mind’s-eye content—either during intentional imagery, stimulus-induced hallucinations, or inferring subject experiences from neural activity (Afrasiyabi et al., 2024, Chkhaidze et al., 11 Jul 2025, Seoane et al., 2014).

2. Neuroimaging and Decoding of Mental Imagery

The application of the Mind’s Eye Paradigm in neuroscience focuses on linking high-dimensional neural activity (fMRI, EEG) to internal visual experiences, including both imaginal and hallucinatory content.

Multimodal Encoder–Decoder Mapping

Afrasiyabi et al. advance a three-branch encoder–decoder model that maps fMRI activations, elicited either by video stimuli or text-based emotion prompts, into a shared low-dimensional latent. This model comprises:

• Video encoder–decoder ($g_v, f_v$): 2D UNet CNN encodes frames $x_v$ to embedding $z_v$; decoder reconstructs frames with loss $L_v$.
• Video-stimulated fMRI branch ($g_f, f_f$, $\operatorname{MAP}_{f \rightarrow v}$): 1D-CNN compresses fMRI vectors $x_f$ to $z_f$, a cross-modal map projects $z_f$ onto video space; MSE and cross-modal alignment losses ($L_f$).
• Text-stimulated fMRI branch ($x_v$0, $x_v$1): Same as $x_v$2, but processes fMRI during imagination, aligns with emotion-prototype centroids, and uses distribution-matching cross-entropy ($x_v$3).

Losses are jointly optimized: $x_v$4. Quantitatively, top-1 video retrieval accuracy from fMRI is 45%, and text-elicited fMRI-to-emotion classification is 62% (chance = 10%). Qualitatively, the model plausibly reconstructs the semantic gist and coarse layout of imagined content. Embedding-space visualizations confirm successful latent alignment (Afrasiyabi et al., 2024).

RSVP-ERP BCI Image Reconstruction

Seoane et al. reconstruct user mental images using EEG-based classification of ERPs triggered by the rapid serial presentation (RSVP) of polygon primitives. Each ERP burst presents a target shape among distractors; classifier decisions accumulate the selected primitives onto a canvas, reconstructing the image. Weighted selection accuracy (fraction of visual information) is ~80.5%, though perfect reconstructions occur in only 25% of trials (Seoane et al., 2014).

Phenotyping Individual Imagery

Chkhaidze et al. employ the “Ganzflicker” paradigm to induce hallucinations and collect free-form text reports. NLP topic modeling and vision–LLMs (CLIP, SigLIP) reveal clear stratification by imagery vividness (e.g., strong imagers report faces/scenes, weak imagers produce simple patterns). Vision–language embeddings best preserve these group differences (Spearman ρ = .76), supporting a layered model of imagery in which only high-vividness individuals engage higher-order visual cortices (Chkhaidze et al., 11 Jul 2025).

3. Grounding Reasoning in Simulation and Internal Visualization

The Mind’s Eye Paradigm extends to artificial systems as frameworks that ground reasoning in perceptual simulation.

Simulation-Augmented LLM Reasoning

The “Mind’s Eye” framework of Wang et al. integrates physics simulation into LM reasoning as follows:

1. Text-to-code conversion: LM generates a MuJoCo XML physics scene from a natural-language physics question.
2. Simulation: MuJoCo executes the scenario, outputs quantitative outcomes (e.g., speeds, energies).
3. Prompt augmentation: Results are distilled into textual “hints.”
4. Grounded inference: Foundation LMs receive the question and hint, providing the answer.

Grounded LMs achieve up to +46 pp improvement in few-shot accuracy over pure text baselines (GPT-3 175B: 84.2% vs. 38.2%). Providing mismatched or corrupted simulation outputs negates the advantage, demonstrating the necessity of correct perceptual grounding (Liu et al., 2022).

Visualization-of-Thought Prompting in LLMs

Wu et al. operationalize the paradigm with “Visualization-of-Thought” (VoT) prompting: LLMs interleave chain-of-thought reasoning steps with explicit ASCII-style visualizations, forming an evolving internal sketchpad. On tasks requiring spatial reasoning—navigation, grid-based planning, polyomino tiling—VoT outperforms both chain-of-thought and vision-augmented models (e.g., GPT-4 VoT next-step prediction: 54.68% vs. GPT-4 CoT 47.18%). Only 25% of VoT-generated sketches are perfectly accurate to the true state, but spatial understanding is robust to visualization errors due to self-correction and attention to intermediate state representations (Wu et al., 2024).

4. Internal Projection and Robustness in Artificial Networks

In adversarially robust computer vision, the paradigm manifests as “render-and-verify” architectures:

• Triple-stage Mind’s Eye architecture: Estimator $x_v$5 predicts parameters $x_v$6 from an image $x_v$7 and class $x_v$8; Projector $x_v$9 synthesizes an image $z_v$0; Comparator $z_v$1 judges local similarity between patches of $z_v$2 and $z_v$3, yielding a global similarity score $z_v$4. The system only predicts $z_v$5 if $z_v$6.
• Losses: Separate estimation, projection, and comparison losses are optimized; inference involves maximizing $z_v$7 across classes.
• Performance: Direct classifiers are easily defeated by imperceptible adversarial perturbations (median $z_v$8); Mind’s Eye models withstand >300 FGSM steps and require perceptible distortions ($z_v$9) for attack success (Berntsen et al., 2016).

Ablations establish that generative internal projection and patch-based comparison block gradient-based exploitation, providing the bulk of adversarial improvement.

5. Limitations, Trade-offs, and Open Challenges

Current implementations of the Mind’s Eye Paradigm share several limitations:

• Resolution and fidelity: Autoencoder-based neural reconstructions and BCI sketches remain low-resolution, lacking fine detail (Afrasiyabi et al., 2024, Seoane et al., 2014).
• Intrinsic subject variability: Both neural and behavioral data require extensive per-subject calibration; transfer learning or meta-learning for few-shot mind’s-eye decoding is an active area (Afrasiyabi et al., 2024).
• Task/Domain specificity: Internal projection models often require explicit models (e.g., 3D meshes) or prompt designs tailored to the target domain, limiting scalability (Berntsen et al., 2016, Wu et al., 2024).
• Simulator fidelity and scope: Physics-augmented reasoning frameworks are bounded by the capabilities of current simulators; generalization beyond textbook mechanics awaits broader simulation coverage (Liu et al., 2022).
• Visualization accuracy vs spatial understanding: In LLMs, sketches may be incomplete or partially wrong, but overall spatial reasoning may nevertheless succeed due to redundancy in representation (Wu et al., 2024).

6. Advances and Future Directions

Progress in the Mind’s Eye Paradigm is driving rapid developments across fields:

• Neural decoding: Transition from reconstructing literal perception to modeling “imagination” and internally generated content, using shared latent spaces and distributional alignment (Afrasiyabi et al., 2024).
• Model robustness: Internal generative projection architectures demonstrate that compelling models to “prove” their classification via synthetic rendering confers dramatically increased adversarial resistance (Berntsen et al., 2016).
• Grounded reasoning: Simulation-augmented and visualization-prompted reasoning deliver state-of-the-art results with smaller LLMs, suggesting efficient use of world models and sketch-like representations (Liu et al., 2022, Wu et al., 2024).
• Phenotyping and profiling: Automatic content analysis (topic modeling, NLP-embedding) of hallucination samples enables scalable assessment of individual imagery capacity, pointing toward neurocognitive stratification and personalized interfaces (Chkhaidze et al., 11 Jul 2025).

Ongoing work seeks to:

• Incorporate perceptual or adversarial losses to increase reconstruction sharpness (Afrasiyabi et al., 2024).
• Move from emotion-label prompts to open-form language input and output in neural decoders (Afrasiyabi et al., 2024).
• Develop end-to-end differentiable renderers for adversarially robust vision (Berntsen et al., 2016).
• Extend ASCII-based visualization prompts to 3D and continuous-space reasoning, closing the gap between mental sketches and world models (Wu et al., 2024).
• Employ multimodal fusion (fMRI, MEG, EEG) for temporally resolved neural decoding (Afrasiyabi et al., 2024).

7. Comparative Summary of Mind’s Eye Paradigm Variants

Paradigm Instance	Input/Modality	Output/Task	Core Mechanism
Multimodal fMRI decoder (Afrasiyabi et al., 2024)	Video/Text→fMRI	Image/Video reconstruction of mental content	Shared latent, cross-modal alignment
RSVP-ERP BCI (Seoane et al., 2014)	EEG, polygon RSVP	Image reconstruction	ERP-based selection, canvas accumulation
Ganzflicker hallucination (Chkhaidze et al., 11 Jul 2025)	Flicker-induced	Content profiling of imagery	NLP topic modeling, embedding analysis
Physics-grounded LMs (Liu et al., 2022)	Natural language	Physics Q&A with simulation grounding	Simulation-augmented prompt
VoT prompting in LLMs (Wu et al., 2024)	Language prompt	Spatial reasoning with internal sketches	Interleaved reasoning and visualization
Internal CNN projection (Berntsen et al., 2016)	Image	Adversarially robust object recognition	Estimate–render–compare pipeline

These threads collectively define the contemporary scope and impact of the Mind’s Eye Paradigm across neuroscience, AI, and cognitive modeling.