- The paper presents a unified framework that integrates vision, language, and neural data via discrete diffusion, enabling multi-task synergy across encoding and decoding tasks.
- It introduces a Brain Tokenizer with MNI152 registration and VQ-VAE design, converting continuous fMRI signals into discrete tokens for robust cross-modal alignment.
- Empirical evaluations show state-of-the-art performance in visual reconstruction, language reasoning, and neural encoding, demonstrating emergent synergistic effects.
Mind-Omni: Unified Tri-Modal Brain-Vision-Language Modeling via Discrete Diffusion
Motivation and Context
The modeling of interactions between external stimuli (images, text) and internal neural representations is central in neural decoding, neural encoding, and foundation modeling for BCIs. Existing approaches largely rely on highly specialized models, each optimized for narrow tasks and specific directions (decoding or encoding). This paradigm restricts model versatility and precludes investigation into inter-task synergies, crucial for developing comprehensive neural foundation models. Mind-Omni introduces a unified solution, simultaneously leveraging vision, language, and neural modalities within a discrete diffusion modeling framework.
Framework and Architectural Innovations
Mind-Omni addresses the challenges of input heterogeneity, modality disparity, and task interdependence through three methodological innovations:
Standardized Neural Representation: Mind-Omni employs MNI152 registration to normalize fMRI data across subjects, eliminating the need for subject-dependent modeling and scaling multi-subject integration.
Brain Tokenizer: Mind-Omni introduces a VQ-VAE-style Brain Tokenizer that converts continuous fMRI signals into discrete semantic tokens. This tokenizer is supervised not only by reconstruction/error minimization, but also by multi-level alignment: (1) coarse semantic alignment (InfoNCE, feature distillation) pulls brain tokens into CLIP based vision-language embedding space; (2) fine-grained alignment achieves token-level semantic congruence via masked token prediction; (3) perceptual alignment ensures reconstructed brain signals are decodeable to actual visual/text CLIP features.
Unified Discrete Diffusion Modeling: Built on Muddit backbone, the tri-modal DiT architecture allows synchronous masked prediction across multiple modalities, freeing generation from autoregressive causal structure and enabling permutation invariance. All seven encoding/decoding tasks are reformulated as masked token prediction under shared continuous-time negative ELBO objectives.
Unified Multi-task Synergy
Mind-Omni unifies seven encoding and decoding tasks:
- I→B, T→B, I{content}T→B (encoding)
- B→I, B→T, B→I{content}T, BQA (decoding/question answering)
Each is realized via specific task conditioning and target masking in the DiT framework. The architecture is trained progressively: initial alignment and joint encoding/decoding, followed by multi-task fine-tuning and instruction tuning on a curated BQA dataset using MLLM-generated captions and visual Q&A.
Quantitative and Qualitative Evaluation
Mind-Omni is empirically benchmarked against specialist and prior unified models (e.g., BraVL, MoPoE, UMBRAE, MindSimulator). Key findings include:
- Multi-task Versatility: Mind-Omni, with ~442M parameters, achieves competitive SOTA across all seven tasks, demonstrating superior task coverage and multi-task transfer over larger specialist models.
- Visual Reconstruction: On visual decoding (B→I, B→I{content}T) metrics, Mind-Omni achieves high SSIM, PixCorr, and semantic CLIP matching scores, rivaling MindSimulator and OneLLM. Joint decoding enhances both image and textual outputs, suppressing artifacts and enriching semantic details.
- Language and Reasoning Tasks: For BQA and detailed description tasks, Mind-Omni outperforms specialist models like UMBRAE and OneLLM (on BLEU, ROUGE, CIDEr, and LLM-as-judge metrics) despite being self-contained (no external LLM calls), demonstrating deep brain-grounded semantic reasoning.
- Neural Encoding: Mind-Omni's encoding performance (PCC, MSE, RSA) outpaces prior unified models in both voxel-level and semantic-level spaces, indicating richer brain-semantic alignment.
- Cross-modal Synergy: Joint image-text encoding/decoding consistently yields higher accuracy and distributed activations compared to unimodal tasks, revealing emergent "1+1>2" effects that mirror semantic integration in real neural processing.
Computational Neuroscience: Testbed Capabilities
Mind-Omni serves as a computational testbed, replicating established category-selectivity (EBA, OFA/FFA, PPA/OPA) in the human visual cortex via predicted fMRI projections. Beyond canonical categories, the framework enables synthesis and cortical mapping for novel concept-selective stimuli, aligning with distributed semantic representations theorized in neuroscience.
Design Principles and Ablation Insights
Ablation studies dissect Brain Tokenizer codebook size/dimensions, demonstrating critical impacts of semantic and perceptual alignment losses on retrieval performance and codebook diversity; codebook collapse is mitigated through careful architecture and multi-level objectives. Progressive multi-stage training and leveraging existing priors are shown to accelerate convergence and stability in data-constrained neuroimaging regimes. Enhanced caption datasets from MLLMs promote semantic richness and improve generalization across description and reasoning tasks.
Implications and Future Directions
Mind-Omni substantiates the viability of unified, multi-task foundation models for neural activity, marking a shift away from narrowly specialized pipelines. The empirical evidence for synergistic gains in joint modality tasks suggests architectures that mirror biological cognition will enable more robust, generalizable, and interpretable neural modeling. Practical implications include robust computational tools for neuroscientific exploration, improved BCIs, and scalable platforms for integrating heterogeneous neural data types.
Ongoing challenges remain—high-fidelity image reconstruction is limited by the lossy compression inherent in current VQ-VAE tokenization of fMRI, and the neural plausibility of underlying DiT encoders can be further enhanced (e.g., through RAE/VAVAE architectures with superior brain alignment). Growth of model and data scale, inclusion of EEG/MEG modalities, and optimization of unified encoders will drive future progress toward comprehensive neural foundation models capable of brain-wide understanding and reasoning.
Conclusion
Mind-Omni introduces a unified tri-modal framework that simultaneously models brain, vision, and language via discrete diffusion and semantic tokenization. Equipped with multi-task capabilities and synergistic learning, it demonstrates competitive and sometimes superior performance to single-task specialists across a spectrum of neural modeling tasks, including advanced reasoning. Its utility as a computational testbed for conceptual neuroscientific exploration further highlights its promise. The blueprint established herein informs the scalable development of future neural foundation models and lays the groundwork for deeper integration of computational neuroscience and AI (2605.29591).
