BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language

Abstract: Modeling the bidirectional correspondence between external sensory stimuli and internal neural activity has emerged as a critical frontier in neuroscience. However, existing approaches predominantly treat brain encoding and decoding as isolated tasks, relying heavily on unimodal alignment and external priors while overlooking the brain's intrinsic nature as a multimodal integration system. To address these limitations, we propose BrainJanus, the first unified brain model that integrates brain, vision, and language within a single framework. Specifically, we introduce a Unified Brain Tokenizer to quantize continuous neural dynamics into discrete tokens aligned with visual and linguistic representations in a shared Omni space. Building on this, we utilize an All-in-One autoregressive architecture that leverages next-token prediction to enable seamless any-to-any generation, which encompasses image-to-brain and text-to-brain encoding, and brain-to-image and brain-to-text decoding. Extensive experiments demonstrate that BrainJanus achieves superior performance across diverse benchmarks. Furthermore, our framework exhibits zero-shot generalization and preserves interpretable biological topography, highlighting its potential as a general-purpose brain modeling paradigm. The code is available at \href{https://github.com/HaitaoWuTJU/BrainJanus}{GitHub}.

• The paper introduces BrainJanus, a unified autoregressive model that integrates brain signals, images, and text within a shared token space.
• The model achieves significant improvements in brain-to-text (BERTScore +7.21) and brain-to-image (CLIP +1.5) decoding, surpassing previous baselines.
• The framework demonstrates robust zero-shot generalization, paving the way for advanced brain-computer interfaces and neural prosthetics.

Unified Modeling of Brain, Vision, and Language via BrainJanus

Motivation and Background

Bidirectional modeling between sensory stimuli and neural activity—the mapping from images/text to brain responses and vice versa—remains a foundational challenge in computational neuroscience and AI. Previous approaches predominantly cast encoding (stimulus-to-brain) and decoding (brain-to-stimulus/text) as isolated tasks, often relying on unimodal alignments (e.g., CLIP for vision) and external priors, neglecting the brain’s intrinsic multimodal integration. Despite incremental advances with task-specific pipelines and patchwork frameworks, current methods are fundamentally hampered by fragmentary semantic exploitation, poor cross-modal generalization, and overdependence on large pretrained models.

Model Architecture

BrainJanus introduces a unified autoregressive architecture that bridges brain signals, visual information, and linguistic content in a single token space—termed the Omni space. Core innovations include:

• Unified Brain Tokenizer: Employs VQ-VAE-style discretization to quantize continuous neural dynamics (e.g., fMRI/EEG) into discrete tokens, aligning them with visual and linguistic tokens for seamless integration.
• All-in-One Autoregressive Transformer: Processes arbitrarily interleaved token sequences across modalities using next-token prediction. This enables four primary tasks: image-to-brain encoding, text-to-brain encoding, brain-to-image decoding, and brain-to-text decoding within a single transferable backbone.

Each modality-specific tokenizer projects raw data into the shared Omni space with fixed-dimensional token embeddings. This facilitates flexible any-to-any cross-modal generation and understanding, directly leveraging multimodal token compositionality.

Experimental Results

Empirical validation uses the NSD dataset, which contains high-resolution fMRI recordings aligned to COCO images and captions, supplemented with detailed synthetic descriptions from advanced multimodal LLMs.

Brain-to-Text Decoding

BrainJanus achieves a BERTScore of 38.12 and a CLIP score of 96.2%, surpassing all previous baselines (e.g., MindEye2, UMBRAE, MindLLM) by significant margins (BERTScore +7.21, CLIP +1.5). Textual reconstructions are more semantically faithful, capturing detailed object and action attributes absent in prior outputs.

Brain-to-Image Decoding

Autoregressive decoding yields a CLIP semantic similarity of 94.4%, outperforming diffusion-based models (MindEye2, UMBRAE, Takagi & Nishimoto) despite eschewing direct diffusion priors. The generated reconstructions display higher structural fidelity and richer preservation of low-level information compared to caption-to-image baselines.

Brain Encoding (Stimulus-to-Neural)

BrainJanus synthesizes fMRI signals from visual/textual inputs, maintaining interpretable cortical topography and inter-subject variability. To ensure biological validity and mitigate information leakage, training avoids direct alignment with visual embeddings, relying solely on cross-entropy loss for neural prediction. Semantic evaluation demonstrates superior preservation of meaningful representations, with metric scores reflecting genuine learning rather than trivial storage.

Zero-shot and Cross-task Generalization

Unified multi-task learning yields robust zero-shot generalization. Models trained on singular modality translation tasks (e.g., fMRI-to-text) seamlessly transfer to others (e.g., fMRI-to-image, image-to-fMRI), confirming the strength of underlying shared representations.

Ablation Studies

Compression ratio and codebook size trade-off studies show that lower compression and larger codebooks foster semantic preservation at the cost of longer token sequences and increased generation complexity. Semantic filtering emerges as an inadvertent benefit, attenuating high-frequency noise.

Theoretical and Practical Implications

The unified tokenization and autoregressive paradigm substantively challenges the fragmentary modality-specific conventions, establishing a scalable framework for true multimodal integration. By aligning biological and digital representations, BrainJanus models the cortex as an omni-modality sparse token manifold, supporting joint cross-modal reasoning and generation. The explicit avoidance of direct visual embedding alignment circumvents evaluation hacking (semantic leakage), addressing a critical methodological flaw in existing encoding-decoding research.

Practically, this architecture unlocks new avenues for general-purpose brain-computer interfaces, multimodal neural prosthetics, and cognitive modeling. The interpretable representations and zero-shot capabilities hold promise for extending brain decoding tasks beyond vision into more abstract domains (e.g., auditory or cognitive tasks).

Limitations and Future Directions

Current deployment is limited to visual cortex fMRI within NSD; extension to full-brain activity and other modalities (e.g., MEG, EEG, intracranial recordings) is requisite for broader applicability. While generative quality is maximized, strict biological faithfulness may be compromised due to reliance on autoregressive priors, potentially introducing hallucinations. Scalability and robustness across heterogeneous subject populations and diverse neural datasets warrant further investigation.

Future developments may include enlarging the codebook, increasing modality diversity, integrating hierarchical brain regions, and developing more rigorous biologically-constrained evaluation protocols to further ground neural synthesis.

Conclusion

BrainJanus presents a unified, autoregressive model for seamless transformation and interpretation across brain, vision, and language, introducing discrete brain tokenization and shared token space integration. The superior numerical results across decoding and encoding benchmarks, combined with interpretable neural representations, define BrainJanus as a comprehensive, extensible foundation for multimodal neural computing. This work lays groundwork for future expansion toward fully general-purpose neural signal modeling, multimodal reasoning, and biologically-constrained AI.