BrainJanus: Bidirectional Brain-Vision-Language Model

• BrainJanus is a bidirectional framework linking neural activity with cognitive domains such as tasks, language, and vision through forward and reverse inference.
• It employs both ontology-driven and autoregressive methods to map neural signals to complex outputs, enabling cross-modal synthesis and zero-shot cognitive decoding.
• Implementations like Text2Brain and the 2026 model demonstrate multimodal tokenization and any-to-any generation, addressing the limitations of one-directional methods.

Searching arXiv for the core papers on BrainJanus and closely related predecessors. BrainJanus denotes a Janus-like framework for linking neural activity to representational domains in both directions: from tasks, language, or sensory stimuli to brain activation, and from brain activation back to cognitive content, text, or images. In the literature summarized under this label, the term refers both to a conceptual program of bidirectional inference and to specific implementations with different scopes. An ontology-driven image-based framework established a bidirectional task↔brain mapping across studies (Schwartz et al., 2013); Text2Brain realized the language→brain half by synthesizing 3D activation maps from free-form text queries (Ngo et al., 2022); and the 2026 model "BrainJanus" defined a unified autoregressive system spanning brain, vision, and language, with any-to-any generation in a shared discrete token space (Wu et al., 29 Jun 2026).

1. Conceptual definition and scope

The central idea of BrainJanus is that brain modeling should not be restricted to a single direction of inference. In the earlier ontology-driven formulation, the two faces of the system are explicit: a forward mode that estimates which regions are recruited by a task or ontology term, and a reverse mode that infers which task or term best explains a new activation map. The forward problem is posed as estimating $P(X_i \neq 0 \mid T)$ for each voxel $i$, whereas the reverse problem inverts the mapping through Bayes’ rule,

$P(T \mid X) = \frac{P(X \mid T) P(T)}{P(X)}.$

This formulation treats bidirectionality as a methodological requirement for principled reverse inference rather than as a purely engineering convenience (Schwartz et al., 2013).

The later 2026 formulation generalizes this Janus-like structure beyond task labels. BrainJanus is described there as a unified model that integrates brain, vision, and language within a single framework, using a shared Omni space of discrete tokens and a single decoder-only Transformer trained with next-token prediction. In that setting, bidirectionality becomes any-to-any generation: image-to-brain, text-to-brain, brain-to-image, and brain-to-text are all handled by one autoregressive model without task-specific heads or separate models (Wu et al., 29 Jun 2026).

A useful distinction within the BrainJanus literature is therefore between partial and full realizations. Text2Brain is a concrete realization of only one half of the idea: it maps natural language directly to brain activation patterns, but does not implement brain→text decoding. By contrast, the ontology-driven 2013 framework and the 2026 unified autoregressive model are explicitly bidirectional, although they differ substantially in data type, representation, and modeling assumptions (Ngo et al., 2022).

2. Ontology-driven bidirectionality before free-form multimodality

The 2013 framework introduced a large-corpus, ontology-driven approach for accumulating knowledge toward a bidirectional link between observed brain activity and the corresponding function. Its corpus comprised 19 task fMRI studies, 486 subjects, 131 activation map types, and 3,826 individual statistical maps. All studies were reprocessed with SPM and spatially normalized to a common template space, reducing inter-study pipeline variability and enabling voxel-wise analyses across studies. Experimental conditions were annotated with the Cognitive Paradigm Ontology (CogPO), using standardized terms across stimulus modality, explicit stimulus, instructions, and overt response, with multi-label structure and retention of terms present in at least two studies (Schwartz et al., 2013).

Forward inference in this system used a GLM,

$x = Y \beta + \epsilon,$

with term-versus-rest contrasts and FWER correction at 5%. Reverse inference used discriminative modeling directly on images, rather than independent voxelwise inversion. The main reverse model was an $\ell_2$-regularized logistic regression per term within each ontology category, trained after spatially-constrained Ward clustering to approximately 15,000 parcels and one-way ANOVA feature selection retaining the top 30% most discriminative parcels for each term. Class imbalance was handled by sample weights inversely proportional to term frequency and by a test-time probability correction,

$P_{\text{biased}} = \rho_{\text{term}} P.$

The evaluation protocol was stringent: leave-one-study-out cross-validation with nested 10-fold stratified shuffle splits for regularization tuning, plus leave-one-laboratory-out validation. Weighted logistic regression outperformed K-NN and naive Bayes, balancing precision and recall above chance while generalizing to completely unseen studies and laboratories. Reverse maps recovered discriminative networks such as motor cortex for the instruction “move,” auditory cortex for “words,” fusiform face area for “face,” and intra-parietal sulci and frontal eye fields for “saccades.” The framework therefore established a form of zero-shot cognitive decoding grounded in ontology labels and full statistical images rather than only peak coordinates (Schwartz et al., 2013).

Within the BrainJanus lineage, this work is significant because it framed bidirectional brain–cognition mapping as an inference problem over harmonized corpora. It also defined several persistent constraints that later systems continue to address: study confounds, long-tail label imbalance, ontology granularity, and the distinction between forward regional recruitment and reverse discriminative decoding.

3. Text2Brain and the language→brain half of the Janus idea

Text2Brain extended the BrainJanus program into unrestricted natural language by replacing ontology terms and fixed vocabularies with free-form text queries. The motivating claim was that keyword-based tools such as Neurosynth and Neuroquery are constrained by synonymy, evolving terminology, longer descriptions whose semantics depend on word order or context, and out-of-vocabulary words or rare phrases. Text2Brain addressed these limits by learning contextualized semantic embeddings and directly synthesizing volumetric activation maps from open-ended descriptions (Ngo et al., 2022).

The model is an encoder–decoder system trained end-to-end. Its transformer-based text encoder is based on SciBERT and produces a 768-dimensional contextualized embedding using bidirectional self-attention and subword tokenization. That embedding is projected through a fully connected layer and reshaped into a low-resolution $4 \times 5 \times 4$ 3D tensor with 64 channels, which is then upsampled by three transposed 3D convolutional layers with 32, 16, and 8 channels to yield a whole-brain prediction in MNI152 space. Input text can be drawn from titles, abstracts, captions, keywords, or discussion sentences, and inference supports queries up to roughly 140 characters in under a second on a single GPU (Ngo et al., 2022).

Its training data were derived from 13,000 neuroimaging articles from the Neuroquery dataset with reported activation foci in MNI152 coordinates. For each table of reported foci, a coordinate-based target map was constructed by placing Gaussian spheres centered at each reported peak coordinate, using a Gaussian kernel with full-width at half maximum of 9 mm. Table-specific maps were paired with the first sentence of the corresponding table caption, while article-average maps were randomly paired with article title, an author-provided keyword, the abstract, or a randomly chosen subset of discussion sentences. This many-to-one text–image pairing was designed to encourage learning of synonymy and context dependence. Optimization used MSE reconstruction loss, Adam, 2000 epochs, batch size 24, learning rates $1 \times 10^{-5}$ for the SciBERT encoder and $3 \times 10^{-2}$ for the 3D decoder, with approximately 75 hours of training on a single Nvidia RTX GPU (Ngo et al., 2022).

Evaluation used thresholded Dice similarity across multiple percentage thresholds from 5% to 30% of top voxels, summarized by Dice AUC, with

$$\mathrm{Dice}(x)=\frac{2\,|\mathrm{Prediction}(x)\cap \mathrm{Target}(x)|}{|\mathrm{Prediction}(x)|+|\mathrm{Target}(x)|}.$$

On 1,000-article title prediction, Text2Brain exceeded Neuroquery and Neurosynth on both “easy” and “hard” test sets; on IBC contrast descriptions it achieved mean Dice AUC $i$0 versus $i$1 for Neuroquery and $i$2 for Neurosynth; on HCP contrasts it achieved $i$3 versus $i$4 and $i$5 respectively. It also showed stable default-network patterns across the “self-generated thought” family of synonymous descriptions and higher concept-matching accuracy on Cognitive Atlas definitions and aliases. In this sense, Text2Brain operationalized the language→activation half of BrainJanus while remaining complementary to a future brain→language component (Ngo et al., 2022).

4. Unified BrainJanus across brain, vision, and language

The 2026 model "BrainJanus" generalized the Janus principle from task labels and free-form text queries to a single multimodal generative architecture. It is presented as a unified, autoregressive model that learns a shared discrete token space spanning neural activity, images, and language. Two technical components are central: the Unified Brain Tokenizer, which discretizes continuous fMRI signals into tokens aligned with visual and linguistic representations, and an All-in-One autoregressive architecture that performs next-token prediction over interleaved multimodal token sequences (Wu et al., 29 Jun 2026).

The Unified Brain Tokenizer follows a VQ-VAE-style design with encoder, codebook, vector quantizer, and decoder. For codebook $i$6, with $i$7, quantization is defined by

$i$8

and the tokenizer loss is

$i$9

with the reconstruction instantiated as MSE in implementation. The reported hyperparameters are codebook size $P(T \mid X) = \frac{P(X \mid T) P(T)}{P(X)}.$0, embedding dimension $P(T \mid X) = \frac{P(X \mid T) P(T)}{P(X)}.$1, compression ratio approximately $P(T \mid X) = \frac{P(X \mid T) P(T)}{P(X)}.$2, commitment weight $P(T \mid X) = \frac{P(X \mid T) P(T)}{P(X)}.$3, entropy regularization weight $P(T \mid X) = \frac{P(X \mid T) P(T)}{P(X)}.$4, AdamW at $P(T \mid X) = \frac{P(X \mid T) P(T)}{P(X)}.$5, batch size 256, 100 epochs, and cross-subject pretraining on 8 NSD subjects (Wu et al., 29 Jun 2026).

The Omni space is defined as

$P(T \mid X) = \frac{P(X \mid T) P(T)}{P(X)}.$6

with modality-specific tokenizers $P(T \mid X) = \frac{P(X \mid T) P(T)}{P(X)}.$7, $P(T \mid X) = \frac{P(X \mid T) P(T)}{P(X)}.$8, and $P(T \mid X) = \frac{P(X \mid T) P(T)}{P(X)}.$9 mapping brain, vision, and language into this space. A single decoder-only Transformer, initialized from Janus-7B, then models the joint distribution with the autoregressive objective

$x = Y \beta + \epsilon,$0

and factorization

$x = Y \beta + \epsilon,$1

Conditioning is purely prefix-based: image tokens can condition brain-token generation for $x = Y \beta + \epsilon,$2, text tokens can condition $x = Y \beta + \epsilon,$3, and brain tokens can condition $x = Y \beta + \epsilon,$4 or $x = Y \beta + \epsilon,$5. The backbone uses hidden size 4096 and LoRA adapters on attention projections with rank scaling 16 and dropout 0.2; modality-specific lightweight aligners project tokenizer outputs into the model embedding size (Wu et al., 29 Jun 2026).

Training used the Natural Scenes Dataset with 7T fMRI and natural images from COCO. Eight subjects were used for tokenizer pretraining and joint autoregressive modeling, while the main evaluation used subjects 01, 02, 05, and 07, each with 9,000 unique training images and 1,000 shared test images, each test image presented three times. Brain signals were single-trial GLM beta maps at 1.8 mm resolution restricted to visual ROIs; example voxel counts were 15,724 for subj01, 14,278 for subj02, 13,039 for subj05, and 12,682 for subj07. Beyond original COCO captions, the work used Qwen3-VL-235B-A22B-Instruct to synthesize detailed captions for 73k images. The autoregressive model was trained with AdamW, cosine schedule and warmup, peak learning rate $x = Y \beta + \epsilon,$6, batch size 16, 15 epochs, ZeRO Stage-2, BF16, and 8 A100 80GB GPUs, with tokenizers frozen during AR supervised fine-tuning (Wu et al., 29 Jun 2026).

5. Empirical profile: performance, zero-shot transfer, and topography

Across the BrainJanus literature, empirical evaluation serves two distinct purposes: verifying bidirectional predictability and testing whether the learned mappings preserve structure that is neuroscientifically meaningful rather than only statistically convenient. The 2013 ontology-driven system demonstrated that reverse decoding can generalize across unseen studies and laboratories and can recover discriminative networks that are more specific than forward maps for several terms, especially when weighted logistic regression is used instead of K-NN or naive Bayes (Schwartz et al., 2013).

Text2Brain contributed a different kind of evidence. It showed that open-ended language can drive whole-brain synthesis more effectively than fixed-vocabulary baselines across article titles, task descriptions, representative meta-analytic domains, and synonym families. The contrast with Neurosynth and Neuroquery is particularly important because both baselines can fail on OOV inputs or degrade on less common phrasings, whereas Text2Brain does not rely on a fixed vocabulary and was reported never to “fail” due to OOV words in those evaluations. Its ablations further indicated that fine-tuning from pretrained SciBERT improves performance relative to training the encoder from scratch, that 9 mm FWHM preprocessing strikes a favorable balance relative to 5 mm and 15 mm alternatives for IBC task prediction, and that shorter descriptions generally yield better predictions although Text2Brain degrades less than keyword-based baselines on longer inputs (Ngo et al., 2022).

The 2026 BrainJanus model shifted the empirical emphasis from coordinate-based brain maps to multimodal generation quality. For brain-to-text decoding, the reported metrics include BLEU-1..4, METEOR, ROUGE, CIDEr, SPICE, BERTScore, CLIP-Text, and CLIP-Image. On the detailed Qwen ground-truth setting, it achieved BERTScore 38.12 and CLIP 96.2%, surpassing prior SOTA by 7.21 and 1.5 percentage points; with standard COCO ground truth it also achieved top scores across BLEU and CLIP, including CLIP 94.8%. For brain-to-image decoding, it attained the highest CLIP image–image similarity, 94.4%, and subject-averaged low-level scores including Pixel Correlation 0.173 and SSIM 0.292. The model also exhibited zero-shot generalization: training only on brain→text enabled reasonable brain→image reconstruction, and vice versa, because all tasks share the same token space and autoregressive backbone. Synthetic fMRI and tokenizer reconstructions preserved coherent patterns across visual ROIs such as Early visual, OPA, PPA, and EAT, supporting the claim that interpretable biological topography is retained in the discrete codes (Wu et al., 29 Jun 2026).

A recurring theme across these results is that bidirectionality is not only about symmetric input and output channels. In the ontology-driven and text-driven systems, improved performance is associated with stronger handling of semantic variation, rare labels, and cross-study heterogeneity. In the 2026 model, it is associated with a shared tokenization and parameter-sharing strategy that permits cross-task transfer without task-specific heads.

6. Limitations, misconceptions, and adjacent Janus-like directions

A common misconception is that any model linking text and neuroimaging already constitutes a full BrainJanus system. The available work does not support that simplification. Text2Brain implements only the text→brain synthesis pathway and does not perform brain→text decoding; it is therefore best understood as a one-directional bridge and as a complement to a fuller bidirectional framework rather than a complete realization of it (Ngo et al., 2022).

A second misconception is that reverse inference can be made reliable from isolated activations or single studies. The ontology-driven literature emphasizes the opposite: reverse inference is prone to fallacies when based on single studies, because activation of a region does not by itself demonstrate selective engagement of a process. The proposed remedy is a large-corpus approach with harmonized preprocessing, ontology labeling, explicit control of class imbalance, and evaluation on unseen studies or laboratories (Schwartz et al., 2013).

The present BrainJanus implementations also have substantive technical limitations. Text2Brain is trained on coordinate-based maps derived from reported peak foci rather than raw statistical images, inheriting literature and reporting biases; it does not attribute predicted clusters to specific words or phrases; it can confuse contrast direction; and it does not report explicit uncertainty quantification. The 2013 framework is limited by design-matrix collinearity, study confounds, persistent difficulty with rare paradigms, and the fact that CogPO captures paradigm features rather than high-level cognitive processes. The 2026 BrainJanus model is restricted experimentally to fMRI in visual cortex, may prioritize plausible semantics over exact biological fidelity in autoregressive generation, and raises privacy and consent issues because decoding internal brain states can threaten mental privacy (Ngo et al., 2022, Schwartz et al., 2013, Wu et al., 29 Jun 2026).

An adjacent direction is provided by "Brain Harmony: A Multimodal Foundation Model Unifying Morphology and Function into 1D Tokens" (Dong et al., 29 Sep 2025). That work does not use the term BrainJanus, but it explicitly presents a Janus-like unification of structure and function through two modality-specific encoders, geometry-aware and TR-aware functional tokenization, and a multimodal Harmonizer with 128 learnable hub tokens. Its staged objectives combine MAE-based structural pretraining, JEPA-based functional pretraining, and hub-token fusion losses, with reported gains across ABIDE-I/II, ADHD-200, PPMI, ADNI, HCP-A, and an Asian clinical cohort. This suggests a broader interpretation of BrainJanus as a family of systems organized around shared latent representations, bidirectional or dual-view mappings, and modality-bridging inductive biases rather than a single architecture alone (Dong et al., 29 Sep 2025).

Taken together, the literature defines BrainJanus as both a concept and an evolving technical program. Its earliest formulations emphasized principled inversion of task–brain mappings; its text-based instantiations extended the forward direction to unrestricted language; and its latest unified model treated brain, vision, and language as tokenized modalities within one autoregressive system. The continuing open problems are correspondingly clear: robust whole-brain coverage, uncertainty estimation, improved logical and compositional reasoning over contrasts and descriptions, stronger interpretability linking regions to symbols, and safeguards for privacy in increasingly capable decoding systems.