A foundation model of vision, audition, and language for in-silico neuroscience

Abstract: Cognitive neuroscience is fragmented into specialized models, each tailored to specific experimental paradigms, hence preventing a unified model of cognition in the human brain. Here, we introduce TRIBE v2, a tri-modal (video, audio and language) foundation model capable of predicting human brain activity in a variety of naturalistic and experimental conditions. Leveraging a unified dataset of over 1,000 hours of fMRI across 720 subjects, we demonstrate that our model accurately predicts high-resolution brain responses for novel stimuli, tasks and subjects, superseding traditional linear encoding models, delivering several-fold improvements in accuracy. Critically, TRIBE v2 enables in silico experimentation: tested on seminal visual and neuro-linguistic paradigms, it recovers a variety of results established by decades of empirical research. Finally, by extracting interpretable latent features, TRIBE v2 reveals the fine-grained topography of multisensory integration. These results establish artificial intelligence as a unifying framework for exploring the functional organization of the human brain.

• The paper introduces TRIBE v2, a transformer-based multimodal encoder that predicts cortical and subcortical fMRI responses from video, audio, and text using more than 1,000 hours of data from 720 subjects.
• TRIBE v2 significantly outperforms matched linear encoders, scales with additional training data, and achieves zero-shot generalization to unseen subjects, with fine-tuning further improving prediction accuracy.
• The model reproduces canonical visual and language networks in silico and reveals interpretable multimodal cortical topographies, while remaining limited by fMRI resolution, missing sensory modalities, and its passive-stimulus framework.

Overview

TRIBE v2, introduced by d'Ascoli et al. (2605.04326), is a tri-modal encoding model that predicts high-resolution fMRI responses from video, audio, and text stimuli. Building on TRIBE v1, which won the 2025 Algonauts challenge out of 263 teams, the authors scale the approach to whole-brain (cortical surface plus subcortical) prediction across a curated corpus of over 1,000 hours of fMRI from 720 subjects. The paper's central claims are threefold: the model supersedes optimized linear encoders by a significant margin, it generalizes zero-shot to novel subjects and non-naturalistic experimental protocols, and it supports in-silico replication of canonical neuroimaging experiments.

Architecture and training data

The model consumes frozen embeddings from state-of-the-art pretrained backbones: Llama-3.2-3B for timestamped text, Wav2Vec-BERT-2.0 for audio, and Video-JEPA-2-Giant for video, all resampled to a 2 Hz stimulus grid and compressed to a shared 384-dimensional latent per modality ($D_{model}=1152$). An 8-layer transformer encoder aggregates information over 100-second windows, followed by adaptive pooling to 1 Hz and a subject-conditional linear projection to 20,484 cortical vertices (fsaverage5) plus 8,802 subcortical voxels (Harvard-Oxford atlas). Two design choices are notable: modality dropout ($p=0.3$) enables predictions when modalities are missing, and an "unseen subject" layer, activated with probability 0.1 during training via subject dropout, enables zero-shot group-level prediction.

Training uses four "deep" datasets (25 subjects, ~452 hours), including Courtois NeuroMod (~80 hours per subject of Friends episodes and movies), Lebel2023 podcasts, BoldMoments clips, and Wen2017 silent videos. Four "broad" datasets held out for testing (695 subjects, ~666 hours) include NNDb, LPP, Narratives, and HCP movie-watching data at 7T. Feature extraction required 24 hours on 128 V100 GPUs; model training fits on a single V100 within a day — a practical point relevant to reproducibility.

Encoding performance and scaling

Across naturalistic conditions, TRIBE v2 yields statistically significant predictions across most of cortex, with task-dependent spatial profiles: temporal peaks for podcast listening, occipital peaks for video, and combined coverage for multimodal stimuli. Subcortical scores are two- to three-fold lower but remain significant in most regions. Against a rigorously matched baseline — a "Deep FIR" linear encoder fed with identical pretrained embeddings and trained at comparable scale — TRIBE v2 is significantly better on all datasets ($q(\text{FDR}) < 10^{-4}$, paired t-tests across subjects). Because input features and optimization are controlled, this gap isolates the contribution of nonlinear temporal integration rather than feature quality.

A log-linear increase in encoding accuracy with training hours is observed on Courtois NeuroMod without plateau, mirroring scaling laws reported elsewhere in fMRI encoding. This implies that current predictive ceilings are data-limited rather than architecture-limited, though the claim rests on a single dataset and a limited range of hours.

Generalization to unseen subjects

On the four held-out broad datasets, zero-shot TRIBE v2 predictions correlate with the group-averaged response more strongly than most individual subjects' own recordings do — a strong claim with practical consequence: the model can substitute for pilot data in study design. The effect is largest on HCP (7T, highest SNR), where $R_{group}$ approaches 0.4, roughly double the median subject's group-predictivity. With fine-tuning on at most one hour of held-out data per subject, encoding scores improve significantly ($q<10^{-4}$) and exceed linear encoders trained from scratch by two- to four-fold. Low-rank factorization of the subject block (rank 128) makes this feasible despite the large number of test subjects.

In-silico experimentation

The paper's most consequential evaluation moves beyond naturalistic stimuli to controlled protocols from the Individual Brain Charting (IBC) dataset, run entirely through the model in unseen-subject mode. For vision, flashed images (1 s every 8 s) produce the expected hemodynamic time course peaking ~5 s post-onset, and contrast maps recover FFA for faces, PPA for places, EBA for bodies, and VWFA for written characters, with significant spatial correlation between predicted and empirical z-scored contrast maps across the 360 Glasser parcels. For language, the model recovers the core language network in speech-versus-non-speech contrasts, TPJ and MTG in emotional-versus-physical pain contrasts, and correctly predicts left-lateralization patterns distinguishing semantic (sentences vs. word lists) from syntactic (complex vs. simple sentences) processing. One methodological caveat: French IBC sentence stimuli were translated to English and re-rendered through text-to-speech before word-timing extraction, so the in-silico language experiments deviate from the original stimuli in language and acoustic realization.

Interpretability and multimodal topography

ICA applied to the unseen-subject projection layer yields five components that each correlate highly with a single canonical functional network — primary auditory cortex, language network, motion area V5, default mode network, and visual system — as validated against NeuroSynth meta-analytic maps. This indicates the learned latent-to-cortex mapping is not entangled but decomposes into neuroscientifically meaningful factors.

Modality ablations show video > audio > text in overall unimodal performance, with complementary spatial domains: audio near auditory cortex, video in occipital/parietal regions, text in language areas and much of prefrontal cortex. Multimodal gains over the best unimodal encoder reach up to 50% at the temporo-parietal-occipital junction, with RGB overlay maps revealing bimodal integration zones (e.g., text+audio in superior temporal lobe; video+audio in ventral/dorsal visual cortices and hippocampus). These maps align qualitatively with the multisensory integration literature while providing finer-grained topography, though the authors note meaningful inter-subject variability in these organizations.

Limitations and open questions

The authors are explicit about several constraints. fMRI's spatio-temporal resolution bounds what any such model can capture; millisecond neural dynamics are inaccessible. Input features cover only vision, audition, and semantics, omitting somatosensation, olfaction, balance, and their integration. More fundamentally, the model treats the brain as a passive stimulus-driven system: it does not model active behavior, agency, development, or clinical pathology, and its training population reflects the usual WEIRD sampling bias. The video embedding pipeline discards spatial positional information via patch-token averaging, which the authors expect degrades low-level retinotopic visual-area encoding — a specific architectural trade-off left unresolved. Whether the log-linear scaling trend extends to substantially larger corpora, and whether in-silico replication generalizes to paradigms involving task instructions, decision-making, or working memory, remain open questions the paper does not answer.

Conclusion

TRIBE v2 demonstrates that a single transformer-based encoder, trained jointly on heterogeneous multimodal fMRI data, can match or exceed specialized pipelines, generalize to hundreds of unseen subjects, and reproduce decades of established functional-localizer results in silico. Its combination of predictive accuracy, zero-shot transfer, and interpretable decomposition positions encoding foundation models as a practical instrument for piloting experiments and probing cortical organization, contingent on addressing the passive-observer framing and modality gaps the authors themselves identify.