TRIBE v2: Tri-Modal Brain Encoder

• TRIBE v2 is a tri-modal foundation model that predicts high-resolution brain activity using frozen unimodal backbones and a trainable multimodal temporal encoder.
• It extends previous models by shifting from parcel-level to fsaverage5 surface mapping with 20,484 cortical vertices and 8,802 subcortical voxels from over 1,100 hours of fMRI data.
• The model demonstrates significant improvements over linear baselines and supports zero-shot transfer and in-silico experimentation for controlled neuroscience studies.

TRIBE v2 is a tri-modal foundation model for in-silico neuroscience that predicts human brain activity from video, audio, and language inputs. It is presented as a successor to TRIBE, extending a multimodal brain-encoding framework from parcel-level prediction toward high-resolution cortical-surface and subcortical prediction, broader cross-dataset training, and explicit support for zero-shot transfer to novel stimuli, tasks, and subjects. In the formulation reported for TRIBE v2, the model operates on frozen pretrained unimodal backbones, a trainable multimodal temporal encoder, and subject-conditioned or unseen-subject readouts, with predicted targets on the fsaverage5 cortical surface and in subcortical voxel space. Its stated scientific purpose is not limited to held-out encoding accuracy: it is also intended as a reusable substrate for in-silico experimentation, interpretability analyses, and downstream reuse in adjacent problems such as feature visualization and synthetic fMRI augmentation (d'Ascoli et al., 5 May 2026).

1. Historical position and model lineage

TRIBE v2 emerged from the earlier TRIBE system, the TRImodal Brain Encoder introduced for whole-brain fMRI response prediction during naturalistic movie watching. The original TRIBE combined transcript, audio, and video features, aligned them on a common temporal grid, modeled long-range temporal context with a transformer, and predicted z-scored BOLD responses in 1,000 non-overlapping Schaefer cortical parcels. That earlier system won the Algonauts 2025 multimodal brain encoding competition with a public leaderboard score of $0.2146$, and its central empirical claim was that multimodal fusion was especially beneficial in associative cortex relative to unimodal baselines (d'Ascoli et al., 29 Jul 2025).

Relative to that earlier parcel-level model, TRIBE v2 is explicitly described as predicting cortical activity on the fsaverage5 surface with 20,484 cortical vertices and subcortical activity with 8,802 voxels from 8 subcortical regions. The paper also states that TRIBE v2 is trained on 1,117.7 hours of fMRI across 720 subjects, whereas the earlier TRIBE paper emphasizes a smaller training setting centered on the Algonauts/CNeuroMod context. The v2 paper presents this as a shift from a competition-oriented multimodal encoder toward a broader “foundation model” intended to support cross-study generalization and in-silico neuroscience (d'Ascoli et al., 5 May 2026).

The naming can be misleading outside this literature. The paper "A Tribe Competition-Based Genetic Algorithm for Feature Selection in Pattern Classification" introduces TCbGA, not TRIBE or TRIBE v2, and uses “tribe” only for GA subpopulations (Ma et al., 2017). Likewise, "Truth or Tribe: How In-group Favoritism Prioritize Facts in Persona Agents" introduces a Truth or Tribe simulation framework, not a brain encoder called TRIBE v2 (Lei et al., 2 May 2026). Within computational neuroscience and brain encoding, however, “TRIBE v2” refers to the multimodal fMRI-prediction model introduced as a foundation model of vision, audition, and language (d'Ascoli et al., 5 May 2026).

2. Architecture and computational formulation

TRIBE v2 has a two-stage architecture: frozen modality-specific feature extraction followed by a trainable multimodal temporal brain encoder. The frozen backbones are Llama-3.2-3B for text, Wav2Vec-Bert-2.0 for audio, and Video-JEPA-2-Giant for video. Text is contextualized with up to $k=1024$ preceding words, audio is processed in 60-second chunks and resampled from 50 Hz, and video is sampled on a common 2 Hz grid using 64 frames spanning the preceding 4 seconds. The extracted modality features are grouped across backbone layers, projected into a shared 384-dimensional space per modality, concatenated into an 1152-dimensional multimodal representation, and passed through an 8-layer, 8-head Transformer encoder over 100-second windows (d'Ascoli et al., 5 May 2026).

The architecture reported in the v2 paper can be summarized briefly as follows.

Component	Specification	Function
Text backbone	Llama-3.2-3B	Transcript features
Audio backbone	Wav2Vec-Bert-2.0	Soundtrack features
Video backbone	Video-JEPA-2-Giant	Visual features
Shared modality width	$384$	Per-modality projection
Multimodal width	$1152$	Concatenated representation
Temporal model	8-layer, 8-head Transformer	Cross-time integration
Cortical output	fsaverage5, $20{,}484$ vertices	High-resolution cortex
Subcortical output	$8{,}802$ voxels	Harvard-Oxford subcortex

The model aligns all modality streams on a common stimulus timeline at $f_{\text{stim}} = 2\,\text{Hz}$, resamples fMRI targets to $f_{\text{fMRI}} = 1\,\text{Hz}$, and applies a 5-second offset to account for hemodynamic delay. In the notation reconstructed from the paper, the modality embeddings are concatenated as

$$h_t = \left[h_t^{(\text{text})};\, h_t^{(\text{audio})};\, h_t^{(\text{video})}\right] \in \mathbb{R}^{1152},$$

contextualized by a transformer,

$$z_{1:\tau} = \mathrm{TransformerEncoder}\!\left(h_{1:\tau} + p_{1:\tau} + s\right),$$

and adaptively average pooled to the fMRI rate before readout. The main subject-specific readout is

$k=1024$0

with a distinct unseen-subject linear layer

$k=1024$1

Training uses mean squared error and includes modality dropout with probability $k=1024$2, subject to at least one modality remaining active (d'Ascoli et al., 5 May 2026).

A notable architectural compromise concerns the video pathway. The model averages over patch tokens from Video-JEPA-2-Giant before temporal encoding, which removes spatial and retinotopic detail. The authors explicitly identify this as a likely cause of weaker performance in low-level visual areas with fine spatial organization, even while the model improves substantially in higher-order multimodal cortices (d'Ascoli et al., 5 May 2026).

3. Data regime, targets, and training setup

TRIBE v2 is trained on four naturalistic datasets and evaluated on four held-out studies. The training side comprises CNeuroMod, BoldMoments, Lebel2023, and Wen2017, totaling 25 subjects and 451.6 hours of fMRI. The held-out evaluation side comprises NNDb, LPP, Narratives, and HCP, totaling 695 subjects and 666.1 hours of fMRI. Across the full corpus, the paper reports 1,117.7 hours of fMRI, 121.1 hours of video, 142.4 hours of audio, and 71k sentences (d'Ascoli et al., 5 May 2026).

The targets are high-dimensional and explicitly neuroanatomical. Cortical signals are projected to the fsaverage surface and retained at fsaverage5 resolution; subcortical signals are extracted from 8 subcortical regions defined by the Harvard-Oxford atlas: hippocampus, lateral ventricles, amygdala, thalamus, caudate, putamen, pallidum, and accumbens. Each vertex time series is z-scored within session and detrended, and BOLD signals are linearly resampled to 1 Hz. The paper emphasizes that detrending materially matters because slow drifts can otherwise inflate apparent encoding scores (d'Ascoli et al., 5 May 2026).

The training and evaluation studies vary by modality composition and scanner regime. A concise summary is sufficient for orientation.

Split	Datasets	Summary
Training	CNeuroMod, BoldMoments, Lebel2023, Wen2017	25 subjects, 451.6 h fMRI
Held-out test	NNDb, LPP, Narratives, HCP	695 subjects, 666.1 h fMRI

Optimization uses AdamW, batch size 16, learning-rate warmup to $k=1024$3 over the first 10% of steps, cosine decay, and early stopping with patience 3 epochs. Feature extraction is cached; the paper reports approximately 24 hours on 128 V100 GPUs for feature extraction and about 24 hours on a single V100 32GB GPU for training the brain encoder itself (d'Ascoli et al., 5 May 2026).

The evaluation metric is the Pearson correlation between predicted and measured fMRI responses. In the paper’s notation, the encoding score is

$k=1024$4

The paper also uses parcel- or ROI-aggregated views of these predictions, but the supervised targets themselves are vertexwise and voxelwise rather than parcelwise (d'Ascoli et al., 5 May 2026).

4. Encoding performance, transfer, and in-silico experimentation

The principal empirical claim is that TRIBE v2 significantly outperforms an optimized linear FIR-style baseline across all held-out datasets, with $k=1024$5. The abstract characterizes the improvement as “several-fold,” while the body text is more specific in the held-out-subject fine-tuning setting: with at most one hour of data from a new subject, fine-tuned TRIBE v2 yields a two- to four-fold improvement over a linear encoder trained from scratch on that subject (d'Ascoli et al., 5 May 2026).

The model is also designed for zero-shot transfer. In unseen-subject mode, the paper compares TRIBE v2 predictions against held-out group-average responses and reports that on the HCP dataset the model reaches an $k=1024$6 near 0.4, described as a two-fold improvement over the median subject’s group-predictivity. The broader interpretation is that the released unseen-subject pathway can serve as an estimator of expected group responses in studies where empirical sampling is limited or absent (d'Ascoli et al., 5 May 2026).

A central aspect of the v2 paper is its use for controlled in-silico neuroscience rather than only naturalistic encoding benchmarks. On visual localizer paradigms from the Individual Brain Charting dataset, TRIBE v2 recovers classic category-selective regions including FFA for faces, PPA for places, EBA for bodies, and VWFA for written characters. On language paradigms, it reproduces reported effects involving A5, STS, Broca’s area (45), TPJ, and MTG, including contrasts such as speech versus non-speech, emotional pain versus physical pain, sentences versus word lists, and complex versus simple sentences. The paper states that predicted and empirical contrast maps show significant spatial correlation across HCP parcels, although the detailed numerical map correlations are not provided in the excerpted text (d'Ascoli et al., 5 May 2026).

TRIBE v2 is also used to probe multisensory cortical organization. Retraining with modality subsets shows that text, audio, and video dominate different territories: audio near auditory cortex, video in occipital and parietal cortex, and text in language cortices and large parts of prefrontal cortex. Comparing the full multimodal model to the best unimodal alternative reveals the largest multimodal gains around the temporal-parietal-occipital junction, where the paper reports gains of up to 50% increase in encoding score. ICA applied to the unseen-subject readout further yields components corresponding to primary auditory cortex, language network, motion detection area, default mode network, and visual system, which the authors compare against NeuroSynth maps (d'Ascoli et al., 5 May 2026).

5. Follow-on analyses and downstream uses

Subsequent work has used TRIBE v2 not merely as an encoder benchmark, but as a reusable computational object. One line of work treats it as a differentiable brain encoder for interpretability. The paper "Feature Visualization Recovers Known Cortical Selectivity from TRIBE v2" freezes TRIBE v2 and V-JEPA 2, then optimizes images to maximize predicted activation in seven visual ROIs. Under shared hyperparameters, the optimized images recover a progression of increasing spatial scale and complexity across V1 $k=1024$7 V4, face-like structure for FFA, rectilinear line patterns for PPA, and radial or streak-like “frozen-motion” patterns for MT despite static-only optimization. The optimized FFA stimulus yields predicted activation $k=1024$8, compared with $k=1024$9 for a real face photograph, which the authors interpret as a super-stimulus effect rather than a canonical face prototype (Bladon et al., 13 May 2026).

Another line of work uses TRIBE v2 as a source of synthetic training data for inverse problems. "Boosting Brain-to-Image Decoding with TRIBE v2 Data Augmentation" uses a pretrained TRIBE v2 model to generate synthetic cortical responses to static images converted into short still videos, and mixes those synthetic responses with real fMRI to train brain-to-image decoders. On NSD and BOLD5000, the paper reports up to 68% improvement in Top-10 image-retrieval accuracy relative to matched real-only baselines, with strong dependence on the real/synthetic ratio, dataset, and decoder class. It also reports that synthetic-only decoders can perform above chance in some settings, suggesting that TRIBE v2 can support a restricted form of zero-shot brain-to-image decoding (Benchetrit et al., 4 Jun 2026).

TRIBE v2 has also been used as a cortical evaluation layer in in-silico studies outside conventional encoding benchmarks. In "Neurological Plausibility of AI-Generated Music for Commercial Environments," the public TRIBE v2 weights and code path are used in an audio-only configuration with non-audio modalities disabled, producing fsaverage5 cortical predictions for five prompt-conditioned instrumental tracks. In that study, the Fast Bright Major Pop condition achieves the highest whole-cortex mean activation $384$0, the highest prefrontal composite $384$1, and the top parcel means in IFJa $384$2, IFJp $384$3, A5 $384$4, and area 45 $384$5. The authors present this as a cortical proxy for salience- and valuation-related processing, while explicitly rejecting any stronger claim about subcortical reward circuitry or consumer behavior (Sufi, 5 Apr 2026).

Not all downstream repurposing attempts have been positive. "A global predicted-fMRI drive signal from TRIBE does not predict YouTube replay heatmaps" evaluates the released checkpoint referred to as facebook/tribev2 on 48 YouTube videos and finds that a global predicted-fMRI drive signal does not track replay heatmaps: the pooled position-controlled partial correlation is $384$6 with 95% CI $384$7, $384$8, $384$9. The paper concludes that the released subject-averaged model is not useful, in that setup, as an off-the-shelf predictor of moment-level re-watch behavior, and it emphasizes temporal confounds that can create spurious positive results under weaker controls (Sahu et al., 1 Jul 2026).

6. Interpretation, limitations, and versioning caveats

TRIBE v2 is presented as a foundation model, but its scope is explicitly bounded. The paper states that it models the brain as a passive observer of stimuli rather than an acting agent, and that it remains constrained by the spatio-temporal resolution of fMRI. It covers video, audio, and language, but not olfaction, vestibular processing, somatosensation, or broader embodied multisensory functions. It also acknowledges that subcortical predictions are weaker and more uniform than cortical predictions, approximately two- to three-fold below cortical scores (d'Ascoli et al., 5 May 2026).

There are also architectural and interpretive caveats. Averaging video patch tokens before temporal encoding discards positional structure and is expected to deteriorate encoding in retinotopically organized visual cortex. The latent analyses and successful in-silico replications support neuroscientific alignment, but they do not amount to a mechanistic or causal explanation of brain function. The authors explicitly note the continuing “black box” character of deep networks, even when ICA and modality ablations yield recognizable systems-level maps (d'Ascoli et al., 5 May 2026).

A further nuance concerns released checkpoints versus the full subject-conditioned formulation. The core TRIBE v2 paper includes both a subject block and an unseen-subject head. By contrast, the downstream replay-heatmap evaluation states that the released facebook/tribev2 checkpoint it used was trained with average_subjects = True, had no subject-specific parameters, n_subjects = 0, and produced bit-identical predictions across the four Algonauts subjects. This suggests that some public release pathways expose a subject-averaged cortical predictor rather than the full range of subject-conditioned behavior described in the architecture paper. A plausible implication is that downstream failures or successes may depend materially on which inference path or checkpoint is used (Sahu et al., 1 Jul 2026).

The downstream literature reinforces the need for careful interpretation. The commercial music study uses TRIBE v2 only in a cortical-only public inference pathway and therefore cannot support claims about subcortical reward structures such as the nucleus accumbens or amygdala (Sufi, 5 Apr 2026). The replay-heatmap study shows that strong brain-encoding performance does not automatically transfer to behavioral engagement prediction (Sahu et al., 1 Jul 2026). The feature-visualization study shows that optimized inputs can become adversarial super-stimuli rather than natural exemplars (Bladon et al., 13 May 2026). Together, these results suggest that TRIBE v2 is best understood as a powerful predictive and experimental substrate whose outputs are scientifically informative but not self-validating.

In that sense, TRIBE v2 occupies a specific place in contemporary computational neuroscience. It is broader than a conventional modality-specific encoding model, more experimentally reusable than a competition-only predictor, and more technically concrete than a purely conceptual “foundation model” metaphor. At the same time, its strongest claims remain those explicitly supported by the published work: high-resolution multimodal fMRI prediction, transfer across studies and subjects, in-silico recovery of established cortical effects, and reuse as a cortical-response prior in downstream modeling pipelines (d'Ascoli et al., 5 May 2026).