---
title: 'TRIBE v2: Tri-Modal Brain Encoder'
url: https://www.emergentmind.com/topics/tribe-v2
type: topic
---

# TRIBE v2: Tri-Modal Brain Encoder

TRIBE v2 is a tri-modal foundation model for in-silico neuroscience that predicts human brain activity from video, audio, and language inputs. It is presented as a successor to TRIBE, extending a multimodal brain-encoding framework from parcel-level prediction toward high-resolution cortical-surface and subcortical prediction, broader cross-dataset training, and explicit support for zero-shot transfer to novel stimuli, tasks, and subjects. In the formulation reported for TRIBE v2, the model operates on frozen pretrained unimodal backbones, a trainable multimodal temporal encoder, and subject-conditioned or unseen-subject readouts, with predicted targets on the fsaverage5 cortical surface and in subcortical voxel space. Its stated scientific purpose is not limited to held-out encoding accuracy: it is also intended as a reusable substrate for in-silico experimentation, interpretability analyses, and downstream reuse in adjacent problems such as feature visualization and synthetic fMRI augmentation [2605.04326].

## 1. Historical position and model lineage

TRIBE v2 emerged from the earlier TRIBE system, the TRImodal Brain Encoder introduced for whole-brain fMRI response prediction during naturalistic movie watching. The original TRIBE combined transcript, audio, and video features, aligned them on a common temporal grid, modeled long-range temporal context with a transformer, and predicted z-scored BOLD responses in 1,000 non-overlapping Schaefer cortical parcels. That earlier system won the Algonauts 2025 multimodal brain encoding competition with a public leaderboard score of \(0.2146\), and its central empirical claim was that multimodal fusion was especially beneficial in associative cortex relative to unimodal baselines [2507.22229].

Relative to that earlier parcel-level model, TRIBE v2 is explicitly described as predicting cortical activity on the fsaverage5 surface with **20,484 cortical vertices** and subcortical activity with **8,802 voxels** from **8 subcortical regions**. The paper also states that TRIBE v2 is trained on **1,117.7 hours of fMRI** across **720 subjects**, whereas the earlier TRIBE paper emphasizes a smaller training setting centered on the Algonauts/CNeuroMod context. The v2 paper presents this as a shift from a competition-oriented multimodal encoder toward a broader “foundation model” intended to support cross-study generalization and in-silico neuroscience [2605.04326].

The naming can be misleading outside this literature. The paper "A Tribe Competition-Based Genetic Algorithm for Feature Selection in Pattern Classification" introduces **TCbGA**, not TRIBE or TRIBE v2, and uses “tribe” only for GA subpopulations [1704.08818]. Likewise, "Truth or Tribe: How In-group Favoritism Prioritize Facts in Persona Agents" introduces a **Truth or Tribe simulation framework**, not a brain encoder called TRIBE v2 [2605.01329]. Within computational neuroscience and brain encoding, however, “TRIBE v2” refers to the multimodal fMRI-prediction model introduced as a foundation model of vision, audition, and language [2605.04326].

## 2. Architecture and computational formulation

TRIBE v2 has a two-stage architecture: frozen modality-specific feature extraction followed by a trainable multimodal temporal brain encoder. The frozen backbones are **Llama-3.2-3B** for text, **Wav2Vec-Bert-2.0** for audio, and **Video-JEPA-2-Giant** for video. Text is contextualized with up to \(k=1024\) preceding words, audio is processed in 60-second chunks and resampled from 50 Hz, and video is sampled on a common 2 Hz grid using **64 frames spanning the preceding 4 seconds**. The extracted modality features are grouped across backbone layers, projected into a shared **384-dimensional** space per modality, concatenated into an **1152-dimensional** multimodal representation, and passed through an **8-layer, 8-head Transformer encoder** over **100-second** windows [2605.04326].

The architecture reported in the v2 paper can be summarized briefly as follows.

| Component | Specification | Function |
|---|---|---|
| Text backbone | Llama-3.2-3B | Transcript features |
| Audio backbone | Wav2Vec-Bert-2.0 | Soundtrack features |
| Video backbone | Video-JEPA-2-Giant | Visual features |
| Shared modality width | \(384\) | Per-modality projection |
| Multimodal width | \(1152\) | Concatenated representation |
| Temporal model | 8-layer, 8-head Transformer | Cross-time integration |
| Cortical output | fsaverage5, \(20{,}484\) vertices | High-resolution cortex |
| Subcortical output | \(8{,}802\) voxels | Harvard-Oxford subcortex |

The model aligns all modality streams on a common stimulus timeline at \(f_{\text{stim}} = 2\,\text{Hz}\), resamples fMRI targets to \(f_{\text{fMRI}} = 1\,\text{Hz}\), and applies a **5-second** offset to account for hemodynamic delay. In the notation reconstructed from the paper, the modality embeddings are concatenated as
\[
h_t = \left[h_t^{(\text{text})};\, h_t^{(\text{audio})};\, h_t^{(\text{video})}\right] \in \mathbb{R}^{1152},
\]
contextualized by a transformer,
\[
z_{1:\tau} = \mathrm{TransformerEncoder}\!\left(h_{1:\tau} + p_{1:\tau} + s\right),
\]
and adaptively average pooled to the fMRI rate before readout. The main subject-specific readout is
\[
\hat{y}_{t}^{(s)} = W^{(s)} \bar{z}_{t},
\]
with a distinct unseen-subject linear layer
\[
\hat{y}_{t}^{(\text{unseen})} = W^{(\text{unseen})} \bar{z}_{t}.
\]
Training uses mean squared error and includes **modality dropout** with probability \(p = 0.3\), subject to at least one modality remaining active [2605.04326].

A notable architectural compromise concerns the video pathway. The model averages over patch tokens from Video-JEPA-2-Giant before temporal encoding, which removes spatial and retinotopic detail. The authors explicitly identify this as a likely cause of weaker performance in low-level visual areas with fine spatial organization, even while the model improves substantially in higher-order multimodal cortices [2605.04326].

## 3. Data regime, targets, and training setup

TRIBE v2 is trained on four naturalistic datasets and evaluated on four held-out studies. The training side comprises **CNeuroMod**, **BoldMoments**, **Lebel2023**, and **Wen2017**, totaling **25 subjects** and **451.6 hours of fMRI**. The held-out evaluation side comprises **NNDb**, **LPP**, **Narratives**, and **HCP**, totaling **695 subjects** and **666.1 hours of fMRI**. Across the full corpus, the paper reports **1,117.7 hours of fMRI**, **121.1 hours of video**, **142.4 hours of audio**, and **71k sentences** [2605.04326].

The targets are high-dimensional and explicitly neuroanatomical. Cortical signals are projected to the **fsaverage** surface and retained at **fsaverage5** resolution; subcortical signals are extracted from **8 subcortical regions** defined by the Harvard-Oxford atlas: hippocampus, lateral ventricles, amygdala, thalamus, caudate, putamen, pallidum, and accumbens. Each vertex time series is **z-scored within session** and detrended, and BOLD signals are linearly resampled to **1 Hz**. The paper emphasizes that detrending materially matters because slow drifts can otherwise inflate apparent encoding scores [2605.04326].

The training and evaluation studies vary by modality composition and scanner regime. A concise summary is sufficient for orientation.

| Split | Datasets | Summary |
|---|---|---|
| Training | CNeuroMod, BoldMoments, Lebel2023, Wen2017 | 25 subjects, 451.6 h fMRI |
| Held-out test | NNDb, LPP, Narratives, HCP | 695 subjects, 666.1 h fMRI |

Optimization uses **AdamW**, batch size **16**, learning-rate warmup to \(10^{-4}\) over the first **10%** of steps, cosine decay, and early stopping with patience **3 epochs**. Feature extraction is cached; the paper reports approximately **24 hours on 128 V100 GPUs** for feature extraction and about **24 hours on a single V100 32GB GPU** for training the brain encoder itself [2605.04326].

The evaluation metric is the Pearson correlation between predicted and measured fMRI responses. In the paper’s notation, the encoding score is
\[
R(\hat{y}, y) = \frac{\sum_t (\hat{y}_t - \bar{\hat{y}})(y_t - \bar{y})}
{\sqrt{\sum_t (\hat{y}_t - \bar{\hat{y}})^2}\sqrt{\sum_t (y_t - \bar{y})^2}}.
\]
The paper also uses parcel- or ROI-aggregated views of these predictions, but the supervised targets themselves are vertexwise and voxelwise rather than parcelwise [2605.04326].

## 4. Encoding performance, transfer, and in-silico experimentation

The principal empirical claim is that TRIBE v2 significantly outperforms an optimized linear FIR-style baseline across all held-out datasets, with \(q(\mathrm{FDR}) < 10^{-4}\). The abstract characterizes the improvement as “several-fold,” while the body text is more specific in the held-out-subject fine-tuning setting: with at most one hour of data from a new subject, fine-tuned TRIBE v2 yields a **two- to four-fold improvement** over a linear encoder trained from scratch on that subject [2605.04326].

The model is also designed for zero-shot transfer. In unseen-subject mode, the paper compares TRIBE v2 predictions against held-out group-average responses and reports that on the **HCP** dataset the model reaches an \(R_{\text{group}}\) near **0.4**, described as a **two-fold improvement** over the median subject’s group-predictivity. The broader interpretation is that the released unseen-subject pathway can serve as an estimator of expected group responses in studies where empirical sampling is limited or absent [2605.04326].

A central aspect of the v2 paper is its use for controlled in-silico neuroscience rather than only naturalistic encoding benchmarks. On visual localizer paradigms from the **Individual Brain Charting** dataset, TRIBE v2 recovers classic category-selective regions including **FFA** for faces, **PPA** for places, **EBA** for bodies, and **VWFA** for written characters. On language paradigms, it reproduces reported effects involving **A5**, **STS**, **Broca’s area (45)**, **TPJ**, and **MTG**, including contrasts such as speech versus non-speech, emotional pain versus physical pain, sentences versus word lists, and complex versus simple sentences. The paper states that predicted and empirical contrast maps show significant spatial correlation across HCP parcels, although the detailed numerical map correlations are not provided in the excerpted text [2605.04326].

TRIBE v2 is also used to probe multisensory cortical organization. Retraining with modality subsets shows that text, audio, and video dominate different territories: audio near auditory cortex, video in occipital and parietal cortex, and text in language cortices and large parts of prefrontal cortex. Comparing the full multimodal model to the best unimodal alternative reveals the largest multimodal gains around the **temporal-parietal-occipital junction**, where the paper reports gains of **up to 50% increase in encoding score**. ICA applied to the unseen-subject readout further yields components corresponding to primary auditory cortex, language network, motion detection area, default mode network, and visual system, which the authors compare against NeuroSynth maps [2605.04326].

## 5. Follow-on analyses and downstream uses

Subsequent work has used TRIBE v2 not merely as an encoder benchmark, but as a reusable computational object. One line of work treats it as a differentiable brain encoder for interpretability. The paper "Feature Visualization Recovers Known Cortical Selectivity from TRIBE v2" freezes **TRIBE v2** and **V-JEPA 2**, then optimizes images to maximize predicted activation in seven visual ROIs. Under shared hyperparameters, the optimized images recover a progression of increasing spatial scale and complexity across **V1 \(\rightarrow\) V4**, face-like structure for **FFA**, rectilinear line patterns for **PPA**, and radial or streak-like “frozen-motion” patterns for **MT** despite static-only optimization. The optimized FFA stimulus yields predicted activation **\(0.339\)**, compared with **\(0.080\)** for a real face photograph, which the authors interpret as a super-stimulus effect rather than a canonical face prototype [2605.13904].

Another line of work uses TRIBE v2 as a source of synthetic training data for inverse problems. "Boosting Brain-to-Image Decoding with TRIBE v2 Data Augmentation" uses a pretrained TRIBE v2 model to generate synthetic cortical responses to static images converted into short still videos, and mixes those synthetic responses with real fMRI to train brain-to-image decoders. On **NSD** and **BOLD5000**, the paper reports **up to 68% improvement in Top-10 image-retrieval accuracy** relative to matched real-only baselines, with strong dependence on the real/synthetic ratio, dataset, and decoder class. It also reports that synthetic-only decoders can perform above chance in some settings, suggesting that TRIBE v2 can support a restricted form of zero-shot brain-to-image decoding [2606.06345].

TRIBE v2 has also been used as a cortical evaluation layer in in-silico studies outside conventional encoding benchmarks. In "Neurological Plausibility of AI-Generated Music for Commercial Environments," the public TRIBE v2 weights and code path are used in an **audio-only** configuration with non-audio modalities disabled, producing fsaverage5 cortical predictions for five prompt-conditioned instrumental tracks. In that study, the **Fast Bright Major Pop** condition achieves the highest whole-cortex mean activation \((0.0402)\), the highest prefrontal composite \((0.0704)\), and the top parcel means in **IFJa \((0.1102)\)**, **IFJp \((0.0995)\)**, **A5 \((0.0188)\)**, and **area 45 \((0.0015)\)**. The authors present this as a cortical proxy for salience- and valuation-related processing, while explicitly rejecting any stronger claim about subcortical reward circuitry or consumer behavior [2604.04025].

Not all downstream repurposing attempts have been positive. "A global predicted-fMRI drive signal from TRIBE does not predict YouTube replay heatmaps" evaluates the released checkpoint referred to as **`facebook/tribev2`** on 48 YouTube videos and finds that a global predicted-fMRI drive signal does not track replay heatmaps: the pooled position-controlled partial correlation is **\(+0.058\)** with 95% CI **\([-0.04, 0.15]\)**, \(t(47)=1.21\), \(p=0.23\). The paper concludes that the released subject-averaged model is not useful, in that setup, as an off-the-shelf predictor of moment-level re-watch behavior, and it emphasizes temporal confounds that can create spurious positive results under weaker controls [2607.01400].

## 6. Interpretation, limitations, and versioning caveats

TRIBE v2 is presented as a foundation model, but its scope is explicitly bounded. The paper states that it models the brain as a **passive observer** of stimuli rather than an acting agent, and that it remains constrained by the spatio-temporal resolution of fMRI. It covers video, audio, and language, but not olfaction, vestibular processing, somatosensation, or broader embodied multisensory functions. It also acknowledges that subcortical predictions are weaker and more uniform than cortical predictions, approximately **two- to three-fold below cortical scores** [2605.04326].

There are also architectural and interpretive caveats. Averaging video patch tokens before temporal encoding discards positional structure and is expected to deteriorate encoding in retinotopically organized visual cortex. The latent analyses and successful in-silico replications support neuroscientific alignment, but they do not amount to a mechanistic or causal explanation of brain function. The authors explicitly note the continuing “black box” character of deep networks, even when ICA and modality ablations yield recognizable systems-level maps [2605.04326].

A further nuance concerns released checkpoints versus the full subject-conditioned formulation. The core TRIBE v2 paper includes both a subject block and an unseen-subject head. By contrast, the downstream replay-heatmap evaluation states that the released **`facebook/tribev2`** checkpoint it used was trained with **`average_subjects = True`**, had **no subject-specific parameters**, **`n_subjects = 0`**, and produced bit-identical predictions across the four Algonauts subjects. This suggests that some public release pathways expose a subject-averaged cortical predictor rather than the full range of subject-conditioned behavior described in the architecture paper. A plausible implication is that downstream failures or successes may depend materially on which inference path or checkpoint is used [2607.01400].

The downstream literature reinforces the need for careful interpretation. The commercial music study uses TRIBE v2 only in a **cortical-only** public inference pathway and therefore cannot support claims about subcortical reward structures such as the **nucleus accumbens** or **amygdala** [2604.04025]. The replay-heatmap study shows that strong brain-encoding performance does not automatically transfer to behavioral engagement prediction [2607.01400]. The feature-visualization study shows that optimized inputs can become adversarial super-stimuli rather than natural exemplars [2605.13904]. Together, these results suggest that TRIBE v2 is best understood as a powerful predictive and experimental substrate whose outputs are scientifically informative but not self-validating.

In that sense, TRIBE v2 occupies a specific place in contemporary computational neuroscience. It is broader than a conventional modality-specific encoding model, more experimentally reusable than a competition-only predictor, and more technically concrete than a purely conceptual “foundation model” metaphor. At the same time, its strongest claims remain those explicitly supported by the published work: high-resolution multimodal fMRI prediction, transfer across studies and subjects, in-silico recovery of established cortical effects, and reuse as a cortical-response prior in downstream modeling pipelines [2605.04326].

Source: https://www.emergentmind.com/topics/tribe-v2