Hierarchical Multimodal Recurrent Ensemble (fMRI)

• The paper presents a hierarchical multimodal recurrent ensemble that maps pretrained video, audio, and text embeddings to fMRI responses, achieving an overall Pearson correlation of 0.2094.
• It employs modality-specific bidirectional RNNs and a post-RNN layer to fuse representations, effectively modeling temporal dependencies in naturalistic movie stimuli.
• Subject-specific output heads and ensemble averaging across 100 models enhance prediction robustness and scalability for distributed cortical responses.

Searching arXiv for the specified paper and closely related benchmark context. A hierarchical multimodal recurrent ensemble is a brain-encoding architecture designed to predict distributed cortical responses to naturalistic movie stimuli by integrating visual, auditory, and semantic information over time. In the formulation reported for the Algonauts 2025 challenge, the system maps pretrained video, audio, and language embeddings to parcel-wise fMRI BOLD time series using modality-specific bidirectional RNNs, a fused recurrent representation, and lightweight subject-specific output heads. The reported implementation ranked third on the competition leaderboard, achieved an overall Pearson correlation of $r = 0.2094$, and obtained the highest single-parcel peak score, with mean peak parcel score $= 0.63$ (Eren et al., 23 Jul 2025).

1. Problem formulation and benchmark setting

The model addresses the brain encoding problem posed by the Algonauts 2025 challenge: given naturalistic movie stimuli with aligned video, audio, and transcript text, predict fMRI BOLD responses across the cortex. The stated objective is to predict distributed cortical responses to movies by integrating visual information, auditory information, and semantic or language information over time with a recurrent architecture that can model temporal dependencies in naturalistic stimuli (Eren et al., 23 Jul 2025).

The benchmark setup uses the Algonauts 2025 dataset with 65 hours of training data, comprising 55 hours from Friends seasons 1–6 plus the Movie10 set consisting of 4 movies. Validation and model selection involved leaderboard scores based on Friends Season 7 during model-building and an out-of-distribution movie set of about 2 hours during model-selection. fMRI acquisition used $\mathrm{TR} = 1.49\,\mathrm{s}$, and the brain targets were 1000 cortical parcels based on the Schaefer 2018 parcellation, covering early sensory through higher-order association networks. Evaluation was based on Pearson correlation between predicted and true parcel time series, averaged across parcels and subjects.

The reported system uses four subject-specific heads, and the performance table is organized for Subject 1, Subject 2, Subject 3, and Subject 5. This subject-specific formulation is central to the benchmark instantiation: the recurrent backbone is shared across subjects, while the final mapping to parcel responses is individualized.

2. Multimodal representation pipeline

The architecture begins with frozen pretrained encoders that produce modality-specific embeddings. Four visual encoders are used: SlowFast, VideoMAE, Swin Transformer, and CLIP. These are applied to 1.49-second video clips and then time-aligned to the fMRI time points. Three audio encoders are used: HuBERT, WavLM, and CLAP. These features are extracted from 1.49-second windows and aligned to fMRI. Two text models are used on dialogue transcripts: BERT for local semantic features and Longformer for longer-range context (Eren et al., 23 Jul 2025).

For language continuity, the system prepends the previous episode’s transcripts when available, so that the LLM can better capture context at the start of segments. The paper reports this as a targeted adjustment for segment boundaries rather than a wholesale change in the language modeling stack.

This multimodal front end is explicitly modular. The authors describe the method as extensible because new modalities can be added by plugging in additional encoders and modality RNNs, while the recurrent backbone and subject heads remain unchanged. A plausible implication is that the architecture is intended less as a highly specialized one-off design than as a reusable scaffold for naturalistic stimulus encoding.

3. Hierarchical recurrent architecture

The core of the model is hierarchical and recurrent. Each modality sequence $\mathbf{x}_m(t)$ is encoded by its own bidirectional RNN:

$$\mathbf{h}_{m}^{\rightarrow}(t) = \mathrm{RNN}_{m}^{\rightarrow}\bigl(\mathbf{x}_m(t),\,\mathbf{h}_{m}^{\rightarrow}(t-1)\bigr)$$

$$\mathbf{h}_{m}^{\leftarrow}(t) = \mathrm{RNN}_{m}^{\leftarrow}\bigl(\mathbf{x}_m(t),\,\mathbf{h}_{m}^{\leftarrow}(t+1)\bigr)$$

$$\mathbf{h}_m(t) = \bigl[\mathbf{h}_{m}^{\rightarrow}(t)\;;\;\mathbf{h}_{m}^{\leftarrow}(t)\bigr] \in \mathbb{R}^{2H}$$

Each RNN is single-layer, with hidden size $H = 768$, and the forward and backward hidden states are concatenated. This produces time-resolved modality-specific hidden representations (Eren et al., 23 Jul 2025).

Fusion is implemented by simple elementwise averaging over modality-specific hidden states:

$$\bar{\mathbf{h}}(t) = \frac{1}{M}\sum_{m=1}^M \mathbf{h}_m(t)$$

The fused representation remains time-resolved, has dimension $\mathbb{R}^{2H}$, and combines visual, auditory, and linguistic information. The authors explicitly report that this simple average worked best, that it was regularizing, and that learned weights or attention did not improve performance.

After fusion, the sequence is passed through a second recurrent layer, termed a post-RNN, intended to capture cross-modality temporal structure:

$= 0.63$0

This layer produces a unified latent representation $= 0.63$1. The post-RNN can be either an LSTM or a GRU depending on the model variant. The hierarchy therefore has two levels: modality-specific temporal encoding followed by recurrent integration of fused multimodal state.

4. Subject adaptation, objective function, and curriculum

Prediction is performed through subject-specific linear heads. For subject $= 0.63$2,

$= 0.63$3

with $= 0.63$4 and $= 0.63$5. During training, each sample is routed through the head corresponding to that subject. The paper emphasizes that this design is parameter-efficient and flexible enough to account for subject-specific scaling and idiosyncrasies while retaining a shared recurrent backbone (Eren et al., 23 Jul 2025).

Training uses a composite MSE-correlation objective:

$= 0.63$6

where $= 0.63$7 is the Pearson correlation between predicted and true parcel responses at time $= 0.63$8:

$= 0.63$9

with

$\mathrm{TR} = 1.49\,\mathrm{s}$0

and $\mathrm{TR} = 1.49\,\mathrm{s}$1 parcels. The paper characterizes this as a composite MSE-correlation loss or MSE+NegCorr style training. The stated intuition is to fit amplitudes with MSE while encouraging pattern similarity with correlation.

A further training mechanism is a dynamic loss-weighting curriculum inspired by hierarchical brain processing. Early-processing parcels are defined as Visual and Somatomotor networks, while late-processing parcels are all remaining parcels. The loss is split into $\mathrm{TR} = 1.49\,\mathrm{s}$2 for early ROI loss and $\mathrm{TR} = 1.49\,\mathrm{s}$3 for late ROI loss. At the start of training, $\mathrm{TR} = 1.49\,\mathrm{s}$4 and $\mathrm{TR} = 1.49\,\mathrm{s}$5, and these weights are linearly annealed toward $\mathrm{TR} = 1.49\,\mathrm{s}$6 and $\mathrm{TR} = 1.49\,\mathrm{s}$7. The reported effect is improved convergence on more difficult ROIs, a small boost in overall correlation, and particular benefit for mid-level and higher cortical areas, although the hardest prefrontal cortex prediction problem remained unresolved.

5. Training protocol, filtering, and ensemble construction

The implementation is reported in Python 3.10 and PyTorch 2.7, with Adam, learning rate $\mathrm{TR} = 1.49\,\mathrm{s}$8, and batch size 4. Each movie episode paired with its fMRI time series is treated as one sample. Early stopping is based on cross-validation score, and cross-validation is group-wise leave-one-movie-out on Movie10. The authors note that cross-validation improvements tracked leaderboard improvements closely (Eren et al., 23 Jul 2025).

A post hoc filtering step is also described. The procedure is: train on all movie segments, evaluate predictions per segment, and remove segments with near-zero correlation. The explicitly removed segments are Friends season 6 18b, 19a, and 19b for subject 1; Friends season 5 13a; and Movie10 bourne01 for subject 2. The stated purpose is to reduce noise and improve robustness.

The final submission is an ensemble of 100 models. It consists of five configurations: Base RNN architecture with MSE-only loss; Base architecture with MSE + correlation loss; GRU post-RNN with MSE loss; GRU post-RNN with MSE + correlation loss; and the early-vs-late ROI curriculum variant. For each configuration, 20 independent random seeds were trained, each for 5 epochs, yielding $\mathrm{TR} = 1.49\,\mathrm{s}$9 models. At inference time, predictions from all ensemble members are averaged for each subject and time point. The authors report that individual models were similar in score, averaging reduced variance, and ensemble averaging gave about 2% improvement while increasing robustness and consistency across subjects.

6. Empirical results, ablations, and interpretive significance

The reported competition outcome is third place on the Algonauts 2025 leaderboard, with overall score $\mathbf{x}_m(t)$0. The abstract further emphasizes the highest single-parcel peak score among all participants, with mean peak parcel score $\mathbf{x}_m(t)$1, and notes particularly strong gains for the most challenging subject, Subject 5 (Eren et al., 23 Jul 2025).

Per-subject ensemble performance is reported as 0.289 for Subject 1, 0.260 for Subject 2, 0.283 for Subject 3, and 0.247 for Subject 5, with average 0.270. The corresponding single-model averages are 0.264 for LSTM-GRU MSE+NegCorr, 0.265 for LSTM-LSTM MSE, 0.264 for LSTM-GRU MSE, 0.264 for LSTM-LSTM MSE+NegCorr, and 0.265 for the early-late curriculum. The ensemble is best across all subjects. Subject 5 remains the hardest, with the lowest values across models, but still improves under ensembling.

Ablation analysis compares three alternatives against the full model: a single-head variant that keeps modality encoders but collapses subject-specific heads; a unified-modality variant that uses one shared RNN instead of modality-specific encoders; and a unified-modality plus single-head variant that removes both modality separation and subject specificity. All ablations are worse than the full model. The reported interpretation is that both modality-specific encoders and subject-specific heads are beneficial.

The paper also highlights qualitative subject- and region-level findings. Subject 5 is described as the most challenging, yet the model performed best on this subject relative to competitors. Performance is described as especially strong in auditory predictions. The weakest region is the prefrontal cortex, likely due to insufficient language or context modeling, and even adding previous episode transcripts helped only modestly.

The broader significance assigned to the method is that it serves as a simple, extensible baseline for multimodal brain encoding on naturalistic stimuli. The paper suggests that multimodal integration matters, temporal modeling matters, subject-specific adaptation matters, simple fusion can be sufficient, ensembling remains highly effective, and curriculum learning is promising though modest in effect. It also suggests that higher-order cortex remains hard, especially prefrontal areas, and that better language or context modeling and longer-range modeling may be required. Within that framing, the hierarchical multimodal recurrent ensemble is less a claim of architectural maximalism than a demonstration that a straightforward recurrent, multimodal, and subject-adaptive design can remain competitive on a large-scale benchmark.