MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding

Abstract: Recent progress in task-optimized neural networks has established encoding models as a powerful tool for predicting brain responses to naturalistic stimuli, yet most existing approaches rely on unimodal representations. The emergence of omni-modal foundation models and rich multimodal neural datasets enables encoding models that jointly integrate visual, auditory, and linguistic information across subjects. We introduce MIRAGE, a brain encoding framework for predicting whole-brain fMRI responses to naturalistic audiovisual stimuli. MIRAGE achieves state-of-the-art performance via a native multimodal backbone and adaptive feature gating across layers. These representations are then combined with a transformer-based brain encoder and a subject-specific linear head over the cortical parcels. Controlled comparisons show that natively multimodal features consistently outperform post-hoc aggregation of independent unimodal features, across architectural levels and backbones. Beyond predictive accuracy, the learned attention weights are directly inspectable to interpret the modality-specific gating profile over the backbone, and each modality traces a distinct anatomical pattern across cortex. Together, these results propose adaptive layer-wise aggregation of natively multimodal features as a generalizable, interpretable, and accurate approach for whole-brain encoding.

• The paper introduces an adaptive multimodal gating framework that fuses vision, audio, and language for whole-brain fMRI encoding using cross-attention queries.
• It leverages a state-of-the-art multimodal model (Qwen3-Omni) to extract deep features, achieving robust performance on in-distribution and out-of-distribution benchmarks.
• Results reveal modality-specific cortical mappings and highlight the critical role of temporal encoding and layer-wise aggregation in accurate brain response prediction.

MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding

Introduction and Motivation

MIRAGE proposes an end-to-end multimodal framework for predicting whole-brain fMRI responses to naturalistic audiovisual stimuli. Unlike prior encoding pipelines that fuse unimodal features only at downstream readouts, MIRAGE leverages a single multimodal foundation model (Qwen3-Omni-30B-A3B-Thinking) to compute natively fused representations across vision, audio, and language. Critically, it introduces adaptive, per-modality layer aggregation via learned cross-attention queries, allowing the model to specialize gating profiles for each modality and region. The work is motivated by the need for depth-aware, fused representation modeling that aligns with the heterogeneous processing hierarchy of cortex and exploits rich multimodal stimulus datasets now available in neuroscience.

Figure 1: MIRAGE architectural flow, showing modality-specific trainable gating over multimodal Qwen3-Omni layers and downstream brain encoding via a temporal transformer.

Methodology

MIRAGE decomposes the modeling pipeline into four explicit components: (a) frozen multimodal feature extraction, (b) modality-specific adaptive layer aggregation through cross-attention pools, (c) concatenative modality fusion, and (d) temporal brain encoding with subject-specific linear readouts over cortical parcels. For each input, time-aligned visual, auditory, and text data are processed through Qwen3-Omni, exposing the hidden states of all backbone layers. Each modality-specific Layer Gating module uses a small bank of learned queries in a cross-attention block, resulting in pooled feature vectors per modality per timestep. These are further concatenated and encoded temporally. The entire pipeline, excluding the backbone, is optimized via mean-squared error between predicted and measured BOLD responses on parcel-wise fMRI data.

Empirical Results

SOTA Performance and Benchmark Comparison

MIRAGE achieves state-of-the-art performance across all splits of the CNeuroMod/Algonauts 2025 data, including the challenging out-of-distribution (OOD) benchmark. Native multimodal fusion consistently outperforms post-hoc fusion of unimodal features at every architectural level, with ensemble MIRAGE reaching a mean Pearson $r$ of 0.323 (in-distribution) and 0.227 (OOD Movies)—a significant improvement over prior ensembles. The drop in predictive accuracy under distribution shift is notably smaller for MIRAGE (30%) compared to post-hoc fusion baselines (40%), evidencing generalizable feature learning and robustness.

Figure 2: (a) Method comparison illustrating MIRAGE’s superiority in BOLD prediction; (b) Backbone ablation, highlighting native fusion advantage over post-hoc strategies.

Modality Contributions and Cortical Distribution

MIRAGE provides fine-grained mapping of modality contributions across the cortex. Leave-one-modality-out ablation identifies dominant modality per parcel, confirming vision dominance in occipitotemporal cortex, audio in superior temporal regions, and text in lateral-temporal/inferior-frontal areas, with multimodal integration prominent in association cortex. Pairwise and trimodal inputs improve prediction over single modalities, supporting the hypothesis of complementary information coding.

Figure 3: Per-parcel accuracy, modality dominance maps, and quantitative modality combination analysis across cortex.

Component Attribution and Architectural Gains

Direct comparison between MIRAGE and matched linear baselines (using identical native-fusion features) isolates gains from the brain encoder and gating modules. The temporal encoder accounts for the largest performance increase, corroborating the importance of modeling temporal dynamics in fMRI prediction. Gains are maximized in Visual and Dorsal Attention networks, and minimized in limbic and sensorimotor regions, tracking the cortical engagement patterns during naturalistic audiovisual tasks.

Figure 4: Parcel-wise Pearson $r$ improvements and network-level analysis demonstrating MIRAGE’s strengths in task-relevant cortical territories.

Layer-Wise Modality Preferences

Inspection of MIRAGE's cross-attention weights reveals distinct layer preferences per modality within Qwen3-Omni. Vision sharply clusters around mid-depth layers (25–30), text distributes across mid-to-late layers (with subclusters in 25–30 and 35–40), and audio exhibits a broader distribution over mid-depth layers. All modalities largely ignore early layers, indicating that brain-aligned features emerge in deeper processing stages. Per-head and per-query analyses confirm modality-specific specialization rather than diffuse averaging.

Figure 5: Layer-wise cross-attention profiles for vision, text, and audio modalities.

Ablation of Gating Query Number

Performance monotonically increases with the number of cross-attention queries per modality, plateauing around $n_q = 24$. Single-query pooling underfits the representational structure required for accurate brain prediction, but larger $n_q$ yields diminishing returns beyond 12–24.

Figure 6: Validation Pearson correlation vs. number of cross-attention queries per modality.

Broader Implications and Future Directions

MIRAGE demonstrates that native fusion within multimodal foundation models, paired with adaptive layer aggregation, yields superior and interpretable encoding models for whole-brain fMRI. This confirms that cross-modal interactions learned during pretraining are more brain-relevant than post-hoc unimodal fusion. Practically, the approach suggests broader applicability to other neural recording modalities (EEG, MEG, ECoG) and motivates richer subject variability modeling. The architectural transparency and interpretability also afford critical neuroanatomical insights into modality-specific computation.

Future extensions could involve hypernetwork-based subject adaptation, leveraging the modality-agnostic design for generalized neural interface development, and fine-grained parcel-level layer attribution for deeper alignment analyses.

Conclusion

MIRAGE establishes adaptive, multimodal layer gating as a generalizable and accurate method for whole-brain neural encoding from naturalistic stimuli. Native multimodal feature fusion outperforms post-hoc strategies consistently, with adaptive cross-attention providing interpretable modality and layer selectivity. The architecture is modular, scalable, and transparent, offering both practical and theoretical advances in model-neuroscience alignment and brain response prediction.

Figure 7: Subject-averaged encoding-accuracy maps highlighting MIRAGE’s gains on both in-distribution and out-of-distribution sets.