Algonauts Project Benchmarking

• Algonauts Project is a benchmarking initiative that quantitatively evaluates brain-inspired models using multimodal neural data to predict sensory responses.
• It evolved from static image-based challenges in 2019 to sophisticated multimodal tasks in 2025, integrating diverse datasets and modalities like fMRI and MEG.
• The project employs various computational approaches including linear encoding, end-to-end architectures, multimodal fusion, and ensembling techniques to enhance predictive accuracy.

The Algonauts Project is a biennial, open benchmarking initiative at the interface of computational neuroscience and artificial intelligence, dedicated to quantitatively evaluating and advancing models that predict human brain responses to complex naturalistic sensory inputs. Structurally, the project is organized around public challenges, each defined by a rigorously curated multimodal neural dataset, clear evaluation protocols, and a transparent, automated leaderboard infrastructure that enables direct comparison of diverse computational modeling approaches (Cichy et al., 2019, Gifford et al., 2023, Gifford et al., 2024).

1. Founding Principles and Historical Evolution

The central premise of the Algonauts Project is that progress in both biological and artificial intelligence can be accelerated through systematic, quantitative confrontation: models inspired by the brain are tested for their ability to explain empirically observed neural data, and the resulting insights feed back into the refinement of architectures for both AI and neuroscientific theory (Cichy et al., 2019).

The project originated in 2019 with “Explaining the Human Visual Brain,” focusing on measuring and modeling fMRI and MEG data from subjects viewing static images. Subsequent challenges have progressively escalated in complexity and ecological validity:

• 2019: Predict fMRI/MEG responses to controlled object-images using representational dissimilarity matrices (RDMs) (Cichy et al., 2019, Jacob et al., 2020, Gaziv, 2019, Fonseca, 2019).
• 2021: Transition to dynamic stimuli—short video clips with whole-brain fMRI (Cichy et al., 2021).
• 2023: Predict responses to tens of thousands of natural scenes (NSD), marking a new scale in both stimuli and neural measurements (Gifford et al., 2023, Nguyen et al., 2023).
• 2025: Multimodal, long-form naturalistic movies (∼80 h per subject), requiring multimodal (video, audio, text) encoding and out-of-distribution (OOD) generalization (Gifford et al., 2024, Scotti et al., 14 Aug 2025, Eren et al., 23 Jul 2025, Scholz et al., 2 Oct 2025).

This progression reflects both empirical advances in neural data acquisition and methodological developments in multimodal AI and machine learning.

2. Data Modalities and Challenge Designs

Each Algonauts challenge is defined by its dataset, which constrains the modeling and evaluation paradigm:

Edition	Primary Stimuli	Modality	Neural Data	Targets (readouts)
2019	Static images	fMRI, MEG	15 subjects, 92+118 images	ROIs: EVC, IT (fMRI); early/late (MEG)
2021	Short video clips (3 s)	fMRI	10 subjects, 1102 videos	Voxelwise/ROI-level BOLD
2023	Natural Scenes Dataset (NSD, >70k)	fMRI (7 T)	8 subjects, 73,000 images	Vertexwise BOLD on fsaverage
2025	Multimodal movies, TV+film	fMRI (3 T)	4 subjects, 80+ hr per subj	1000 whole-brain parcels

Across editions, the input feature space has progressed from static, single-modality to rich, time-aligned, multimodal representations, with increasing data volume, stimulus diversity, and requirements for temporal and cross-modal integration (Gifford et al., 2024, Gifford et al., 2023, Cichy et al., 2021).

3. Computational Modeling Frameworks

The Algonauts Project has catalyzed a spectrum of computational approaches for stimulus-to-brain mapping, unified by a common evaluation infrastructure:

• Linear encoding: Extract pretrained features (e.g., AlexNet, ResNet, ViT), then fit parcel-/voxel-wise ridge regressions (Gifford et al., 2023, Gaziv, 2019).
• End-to-end architectures: Directly train CNN, RNN, or transformer models to map sensory input to brain responses, often via MSE or correlation-based losses (Eren et al., 23 Jul 2025, Jacob et al., 2020, Nguyen et al., 2023).
• Multimodal fusion: Fuse modality-specific embeddings (video, audio, transcript) through concatenation, attention, or recurrent pooling. Top-performing 2025 models universally leverage modality-specific encoders (e.g., VideoMAE, Whisper, LLaVA, BERT), with downstream fusion via transformers or RNNs (Scotti et al., 14 Aug 2025, Eren et al., 23 Jul 2025, Scholz et al., 2 Oct 2025).
• Ensembling: Combination of hundreds of base models by weighted averaging, stacking, or parcel-specific weighting. This consistently outperforms any single architecture, with brute-force ensembling yielding a mean OOD performance gain of ~0.005–0.007 in Pearson’s r̄ (Scotti et al., 14 Aug 2025, Eren et al., 23 Jul 2025, Scholz et al., 2 Oct 2025).
• Regularization and transfer: Use of pretrained and “stimulus-tuned” feature extractors, LoRA finetuning, curriculum loss schedules, and explicit modeling of subject-specific idiosyncrasies via dedicated output heads or adapters (Scholz et al., 2 Oct 2025, Nguyen et al., 2023, Eren et al., 23 Jul 2025).

Model training protocols emphasize split-by-subject/scene cross-validation, careful OOD validation, and matching of the target BOLD dynamic range to the model output (Gifford et al., 2024, Jacob et al., 2020).

4. Evaluation Metrics and Leaderboard Protocols

Evaluation is uniformly based on the predictive alignment between model outputs and held-out neural data, with challenge-specific primary metrics:

• 2019 (RDM-based): Compare model-predicted and empirical RDMs via Spearman correlation and noise-normalized explained variance ($R^2_\text{norm}$) (Cichy et al., 2019, Jacob et al., 2020, Fonseca, 2019, Gaziv, 2019).
• 2021-2025 (direct mapping): For each subject and spatial target (parcel/vertex), compute Pearson’s $r$ between predicted and observed time-series. Final score: mean $r$ across all spatial targets and subjects, or mean noise-normalized $R^2$ for NSD (Gifford et al., 2023, Gifford et al., 2024, Scotti et al., 14 Aug 2025, Eren et al., 23 Jul 2025).
• Public leaderboard: Automated scoring infrastructure (Codabench, custom scripts), immediate feedback, indefinite post-challenge phase, and OOD-only final ranking in 2025 (Gifford et al., 2024, Scotti et al., 14 Aug 2025).

The challenges incorporate noise ceilings—measured as inter-subject or repeat-based consistency—to contextualize raw prediction scores and promote accurate comparisons across tasks and data (Cichy et al., 2019, Gifford et al., 2023).

5. Notable Algorithms and Empirical Findings

The Algonauts challenges have revealed several robust modeling and neuroscientific observations:

• Late-stage features and predictiveness: Even for primary visual/early cortical regions, the most predictive features often arise from late layers of deep neural networks. Adaptive, channel-wise gating of ResNeXt or AlexNet consistently upweights the deepest stages (Gaziv, 2019, Jacob et al., 2020).
• Multimodal integration: OOD prediction of higher-order association cortex depends critically on the integration of video, audio, and linguistic information; unimodal models cannot achieve leading scores in association parcels (Scotti et al., 14 Aug 2025).
• Pretrained-fusion outperforms end-to-end: Across recent editions, the dominant recipe is pretrained encoders with shallow fusion and subject-specific heads, rather than from-scratch, domain-specific architectures (Nguyen et al., 2023, Eren et al., 23 Jul 2025, Scotti et al., 14 Aug 2025).
• Ensembling and parcel-weighted stacking: Constructing large, diverse model ensembles, and tuning their weights parcel-wise, yields systematic incremental gains and robustness, particularly in low-SNR fMRI regions (Eren et al., 23 Jul 2025, Scholz et al., 2 Oct 2025).
• Curriculum loss: Training schedules that dynamically reweight early vs. late parcel loss terms improve performance in high-order cortical targets (Eren et al., 23 Jul 2025).
• Unsupervised predictive coding: Networks trained to predict future states (PredNet) can outperform image-classification baselines for both fMRI and MEG alignment, challenging assumptions about the necessity of supervised learning for biological plausibility (Fonseca, 2019).

Typical leaderboard scores in OOD settings (2025) are mean $r̄ \approx 0.209 – 0.214$ for top ensembles, with single parcel peaks (e.g., V1) reaching $r \approx 0.6$ while DMN or ATL parcels remain at $0.1 – 0.2$ (Scotti et al., 14 Aug 2025, Eren et al., 23 Jul 2025).

6. Community Practices, Open Science, and Reproducibility

The Algonauts Project mandates open reporting and public code release for winning entries. Finalists typically deposit preprint methods reports and public repositories are maintained for major solutions (e.g., https://github.com/uark-cviu/Algonauts2023; https://github.com/erensemih/Algonauts2025_ModalityRNN) (Nguyen et al., 2023, Eren et al., 23 Jul 2025).

Strict OOD validation, embargoed neural data splits, and limited leaderboard submissions in final test phases are enforced to discourage overfitting. Leaderboards persist beyond the challenge window to enable ongoing benchmarking (Gifford et al., 2024, Scotti et al., 14 Aug 2025).

Comprehensive development kits, reproducible data loaders, and extensive code bases have become standard, supporting both educational use and large-scale method comparisons (Nguyen et al., 2023).

7. Current Limitations and Future Directions

Despite substantial advances, several open problems remain:

• Subject specificity: Most models are still subject-adapted; cross-subject or universal encoding remains unsolved (Scotti et al., 14 Aug 2025).
• Temporal modeling limitations: Owing to rigid fMRI resolution, current approaches are not fully temporally continuous; explicit hemodynamic response functions are often omitted but may provide further gains (Scotti et al., 14 Aug 2025).
• Scaling and integration: Scaling laws with respect to data, model, or subject size remain ambiguous, with no observed power-law scaling yet (Scotti et al., 14 Aug 2025).
• Extension beyond passive viewing: Future challenges are anticipated to incorporate active tasks, MEG/EEG integration, and closed-loop model-stimulus co-evolution, as well as datasets covering decision-making, attention, and other cognitive domains (Gifford et al., 2024, Scotti et al., 14 Aug 2025).

The project aspires to advance both human neuroscience—by constraining and testing mechanistic hypotheses—and AI, by leveraging brain-inspired structure as inductive bias for more robust, generalizable artificial systems (Cichy et al., 2019, Gifford et al., 2024).