Boosting Brain-to-Image Decoding with TRIBE v2 Data Augmentation

Abstract: Brain decoding is limited by the availability of labeled neural data, and remains challenging in low-data regimes. To address this issue, we investigate whether and when brain decoding can be boosted by augmenting small fMRI datasets with synthetic data generated by a pretrained model of fMRI responses to stimuli. We use TRIBE v2, a large encoding model pretrained on more than 1000 hours of fMRI responses to video, audio and language. For each dataset, we evaluate systematic grids that show how the performance of image decoders varies with the amount of synthetic data used for training. Our results, based on two datasets (the 7T fMRI Natural Scenes Dataset and 3T fMRI BOLD5000), show up to 68% improvement in Top-10 image-retrieval accuracy compared to decoders trained only on real data. Importantly, the proportion of augmented data required to reach a given image decoding performance needs to be adjusted depending on the data source. Surprisingly, image decoders trained exclusively on synthetic fMRI can perform above chance in some settings, suggesting that TRIBE v2 can support zero-shot brain-to-image decoding. Together, these results show how large-scale models of the fMRI responses to sight, sound and language may provide a foundation to improve the data efficiency for image decoding.

• The paper introduces a TRIBE v2-based synthetic augmentation protocol that significantly improves image retrieval and reconstruction by leveraging limited real fMRI data.
• It demonstrates that optimal synthetic-to-real ratios can boost performance, achieving up to 68% higher Top-10 retrieval accuracy and nearly full-dataset results with reduced real data.
• The paper highlights trade-offs in augmentation strategies for Ridge and deep decoders, emphasizing benefits in low-data regimes and challenges with over-augmentation.

Model-Based fMRI Augmentation for Brain-to-Image Decoding with TRIBE v2

Introduction

The paper "Boosting Brain-to-Image Decoding with TRIBE v2 Data Augmentation" (2606.06345) systematically investigates the utility of model-based, image-conditioned synthetic data augmentation for improving brain-to-image decoding, specifically in scenarios where labeled neural data are scarce. Leveraging TRIBE v2, a foundation model that predicts fMRI responses to stimuli across modalities, the study demonstrates augmentation protocols capable of significantly enhancing image retrieval and reconstruction performance by extending training data with population-based synthetic fMRI responses. Comprehensive benchmarking is conducted on two public fMRI datasets: the 7T Natural Scenes Dataset (NSD) and the 3T BOLD5000, with in-depth analysis of synthetic-to-real data ratios, decoder class, and data regime.

TRIBE v2 Augmentation Protocol

TRIBE v2 is a tri-modal encoding model pretrained with over 1,000 hours of fMRI data, spanning visual (V-JEPA 2), auditory, and linguistic stimuli. Notably, while TRIBE v2 is designed for dynamic stimulus encoding, the protocol adapts it to static images by converting each image into a short, motionless video. This enables the model to generate synthetic cortical fMRI responses for new images, which are then used to augment real datasets for downstream decoding tasks.

Figure 1: TRIBE v2 image-conditioned fMRI augmentation protocol, mapping unseen images to synthetic fMRI responses using a pretrained model and mixing these with real data to train decoders.

Synthetic augmentation is parameterized by two variables: the percentage of real fMRI retained ($p$), and an augmentation factor ($a$) controlling the relative number of synthetic pairs. Decoders are then trained and evaluated on held-out real fMRI, with performance systematically mapped across this operating grid.

Experimental Framework and Datasets

The methodology spans both retrieval (image embedding prediction) and reconstruction (pixel or semantic image recovery) tasks. For retrieval, decoders map fMRI responses to DINOv2-small embeddings, with evaluation via Top-$K$ image retrieval. For reconstruction, DynaDiff is used, enabling direct generation of images from fMRI.

Data sources:

• NSD: 7T fMRI, high spatial resolution, $\sim$10,000 unique images per full subject (subjects 1, 2, 5, 7 considered).
• BOLD5000: 3T fMRI, $\sim$5,000 unique images per subject, more heterogeneous and lower resolution than NSD.

Preprocessing aligns both real and TRIBE-synthetic fMRI on a common cortical mesh (fsaverage5), allowing seamless mixing for supervised decoder training.

Figure 2: Schematic depiction of the Natural Scenes Dataset (NSD) acquisition protocol for image-fMRI mapping.

Performance of Synthetic Data Augmentation

The core results are visualized as operating grids revealing how decoding accuracy varies with the fraction of real and synthetic data.

Key numerical findings:

• Top-10 image retrieval accuracy improves by up to 68% over real-only baselines in low- and medium-data regimes (notably BOLD5000 with Ridge decoders).
• TRIBE augmentation enables achieving $\sim$90% of full-dataset performance using only half the real data in NSD; with BOLD5000, similar accuracy is reached with merely 30% of the real data.
• Exclusive training on TRIBE-synthetic data yields above-chance performance in some regimes, demonstrating substantive transfer of visual content from the population encoding model to the decoding target.
• Optimal synthetic/real ratios are dataset- and decoder-dependent; excessive augmentation can saturate or degrade performance, indicating a non-trivial balance.

  Figure 3: Image reconstruction metrics on NSD with DynaDiff; TRIBE augmentation increases low- and high-level image similarity up to a critical augmentation factor, after which gains saturate.

Importantly, augmentation with random noise, as a control, does not confer similar benefits—validating that the generative signal, not just additional variance, enables decoding gains.

Decoder Class and Data-Regime Specificity

Both linear (Ridge regression) and deep residual MLP ("MindEye"-style) decoders were evaluated:

• Ridge models manifest the largest and most consistent relative improvements from TRIBE augmentation, with significant reduction in real fMRI requirements to reach fixed performance.
• Deep decoders, which are already high-performing, show more modest and variable improvements; their performance is harder to surpass as most gains are realized in the real-only matched setting.

Synthetic augmentation is most beneficial when operating with limited real data (low $p$). As $p$ increases, the marginal benefit declines and can invert at large augmentation factors, highlighting the regularization effect of diverse, albeit subject-agnostic, synthetic samples.

Zero-Shot Decoding and Transferability

A salient result is that decoders trained solely on TRIBE-synthetic fMRI responses (i.e., zero real data from the target subject and paradigm) achieve better-than-chance Top-10 retrieval in certain instances, particularly with BOLD5000 and deep architectures. This suggests a nontrivial alignment between the population-level encoding signal and individual test subject data, even in out-of-distribution contexts.

Figure 4: Subject-level decoder performance stratified by augmentation regime, revealing heterogeneity in synthetic data transfer across individuals.

Implications, Limitations, and Future Directions

These findings imply that foundation models such as TRIBE v2 can serve as generative priors, greatly improving data efficiency in neuroimaging/AI pipelines. Careful calibration of synthetic-to-real ratios is critical, as mismatched or over-augmented regimes can harm performance. The approach is fundamentally limited by its population-level, subject-agnostic nature, with the strongest results requiring some subject-specific real fMRI. The methodology is demonstrated for image decoding but can extend to multimodal (e.g., audio, language) settings, as TRIBE v2 is a general encoding model.

Prospective research avenues include:

• Subject-adaptive synthetic data generation (personalizing TRIBE outputs).
• Joint training approaches leveraging both synthetic augmentation and multi-subject pretraining.
• Extending operating-grid analysis to other neuroimaging paradigms and generative models.
• More principled strategies for synthetic instance selection and domain shift correction.

Conclusion

The paper demonstrates a robust, principled framework for leveraging large-scale, model-based synthetic fMRI augmentation to address data scarcity in brain-to-image decoding. With up to 68% improvement in retrieval accuracy in constrained regimes, the results underscore the utility of foundation encoding models as synthetic data generators, provided strict calibration and domain considerations are observed. The approach facilitates reduced scan-time requirements, potentially democratizing research in neural decoding, while highlighting open challenges in tailoring synthetic augmentation to individual and task-specific contexts.