Task-guided cross-subject latent alignment: a multi-encoder-decoder VAE

Abstract: Aligning neural activity across subjects offers the promise of discovering shared computational principles and generalizable decoders. However, traditional alignment methods require shared stimuli across subjects, a constraint that limits applicability to naturalistic paradigms with limited or non-overlapping data. We introduce a Multi-Encoder-Decoder Variational Autoencoder (MED-VAE) that achieves cross-subject alignment without shared stimuli by anchoring representations to a common scaffold provided by a pretrained ANN. Using the Natural Scenes Dataset, we show that MED-VAE creates common latent spaces with superior semantic organisation, achieving higher cross-subject alignment than common methods while maintaining robust generalisation to held-out stimuli where traditional methods degrade. Reconstructing from these common spaces back to each subject's original neural space, MED-VAE preserves equal stimulus-driven signal in its cross-subject latent space. Finally, we show that this superior alignment directly enables cross-subject neural prediction, as demonstrated via cross-subject image decoding. In summary, we introduce a framework to identify generalisable common subspaces for cross-subject predictions and downstream tasks, demonstrated here for visual cortex responses to static images.

• The paper introduces MED-VAE, a novel multi-encoder-decoder VAE that leverages a pretrained ANN to align fMRI data across subjects without requiring shared stimuli.
• The methodology integrates cross-modal reconstructions and KL-regularized latent alignments, yielding superior semantic organization and improved neural prediction metrics.
• Empirical results show enhanced category clustering (silhouette ≈ 0.34) and robust cross-subject prediction, enabling efficient subject integration for population-level analysis.

Task-Guided Cross-Subject Latent Alignment with MED-VAE

Introduction

Cross-subject neural alignment aims to transform fMRI responses from multiple individuals into a common representational space, which is essential for population-level inference, understanding universal coding mechanisms, and enabling generalizable brain-machine interfaces. Existing methodologies such as Shared Response Model (SRM), Procrustes analysis, and Hyperalignment require that subjects are exposed to shared stimuli. This dependence is restrictive, especially for naturalistic or large-scale datasets where stimulus overlap is minimal or absent.

This work introduces MED-VAE, a Multi-Encoder Multi-Decoder Variational Autoencoder that eliminates the need for shared stimuli by leveraging a pretrained artificial neural network (ANN) as a shared computational scaffold. MED-VAE provides an ANN-guided alignment of fMRI data across subjects, creating a common latent space encoding task-relevant visual information and supporting direct reconstruction into each subject’s native neural space. The approach is benchmarked against SRM and Procrustes on the Natural Scenes Dataset (NSD), demonstrating superior semantic organization, cross-subject alignment, and generalization.

Figure 1: The MED-VAE architecture uses subject-specific fMRI encoders/decoders and shared ANN encoder/decoder to enforce alignment in a common latent space, enabling direct cross-subject comparisons even with non-overlapping stimuli.

MED-VAE Architecture and Loss Function

The MED-VAE framework consists of subject-specific fMRI encoders and decoders, as well as shared ANN encoder and decoder modules. Each subject’s fMRI response is projected into a shared $d$-dimensional latent space by their individual encoder. The ANN encoder maps ResNet-50 features (extracted from the visual stimuli) into the same latent space. Each fMRI decoder reconstructs subject-specific voxels from latent codes, while the shared ANN decoder reconstructs ResNet-50 features, enforcing the cross-modal compatibility of representations.

The total loss integrates four pathways: within-modality fMRI (fMRI→latent→fMRI), cross-modal ANN→fMRI, within-modality ANN, and cross-modal fMRI→ANN reconstructions, each regularized by KL divergence on the latent variables. Alignment pressure is governed by a weighting hyperparameter $w$, which tunes the influence of the shared ANN decoder pathway, balancing alignment with subject-specific reconstruction.

At inference, only the subject's fMRI encoder is needed—ANN scaffolding is used exclusively during training, making the framework suitable for practical downstream analysis.

Semantic Structure and Cross-Subject Alignment

Empirically, the latent spaces constructed by MED-VAE exhibit strong semantic organization. Evaluation on multi-label natural scene fMRI data reveals:

• Silhouette scores for category clustering: MED-VAE substantially outperforms SRM and Procrustes, with mean silhouette ≈ 0.34 compared to ≈ 0.22 for alternatives.
• Category decoding: Leave-one-subject-out classifiers trained on the latent space achieve higher exact-match and balanced accuracy for MED-VAE, indicating that the latent geometry captures generalizable, categorical structure.

This semantic organization arises from the ANN scaffold, which is pretrained on object recognition (ResNet-50, ImageNet) and hence induces latent representations that reflect visual category and hierarchical feature structure intrinsic to both the ANN and the human ventral visual stream.

Generalization and Functional Alignment

Alignment quality is measured using component-wise correlation and Representational Similarity Analysis (RSA) across non-overlapping subject data. MED-VAE achieves:

• Component-wise latent correlation: 0.64 (MED-VAE) vs. 0.52 (SRM) / 0.45 (Procrustes)
• RSA: 0.63 (MED-VAE) vs. 0.41 (SRM/Procrustes)

Analysis of the generalization gap (alignment on held-out stimuli vs. shared-training stimuli) demonstrates that SRM/Procrustes degrade on unseen images, with performance declining further as latent dimensionality increases. In contrast, MED-VAE maintains stable high alignment, as its inductive bias derives from task-level structure encoded in the ANN rather than from the statistical relationships in a limited training subset.

Cross-subject retrieval accuracy in the latent space illustrates that MED-VAE preserves stimulus identity more than twofold higher than baselines (top-1 accuracy of 46% vs. ≈ 20%), validating the functional utility of its shared space for applications such as cross-individual decoding.

Signal Preservation and Noise Analysis

Within-trial reconstruction metrics indicate that classical methods seemingly outperform MED-VAE (e.g., within-subject self-reconstruction correlations of 0.71–0.75 vs. 0.61 for MED-VAE), but cross-trial evaluation—where input and target are independent trials of the same stimulus—equates the models, showing no significant differences (~0.18–0.19 for all). This demonstrates that the within-trial advantage of classical models is due to the reproduction of trial-specific noise, not signal.

Noise-ceiling normalized reconstruction metrics confirm this: classical methods exceed the theoretical maximum for stimulus-related variance (>130%), a direct sign of overfitting to noise, whereas MED-VAE remains at ceiling (≈ 100%), indicating principled noise suppression without loss of stimulus information.

Cross-Subject Neural Prediction and Image Decoding

Superior latent alignment directly translates to improved cross-subject neural prediction: using one subject's encoder and another's decoder, MED-VAE achieves higher voxel-wise prediction accuracy for all subject pairs.

Downstream tasks, such as visual image decoding using a pre-trained MindEye2 decoder, exhibit higher performance when the input is generated via MED-VAE latent alignment, particularly on high-level semantic metrics and retrieval accuracy. Notably, at low-dimensional bottlenecks (32D), MED-VAE's output retains more semantic fidelity than classical baselines at 512D, indicating greater efficiency and robustness of the ANN-guided common space.

Practical Implications and Extensions

The MED-VAE approach allows:

• Population-level alignment without shared stimuli, enabling aggregation of fMRI data across protocols and studies with heterogeneous stimulus sets.
• Rapid subject integration: A new individual can be mapped into the shared space with only a subject-specific encoder/decoder trained on as little as 10% of their data, with minimal loss of alignment quality and without affecting the group manifold.
• Robustness to scaffold architecture: Alternative scaffolds (e.g., CLIP ViT-L/14) yield similar, though slightly lower, alignment, confirming generalizability beyond a single ANN.

This architecture is naturally extensible to different cognitive domains with suitable scaffolds (language, audition, multimodal cognition) and can accommodate subjects with different ROI definitions, voxel counts, and potentially different scanners.

Limitations and Future Directions

Residual limitations include the reliance on the representational adequacy of the chosen ANN scaffold—cross-subject alignment is fundamentally capped by the brain-ANN correspondence. Extending the approach to genuinely heterogeneous multi-site and multi-protocol datasets, systematically comparing scaffold architectures, and direct behavioral readout from the latent space are natural next steps.

Conclusion

MED-VAE introduces a principled, task-guided architecture for cross-subject latent alignment, bypassing the need for explicit stimulus overlap, and yielding functionally meaningful, semantically organized common spaces. This not only supports robust neuroscience inference on large, heterogeneous datasets but also enables a range of practically relevant downstream tasks, including cross-subject neural prediction and image decoding.

Figure 1: Schematic of the Multi-Encoder Multi-Decoder VAE architecture, which enables cross-subject latent alignment without explicit stimulus matching by using an ANN scaffold for shared representational structure.