Beyond Prediction Accuracy: Target-Space Recovery Profiles for Evaluating Model-Brain Alignment

Abstract: Artificial vision models are often evaluated against the human visual cortex by measuring how accurately their internal representations predict brain responses. However, prediction accuracy alone does not indicate which dimensions of the target brain's response space are recovered. Here, we introduce a unified framework for evaluating both model-brain and brain-brain alignment by identifying the response dimensions recovered by prediction. Using repeated fMRI measurements, we first identify target-brain response dimensions that can be reproducibly predicted across independent trial splits. We then predict target-brain responses from either another subject's brain responses or a vision model's internal representations, and quantify how strongly each of these reproducible response dimensions is recovered. Applying this framework to a subset of the Natural Scenes Dataset, in which eight subjects viewed the same natural images during fMRI, we find that the early-to-intermediate visual-cortex responses contain a low-dimensional set of reproducible dimensions. Brain-to-brain comparisons identify which of these dimensions are consistently recoverable from other subjects' brains, providing a diagnostic human reference rather than only a scalar benchmark. In some cases, pretrained and randomly initialized models achieve similar prediction accuracy while showing distinct recovery profiles across these response dimensions. These results show that prediction accuracy alone can mask model-brain mismatches. By making explicit which reproducible brain response dimensions are recovered by prediction, our framework provides a more diagnostic evaluation of alignment between artificial vision models and the human visual cortex.

• The paper presents a predictive subspace framework that quantifies model–brain alignment by examining the overlap with reproducible brain response dimensions.
• It employs split-half cross-validation and regularized low-rank linear mappings to derive metrics such as Directional and Top-K Reference Coverage.
• Empirical results reveal that pretrained models recover broader, more human-like subspaces compared to random initializations, despite similar prediction accuracies.

Beyond Prediction Accuracy: Recovery Profiles for Model–Brain Alignment

Introduction and Conceptual Framework

This work introduces a new framework for evaluating the alignment between artificial vision models and the human visual cortex—specifically, it moves beyond conventional measures of prediction accuracy to characterize the specific brain response dimensions that are recovered by different sources, whether they are other human brains (brain-to-brain) or artificial models (model-to-brain). The foundational claim is that high scalar prediction accuracy does not suffice for diagnostic evaluation: it does not reveal which target-space directions (i.e., reproducible dimensions in a brain ROI) are matched.

The proposed predictive subspace framework leverages repeated fMRI measures from the Natural Scenes Dataset (NSD) to define a data-driven "reproducible target reference": the subset of ROI response space dimensions that are reliably predicted across independent splits of repeated stimulus presentations. Any model or brain source is then evaluated not merely via accuracy, but also by the degree—and pattern—with which its induced predictive subspace overlaps with these robust, biologically meaningful dimensions.

Figure 1: Overview of the predictive subspace framework, with a pipeline for estimating the reproducible target reference and quantifying the overlap between source-induced predictive subspaces and this reference across models and subjects.

Methodology: Predictive Subspace and Recovery Profile

Formally, for each subject–hemisphere–ROI ("target unit"), the reproducible target reference is constructed by split-half cross-prediction across repeated trials, resulting in a soft subspace (captured via the eigenspectrum of the average of split-specific projection operators). The principal axes (target-reference directions) and their reference weights quantify the structure and reliability of the response space.

To assess a source (another brain or a model), a regularized low-rank linear mapping is fit from source representations to the target ROI. The prediction quality is summarized not only by the held-out correlation ("accuracy") but, crucially, by how the resulting predictive subspace aligns with the reference axes—via axis-wise directional coverage and prefix-weighted "top-$k$" coverage metrics.

Key structural diagnostics include:

• Directional Reference Coverage ($DirCov$): Degree to which each reference direction $u_j$ is present in the predictive subspace.
• Top-$k$ Reference Coverage ($TopKCov$): Prefix-weighted average overlap over the $k$ leading, most reproducible directions.
• Profile Mean: Overall average of the $TopKCov$ across a fixed number of directions (typically $K=10$).

  Figure 2: The reference eigenspectrum is dominated by a small number of directions, and restricting readouts to this predictive subspace preserves prediction accuracy as well as outperforming source-side matched-rank PCA controls.

The framework allows for direct, quantitative comparison between the recovery patterns afforded by brain-to-brain and model-to-brain mappings—enabling both reference ceiling estimation and deeper analyses of which brain-relevant features are accessible to models.

Empirical Results: Recovery Profiles and Model Differences

Low-Dimensional Structure in Visual Cortex

Repeated-measure approaches confirm that response spaces in early-to-intermediate visual ROIs are inherently low-dimensional: a few reference directions cover the vast majority of reproducible variance (e.g., top 3 dimensions explain $\sim$89% of the reference weight; median effective rank $\sim$5.1).

Brain-to-Brain Recovery as a Structured Human Reference

Brain-to-brain recovery profiles reveal systematic decline from near-complete coverage for the leading direction ($DirCov$0; 0.959) to diminished coverage as lower-weight dimensions are incorporated ($DirCov$1; 0.868). This structured profile provides a richer diagnostic than scalar accuracy or noise ceilings.

Figure 3: Brain-to-brain top-$DirCov$2 reference-coverage curves across visual ROIs and their relation to prediction accuracy, illustrating the non-identity between these metrics.

Model–Brain Discrepancies: Prediction Accuracy vs. Structure

Crucially, the analyses reveal that sources (models and brains) with equivalent prediction accuracy can exhibit markedly different recovery profiles. For models, the dichotomy between pretrained and randomly-initialized representations is especially apparent: while both can achieve similar held-out correlation, only pretrained models (especially those trained on ImageNet) recover a broader and more human-like distribution of target-reference dimensions (mean profile coverage difference in top-$DirCov$3 reference: +0.177).

This is robust to model architecture, ROI, and random initialization (random ViT seeds have minimal effect relative to pretraining). Pretraining broadens coverage not only of the leading dimension but also of additional subspace directions (top-$DirCov$4 difference: +0.137 at $DirCov$5, +0.178 to +0.189 for $DirCov$6–10).

Figure 4: Models with matched prediction accuracy have substantially different coverage curves, and pretraining considerably broadens reference dimension recovery.

Controls and Robustness

Thorough validation demonstrates that the observed effects are not artifacts of trial splits, response variance, or low-rank PCA approximations. Controls using matched-rank PCA, random target directions, and dataset shift (via NSD-synthetic) consistently show that recovery profiles are highly diagnostic and model-specific.

Figure 5: Robustness analyses confirm that observed recovery profiles are not explained by simple variance models or data-specific artifacts.

Theoretical Implications and Scope

This framework fundamentally reframes model–brain evaluation, prioritizing not just "how much" is predicted, but "which" brain-like structures are accessible. It builds on and extends prior work on communication subspaces, functional alignment (e.g., SRM, hyperalignment), and spectral decompositions of model–brain alignment. By explicitly operationalizing target-space recovery, it enables separation of conditional model–brain alignment (ROI, dataset, preprocessing-specific) from unwarranted claims of global brain-likeness.

Model Evaluation Criteria

Scalar prediction scores are insufficient: two sources may have identical scores yet recover orthogonal subspaces of the biologically relevant signal. The recovery profile localizes these discrepancies, specifying which parts of the target reference each model accesses and which it fails to recover. This raises the standard for mechanistic claims of brain-model homology and provides a path for more granular, interpretable diagnostics—facilitating substantive model improvement.

Extensions and Limitations

The framework is adaptable to any repeated-measurement neural dataset with response variability, but its deployment is most natural in domains where meaningful subspace dimensions can be extracted (high-dimensional brain activity rather than categorical labels or single-dimensional outputs).

Limitations include reliance on repeated measures, the linearity and rank constraints of the predictive mapping, and focus on early-to-intermediate visual areas; extensions to higher-order cortex or other modalities will require validating reference stability and subspace interpretability.

Figure 6: Axis-wise directional coverage reveals that pretraining increases coverage for leading reference directions, but late directions remain variable and weakly captured by models.

Practical and Future Directions

Practically, the recovery profile enables model selection and diagnosis based on criteria aligned with robust population-level brain representations. The approach provides an evaluative reference (brain-to-brain recovery) without relying on arbitrary noise-ceiling heuristics, thereby supporting the evaluation of transfer, robustness, and generalization under dataset shifts.

Theoretically, the framework motivates a spectrum of future research:

• Layer-wise and Objective Diagnostics: Extending model surveys to include objective- and layer-specific analyses, revealing which architectural choices optimize human-relevant subspace recovery (see, e.g., effects of self-supervision, CLIP/DINO objectives).
• Beyond Vision: Application to multimodal or higher-level cognition, probing whether models trained on fundamentally different objectives recover comparable subspaces in language or memory circuits.
• Interpretability: Future work could aim to semantically decode the leading reference directions, linking predictive subspaces directly to computational or cognitive functions.

Conclusion

This study demonstrates that recovery profiles of reproducible target-reference subspaces provide a more nuanced, diagnostically valid methodology for model–brain alignment analysis than scalar prediction accuracy alone. Many models with high accuracy differ in whether, and how, they recover meaningful dimensions of brain activity—a difference that straightforward accuracy metrics obscure. This reframing enables both comparison and mechanistic interpretation of models, setting a new standard for evaluating neural plausibility and guiding principled model development in NeuroAI.

Figure 4: Models differ in recovered dimensions beyond prediction accuracy, with pretraining systematically broadening the set of target-reference dimensions accessed by the model.