Only Brains Align with Brains: Cross-Region Alignment Patterns Expose Limits of Normative Models

Abstract: Neuroscientists and computer vision researchers use model-brain alignment benchmarks to compare artificial and biological vision systems. These benchmarks rank models according to alignment measures such as the similarity of representational geometry or the predictability of neural responses from model activations. However, recent works have identified a number of problems with these rankings, among them their lack of discriminative power and robustness, raising the conceptual question of what it means for a model to be brain-aligned. Here we introduce alignment patterns -- characteristic functional relationship profiles of each brain region to all others -- and propose that models should reproduce these patterns to qualify as brain-aligned. First, we apply a standard benchmarking pipeline to a broad spectrum of vision models of the BOLD Moments video fMRI dataset across visual regions of interest (ROIs). We find diverse models appear equivalent in their brain alignment, reflecting the lack of discriminative power of conventional alignment benchmarking pipelines. In contrast, alignment pattern analysis (APA) is a second-order structural consistency test: a model aligned to a given ROI should reproduce that ROI's characteristic cross-region alignment profile. Applying APA, we find that, while these patterns are highly stable across brains of different subjects, even top-ranked models often fail to capture them. Finally, we argue for a clearer distinction between the criteria a model must meet to serve as a tool versus as a computational model for human visual cortex. Conventional alignment measures may be sufficient for identifying neurally predictive models, but claims about computational or algorithmic similarity may require a stronger basis of evidence, including the reproducibility of relational alignment patterns.

• The paper demonstrates that conventional alignment measures (RSA, LP) inadequately discriminate models with distinct architectures and objectives.
• It introduces alignment pattern analysis (APA) to evaluate relational cross-region profiles, offering a refined benchmark for assessing brain-like models.
• The study reveals that high alignment scores may result from non-biological representations, challenging normative claims about 'brain-likeness' in vision models.

Cross-Region Alignment Patterns Reveal Limits of Normative Model-Brain Alignment

Introduction

This paper interrogates the validity and discriminative power of conventional model-brain alignment benchmarks in visual neuroscience and computer vision. The authors demonstrate that widely used alignment metrics, such as representational similarity analysis (RSA) and linear predictivity (LP), inadequately distinguish models that are fundamentally distinct in architecture and objective but yield similar alignment scores to brain regions. They introduce alignment pattern analysis (APA)—a second-order relational criterion based on the reproducibility of cross-region alignment profiles—as a necessary extension of pointwise alignment measures. This framework exposes the limits of normative claims about "brain-likeness" and offers a refined basis for discriminating computational models from mere neural predictors.

Model-Brain Alignment and Benchmarking Limitations

The authors evaluate a large suite of state-of-the-art vision models, spanning supervised, self-supervised, and multimodal approaches—including Taskonomy, ConvNext, SimCLR, CLIP, V-JEPA, video transformers, and 3D foundation models—against the BOLD Moments video fMRI dataset. Alignment scores are computed across visual cortical ROIs using RSA and LP. Consistent with prior studies, self-supervised models such as V-JEPA and variants of CLIP rank highly under both metrics (2604.21780). However, models with starkly different training regimes, architectures, and objectives cluster as "effectively equivalent" in alignment score distributions. The authors operationalize equivalence via bootstrap confidence intervals, revealing a lack of discriminative power in existing benchmarking pipelines.

Furthermore, normalization of scores relative to intersubject (brain-brain) alignment, as proposed by the NeuroAI Turing Test, shows that LP is saturated in many regions—model-brain scores reach or surpass brain-brain alignment—whereas RSA does not saturate, indicating further room for nuanced assessment.

Alignment Pattern Analysis: A Relational Structural Consistency Criterion

APA evaluates not only the degree to which a model aligns with a single ROI, but also whether it recapitulates the characteristic cross-region alignment profile of that ROI as observed in human brains. These profiles are highly reproducible fingerprints for each visual area, stable across subjects and distinct across regions, as demonstrated by fMRI-derived alignment patterns and corroborated via structural connectivity data. APA thus constitutes a relational extension of the NeuroAI Turing Test, wherein a model is only considered "brain-aligned" if its relational alignment pattern similarity (APS) to brain patterns matches the intersubject (brain-brain) APS.

Applying APA, the authors show that equivalently aligned models under conventional metrics diverge sharply in their alignment patterns. Top-ranking predictive models, particularly the self-supervised V-JEPA suite, often fail to capture the relational fingerprints of key visual regions. Models that optimize global predictive scores can exhibit non-biological cross-region alignments, undermining claims about computational or algorithmic similarity to cortex.

Discrimination Between Tools and Computational Models

The findings underscore a critical distinction between models that function as neurally predictive tools and those that instantiate plausible computational mechanisms of the visual cortex. Alignment measures such as LP, when saturated, merely indicate predictive utility—not mechanistic similarity. APA reveals that achieving high predictive scores across multiple regions does not imply correspondence to the hierarchical or functional architecture of primate vision. For instance, models that yield comparably strong alignment to both early and late visual regions violate known constraints of hierarchical complexity gradients, as elucidated in ventral stream literature.

APA provides a quantifiable mechanism for enforcing this distinction, narrowing the pool of candidate "brain-like" models and ruling out classes of models (e.g., V-JEPA) that fail the relational criterion, despite pointwise score equivalence. The authors caution that interpretive claims based on normalized scores must be scrutinized for meaningfulness, particularly when unexplained variance persists or the normalization reference is suboptimal.

Theoretical Implications and Prospects

This work frames model-brain alignment as an underdetermined problem, subject to the contravariance principle: task constraints (increased task difficulty or ecological validity) should restrict the space of viable solutions, promoting representational convergence. Nonetheless, current performance scaling in artificial vision models increasingly decouples from mechanistic similarity, as optimization trajectories favor alternative, non-biological implementations. As a result, observed intermediate alignment may stem from convergence in generic, task-agnostic representations rather than task-critical computations.

APA strengthens the theoretical foundation for benchmarking, applying stricter constraints independent of task difficulty, and operationalizes the demand for mechanism-level evidence. The results call for rigorous causal evaluation, explicit scientific commitments about the evidentiary power of alignment measures, and renewed emphasis on relational benchmarks as models grow in scale and predictive utility.

Conclusion

The paper demonstrates that pointwise model-brain alignment benchmarks lack discriminative adequacy and are insufficient for meaningful normative claims about computational similarity. Alignment pattern analysis offers a robust relational test, sharply differentiating models in terms of their ability to recapitulate characteristic cross-region alignment profiles. This relational criterion is essential for progressing towards explanatory models of visual cognition, distinguishing predictive tools from mechanistically plausible computational models. As vision models become increasingly powerful, APA and similar relational benchmarks will be pivotal in maintaining coupling between performance metrics and scientific insight into the brain's underlying computational principles (2604.21780).