Feature Visualization Recovers Known Cortical Selectivity from TRIBE v2

Abstract: Brain encoder models predict cortical fMRI responses from the internal activations of pretrained vision and language networks, and are typically evaluated by held-out prediction accuracy. This is a useful signal for training but a poor one for interpretation: it tells us an encoder fits the data without telling us whether it has internalized the functional organization of the brain. We propose feature visualization -- gradient ascent on the encoder's predicted activation for a target region of interest (ROI) -- as a complementary interpretability technique, and apply it to TRIBE v2 composed with V-JEPA 2 (ViT-G, 40 layers), holding both frozen and synthesizing still images for seven regions spanning the ventral and dorsal visual hierarchies. Under identical hyperparameters, the probe recovers a visible progression of increasing spatial scale and feature complexity across V1 to V4, matching the ventral-stream hierarchy. It also produces three distinctive downstream regimes: radial "frozen-motion" streaks for the middle temporal area (MT) despite static-only optimization, face-like features for the fusiform face area (FFA), and consistent rectilinear line patterns for the parahippocampal place area (PPA). Optimized FFA stimuli drive the predicted region ~4x as much as a natural face photograph, consistent with feature visualization producing adversarial super-stimuli rather than canonical exemplars. The probe is simple, differentiable, and applicable to any brain encoder with a differentiable backbone, allowing for qualitative evaluation of brain encoders.

• The paper demonstrates that feature visualization recovers canonical cortical selectivity by optimizing stimuli for specific ROIs using TRIBE v2.
• It utilizes gradient ascent in Fourier space to generate images that elicit targeted fMRI-predicted responses from visual cortical regions.
• Quantitative results show high activation lifts, including a super-stimulus in FFA with ~4× increased activation compared to natural faces.

Feature Visualization as an Interpretability Probe for Brain Encoders

Introduction

This paper addresses a core challenge in neural representational modeling: held-out prediction accuracy is insufficient to claim that a brain encoder internalizes the functional organization of cortex. By employing feature visualization—pixel-space gradient ascent to synthesize inputs maximizing predicted activation for a target cortical region—the study proposes a qualitative probe that interrogates whether an encoder’s internal representations align with canonical neuroanatomical selectivities.

Methodology

The approach utilizes TRIBE v2, a brain encoder mapping V-JEPA 2 (ViT-G, 40 layers) feature embeddings to fMRI-predicted cortical activations. Targeting seven ROIs (V1, V2, V3, V4, MT, FFA, PPA) spanning ventral and dorsal visual pathways, the probe optimizes still images via gradient ascent in Fourier space under a “global lift” loss (target ROI mean minus mean elsewhere in cortex, plus mild spectral energy regularization). Each ROI is represented as a parcel in the HCP-MMP1 parcellation; optimization runs 3000 steps per seed, producing five restarts per ROI.

Figure 1: All 35 optimized stimuli across seven ROIs and five random-seed restarts, visually demonstrating progression from oriented edges in V1 to mid-level structure in V4, motion-like textures in MT, face features in FFA, and rectilinear patterns in PPA.

The generated stimuli are evaluated against random-noise baselines, reporting activation lift and cross-ROI selectivity matrices. Additional benchmarks compare optimized FFA stimuli directly to natural face photographs, quantifying the differential in predicted activation.

Results

Qualitative Recovery of Canonical Tuning

The probe recovers established cortical selectivities:

• V1–V4: Clear gradient from fine-oriented edges to mid-level contour and curvature, mirroring the ventral stream hierarchy.
• MT: Radial, diagonal “frozen-motion” streaks emerge, though only static images are optimized, indicating encoder sensitivity to implied motion.
• FFA: Multimodal face-like structures—eyes, noses, mouths—appear systematically, even in stochastic restarts.
• PPA: Consistent rectilinear, parallel line patterns across restarts, though lacking category-specific scene features.

These outputs are entirely determined by the target ROI; no region-specific hyperparameters or priors are used, maintaining methodological rigor.

Figure 2: Cross-ROI activation comparison for the optimized FFA stimulus vs. random noise, highlighting significant FFA activation and suppression of PPA.

Quantitative Selectivity and Super-stimulus Phenomenon

Activation matrices reveal strong selectivity and inter-region relationships. For example, V3-optimized stimuli drive V4 harder than V3 itself, echoing anatomical overlap. MT achieves the highest activation lift (+0.700), facilitated by a strongly negative baseline from random noise.

Optimized FFA stimuli yield activation ~4× greater than natural face photographs, consistent with feature visualization being adversarial (super-stimulus) rather than exemplar-based. This reflects maximization in model space, not necessarily naturalistic category representation.

Figure 3: Predicted FFA activation for a natural face photograph ($+0.080$) versus the optimized color-64 FFA stimulus ($+0.343$), demonstrating the magnitude of the super-stimulus effect.

Discussion

Implications for Interpretability and Functional Mapping

The results show that TRIBE v2, trained solely to minimize prediction error on naturalistic video fMRI, reconstructs canonical region tuning when inverted via feature visualization. The methodology enables direct inspection of “what each region looks for,” supplementing prediction accuracy with mechanistic interpretability.

The MT result is also crucial—static feature optimization recovers implied-motion signatures, suggesting V-JEPA 2’s backbone encodes motion-related directions even absent temporal information. This aligns with findings that static photographs can evoke motion-selective responses in humans (e.g., Kourtzi et al. 2000), allowing the model to distinguish “motion evidence” from actual movement.

Limitations

Interpretation of optimized stimuli is constrained by their extremal, potentially adversarial nature. Activation maxima do not always correspond to human-recognizable category maxima; robust qualitative evaluation demands inspection across restart ensembles rather than raw activation selection. Results are also limited by the static-only input paradigm and compute-driven restart caps.

Future Directions

Practical extensions include optimizing temporal content for motion-rich regions, expanding ROI sweeps across the full cortical parcellation, clustering regional tuning for functional similarity, and direct comparison to held-out natural fMRI or human observer ratings. Formal statistical rigor requires more extensive restart ensembles and significance testing.

Conclusion

Feature visualization provides a structurally informative, differentiable qualitative probe for brain encoders, revealing whether learned mappings align with neurobiological function. Applied to TRIBE v2 across broad visual hierarchies, the probe recovers recognizable functional structure and inter-region selectivity, notably generating strong, interpretable outputs for motion and face regions. Future iterations will further distinguish encoders that merely predict neural data from those that replicate underlying representations, advancing mechanistic interpretability in neural modeling.