Visual Cortex Flow Architecture (VCFlow)

• VCFlow is a unified framework that models visual information flow by integrating dual-stream cortical processing with deep neural network alignment methodologies.
• It employs techniques such as Inter-Subject Representational Similarity Analysis and directed connectivity to reveal latent geometries between brain activity and DNN representations.
• The architecture enables subject-agnostic fMRI-to-video decoding using tri-partitioned ROI processing, multi-level feature extraction, and a hierarchical explicit decoder with contrastive learning.

The Visual Cortex Flow Architecture (VCFlow) defines a convergent, hierarchical computational motif for understanding and modeling the flow of visual information in both biological and artificial systems. VCFlow is characterized by a dual-stream architecture that mirrors the neuroanatomical ventral ("what") and dorsal ("where/how") visual pathways, and it incorporates both diagnostic tools for representational analysis as well as practical frameworks for high-dimensional brain decoding, such as subject-agnostic fMRI-to-video reconstruction. It integrates insights from cortical functional organization, deep neural network (DNN) representational hierarchies, and modern feature-level contrastive learning, yielding both a precise computational phenotype for visual cognition and a performant architecture for scalable neurotechnology applications (Marcos-Manchón et al., 18 Jul 2025, Lu et al., 4 Nov 2025).

1. Foundation: Unified Framework for Representational Flow

The foundational VCFlow framework comprises three methodological pillars: Inter-Subject Representational Similarity Analysis (IS-RSA), Model–Brain Representational Similarity Alignment, and Representational Directed Flow connectivity.

1. Inter-Subject Alignment (IS-RSA): For each cortical parcel $r$ and participant $p$, the trial × voxel matrix $X_{p,r} \in \mathbb{R}^{n \times v_{p,r}}$ is reduced to a Representational Dissimilarity Matrix (RDM):

$D(X_{p,r})_{ij} = 1 - \rho(x_i, x_j)$

where $\rho$ is the Pearson correlation. The vectorized upper triangle of these RDMs is used in the RSA metric:

$$\mathrm{RSA}(X, Y) = \rho(\mathrm{vec}_u\, D(X),\, \mathrm{vec}_u\, D(Y))$$

IS-RSA computes cross-subject stimulus-driven similarity, isolating sites with consistent geometry.

1. Model–Brain Alignment: For model $m$, each layer’s activation $Z_{m,l,s} \in \mathbb{R}^{n_s \times d_m}$ is mapped via RSA to corresponding brain RDMs. The alignment metric for parcel $r$:

$$\mathcal{A}^{\mathrm{BM}}_{p,r,m} = \frac{1}{|\mathcal{T}_p|} \sum_{s \in \mathcal{T}_p} \operatorname{sign\,max}_l\,\mathcal{C}_{p,r,s,m}(l)$$

where $$\mathcal{C}_{p,r,s,m}(l) = \mathrm{RSA}(X_{p,r,s}, Z_{m,l,s})$$. Hierarchical depth $d^*_{p,r,m}$ quantifies the DNN layer most aligned with each parcel.

1. Representational Connectivity and Flow: Pairwise IS-RSA matrices across parcels are used to construct directed representational graphs, wherein the directionality is inferred by hierarchical alignment depth, mapping the "flow" of information from early to late representations.

A central result is the power-law fit between IS-RSA and vision-model RSA ($R^2=0.94$), indicating a unified latent geometry across individuals and models.

2. Two-Stream Cortical Organization and Alignment with Artificial Hierarchies

VCFlow delineates two major functional streams, each with specific anatomical, functional, and model-alignment properties:

• Medial-Ventral Stream (Scene Structure):

Trajectory: V1–V4 $\to$ VMV1–3 $\to$ PHA1–3 $\to$ prefrontal/parahippocampal zones. Alignment: Peaks at intermediate DNN layers (20–50% depth), consistent with encoding mid-level scene semantics. Connectivity: Directed graph edges follow increasing depth, consistent with feed-forward scene integration.

• Lateral-Dorsal Stream (Social/Biological Content):

Trajectory: V1–V4 $\to$ V4t $\to$ MT $\to$ MST $\to$ FST $\to$ TPOJ2–3 (LOTC) $\to$ temporoparietal/social areas. Alignment: Gradually increases towards late DNN layers (80–100% depth), specifically sensitive to animacy and agent-centric content. LLMs align only to the LOTC hub, suggesting specificity for token structure rather than visual-semantic transformation.

These dual streams extend anteriorly into default-mode, parietal, and prefrontal cortices, implying routing of processed visual semantics into systems for memory, decision, and action.

3. VCFlow for Subject-Agnostic Brain Decoding

The subject-agnostic instantiation of VCFlow demonstrates robust generalization for fMRI-to-video decoding, even across unseen test subjects (Lu et al., 4 Nov 2025).

Key architectural steps:

1. Tri-Partitioned ROI Processing: Full-brain fMRI volumes $X \in \mathbb{R}^{B \times S \times V}$ are partitioned into early visual, ventral, and dorsal ROIs, mapping to $$(X_\mathrm{early}, X_\mathrm{ventral}, X_\mathrm{dorsal})$$.
2. Multi-Level Feature Extraction and Alignment (HCAM):
   • Vision Transformer backbone generates global embedding $E_\mathrm{brain}$.
   • Each ROI is projected via MLP into $D$-dimensional space and fused with global embedding using cross-attention.
   • Feature streams are aligned with OpenCLIP embeddings at shallow (early), deep (ventral), and video spaces (dorsal) respectively.
   • BiMixCo contrastive loss and a mean-squared prior loss ensure alignment with CLIP’s semantic space.
3. Subject-Agnostic Redistribution Adapter (SARA):
   • Stack and expand all streams: $F_\mathrm{exp} \rightarrow$ split into $[T_\mathrm{sem}, T_\mathrm{subj}]$ via a Transformer head.
   • Three losses: (a) semantic alignment (to CLIP), (b) inter-subject InfoNCE enforcing invariance, (c) subject-identity classification penalizing residual idiosyncrasy.
   • The aggregate SARA loss:

   $$\mathcal{L}_\mathrm{SARA} = \lambda_\mathrm{align}\mathcal{L}_\mathrm{align} + \lambda_\mathrm{generic}\mathcal{L}_\mathrm{generic} + \lambda_\mathrm{subj}\mathcal{L}_\mathrm{subj}$$

4. Hierarchical Explicit Decoder (HED):

   • Explicit task heads per stream yield segmentation masks, captions, optical flow, and guidance for fusion in a diffusion-based video generator.

Contrastive feature-level learning (InfoNCE with cross-subject positive pairs) ensures that $T_\mathrm{sem}$ encodes subject-invariant visual information.

4. Empirical Results and Performance Benchmarks

VCFlow exhibits several empirical advantages:

• Superior Cross-Subject Generalization:

Trained on a pool of subjects (e.g., 3 of 4 in cc2017), VCFlow generalizes without retraining, with only a 7% (absolute) reduction in 50-way semantic accuracy (from 21.0% to 14.0%) compared to subject-specific baselines. It outperforms prior cross-subject methods GLFA* and NEURONS* by 38–46% in semantic accuracy and 128–189% in SSIM.

• Computational Efficiency:

Full training on four A6000 GPUs requires approximately 3 days. Inference on an unseen subject (8 min fMRI clip) takes 10 seconds (including diffusion decoding).

• Qualitative Fidelity:

Reconstructions reliably capture spatial structure via early-stream features, semantic coherence via ventral stream, and temporal/motion information via dorsal stream.

Metric	VCFlow	NEURONS*	GLFA*
50-way accuracy (%)	14.0	10.1	9.6
SSIM	0.396	0.380	0.137
Video-semantic acc. (%)	18.2	16.1	17.0
CLIP-pcc (temporal cont.)	0.940	0.931	--

*These results reflect averages over three cross-subject splits on cc2017.

5. Diagnostic and Theoretical Implications

The power-law relationship between IS-RSA and model alignment implies a single latent geometry underlying both cross-human and model-brain similarity. This supports convergent encoding of natural scene structure in both the cortex and hierarchical DNNs, rather than unconstrained individual or idiosyncratic representations (Marcos-Manchón et al., 18 Jul 2025).

Kernel Multi-view Canonical Correlation Analysis reveals that first shared subspace axes are interpretable along biologically relevant dimensions: low-level features (early visual), scene-object gradients (ventral), and animacy (LOTC/lateral-dorsal). Partial RSA ablations confirm the functional specificity of these axes, particularly the animacy axis for the LOTC hub.

This convergence suggests that the anatomy of hierarchical DNNs—when appropriately branched and regularized—can serve as accurate computational models of human visual cortex.

6. Blueprint for Future Architectures

The VCFlow architecture defines concrete design principles applicable to future biologically informed vision models:

• Dual-Stream Branching:

Implement early shared backbones branching into ventral (scene/semantic, multi-scale pooling) and lateral (social/animacy, body/face patches, attentive detection) modules.

• Cross-Stream Routing and Weak Inhibition:

Introduce routing gates based on early feature cues; incorporate skip connections that realize empirically observed weak inhibitory inter-stream connectivity.

• Layer-wise Objectives:

Optimize multiple objectives at different depths: low-level reconstruction (early), scene or context classification (ventral), animacy/social-cue decoding (lateral/LOTC), and attenuate residual misalignment with power-law loss terms.

• Explainable Representational Subspaces:

Use multi-view CCA or supervised disentanglement to structure internal feature spaces along interpretable axes (e.g., animacy, scene layout); regularize for stream-specific axes akin to partial RSA.

Adhering to this blueprint aims to simultaneously maximize biological plausibility and performance in complex scene and social understanding tasks, while providing transparent diagnostic axes for interpreting model representations.

7. Significance and Applications

VCFlow operationalizes both a diagnostic toolkit for neuroscientific investigation and a scalable, clinically relevant decoding framework. In the neuroscientific domain, it enables systematic quantification of representational convergence between humans and artificial models, isolates functionally specific axes (e.g., animacy, semantics), and offers a computational scaffold for future connectomic investigations. In neurotechnology, VCFlow enables rapid, high-fidelity brain-to-video decoding deployable in subject-agnostic settings, reducing data requirements from hours to seconds and thus offering a practical path toward clinical utility in cognitive assessment and communication interfaces (Lu et al., 4 Nov 2025).

A plausible implication is that architectures adhering to VCFlow’s dual-stream, layer-wise, explainable design will bridge the gap between cognitive neuroscience and scalable AI, establishing a standardized paradigm for both theoretical modeling and high-impact brain-computer interface systems.