Self-organized MT Direction Maps Emerge from Spatiotemporal Contrastive Optimization

Abstract: The spatial and functional organization of the primate visual cortex is a fundamental problem in neuroscience. While recent computational frameworks like the Topographic Deep Artificial Neural Network (TDANN) have successfully modeled spatial organization in the ventral stream, the computational origins of the dorsal stream's distinct topographies, such as direction-selective maps in the middle temporal (MT) area, remain largely unresolved. In this work, we present a spatiotemporal TDANN to investigate whether MT topography is governed by the same universal principles. By training a 3D ResNet on naturalistic videos via a Momentum Contrast (MoCo) self-supervised paradigm alongside a biologically inspired spatial loss, we demonstrate the spontaneous emergence of brain-like direction maps and topological pinwheel structures. Crucially, we reveal that MT tuning properties, characterized by strong direction selectivity paired with a residual axial component, arise from a strict optimization trade-off between task-driven discriminative pressure and spatial regularization. The model's representations quantitatively match in vivo macaque MT physiological baselines, including direction selectivity index, circular variance, and pinwheel density. These findings unify the computational origins of the ventral and dorsal streams, establishing a general mechanism for cortical self-organization.

• The paper demonstrates that self-supervised spatiotemporal contrastive learning paired with spatial regularization yields MT direction maps with pinwheel structures.
• It employs a 3D ResNet-18 and MoCo instance discrimination to attain approximately 72% direction selectivity, closely mirroring macaque MT properties.
• The model reveals trade-offs between discriminative tuning and spatial smoothness, suggesting a universal optimization principle for cortical self-organization.

Emergent MT Direction Maps via Spatiotemporal Contrastive Optimization

Introduction

The paper "Self-organized MT Direction Maps Emerge from Spatiotemporal Contrastive Optimization" (2605.11718) addresses the computational origins of direction-selective topography in the primate dorsal visual stream, specifically the middle temporal (MT) area. The authors extend the Topographic Deep Artificial Neural Network (TDANN) framework previously validated for spatial organization in the ventral stream to the spatiotemporal domain, modeling MT development with a 3D ResNet backbone trained using self-supervised Motion Contrast (MoCo) and spatial locality constraints. The central claim is that direction-selective MT maps and associated pinwheel structures emerge spontaneously through a strict optimization trade-off between discriminative instance learning and biophysical spatial regularization.

Methodology

Spatiotemporal TDANN Architecture

A 3D ResNet-18 architecture processes naturalistic video inputs, establishing correspondence between layers and primate visuotopic areas based on anatomical parameters. Layer 7 serves as the MT analogue due to its hierarchical placement. Activations are mapped to a simulated 2D cortical sheet; temporal averaging simulates neural firing rates for spatial analysis.

Self-Supervised Training Paradigm

MoCo instance discrimination is employed, optimizing representations to distinguish temporally augmented clips from negatives in the UCF101 dataset. The contrastive loss forces temporal invariance, aligning with the motion-sensitive nature of MT.

Spatial Loss

A biologically inspired spatial loss penalizes dissimilar activation profiles among spatially proximate units based on their cortical sheet locations. The total optimization objective balances contrastive and spatial terms:

$$L_{\text{total}} = L_{\text{contrast}} + \alpha \sum_k L_{\text{spatial},k}$$

where $\alpha$ tunes the regularization regime.

Progressive Training Protocol

To decouple spatial optimization from weight-sharing conflicts inherent in CNNs, the approach incorporates multi-stage training: contrastive pretraining, biologically grounded position initialization, iterative spatial rearrangement, freezing unit positions, weight re-initialization, and joint loss training. This simulates prenatal and experience-dependent developmental processes.

Results: Emergent Direction Selectivity and Topography

MT-Like Tuning Properties

Upon stimulation with drifting gratings across 16 directions, the MT-like layer achieves high functional specificity. Approximately 72% of units show direction selectivity (DSI > 0.5, optimal $\alpha=0.5$), closely matching in vivo macaque MT baselines (~80% selective neurons, median DSI ≈ 0.68 vs. 0.77 biological). Emergent population tuning curves (PTCs) display sharp primary peaks at preferred directions with characteristic secondary rebounds (≈ 0.3), indicating persistent axial bias.

Quantitatively:

• Circular Variance (CV): Model MT-layer median CV ≈ 0.73, aligning with physiological baseline (≈ 0.72).
• Statistical bandwidth (wrapped normal): Model ≈ 93.36°, biological ≈ 91°.

Optimization Trade-offs

Strong discriminative pressure (low $\alpha$) produces sharp tuning and fragmented maps, whereas spatial regularization (high $\alpha$) enforces local smoothness and introduces a bimodal response profile. The geometric FWHM narrows (≈ 68.2°), but statistical bandwidth inflates (≈ 93°) due to axial symmetry, explaining observed divergence in neural tuning metrics.

Pinwheel Structure Emergence

Spatial aggregation and smoothing of unit responses reveal spontaneous formation of pinwheel singularities, with positive and negative topological charges organizing into low-energy dipole pairs. Pinwheel density reaches ≈ 6 mm$^{-2}$ at optimal regularization ($\alpha=0.5$), closely matching the empirical macaque MT baseline (≈ 4.9 mm$^{-2}$). Local direction gradients remain below 20°, confirming map smoothness and comprehensive directional coverage. Trade-offs in spatial constraint further modulate pinwheel density and map stability.

Implications

Theoretical Significance

This research substantiates a universal optimization principle underlying cortical feature map formation, bridging the ventral (static, category/topographic) and dorsal (spatiotemporal, motion-selective) streams. MT tuning, including residual axial bias and pinwheel density, is recapitulated as an emergent property from competing task-driven and spatial constraints, dispensing with hand-coded or connectivity-driven models in favor of self-organizing objectives.

Practical Applications

Unified self-organizing Topographic DANN models bear significance for biologically inspired architectures in machine learning, guiding the development of motion-sensitive neural systems with spatially regularized representations. The framework could inform adaptive sensory processing, neurorobotics, and efficient encoding strategies for dynamic stimuli.

Future Directions

Augmenting the model with recurrent dynamics (ConvRNNs), spiking components, and biologically realistic learning rules (e.g., Hebbian plasticity, predictive coding) can further approach physiological fidelity. Task paradigms involving active behavior will enhance quantitative validation. Cross-domain expansion to auditory or multisensory systems may generalize the underlying optimization principle.

Conclusion

The paper demonstrates that the spatial and functional organization of primate MT emerges from the interplay of self-supervised spatiotemporal representation learning and local spatial regularization. Direction-selective maps and pinwheel structures arise as optimal solutions balancing discriminative instance processing and biophysical efficiency, offering a unified computational framework for cortical self-organization across visual streams. This approach paves the way for integrative and neurobiologically plausible models in both neuroscience and artificial intelligence.