Recurrence is required to capture the representational dynamics of the human visual system (1903.05946v2)

Abstract: The human visual system is an intricate network of brain regions that enables us to recognize the world around us. Despite its abundant lateral and feedback connections, object processing is commonly viewed and studied as a feedforward process. Here, we measure and model the rapid representational dynamics across multiple stages of the human ventral stream using time-resolved brain imaging and deep learning. We observe substantial representational transformations during the first 300 ms of processing within and across ventral-stream regions. Categorical divisions emerge in sequence, cascading forward and in reverse across regions, and Granger causality analysis suggests bidirectional information flow between regions. Finally, recurrent deep neural network models clearly outperform parameter-matched feedforward models in terms of their ability to capture the multi-region cortical dynamics. Targeted virtual cooling experiments on the recurrent deep network models further substantiate the importance of their lateral and top-down connections. These results establish that recurrent models are required to understand information processing in the human ventral stream.

• The paper demonstrates that recurrent neural networks better capture rapid representational transformations in the human visual system compared to feedforward models.
• It employs MEG data and representational dynamics analysis to map evolving visual representations across multiple brain regions.
• Virtual cooling experiments reveal that deactivating lateral and top-down connections significantly impairs object categorization accuracy.

An Examination of Recurrent and Feedforward Models in Capturing Human Visual Dynamics

The research article explores the intricate dynamics of the human visual system, particularly focusing on the importance of recurrent processing in understanding visual perception along the ventral stream. Leveraging magnetoencephalography (MEG) and deep learning models, the paper provides compelling evidence that recurrent neural networks (RNNs) outperform feedforward networks (FNNs) in replicating the rapid representational transformations observed in the human brain during visual object recognition.

The human visual system is often oversimplified as a predominantly feedforward process in traditional studies. This paper challenges that notion by systematically evaluating the bidirectional interactions within and between regions of the ventral visual pathway. Kietzmann et al. employed representational dynamics analysis (RDA) to elucidate how representations within multiple visual areas evolve over the first 300 milliseconds following stimulus onset. The paper mapped the transformations from low-level visual features in early regions (V1-V3) to more complex categorical distinctions in higher regions like IT/PHC.

The empirical results confirmed significant episodic transformations, corroborating the need for recurrent connections to fully capture these dynamics. Granger causality analyses further supported these findings by revealing substantial bidirectional information flows, highlighting feedback pathways complementing the anticipated feedforward interactions.

Recurrent deep neural network models, which incorporate lateral and top-down feedback loops, were systematically compared against parameter-matched feedforward models. RNNs clearly surpassed FNNs in aligning with observed human brain activity. The recurrent architectures demonstrated superior predictive performance in capturing the multi-region cortical dynamics, as evidenced by significantly higher correlation with MEG and fMRI data (average-distance trajectory correlations: 0.95, 0.93, 0.97 for V1-3, V4t/LO, and IT/PHC, respectively).

A particularly innovative aspect of this paper includes virtual cooling experiments on recurrent models, which involved selectively deactivating lateral and top-down connections. The observed marked decline in performance post-deactivation validated the operational significance of recurrent connectivity in object categorization and modeling human ventral stream dynamics.

This research advances existing theoretical frameworks by emphasizing the necessity of recurrent computations for visual recognition, thereby providing critical implications for the design of brain-inspired machine vision systems. The results advocate for the integration of neuroscientific principles, such as recurrence, in developing more effective computer vision models.

Future research endeavors could further delineate the specific functional roles of various types of recurrences and how such mechanisms might be differentially engaged under diverse visual conditions. It would be insightful to explore the scalability of these findings across other sensory modalities and their potential for enhancing adaptive learning in artificial intelligence systems.