The Dynamic Sensorium competition for predicting large-scale mouse visual cortex activity from videos (2305.19654v2)

Abstract: Understanding how biological visual systems process information is challenging due to the complex nonlinear relationship between neuronal responses and high-dimensional visual input. Artificial neural networks have already improved our understanding of this system by allowing computational neuroscientists to create predictive models and bridge biological and machine vision. During the Sensorium 2022, we introduced benchmarks for vision models with static input. However, animals operate and excel in dynamic environments, making it crucial to study and understand how the brain functions under these conditions. Moreover, many biological theories, such as predictive coding, suggest that previous input is crucial for current input processing. Currently, there is no standardized benchmark to identify state-of-the-art dynamic models of the mouse visual system. To address this gap, we propose the Sensorium 2023 Benchmark Competition with dynamic input. It includes the collection of a new large-scale dataset from the primary visual cortex of ten mice, containing responses from over 78,000 neurons to over 2 hours of dynamic stimuli per neuron. Participants in the main benchmark track will compete to identify the best predictive models of neuronal responses for dynamic input. We will also host a bonus track in which submission performance will be evaluated on out-of-domain input, using withheld neuronal responses to dynamic input stimuli whose statistics differ from the training set. Both tracks will offer behavioral data along with video stimuli. As before, we will provide code, tutorials, and strong pre-trained baseline models to encourage participation. We hope this competition will continue to strengthen the accompanying Sensorium benchmarks collection as a standard tool to measure progress in large-scale neural system identification models of the entire mouse visual hierarchy and beyond.

• The paper presents a novel competition that benchmarks models predicting neural responses to dynamic visual stimuli in mice.
• It leverages a large-scale dataset of over 38,000 neurons from five mice, integrating video and behavioral data for robust modeling.
• Evaluations using baseline models and dual-track metrics offer actionable insights into predictive coding and neural variability.

Overview of the SENSORIUM 2023 Competition: Predicting Dynamic Visual Cortex Activity

The "Dynamic Sensorium Competition", as highlighted in this paper, reflects an evolution in the standard approaches to mouse visual cortex research, shifting the focus from static images to dynamic stimuli. The competition's primary goal is to assess and benchmark computational models that predict neuronal responses to dynamic visual inputs, using a large-scale dataset from the primary visual cortex of mice. The initiative is poised to address a significant gap in the current landscape of neural prediction challenges, providing a standardized method for evaluating models capable of processing spatiotemporal stimuli akin to real-world scenarios.

Motivations and Objectives

Recognizing the intricacies of spatiotemporal processing in biological visual systems, this competition is grounded in empirical evidence suggesting that the temporal dynamics inherent in natural environments play a crucial role in visual perception and neural encoding. Historically, vision research has focused extensively on static images, which simplifies the modeling process but fails to capture the complexity of neural responses to moving stimuli and changes over time. The Sensorium 2023 competition thus aims to provide a robust platform for researchers to benchmark models that account for these dynamics, inspired by theories like predictive coding that emphasize the significance of prior information in processing current stimuli.

Dataset and Methodological Framework

An impressive dataset involving over 38,000 neurons from five mice was constructed, allowing for two hours of dynamic stimuli per neuron. This comprehensive dataset illuminates neural responses across varied video stimuli and includes measurements of mouse behavior, such as pupil and locomotion data, to inform model predictions further. The competition includes both a main track, focusing on the predictivity for natural videos, and a bonus track that tests model robustness and generalizability to out-of-domain (OOD) stimuli—such as Gaussian dots or drifting Gabors—unseen in the training phase.

Three baseline models provide reference points for competitors: a GRU-based recurrent model, a spatial-temporal 3D convolutional model, and an ensemble of factorized convolution models. These serve not only as baselines but demonstrate successful architectural paradigms from previous studies and provide insights into dynamic neural processes' potential modeling approaches.

Performance Evaluation

The competition’s evaluation criteria focus on metrics such as single-trial correlation and correlation to average, ensuring that models can capture trial-to-trial variability as well as average response trends. This dual focus addresses the challenge of translating model performance across different experimental noise levels and variability inherent in biological datasets.

Implications and Future Directions

The SENSORIUM 2023 competition is positioned as a key driver for future advancements in understanding the mouse visual cortex, with implications reaching beyond academic curiosity. By challenging the participants to model responses to dynamic stimuli, it accelerates the progress of understanding biological vision and develops methods potentially transferable to artificial systems. These developments hold promise for practical applications, ranging from improved machine vision systems to deeper insights into visual processing disorders.

Additionally, as a scaffold for ongoing innovation, the competition framework invites iterative refinement and the continual introduction of new datasets, fostering a dynamic and evolving benchmark resource. Such competitions expand the community’s methodological toolkit, enlighten underlying neural mechanisms, and pave the way for applications that span across neuroscientific and technological enterprises.

In conclusion, as models become increasingly adept at predicting complex and dynamic neuronal responses, competitions like SENSORIUM 2023 not only push the boundaries of neuroscientific research but also stimulate interdisciplinary engagement—utilizing insights from machine learning, neuroscience, and beyond to decode the brain’s visual processing intricacies.