ANC Workshop - Bryan Li,

Tuesday, 23rd April 2024

Towards a foundation model of the mouse primary visual cortex

Abstract: Accurate models predicting the visual cortex response to natural visual scenes remain a challenge in computational neuroscience. Convolutional neural networks (CNNs) have so far been the architecture of choice for predicting neural responses from visual stimuli. However, the weight-sharing paradigm and localized receptive fields hinder their ability to model long-range spatial and temporal dependencies. Transformers, on the other hand, excel in learning such dependencies via the self-attention mechanism. We therefore tested whether they can serve as an effective predictor of V1 responses to dynamic visual stimuli and provide insights into dynamic visual processing. To this end, we introduce a spatiotemporal Transformer-based architecture to predict neural activations in the mouse primary visual cortex (V1) from videos. In our approach, a spatial Transformer learns a spatial representation of each video frame, followed by a temporal Transformer that captures interactions between frames, ultimately producing a shared visual and behavioral representation across animals. Notably, our convolutional-free method outperforms the previous state-of-the-art CNN model by 6% and is one of the winning approaches in the Sensorium 2023 challenge. This challenge involved predicting V1 responses from thousands of neurons to over 2 hours of dynamic stimuli from 5 animals and is the first model of this kind to predict dynamic V1 responses. Beyond its strong prediction performance, our model also illuminates the modulation of neural responses by behavioral variables. Specifically, we observe that the self-attention weights in the spatial and temporal Transformers correlate with pupil position and locomotive speed, respectively. This finding not only enhances our understanding of the inner workings of the model but also provides a platform to investigate the computation in the visual system in silico.

Event type: Workshop

Date: Tuesday, 23rd April

Time: 11:00

Location: G.03

Speaker(s): Bryan Li, 

Chair/Host: Milad Jabbari