Vision CNNs trained to estimate spatial latents learned similar ventral-stream-aligned representations (2412.09115v2)

Abstract: Studies of the functional role of the primate ventral visual stream have traditionally focused on object categorization, often ignoring -- despite much prior evidence -- its role in estimating "spatial" latents such as object position and pose. Most leading ventral stream models are derived by optimizing networks for object categorization, which seems to imply that the ventral stream is also derived under such an objective. Here, we explore an alternative hypothesis: Might the ventral stream be optimized for estimating spatial latents? And a closely related question: How different -- if at all -- are representations learned from spatial latent estimation compared to categorization? To ask these questions, we leveraged synthetic image datasets generated by a 3D graphic engine and trained convolutional neural networks (CNNs) to estimate different combinations of spatial and category latents. We found that models trained to estimate just a few spatial latents achieve neural alignment scores comparable to those trained on hundreds of categories, and the spatial latent performance of models strongly correlates with their neural alignment. Spatial latent and category-trained models have very similar -- but not identical -- internal representations, especially in their early and middle layers. We provide evidence that this convergence is partly driven by non-target latent variability in the training data, which facilitates the implicit learning of representations of those non-target latents. Taken together, these results suggest that many training objectives, such as spatial latents, can lead to similar models aligned neurally with the ventral stream. Thus, one should not assume that the ventral stream is optimized for object categorization only. As a field, we need to continue to sharpen our measures of comparing models to brains to better understand the functional roles of the ventral stream.

• The paper demonstrates that CNNs trained on synthetic datasets to estimate spatial latents achieve neural alignment scores with the ventral visual stream comparable to models trained on natural images for category recognition.
• Internal representations in CNNs trained on diverse spatial and category latents show significant similarity across layers, suggesting convergence towards common network structures despite varied objectives.
• Models inadvertently learn representations of non-target latents present in the training data variability, indicating data diversity aids in building robust, adaptable internal structures.

Analysis of CNNs for Estimating Spatial and Category Latents

This paper presents a focused paper on the functional role of Convolutional Neural Networks (CNNs) trained to estimate spatial latents, specifically those aligned with the primate ventral visual stream. Traditionally, research on the ventral stream has emphasized its role in object categorization, potentially overlooking its capacity to estimate spatial characteristics such as object position and pose. This investigation leverages synthetic datasets generated by a 3D graphic engine to train CNNs for estimating spatial latents, providing insights into the neural alignment of these models with the primate ventral visual stream.

Key Findings

1. Neural Alignment with Synthetic Datasets:
   • CNNs trained solely on synthetic image datasets show neural alignment scores comparable to those trained on natural images such as ImageNet. Specifically, models trained to estimate spatial latents demonstrate neural alignment scores similar to those obtained from ImageNet-trained models focused on hundreds of categories, despite being trained only on a handful of spatial latents.
2. Similarity in Representations:
   • Internal representations of CNNs trained on different spatial and category latents exhibit significant similarity, particularly in early and middle layers. This finding implies that models trained on varied objectives may converge towards similar networks, challenging the conventional understanding of task-specific architecture requirements in CNNs.
3. Impact of Non-target Latent Variability:
   • Models inadvertently learn representations of non-target latents due to variability in the training data. This suggests that the apparent similarity in representations arises from exposure to diverse latent variables within the dataset, which aids the model in developing a comprehensive and adaptable internal structure.

Implications

The research implies that the canonical separation of the ventral stream into object categorization ("what") and spatial disposition ("where") functions might be overly simplistic. The observed correlation between spatial estimation performance and neural alignment suggests that ventral stream representations are likely multi-dimensional, serving multiple perceptual functions beyond mere categorization.

Speculation on Future Directions

The findings introduce potential pathways for refining computational models to more closely simulate the visual processing characteristics of the ventral stream. Future advancements could explore the enhancement of synthetic datasets in terms of rendering quality and diversity, aiming to achieve even higher neural alignment. The paper opens avenues for understanding how neural models might integrate cross-dimensional latent information into unified representations, potentially informing neural architecture designs in AI that more closely mirror biological vision systems.

The exploration of non-target variability presents opportunities for further investigation into how data diversity impacts model generalization and robustness. These insights could have significant ramifications for improving out-of-distribution performance in vision models, a critical area for developing resilient AI systems.

In summary, the paper contributes valuable insights into the learning dynamics of CNNs vis-a-vis spatial and category latents, suggesting a reconsideration of architectural and task-specific assumptions prevalent in vision modeling research. The implications for understanding the ventral stream as a multi-functional processing unit are profound, warranting continued exploration of integrated visual representations in both cognitive and computational paradigms.