Learning 3D object-centric representation through prediction (2403.03730v1)
Abstract: As part of human core knowledge, the representation of objects is the building block of mental representation that supports high-level concepts and symbolic reasoning. While humans develop the ability of perceiving objects situated in 3D environments without supervision, models that learn the same set of abilities with similar constraints faced by human infants are lacking. Towards this end, we developed a novel network architecture that simultaneously learns to 1) segment objects from discrete images, 2) infer their 3D locations, and 3) perceive depth, all while using only information directly available to the brain as training data, namely: sequences of images and self-motion. The core idea is treating objects as latent causes of visual input which the brain uses to make efficient predictions of future scenes. This results in object representations being learned as an essential byproduct of learning to predict.
- Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11390, 2019.
- Savi++: Towards end-to-end object-centric learning from real-world videos. arXiv preprint arXiv:2206.07764, 2022.
- Neural scene representation and rendering. Science, 360(6394):1204–1210, 2018.
- Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3749–3761, 2022.
- Unsupervised object-centric video generation and decomposition in 3d. Advances in Neural Information Processing Systems, 33:3106–3117, 2020.
- Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- Object-centric learning with slot attention. arXiv preprint arXiv:2006.15055, 2020.
- Information processing with population codes. Nature Reviews Neuroscience, 1(2):125–132, 2000.