VSA4VQA: Scaling a Vector Symbolic Architecture to Visual Question Answering on Natural Images (2405.03852v1)
Abstract: While Vector Symbolic Architectures (VSAs) are promising for modelling spatial cognition, their application is currently limited to artificially generated images and simple spatial queries. We propose VSA4VQA - a novel 4D implementation of VSAs that implements a mental representation of natural images for the challenging task of Visual Question Answering (VQA). VSA4VQA is the first model to scale a VSA to complex spatial queries. Our method is based on the Semantic Pointer Architecture (SPA) to encode objects in a hyperdimensional vector space. To encode natural images, we extend the SPA to include dimensions for object's width and height in addition to their spatial location. To perform spatial queries we further introduce learned spatial query masks and integrate a pre-trained vision-LLM for answering attribute-related questions. We evaluate our method on the GQA benchmark dataset and show that it can effectively encode natural images, achieving competitive performance to state-of-the-art deep learning methods for zero-shot VQA.
- (2016). Neural Module Networks..
- (2021, October). Weakly Supervised Relative Spatial Reasoning for Visual Question Answering. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, QC, Canada: IEEE.
- (2022). Biologically-Based Neural Representations Enable Fast Online Shallow Reinforcement Learning. Proceedings of the Annual Meeting of the Cognitive Science Society.
- (2023, July). Modularized Zero-shot VQA with Pre-trained Models. In A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Findings of the Association for Computational Linguistics: ACL 2023. Toronto, Canada: Association for Computational Linguistics.
- (2020, November). Meta Module Network for Compositional Visual Reasoning. arXiv.
- Choo, F.-X. (2018). Spaun 2.0: Extending the World’s Largest Functional Brain Model. Doctoral Thesis, University of Waterloo.
- Eliasmith, C. (2013). How to Build a Brain: A Neural Architecture for Biological Cognition. Oxford University Press.
- (2023, March). A Neuro-vector-symbolic Architecture for Solving Raven’s Progressive Matrices. arXiv.
- (2019, May). GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. arXiv.
- (2019, July). A neural representation of continuous space using fractional binding. In Annual Meeting of the Cognitive Science Society.
- (2019, July). Representing spatial relations with fractional binding. In Annual Meeting of the Cognitive Science Society.
- Plate, T. A. (2003). Holographic Reduced Representation: Distributed Representation for Cognitive Structures. Center for the Study of Language and Information.
- (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR.
- (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Advances in Neural Information Processing Systems. Curran Associates, Inc.
- (2023, October). What does CLIP know about a red circle? Visual prompt engineering for VLMs. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV). Paris, France: IEEE.
- (2022, March). CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment. arXiv.
- (2012). Spaun: A Perception-Cognition-Action Model Using Spiking Neurons. Proceedings of the Annual Meeting of the Cognitive Science Society.
- (2022, May). ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension. arXiv.
Sponsored by Paperpile, the PDF & BibTeX manager trusted by top AI labs.
Get 30 days freePaper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.