Visual Question Answering: A Survey of Methods and Datasets (1607.05910v1)

Abstract: Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. Given an image and a question in natural language, it requires reasoning over visual elements of the image and general knowledge to infer the correct answer. In the first part of this survey, we examine the state of the art by comparing modern approaches to the problem. We classify methods by their mechanism to connect the visual and textual modalities. In particular, we examine the common approach of combining convolutional and recurrent neural networks to map images and questions to a common feature space. We also discuss memory-augmented and modular architectures that interface with structured knowledge bases. In the second part of this survey, we review the datasets available for training and evaluating VQA systems. The various datatsets contain questions at different levels of complexity, which require different capabilities and types of reasoning. We examine in depth the question/answer pairs from the Visual Genome project, and evaluate the relevance of the structured annotations of images with scene graphs for VQA. Finally, we discuss promising future directions for the field, in particular the connection to structured knowledge bases and the use of natural language processing models.

Visual Question Answering: A Survey of Methods and Datasets

The paper "Visual Question Answering: A Survey of Methods and Datasets" offers an in-depth analysis of the current state of Visual Question Answering (VQA), a task that amalgamates computer vision and NLP to enable machines to respond to questions about images. This paper is structured into two primary sections: an overview of VQA methodologies and an examination of available datasets, each pivotal to advancing this interdisciplinary domain.

In the first section, the authors categorize VQA methods into joint embedding approaches, attention mechanisms, compositional models, and knowledge base-enhanced methods. The joint embedding paradigms often involve convolutional neural networks (CNNs) and recurrent neural networks (RNNs) or LSTMs to represent both images and text in a shared feature space. This foundational approach sets the stage for subsequent advancements in VQA methods.

Attention mechanisms are highlighted as pivotal techniques that improve upon joint embeddings by dynamically focusing on pertinent parts of an image relative to the posed question. This mimics selective human attention, which enables more nuanced image analysis. Compositional models like Neural Module Networks and Dynamic Memory Networks are underscored for their ability to structure computational pathways akin to logical reasoning required to tackle complex queries.

The inclusion of knowledge bases addresses the necessity of external, non-visual data to answer questions requiring common sense, domain-specific knowledge, or encyclopedic information. Integrating databases like DBpedia, ConceptNet, and WebChild enriches VQA systems with the capacity to handle questions that extend beyond the image data per se.

The second section explores datasets, evaluating their role in shaping VQA research. Datasets assess the capability of VQA systems to answer questions involving various levels of reasoning and non-visual knowledge. Key datasets, such as DAQUAR, COCO-QA, VQA-real, and Visual Genome, are contrasted based on criteria including image source, method of question generation, and the complexity of questions. The authors argue for the necessity of comprehensive datasets that pose nuanced questions requiring sophisticated model reasoning, beyond mere recognition tasks.

The paper also presents an analysis of structured annotations available in datasets like Visual Genome. This analysis reveals a significant gap where many questions cannot be answered solely from provided scene graphs, underscoring the requirement for enhanced representations or the incorporation of external knowledge bases.

Implications of this research are broad, highlighting pathways for enhancing VQA task formulations and methodologies. The synergistic potential of integrating richer datasets, structured annotations, and external knowledge bases is emphasized as a direction for future research endeavors. Potential developments in related AI domains, such as autonomous systems, interactive agents, and educational technologies, could benefit substantially from advancements in VQA capabilities.

The speculative outlook on future directions advocates for deeper explorations into memory-augmented networks, natural language processing techniques, and logic-based reasoning to ameliorate the integration of external knowledge. As the domain progresses, consolidating efforts between computer vision and NLP through VQA could yield significant intellectual and practical advancements.