An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2109.05014v2)

Abstract: Knowledge-based visual question answering (VQA) involves answering questions that require external knowledge not present in the image. Existing methods first retrieve knowledge from external resources, then reason over the selected knowledge, the input image, and question for answer prediction. However, this two-step approach could lead to mismatches that potentially limit the VQA performance. For example, the retrieved knowledge might be noisy and irrelevant to the question, and the re-embedded knowledge features during reasoning might deviate from their original meanings in the knowledge base (KB). To address this challenge, we propose PICa, a simple yet effective method that Prompts GPT3 via the use of Image Captions, for knowledge-based VQA. Inspired by GPT-3's power in knowledge retrieval and question answering, instead of using structured KBs as in previous work, we treat GPT-3 as an implicit and unstructured KB that can jointly acquire and process relevant knowledge. Specifically, we first convert the image into captions (or tags) that GPT-3 can understand, then adapt GPT-3 to solve the VQA task in a few-shot manner by just providing a few in-context VQA examples. We further boost performance by carefully investigating: (i) what text formats best describe the image content, and (ii) how in-context examples can be better selected and used. PICa unlocks the first use of GPT-3 for multimodal tasks. By using only 16 examples, PICa surpasses the supervised state of the art by an absolute +8.6 points on the OK-VQA dataset. We also benchmark PICa on VQAv2, where PICa also shows a decent few-shot performance.

Overview of "An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA"

The paper presented in the paper reflects an empirical investigation into the capabilities of the GPT-3 model within the context of knowledge-based visual question answering (VQA). The research addresses the challenge of answering questions that require domain-specific external knowledge beyond what is provided in an image. This typically involves a two-step process where external knowledge is first retrieved and then used for reasoning over the image and question to predict answers. The authors propose an alternative—leveraging GPT-3 as an implicit and unstructured knowledge base, facilitated through image captions as input prompts—demonstrating especially promising results when compared with previous state-of-the-art methods on specific datasets.

Methodological Innovations

The authors introduce a novel application of GPT-3 for knowledge-based VQA without resorting to traditional structured knowledge bases. The key idea is to translate images into text captions that GPT-3 can interpret, treating the model as an implicit knowledge base capable of processing the acquired knowledge. This approach allows GPT-3 to predict answers using only a few in-context examples effectively.

Key methodological contributions include:

1. Prompt Engineering with Captions: The method converts the visual content of images into descriptive textual captions that serve as input to GPT-3. This represents a shift away from explicit knowledge retrieval towards implicit querying of a vast pretrained LLM.
2. Few-Shot Learning Paradigm: The model's adaptation to the VQA task relies on few-shot learning capabilities by incorporating limited in-context examples rather than traditional supervised fine-tuning, significantly reducing the dependency on large labeled datasets.
3. Enhanced Performance through In-context Example Selection: The paper employs techniques such as image-question similarity ranking using models like CLIP to enhance the selection of in-context examples, and multi-query ensemble methods to exploit the available information maximally.

Empirical Results

The proposed method, particularly in its full configuration (PICa-Full), demonstrates significant improvements over baseline models and existing state-of-the-art methods. Notably, with just 16 in-context examples, it achieves an accuracy of 48.0% on the OK-VQA dataset, surpassing supervised methods by 8.6 percentage points. Moreover, the model attains respectable performance on the VQAv2 dataset, showing broader applicability across different specified contexts.

Implications and Speculative Outlook

The paper's findings illuminate the latent potential of LLMs like GPT-3 in domains traditionally constrained by explicit knowledge retrieval efforts. The adaptability of GPT-3 in few-shot settings highlights its role both as a repository of relational and encyclopedia-style knowledge, but also its reasoning capability in synthesizing responses based on incomplete datasets.

Looking forward, exploring the integration of multimodal inputs directly within the architectures of such LLMs could provide further performance boosts. Additionally, expanding this framework to other vision-and-language tasks beyond VQA could foster innovations in domains like autonomous systems, information retrieval, and content generation, broadening the current understanding of AI's multimodal reasoning capabilities.

In conclusion, this paper makes significant strides in exploiting large-scale LLMs for knowledge-intensive multimodal tasks, paving a pathway toward more generalized, robust AI systems capable of dynamic reasoning across data types and formats.