LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation (2310.10404v8)

Abstract: Weakly-Supervised Scene Graph Generation (WSSGG) research has recently emerged as an alternative to the fully-supervised approach that heavily relies on costly annotations. In this regard, studies on WSSGG have utilized image captions to obtain unlocalized triplets while primarily focusing on grounding the unlocalized triplets over image regions. However, they have overlooked the two issues involved in the triplet formation process from the captions: 1) Semantic over-simplification issue arises when extracting triplets from captions, where fine-grained predicates in captions are undesirably converted into coarse-grained predicates, resulting in a long-tailed predicate distribution, and 2) Low-density scene graph issue arises when aligning the triplets in the caption with entity/predicate classes of interest, where many triplets are discarded and not used in training, leading to insufficient supervision. To tackle the two issues, we propose a new approach, i.e., LLM for weakly-supervised SGG (LLM4SGG), where we mitigate the two issues by leveraging the LLM's in-depth understanding of language and reasoning ability during the extraction of triplets from captions and alignment of entity/predicate classes with target data. To further engage the LLM in these processes, we adopt the idea of Chain-of-Thought and the in-context few-shot learning strategy. To validate the effectiveness of LLM4SGG, we conduct extensive experiments on Visual Genome and GQA datasets, showing significant improvements in both Recall@K and mean Recall@K compared to the state-of-the-art WSSGG methods. A further appeal is that LLM4SGG is data-efficient, enabling effective model training with a small amount of training images.

Exploring LLMs for Weakly Supervised Scene Graph Generation

The research paper focuses on the application of LLMs in Weakly-Supervised Scene Graph Generation (WSSGG) to address the challenges of cost and complexity involved in fully-supervised scene graph generation approaches. The work introduces a novel method, LLM4SGG, which leverages LLMs to enhance the process of extracting structured visual knowledge from images by parsing image captions and aligning the parsed content with target entity and predicate classes.

Background and Problem Statement

Scene Graph Generation (SGG) is a crucial task in computer vision that involves identifying objects and their relationships within an image, often requiring extensive data annotation. The traditional fully-supervised SGG methods are heavily dependent on annotated datasets, which are labor-intensive and expensive to produce. To reduce the reliance on such costly annotations, WSSGG has emerged, utilizing easily-obtainable image captions instead.

Nevertheless, two significant issues affect the WSSGG approaches: (1) semantic over-simplification during triplet formation, where complex predicates are reduced to less informative terms, resulting in an imbalanced predicate distribution, and (2) low-density scene graphs due to the inadequate alignment of parsed triplets with the desired entity and predicate classes, leading to insufficient supervision.

Methodology

The LLM4SGG framework addresses these challenges by utilizing LLMs to improve both the extraction and alignment of triplets from captions. The approach is divided into two primary processes:

1. Triplet Extraction (Chain-1): The LLM is used to extract triplets—structures consisting of a subject, predicate, and object—from both original and paraphrased image captions. By leveraging the LLM's language understanding capabilities, fine-grained predicates are better captured, mitigating the semantic over-simplification issue.
2. Class Alignment (Chain-2): LLMs also assist in aligning the extracted triplet components with predefined entity and predicate classes in target datasets, minimizing the discard of useful triplets and thus reducing the low-density scene graph issue.

Importantly, to enhance the effectiveness of triplet extraction and alignment, LLM4SGG incorporates the Chain-of-Thought methodology, which allows step-by-step reasoning, and in-context few-shot learning to adapt LLM capabilities without requiring extensive model fine-tuning.

Experimental Analysis

The approach is validated through experiments on standard datasets like Visual Genome and GQA, showcasing significant improvements in mean Recall@K and Recall@K performance metrics—commonly used indices in scene graph tasks that measure the prediction accuracy of graph components. Compared with baseline methods, LLM4SGG exhibits a marked advantage in achieving data-efficient training, maintaining robustness even with relatively small training datasets.

Implications and Future Directions

LLM4SGG's success opens new avenues for enhancing WSSGG processes and potentially other computer vision tasks through LLMs. By improving triplet formation for scene graphs, the method holds promise for advancing systems that require detailed image understanding, such as autonomous vehicles, robotics, and complex visual question-answering systems.

Future research could explore the use of LLMs for grounding triplets directly in image regions, bypassing traditional object detectors, and possibly integrating vision-language representation models, which translate images into textual data conducive for LLM processing.

In conclusion, LLM4SGG demonstrates the potential of LLMs to address key limitations of weakly-supervised scene graph generation by intelligently parsing and aligning image-caption-based data, thereby facilitating more efficient and expansive scene graph generation frameworks.