Neural Motifs: Scene Graph Parsing with Global Context

Abstract: We investigate the problem of producing structured graph representations of visual scenes. Our work analyzes the role of motifs: regularly appearing substructures in scene graphs. We present new quantitative insights on such repeated structures in the Visual Genome dataset. Our analysis shows that object labels are highly predictive of relation labels but not vice-versa. We also find that there are recurring patterns even in larger subgraphs: more than 50% of graphs contain motifs involving at least two relations. Our analysis motivates a new baseline: given object detections, predict the most frequent relation between object pairs with the given labels, as seen in the training set. This baseline improves on the previous state-of-the-art by an average of 3.6% relative improvement across evaluation settings. We then introduce Stacked Motif Networks, a new architecture designed to capture higher order motifs in scene graphs that further improves over our strong baseline by an average 7.1% relative gain. Our code is available at github.com/rowanz/neural-motifs.

• The paper presents a novel approach to scene graph parsing by leveraging global context with LSTMs to capture recurring structural motifs.
• It reveals that object labels are highly predictive of relation labels, emphasizing the asymmetric dependency in scene understanding.
• Empirical evaluations on the Visual Genome dataset show significant improvements in mean recall using the proposed MotifNet model.

Neural Motifs: Scene Graph Parsing with Global Context

The paper "Neural Motifs: Scene Graph Parsing with Global Context," authored by Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi, focuses on the task of generating structured representations of visual scenes through scene graph parsing. Scene graphs provide semantic summaries of objects within an image and detail their relationships. This task has been acknowledged for its potential to improve various computer vision applications, including image captioning and visual question answering.

Key Contributions

The paper's contributions can be categorized into two primary advancements: an insightful analysis of structural regularities in scene graphs and the introduction of novel models that leverage these regularities.

Analysis of Scene Graph Structure

The authors analyzed motifs—regularly occurring substructures—in the Visual Genome dataset. This dataset contains annotated scene graphs for 100K images and includes over a million instances of objects and 600K relations. The analysis revealed several significant findings:

1. Predictive Relation Labels: Object labels are highly predictive of relation labels, but the inverse is not true. This asymmetry underscores the importance of conditioning relation predictions on object labels.
2. Recurrence of Motifs: There are recurring patterns within the scene graphs, where more than 50% of graphs contain motifs involving at least two relations. These motifs illustrate common object-relation combinations that frequently appear together, such as "person-wearing-shirt."

Baseline and Model Introductions

The authors proposed a strong baseline that predicts the most frequent relations between object pairs, given their labels observed during training. This simple yet effective strategy demonstrated a 3.6% relative improvement over the previous state-of-the-art.

Building on this, the paper introduced the Stacked Motif Network (MotifNet), a new architecture designed to capture higher-order motifs in scene graphs. MotifNet leverages Long Short-Term Memory Networks (LSTMs) to encode global context and directly informs the local predictors of objects and relationships. The model is divided into three stages:

1. Bounding Box Predictions: Utilizing Faster R-CNN to predict bounding regions and object labels.
2. Object Context: Computing contextualized representations of bounding regions using bidirectional LSTMs.
3. Relation Predictions: Predicting relations between bounding regions using globally contextualized object representations, facilitated through outer products of head, tail, and union box features.

Experimental Results

Empirical evaluations on the Visual Genome dataset showed that the proposed baseline improved mean recall significantly, indicating the importance of conditioning relations on object labels. MotifNet further enhanced performance, achieving an additional 7.1% relative gain over the strong baseline due to effective modeling of global context and higher-order structural motifs.

The experimental setup included evaluations across three standard tasks:

• Predicate Classification (PredCls): Predicting edge labels given ground truth boxes and labels.
• Scene Graph Classification (SGCls): Predicting object labels and edge labels given ground truth boxes.
• Scene Graph Detection (SGDet): Predicting boxes, object labels, and edge labels.

In all evaluation settings, MotifNet showcased superior performance, validating the model's robustness in leveraging both local and global context for accurate scene graph parsing.

Implications and Future Directions

The proposed analyses and models provide substantial advancements in understanding and leveraging structural regularities in scene graphs. Practically, the insights contribute to the improved performance of computer vision systems in tasks requiring detailed semantic understanding of images. Theorectically, the findings emphasize the importance of asymmetric dependencies in structured prediction tasks.

Looking forward, future research could explore more advanced ways of integrating global context and potentially employing other forms of recurrent neural networks (RNNs) or transformers to capture even more complex dependencies and motifs. Additionally, efforts could be directed towards improving the scalability and efficiency of graph-based methods for larger and more diverse datasets.

In summary, the paper delivers significant contributions to the domain of scene graph parsing, providing both foundational analyses and novel models that pave the way for more sophisticated and accurate visual understanding systems.