Fine-Grained Scene Graph Generation with Data Transfer (2203.11654v2)

Abstract: Scene graph generation (SGG) is designed to extract (subject, predicate, object) triplets in images. Recent works have made a steady progress on SGG, and provide useful tools for high-level vision and language understanding. However, due to the data distribution problems including long-tail distribution and semantic ambiguity, the predictions of current SGG models tend to collapse to several frequent but uninformative predicates (e.g., on, at), which limits practical application of these models in downstream tasks. To deal with the problems above, we propose a novel Internal and External Data Transfer (IETrans) method, which can be applied in a plug-and-play fashion and expanded to large SGG with 1,807 predicate classes. Our IETrans tries to relieve the data distribution problem by automatically creating an enhanced dataset that provides more sufficient and coherent annotations for all predicates. By training on the enhanced dataset, a Neural Motif model doubles the macro performance while maintaining competitive micro performance. The code and data are publicly available at https://github.com/waxnkw/IETrans-SGG.pytorch.

• The paper introduces IETrans, a method to redistribute general predicates in scene graphs for more informative predictions.
• It employs internal data transfer to resolve semantic ambiguity and external transfer to enrich datasets for underrepresented predicates.
• Results demonstrate significant improvements in mean recall and accuracy on VG-50 and VG-1800, confirming its practical impact.

Fine-Grained Scene Graph Generation with Data Transfer: An Academic Overview

The paper "Fine-Grained Scene Graph Generation with Data Transfer" presents a novel approach to improving Scene Graph Generation (SGG) by addressing two predominant issues: the long-tail distribution of predicates and semantic ambiguity in current datasets. The authors propose a method called Internal and External Data Transfer (IETrans), designed to enhance datasets and minimize data distribution problems.

Key Contributions and Methodology

Scene graph generation involves detecting relations among objects in images, typically formatted as (subject, predicate, object) triplets. The authors recognize the limitations of existing SGG systems due to biased data distributions that result in uninformative predicate predictions. To counter these drawbacks, IETrans is introduced—a framework that restructures datasets to provide balanced and coherent predicate annotations.

1. Internal Data Transfer: This component of IETrans addresses semantic ambiguity, where general predicates like "on" frequently overshadow more informative ones such as "riding" due to human annotation preferences. By analyzing model output confusion matrices, the method automatically transfers instances from these general predicates to their informative counterparts, enhancing the informative predicate representation.
2. External Data Transfer: Unannotated object pairs often labeled as 'NA' due to missed annotations are potential sources for additional data. IETrans identifies these samples and relabels them to bolster training datasets, thus compensating for the long-tail problem by supplying more samples for underrepresented predicate classes.

Performance and Implications

The IETrans method is evaluated on benchmarks including VG-50 and a new VG-1800 dataset created by the authors to challenge SGG with 1,807 predicate categories. The new benchmark ensures a practical test environment by providing adequate samples for tail classes. IETrans demonstrates significant improvements in mean recall and overall scene graph generation accuracy across base models such as Motif, VCTree, and GPS-Net, reaffirming its efficacy in refining SGG systems.

In practice, better scene graphs enable more accurate and nuanced interpretations of images, benefiting applications like visual question answering, image retrieval, and beyond. The authors' approach to utilizing dataset-inherent knowledge, rather than relying solely on external resources, highlights a scalable solution to large-scale SGG challenges.

Future Directions

The paper opens avenues for future research in large-scale visual recognition. By showing how IETrans can be adapted to handle large predicate vocabularies effectively, it encourages further exploration into adaptive data augmentation methods tailored to other visual recognition tasks, such as image classification and semantic segmentation.

In summary, this work contributes a methodologically sound framework to tackle long-standing issues in SGG, offering a plug-and-play solution applicable across diverse models and settings. As AI progresses into more complex real-world scenarios, the principles set forth in this research can guide future advancements in understanding and interpreting visual data.