Knowledge-Embedded Routing Network for Scene Graph Generation (1903.03326v1)

Abstract: To understand a scene in depth not only involves locating/recognizing individual objects, but also requires to infer the relationships and interactions among them. However, since the distribution of real-world relationships is seriously unbalanced, existing methods perform quite poorly for the less frequent relationships. In this work, we find that the statistical correlations between object pairs and their relationships can effectively regularize semantic space and make prediction less ambiguous, and thus well address the unbalanced distribution issue. To achieve this, we incorporate these statistical correlations into deep neural networks to facilitate scene graph generation by developing a Knowledge-Embedded Routing Network. More specifically, we show that the statistical correlations between objects appearing in images and their relationships, can be explicitly represented by a structured knowledge graph, and a routing mechanism is learned to propagate messages through the graph to explore their interactions. Extensive experiments on the large-scale Visual Genome dataset demonstrate the superiority of the proposed method over current state-of-the-art competitors.

Knowledge-Embedded Routing Network for Scene Graph Generation

The paper "Knowledge-Embedded Routing Network for Scene Graph Generation" presents a novel approach to enhancing scene graph generation by integrating statistical correlations into the deep learning architecture. Scene graph generation is crucial for visual recognition tasks, dealing not only with locating and identifying objects within a scene but also with understanding the relationships and interactions between them. The authors focus on addressing the class imbalance problem prevalent in real-world relationships by embedding knowledge from statistical correlations of object pairs and their relationships into a neural network model.

The paper introduces a Knowledge-Embedded Routing Network (KERN) that leverages statistical co-occurrences of objects and their relationships. The key idea is to represent and utilize these statistical correlations via a structured knowledge graph. This approach aims to regularize the semantic space, thereby leading to improved prediction performance, particularly for relationships with fewer samples. The authors incorporate this knowledge graph into a propagation mechanism within a neural network, allowing for message passing that reflects the inherent statistical regularities.

The empirical evaluation, particularly on the Visual Genome dataset, evidences the model's superiority over existing methods. The introduction of the mR@$K$ (mean recall@$K$) metric provides a more holistic evaluation by averaging performance across all relationships, mitigating the usual skew toward frequent relationships seen with standard R@$K$ metrics. The model shows a substantial improvement in mean recall, indicating its robustness against the distribution skew in real-world data. For instance, the model achieves mean mR@50 and mR@100 values of 11.7% and 26.5% without constraints, which are relative improvements of 30.0% and 28.6%, respectively, over the previous state-of-the-art.

The theoretical implications of this research suggest a new avenue for integrating structured knowledge into neural networks to address distribution imbalances, a recurrent challenge in machine learning. Furthermore, the work implies that future models might benefit from explicit incorporation of domain knowledge, formalized through graph structures, to guide learning processes. Practically, KERN could improve the accuracy and reliability of AI systems in applications requiring nuanced understanding from visual data, such as autonomous vehicles or real-time video surveillance.

Looking ahead, this line of research could extend to various domains where statistical correlations are strong predictors of interactions or outcomes. The success of KERN in scene graph generation posits the potential for similar strategies in natural language understanding, robotics, and beyond, where structured knowledge could play a pivotal role in advancing AI capabilities.