Explainable and Explicit Visual Reasoning over Scene Graphs (1812.01855v2)

Abstract: We aim to dismantle the prevalent black-box neural architectures used in complex visual reasoning tasks, into the proposed eXplainable and eXplicit Neural Modules (XNMs), which advance beyond existing neural module networks towards using scene graphs --- objects as nodes and the pairwise relationships as edges --- for explainable and explicit reasoning with structured knowledge. XNMs allow us to pay more attention to teach machines how to "think", regardless of what they "look". As we will show in the paper, by using scene graphs as an inductive bias, 1) we can design XNMs in a concise and flexible fashion, i.e., XNMs merely consist of 4 meta-types, which significantly reduce the number of parameters by 10 to 100 times, and 2) we can explicitly trace the reasoning-flow in terms of graph attentions. XNMs are so generic that they support a wide range of scene graph implementations with various qualities. For example, when the graphs are detected perfectly, XNMs achieve 100% accuracy on both CLEVR and CLEVR CoGenT, establishing an empirical performance upper-bound for visual reasoning; when the graphs are noisily detected from real-world images, XNMs are still robust to achieve a competitive 67.5% accuracy on VQAv2.0, surpassing the popular bag-of-objects attention models without graph structures.

• The paper proposes Explainable and Explicit Neural Modules (XNMs) which use scene graphs to achieve structured and transparent visual reasoning.
• XNMs are highly parameter-efficient, provide explainable reasoning paths via scene graphs, and achieve competitive 67.5% accuracy on VQAv2.0.
• The research points towards a future of modular, interpretable AI systems by showing the value of structured representations and disentangling vision from reasoning.

Explainable and Explicit Visual Reasoning over Scene Graphs

The paper "Explainable and Explicit Visual Reasoning over Scene Graphs" presents an alternative approach to visual reasoning by developing Explainable and Explicit Neural Modules (XNMs) that leverage scene graphs for structured and transparent reasoning. Departing from conventional black-box neural networks, this research proposes a modular architecture that not only enhances explainability but also effectively disentangles reasoning from perceptual tasks.

Summary

The authors criticize traditional end-to-end deep learning models in visual reasoning for being opaque and failing to generalize under dataset biases. Addressing this, they introduce XNMs which are designed to operate over scene graphs—a representation where objects are nodes and their relationships are edges. This approach aims to encapsulate structured knowledge and facilitate explicit reasoning paths.

XNMs differentiate themselves by using scene graphs as inductive biases and being implemented with a concise set of only four meta-types: AttendNode, AttendEdge, Transfer, and Logic. This trim design is highly parameter-efficient, reducing parameter count by up to two orders of magnitude compared to existing models. By leveraging scene graphs, XNMs provide a mechanism to trace reasoning flow via graph attention, allowing explainable intermediate steps visible in the resulting outputs.

Numerical Results and Claims

The paper details strong quantitative results across several benchmark datasets. On the controlled CLEVR and CLEVR-CoGenT datasets, XNMs achieve near-perfect accuracy, providing evidence for their performance upper-bound when using ideal scene graphs and question parsing. On real-world datasets like VQAv2.0, which introduce noise in both vision and language components, XNMs still maintain competitive performance, achieving 67.5% accuracy—surpassing conventional bag-of-objects models.

Implications and Future Directions

The research underscores the benefits of disentangling vision and reasoning, demonstrating that well-defined structured representations like scene graphs can bolster reasoning capabilities. This approach not only pushes modular networks towards higher accuracy but also provides a blueprint for explainable AI, an increasingly critical requirement in machine reasoning systems.

The theoretical implications suggest a potential paradigm shift in AI methodologies—from monolithic end-to-end models to composite, interpretable systems—emphasizing transparency in learned representations and decisions.

On a practical level, XNMs' flexibility in handling varying quality of scene graphs illustrates robustness, promoting their adaptability to different real-world conditions. Going forward, advancements in scene graph detection could further amplify the efficacy of XNMs, suggesting an avenue for research focus that aligns computational perception more closely with human-like cognitive abilities.

In summation, XNMs embody a significant step towards interpretable and effective visual reasoning, potentially catalyzing further research and development into modularity, graph-based knowledge representation, and transparent AI systems.