Visual Coreference Resolution in Visual Dialog using Neural Module Networks (1809.01816v1)

Abstract: Visual dialog entails answering a series of questions grounded in an image, using dialog history as context. In addition to the challenges found in visual question answering (VQA), which can be seen as one-round dialog, visual dialog encompasses several more. We focus on one such problem called visual coreference resolution that involves determining which words, typically noun phrases and pronouns, co-refer to the same entity/object instance in an image. This is crucial, especially for pronouns (e.g., it'), as the dialog agent must first link it to a previous coreference (e.g.,boat'), and only then can rely on the visual grounding of the coreference boat' to reason about the pronounit'. Prior work (in visual dialog) models visual coreference resolution either (a) implicitly via a memory network over history, or (b) at a coarse level for the entire question; and not explicitly at a phrase level of granularity. In this work, we propose a neural module network architecture for visual dialog by introducing two novel modules - Refer and Exclude - that perform explicit, grounded, coreference resolution at a finer word level. We demonstrate the effectiveness of our model on MNIST Dialog, a visually simple yet coreference-wise complex dataset, by achieving near perfect accuracy, and on VisDial, a large and challenging visual dialog dataset on real images, where our model outperforms other approaches, and is more interpretable, grounded, and consistent qualitatively.

Visual Coreference Resolution in Visual Dialog using Neural Module Networks

The paper "Visual Coreference Resolution in Visual Dialog using Neural Module Networks" presents a sophisticated approach to handling coreference resolution within the task of visual dialog. Visual dialog involves question-answering based on images, utilizing a dialog history for context. This paper tackles the challenge of visual coreference resolution, where the model must identify and ground noun phrases and pronouns to specific entities in images to maintain continuity and comprehension across a dialog. Traditional models are either implicitly trained through memory networks or operate at a coarse sentence level. This paper innovates by proposing a neural module network architecture that explicitly resolves coreferences at the phrase level, introducing two significant modules: Refer and Exclude.

Key Contributions

1. Neural Module Network Architecture: The authors propose a modular architecture tailored for visual dialog, enhancing interpretability and resolution granularity by focusing at the word level. This is in contrast to previous approaches that relied either on implicit memory over shared dialog history or coarse question-level processing.
2. Refer and Exclude Modules: These novel modules allow the model to perform fine-grained coreference resolution. The Refer module links pronouns and noun phrases to previously mentioned entities within the dialog history, thereby grounding them accurately in conversation. The Exclude module handles queries that require identifying entities aside from specified ones, enriching the model's ability to discern nuanced dialog requirements.
3. Coreference Pool and Attention Mechanism: The model leverages a dictionary, or coreference pool, which records entities and their corresponding visual attention from all prior dialog interactions. This dictionary is instrumental for the Refer module, enabling it to resolve anaphoric references effectively and diversify its usage in complex dialog scenarios.

Results

The model was evaluated on two datasets:

• MNIST Dialog Dataset: Despite its simplistic visual nature, this dataset poses complex coreference challenges. The proposed model achieved a near-perfect accuracy of 99.3%, surpassing previous benchmarks and showcasing its proficiency in explicit coreference resolution.
• VisDial Dataset: On this more visually and linguistically challenging dataset, the model outperformed existing techniques, demonstrating improved metrics across Mean Reciprocal Rank (MRR), Recall@k, and Mean Rank. This highlights the significance of explicit coreference tracking and interpretation in visual dialog tasks.

Implications

This research marks a substantial step in bridging the gap between human-like comprehension and automated dialog systems by integrating interpretable module networks for coreference resolution. The modular nature allows potential adaptability and scalability to different dialog tasks that require nuanced understanding and entity tracking over multiple conversational turns. As AI continues to evolve, the ability to contextually and accurately identify entities in dialogs will enhance interaction quality in applications such as assistive technologies, natural language interfaces for robotics, and intelligent home systems.

Future Directions

The practical implications of this research could be extended by integrating similar architectures into more diverse datasets and dialog types. Furthermore, future developments could focus on refining module efficiency and expanding coreference resolution capabilities across broader linguistic and visual contexts, potentially integrating with multimodal models that include textual and auditory data. The application of this model in real-time dialog systems also opens avenues for significant exploration in human-machine interaction interfaces.

As AI methodologies continue to advance, visual coreference resolution will play a pivotal role in enhancing dialog comprehension, efficiency, and accuracy, thereby laying the groundwork for more natural and seamless human-computer interactions.