UMIE: Unified Multimodal Information Extraction with Instruction Tuning (2401.03082v1)

Abstract: Multimodal information extraction (MIE) gains significant attention as the popularity of multimedia content increases. However, current MIE methods often resort to using task-specific model structures, which results in limited generalizability across tasks and underutilizes shared knowledge across MIE tasks. To address these issues, we propose UMIE, a unified multimodal information extractor to unify three MIE tasks as a generation problem using instruction tuning, being able to effectively extract both textual and visual mentions. Extensive experiments show that our single UMIE outperforms various state-of-the-art (SoTA) methods across six MIE datasets on three tasks. Furthermore, in-depth analysis demonstrates UMIE's strong generalization in the zero-shot setting, robustness to instruction variants, and interpretability. Our research serves as an initial step towards a unified MIE model and initiates the exploration into both instruction tuning and LLMs within the MIE domain. Our code, data, and model are available at https://github.com/ZUCC-AI/UMIE

• The paper introduces a unified framework that transforms multiple MIE tasks into a generation problem through instruction tuning.
• UMIE adapts to diverse extraction scenarios without task-specific architectures, outperforming state-of-the-art methods across six datasets.
• The research demonstrates robust zero-shot performance and improved interpretability, opening new avenues for multimodal information extraction.

The paper "UMIE: Unified Multimodal Information Extraction with Instruction Tuning" addresses the challenges faced by existing multimodal information extraction (MIE) methods, primarily their reliance on task-specific model structures. This task specificity often leads to limited generalizability and an underuse of shared knowledge across various MIE tasks.

To overcome these limitations, the authors introduce UMIE, a unified multimodal information extractor. UMIE transforms three core MIE tasks into a generation problem through the use of instruction tuning. This approach allows UMIE to effectively extract information from both textual and visual data.

Key contributions and findings of the paper include:

1. Unified Framework: UMIE consolidates multiple MIE tasks under a single unified framework. By framing information extraction as a generative process, the model leverages shared knowledge, enhancing generalizability across different tasks.
2. Instruction Tuning: The authors employ instruction tuning, which enables the model to adapt to different MIE tasks through specific instructions. This method ensures that UMIE can handle diverse MIE scenarios without the need for task-specific architectures.
3. Performance: Extensive experiments demonstrate that UMIE outperforms various state-of-the-art (SoTA) methods across six different MIE datasets spanning three tasks. The results highlight the effectiveness of UMIE’s unified approach in extracting multimodal information.
4. Generalization and Robustness: The paper emphasizes UMIE's strong generalization capabilities, particularly in zero-shot settings. This means the model performs well on new, unseen tasks without requiring additional fine-tuning. Additionally, UMIE shows robustness to variations in instructions, which underscores the flexibility of the instruction tuning paradigm.
5. Interpretability: UMIE provides insights into its decision-making processes, making it more interpretable compared to other models. This is significant for practical applications where understanding model behavior is crucial.
6. Initial Exploration: The research represents an initial step towards developing a truly unified MIE model. It also initiates exploration into the use of instruction tuning and LLMs within the MIE domain.

The authors have made their code, data, and model publicly available, fostering further research and development in the field of multimodal information extraction. This openness aims to encourage collaboration and expedite advancements in creating more unified and generalizable models for various MIE tasks.

Overall, UMIE sets a new benchmark in the field of multimodal information extraction by addressing the limitations of task-specific models and demonstrating the power of a unified approach through instruction tuning.