Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks

Abstract: Text-rich images, where text serves as the central visual element guiding the overall understanding, are prevalent in real-world applications, such as presentation slides, scanned documents, and webpage snapshots. Tasks involving multiple text-rich images are especially challenging, as they require not only understanding the content of individual images but reasoning about inter-relationships and logical flows across multiple visual inputs. Despite the importance of these scenarios, current multimodal LLMs (MLLMs) struggle to handle such tasks due to two key challenges: (1) the scarcity of high-quality instruction tuning datasets for text-rich multi-image scenarios, and (2) the difficulty in balancing image resolution with visual feature sequence length. To address these challenges, we propose Leopard, an MLLM tailored for handling vision-language tasks involving multiple text-rich images. First, we curated about one million high-quality multimodal instruction-tuning data, tailored to text-rich, multi-image scenarios. Second, we proposed an adaptive high-resolution multi-image encoding module to dynamically optimize the allocation of visual sequence length based on the original aspect ratios and resolutions of images. Experiments on a diverse set of benchmarks reveal that our model consistently outperforms state-of-the-art systems, such as Llama-3.2 and Qwen2-VL, in challenging text-rich, multi-image evaluations. Remarkably, our approach achieves outstanding performance using only 1.2M training instances, all of which are fully open-sourced, demonstrating both high efficiency and effectiveness compared to models trained on large-scale in-house data. Our code and data are available at https://github.com/tencent-ailab/Leopard.

• The paper presents a vision-language model that leverages the LEOPARD-INSTRUCT dataset to tackle text-rich, multi-image challenges.
• It employs an adaptive high-resolution encoding mechanism that dynamically optimizes image detail preservation based on aspect ratios and resolution limits.
• Extensive experiments show the model outperforms existing systems by an average of +9.61 points on text-rich, multi-image benchmarks.

An Analytical Overview of LEOPARD: A Vision LLM for Text-Rich Multi-Image Tasks

The paper "LEOPARD: A Vision LLM for Text-Rich Multi-Image Tasks" introduces LEOPARD, a Multimodal LLM (MLLM) designed to overcome specific challenges in handling text-rich, multi-image data scenarios. These challenges include the scarcity of high-quality instruction tuning datasets specific to these scenarios and balancing image resolution with visual sequence length. The model proposes a novel dataset and an advanced encoding mechanism to address these issues.

Key Contributions

The paper highlights two main contributions:

1. LEOPARD-INSTRUCT Dataset: The authors curated approximately one million high-quality multimodal instruction-tuning data samples tailored for text-rich, multi-image scenarios. This extensive dataset addresses the scarcity of relevant data for effectively training models to handle tasks that involve multiple text-rich images simultaneously. The dataset spans domains frequently encountered in modern digital ecosystems, such as multi-page documents, charts, tables, and webpage trajectories.
2. Adaptive High-Resolution Multi-Image Encoding: LEOPARD introduces an adaptive encoding module that dynamically optimizes visual sequence length allocation based on the original image aspect ratios and resolutions. This mechanism ensures high-resolution details are preserved without exceeding the model's sequence length limitations, thus enhancing the model's ability to manage multiple high-resolution images simultaneously.

Experimental Validation

Experiments conducted across a diverse set of benchmarks demonstrate LEOPARD's superior performance in text-rich, multi-image evaluations and competitive results in general vision-language tasks. Specifically, LEOPARD outperforms existing open-source models by an average of +9.61 points across five text-rich, multi-image benchmarks, indicating its efficacy in handling complex scenarios that require understanding interconnected textual and visual information across multiple images. Furthermore, the model's competitive performance on general domain evaluations underscores its capability to generalize beyond the targeted tasks.

Implications and Future Directions

The contributions of this paper hold significant implications for the development of MLLMs. By tackling the difficulties associated with text-rich, multi-image data, LEOPARD can be considered a robust step towards better document and scene understanding in applications such as multi-page document analysis and presentation slide comprehension. The introduction of the LEOPARD-INSTRUCT dataset provides a substantial resource for the research community, potentially serving as a benchmark for future models aiming to address similar challenges.

The adaptive high-resolution multi-image encoding mechanism presents a novel approach that balances computational efficiency with detail preservation, setting a precedent for future research in the area. This approach could be further extended or fine-tuned for specific applications, such as AR/VR environments, where high-resolution details are crucial, or in specialized domains like medical imaging, where multiple high-resolution images are typical.

Looking ahead, future research could explore refining the encoding mechanism to further scale the number of high-resolution images that can be processed concurrently. Additionally, integrating the dataset with broader generalist models could facilitate improvements in models' abilities to understand and interact with real-world scenarios involving complex multimodal inputs.

In summary, the paper on LEOPARD presents a well-defined advancement in the field of text-rich multi-image processing in MLLMs. Through its methodical approach to dataset curation and encoding strategy, it provides valuable insights and resources for ongoing research in multimodal AI.