2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining

Abstract: Compared to image-text pair data, interleaved corpora enable Vision-LLMs (VLMs) to understand the world more naturally like humans. However, such existing datasets are crawled from webpage, facing challenges like low knowledge density, loose image-text relations, and poor logical coherence between images. On the other hand, the internet hosts vast instructional videos (e.g., online geometry courses) that are widely used by humans to learn foundational subjects, yet these valuable resources remain underexplored in VLM training. In this paper, we introduce a high-quality \textbf{multimodal textbook} corpus with richer foundational knowledge for VLM pretraining. It collects over 2.5 years of instructional videos, totaling 22,000 class hours. We first use an LLM-proposed taxonomy to systematically gather instructional videos. Then we progressively extract and refine visual (keyframes), audio (ASR), and textual knowledge (OCR) from the videos, and organize as an image-text interleaved corpus based on temporal order. Compared to its counterparts, our video-centric textbook offers more coherent context, richer knowledge, and better image-text alignment. Experiments demonstrate its superb pretraining performance, particularly in knowledge- and reasoning-intensive tasks like ScienceQA and MathVista. Moreover, VLMs pre-trained on our textbook exhibit outstanding interleaved context awareness, leveraging visual and textual cues in their few-shot context for task solving. Our code are available at https://github.com/DAMO-NLP-SG/multimodal_textbook.

• The paper introduces a novel multimodal textbook corpus for Vision-Language Model pretraining, derived from 22,000 hours of instructional videos to offer richer context than traditional image-text pairs.
• Researchers built this corpus using a taxonomy-driven data collection guided by an LLM, followed by a pipeline to extract and filter keyframes, audio transcripts, and text from videos.
• Pretraining VLMs on this textbook corpus demonstrates notable performance improvements on benchmarks like ScienceQA and MathVista, enhancing capabilities for tasks requiring deep knowledge and reasoning.

Overview of "2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining"

The paper "2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining" introduces a novel approach for Vision-LLMs (VLMs) using a meticulously curated multimodal corpus derived from instructional videos. This corpus serves as a high-quality dataset for pretraining VLMs, addressing current limitations in image-text paired datasets. The methodology and results presented demonstrate enhanced learning capabilities for vision-language tasks.

The authors argue that existing datasets, often sourced from the web, suffer from issues like low knowledge density and weak image-text connections. To mitigate these issues, the researchers propose a dataset comprising 22,000 class hours of educational content spanning over 2.5 years. The dataset is constructed by systematically collecting, processing, and refining video content from instructional video platforms, focusing on core subjects like mathematics and physics.

Key Contributions

1. Multimodal Textbook Corpus: The major contribution of this work is the construction of a multimodal corpus designed to improve VLM pretraining. Unlike conventional datasets focused on simple image-text pairs, this corpus offers interleaved sequences of key video frames and extracted textual data through OCR and ASR technologies. This approach is expected to provide richer context and more coherent content for VLMs.
2. Taxonomy-Driven Data Collection: The process begins with the deployment of a LLM to construct a taxonomy of knowledge points. This taxonomy guides the systematic collection of relevant instructional videos, ensuring that the dataset covers a broad and relevant spectrum of foundational topics.
3. Data Extraction and Filtering: A pipeline has been developed to extract keyframes, audio transcripts (using ASR), and textual content (via OCR). The data is organized in a temporal order to maintain the pedagogical context of the original videos. Several filtering mechanisms ensure only high-quality, informative content is included.
4. Performance Evaluation: Evaluated against benchmarks like ScienceQA and MathVista, the VLMs pretrained on this textbook exhibit notable improvements over existing methods. The models are proficient in handling tasks requiring deep knowledge and reasoning capabilities.

Implications and Future Directions

The introduction of a multimodal textbook has several implications for VLM development. Practically, the approach promises to enhance the capacity of models in educational applications, where understanding complex concepts and reasoning tasks is paramount. Theoretically, this work encourages a shift from traditional paired datasets to more contextually rich corpora.

Future research could explore expanding the taxonomy to incorporate more diverse subject areas and experimenting with additional modalities such as interactive content. Additionally, this methodology could be adapted to other domains, potentially benefiting any field reliant on the nuanced understanding of multifaceted content.

Conclusion

This paper represents a significant step toward advancing the capabilities of VLMs by utilizing a multimodal textbook that mirrors real-world educational practices. The work is a testament to the authors' efforts in addressing the inadequacies of current datasets, providing a robust framework for future developments in AI-driven educational tools. The results achieved highlight the potential of such corpora to improve both the fidelity and applicability of vision-language integration in advanced cognitive tasks.