VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (1904.03493v3)

Abstract: We present a new large-scale multilingual video description dataset, VATEX, which contains over 41,250 videos and 825,000 captions in both English and Chinese. Among the captions, there are over 206,000 English-Chinese parallel translation pairs. Compared to the widely-used MSR-VTT dataset, VATEX is multilingual, larger, linguistically complex, and more diverse in terms of both video and natural language descriptions. We also introduce two tasks for video-and-language research based on VATEX: (1) Multilingual Video Captioning, aimed at describing a video in various languages with a compact unified captioning model, and (2) Video-guided Machine Translation, to translate a source language description into the target language using the video information as additional spatiotemporal context. Extensive experiments on the VATEX dataset show that, first, the unified multilingual model can not only produce both English and Chinese descriptions for a video more efficiently, but also offer improved performance over the monolingual models. Furthermore, we demonstrate that the spatiotemporal video context can be effectively utilized to align source and target languages and thus assist machine translation. In the end, we discuss the potentials of using VATEX for other video-and-language research.

• The paper introduces VaTeX, a dataset featuring over 41,250 videos and 825,000 high-quality English and Chinese captions.
• The study proposes unified models for multilingual video captioning and video-guided machine translation that outperform traditional monolingual approaches.
• Experimental results demonstrate significant improvements by leveraging video context to resolve linguistic ambiguities and enhance translation accuracy.

Overview of VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research

The paper "VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research" introduces VaTeX, an extensive dataset designed to advance research at the intersection of video and natural language processing. With over 41,250 videos and 825,000 captions in both English and Chinese, VaTeX provides a significant resource for tasks requiring high-quality, multilingual video descriptions. Notably, the dataset includes more than 206,000 English-Chinese parallel caption pairs, facilitating studies in multilingual contexts.

Dataset Characteristics

VaTeX stands out in comparison to existing datasets like MSR-VTT, due to its multilingual nature and the diversity of its video and language descriptions. Each video in VaTeX is annotated with 20 captions split equally among English and Chinese, crafted by 20 individual annotators. This ensures a high level of linguistic richness and reduces duplicated captions, a common issue in other datasets. Moreover, VaTeX spans 600 distinct human activities, offering a more comprehensive coverage than its predecessors.

Research Tasks & Models

Two primary tasks are proposed using the VaTeX dataset:

1. Multilingual Video Captioning: The goal here is to generate descriptions in multiple languages for a given video. The dataset’s bilingual nature allows for the development of a unified model that efficiently produces captions in both English and Chinese. The research demonstrates that such models outperform monolingual approaches, benefiting from multilingual knowledge transfer.
2. Video-guided Machine Translation (VMT): This novel task uses videos as contextual information to enhance machine translation between English and Chinese. The paper suggests that video context aids in resolving linguistic ambiguities, particularly for verbs and nouns. The proposed model with video context significantly improves translation accuracy over traditional text-only translation systems.

Experimental Results

Extensive experiments show that models built on VaTeX offer substantial improvements over baseline models. The unified multilingual captioning model shows greater efficiency and effectiveness, indicating the benefits of leveraging shared model components across languages. Additionally, VMT demonstrates its strength in using video context to boost machine translation quality, especially highlighted through noun/verb masking experiments that illustrate context aiding in recovering missing information.

Implications and Future Directions

Practically, VaTeX has significant implications for developing technologies that need to interpret and describe video content in multiple languages, such as automated video subtitling and real-time multilingual video communication tools. Theoretically, VaTeX provides a robust benchmark for testing the limits of cross-modal and multilingual models, encouraging further exploration into unified models that integrate diverse input types.

Future research might focus on extending the VaTeX dataset to include more languages, exploring other multimedia elements such as audio, or utilizing VaTeX for zero-shot learning tasks. Moreover, the dataset could serve as a valuable tool in cognitive science research, helping to understand language processing in multilingual and multimedia contexts.

VaTeX represents a comprehensive resource for researchers looking to bridge the gap between video content and multilingual language understanding, setting a foundation for future advancements in AI.