VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2106.04632v2)

Abstract: Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task. In reality, a truly useful VidL system is expected to be easily generalizable to diverse tasks, domains, and datasets. To facilitate the evaluation of such systems, we introduce Video-And-Language Understanding Evaluation (VALUE) benchmark, an assemblage of 11 VidL datasets over 3 popular tasks: (i) text-to-video retrieval; (ii) video question answering; and (iii) video captioning. VALUE benchmark aims to cover a broad range of video genres, video lengths, data volumes, and task difficulty levels. Rather than focusing on single-channel videos with visual information only, VALUE promotes models that leverage information from both video frames and their associated subtitles, as well as models that share knowledge across multiple tasks. We evaluate various baseline methods with and without large-scale VidL pre-training, and systematically investigate the impact of video input channels, fusion methods, and different video representations. We also study the transferability between tasks, and conduct multi-task learning under different settings. The significant gap between our best model and human performance calls for future study for advanced VidL models. VALUE is available at https://value-benchmark.github.io/.

Overview of the VALUE Benchmark for Video-and-Language Understanding

The paper under discussion introduces the Video-And-Language Understanding Evaluation (VALUE) benchmark, a comprehensive framework for evaluating video-and-language (VidL) systems. VALUE addresses a critical gap in the analysis of VidL systems by incorporating a multi-task benchmark that evaluates models across diverse datasets and tasks, thereby providing a robust platform for tracking advancements in this rapidly evolving domain. The benchmark consists of 11 datasets spanning three key tasks: text-based video retrieval, video question answering (QA), and video captioning.

Key Contributions

VALUE makes several contributions to the field of VidL understanding:

1. Diverse Task Set: Unlike prior efforts that often focus on single tasks, VALUE encompasses three distinct tasks, drawing from datasets that vary in genres, video lengths, data volumes, and difficulty levels. This diversity ensures the benchmarking of systems that claim to generalize across various conditions.
2. Multi-Channel Input: VALUE emphasizes the importance of leveraging multi-channel inputs, including both video frames and subtitles. This allows the benchmark to evaluate models that can synthesize information from both visual and textual sources, a necessity for more genuine comprehension of multimedia data.
3. Rigorous Evaluation: The benchmark reveals a considerable gap between existing model performances and human benchmarks, highlighting substantial opportunities for further advancements in VidL systems. The framework employs standard metrics such as Recall@K, accuracy, and CIDEr-D, facilitating a clear comparison of system performance.
4. Multi-Task Learning Insights: VALUE includes extensive experimentation on the transferability between tasks and the impact of multi-task learning. The insights indicate that although there is task-specific variance, there are shared foundations that multi-task learning can build upon to improve overall performance.

Experimental Results

The benchmark paper evaluates baseline models with varying configurations of video inputs and fusion techniques. Results consistently show that utilizing both video and subtitles (multi-channel input) generally enhances system performance. However, the choice of fusion method—how video and subtitle information are integrated—also plays a pivotal role, with transformer-based models like HERO performing robustly across the evaluated tasks.

In investigating visual feature representations, the authors test combinations of 2D and 3D features, notably from networks like CLIP-ViT and SlowFast. Results demonstrate that integrating both 2D appearance and 3D motion features yields superior results, emphasizing the need for models to capture both static and dynamic elements intricately.

Furthermore, the research evaluates transferability across tasks, affirming that fine-tuning models on specific tasks significantly outperforms zero-shot task application. These findings are crucial for guiding future developments in VidL systems toward more generalized and transferable methodologies.

Future Implications

The VALUE benchmark sets a significant milestone for the VidL research community by providing a comprehensive platform for model evaluation. The substantial gap between model and human performance outlined by the benchmark underlines the complexity of VidL tasks and the limitations of current models. Moving forward, advancements might focus on enhancing model architectures to better fuse and analyze multi-channel data, improve task transferability, and bridge the performance gap identified.

Additionally, VALUE can serve as a compelling catalyst for research on diagnostic and qualitative assessments of VidL systems, offering detailed insights that quantitative metrics may not capture. By advancing the capabilities of VidL systems in these directions, future research will contribute to developing models that perform more reliably and robustly in real-world applications.