Unifying Vision-and-Language Tasks via Text Generation (2102.02779v2)

Abstract: Existing methods for vision-and-language learning typically require designing task-specific architectures and objectives for each task. For example, a multi-label answer classifier for visual question answering, a region scorer for referring expression comprehension, and a language decoder for image captioning, etc. To alleviate these hassles, in this work, we propose a unified framework that learns different tasks in a single architecture with the same LLMing objective, i.e., multimodal conditional text generation, where our models learn to generate labels in text based on the visual and textual inputs. On 7 popular vision-and-language benchmarks, including visual question answering, referring expression comprehension, visual commonsense reasoning, most of which have been previously modeled as discriminative tasks, our generative approach (with a single unified architecture) reaches comparable performance to recent task-specific state-of-the-art vision-and-LLMs. Moreover, our generative approach shows better generalization ability on questions that have rare answers. Also, we show that our framework allows multi-task learning in a single architecture with a single set of parameters, achieving similar performance to separately optimized single-task models. Our code is publicly available at: https://github.com/j-min/VL-T5

Unifying Vision-and-Language Tasks via Text Generation

The paper "Unifying Vision-and-Language Tasks via Text Generation" presents a novel approach to tackling multiple vision-and-language tasks within a single, unified framework. This research seeks to alleviate the complexity of task-specific model design by employing a generative method using multimodal conditional text generation. Two models, VL-T5 and VL-BART, are introduced, extending the capabilities of the T5 and BART LLMs to process visual information.

Methodology

The authors propose transforming various vision-and-language tasks into a text generation problem, unifying both discriminative and generative tasks under a singular architecture. Traditional vision-and-LLMs necessitate specialized architectures and objectives for each task, such as visual question answering (VQA) or image captioning. This work, however, eschews bespoke solutions in favor of learning tasks via text label generation based on visual and textual input.

VL-T5 and VL-BART incorporate an additional visual processing capability by embedding visual inputs as part of the transformer architecture. This setup uses shared parameters across tasks without requiring separate task-specific models, significantly reducing overhead and simplifying the training process.

Results

The unified framework achieves competitive results on seven benchmarks, demonstrating similar performance to the state-of-the-art task-specific models. It shows robustness across a broad set of challenges including VQA, referring expression comprehension, visual commonsense reasoning, among others. Notably, this approach exhibits a superior generalization capability on VQA tasks with rare answers. Here, the generative model has an edge over discriminative counterparts.

Implications and Future Directions

This research implies a shift towards more flexible and generalized approaches in multimodal model design, reducing the need to hand-craft specific solutions for each task. The potential to streamline tasks within a unified architecture may lead to more efficient models, facilitating easier maintenance and scalability.

The paper opens avenues for further exploration in expanding the model to accommodate even more complex tasks or integrating enhancements through task-specific prompts. Future work could explore optimizing the integration of visual information within the text backbone and extending the framework’s applications within more intricate vision-and-language scenarios.

In conclusion, the research presents a compelling case for embracing a unified, generative framework for diverse vision-and-language tasks, aiming to simplify and generalize multi-task learning in the field of AI.