Spoken question answering for visual queries

Abstract: Question answering (QA) systems are designed to answer natural language questions. Visual QA (VQA) and Spoken QA (SQA) systems extend the textual QA system to accept visual and spoken input respectively. This work aims to create a system that enables user interaction through both speech and images. That is achieved through the fusion of text, speech, and image modalities to tackle the task of spoken VQA (SVQA). The resulting multi-modal model has textual, visual, and spoken inputs and can answer spoken questions on images. Training and evaluating SVQA models requires a dataset for all three modalities, but no such dataset currently exists. We address this problem by synthesizing VQA datasets using two zero-shot TTS models. Our initial findings indicate that a model trained only with synthesized speech nearly reaches the performance of the upper-bounding model trained on textual QAs. In addition, we show that the choice of the TTS model has a minor impact on accuracy.

• The paper presents a novel multimodal SVQA model that fuses speech, text, and image inputs to address spoken visual queries.
• It leverages Whisper and CLIP encoders with trainable projection modules and LoRA fine-tuning to achieve effective modality alignment.
• Results demonstrate that the model outperforms unimodal systems even when using synthetic speech data, highlighting its robust performance.

Spoken Question Answering for Visual Queries

The paper "Spoken question answering for visual queries" (2505.23308) presents an innovative extension to existing unimodal models by combining textual, visual, and spoken modalities for question answering systems. This multi-modal approach enables user interaction through both speech and images, tackling the task of spoken visual question answering (SVQA).

Introduction to Multimodal Question Answering

The transition from unimodal to multimodal AI systems represents a significant evolution in the AI domain. Traditional systems have been limited to processing single types of input data, such as text-only question answering or image classification. Multimodal systems integrate several data types, enabling richer, more interactive user experiences. This paper focuses on developing a model capable of interpreting and responding to spoken queries about visual information. The model combines three input modalities: audio, image, and text to conduct SVQA.

Model Architecture

The SVQA model architecture builds on the LLaVA framework by incorporating two modality-specific encoders: Whisper for speech processing and CLIP for image representations. These encoders are aligned with the input embedding space of a LLM using trainable projector modules.

Figure 1: SVQA model components with designated frozen and trainable modules.

The architecture works as follows:

1. Speech Encoder (Whisper): Processes spoken queries into a feature representation suitable for interaction with the LLM.
2. Image Encoder (CLIP-ViT-Large-336px): Extracts visual features from images, aligned with the LLM's processing space.
3. Textual Input: Text, including prompts and potential answers, guides the LLM in generating appropriate responses.

The system supports questions about visual information contained within images, using an audio-based query that might include text-based instructions or multiple-choice formats.

Dataset Creation

Developing SVQA models required synthesizing a dataset capable of handling the three involved modalities. Existing visual question answering (VQA) datasets were converted into spoken formats using two distinct zero-shot TTS systems: StyleTTS2 and F5-TTS. These synthetic datasets overcome the current scarcity of human speech data integrated with visual questions by leveraging diverse voices and styles to prevent overfitting.

Training Strategy

Training of the SVQA model followed two main phases:

1. Speech Projector Pre-training: Utilizing tasks like ASR and audio description tasks on MLS datasets, the speech projector is pre-trained with frozen weights for the encoder and LLM.
2. Joint Fine-tuning: Combines modality-specific encoders and their projectors with LoRA-enabled tuning of the LLM across varied dataset combinations.

The models trained under this framework demonstrate efficient multimodal alignment, fostering robust spoken question answering.

Evaluation and Results

The SVQA models outperform unimodal systems when assessing across standard VQA benchmarks. Despite being trained with synthetic speech data, the results were promisingly close to those achieved by models trained with textual inputs alone. Results show only minor differences between datasets synthesized using distinct TTS models, affirming the practical applicability of synthetic speech for model training and evaluation.

In comparison to systems deploying ASR followed by traditional VQA models, SVQA offers enhanced robustness, particularly where ASR introduces transcription errors.

Discussion and Future Directions

The paper effectively highlights SVQA's capability and identifies potential enhancement areas, such as exploring larger-scale datasets and more comprehensive pre-training for improved transcription accuracy. Future developments could aim at improving the integration of synthetic speech or expanding the dataset size to further bridge the performance gap between SVQA and unimodal systems.

Conclusion

This paper's contributions are notable in the advancement of multimodal AI systems, demonstrating the feasibility and efficacy of SVQA models in enhancing human-machine interactions through richer, more dynamic communication modes. As multimodal systems continue to evolve, the research here serves as a pivotal step toward broader practical applications in speech and visual-based AI technologies.