Towards Open-Vocabulary Audio-Visual Event Localization (2411.11278v3)

Abstract: The Audio-Visual Event Localization (AVEL) task aims to temporally locate and classify video events that are both audible and visible. Most research in this field assumes a closed-set setting, which restricts these models' ability to handle test data containing event categories absent (unseen) during training. Recently, a few studies have explored AVEL in an open-set setting, enabling the recognition of unseen events as ``unknown'', but without providing category-specific semantics. In this paper, we advance the field by introducing the Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) problem, which requires localizing audio-visual events and predicting explicit categories for both seen and unseen data at inference. To address this new task, we propose the OV-AVEBench dataset, comprising 24,800 videos across 67 real-life audio-visual scenes (seen:unseen = 46:21), each with manual segment-level annotation. We also establish three evaluation metrics for this task. Moreover, we investigate two baseline approaches, one training-free and one using a further fine-tuning paradigm. Specifically, we utilize the unified multimodal space from the pretrained ImageBind model to extract audio, visual, and textual (event classes) features. The training-free baseline then determines predictions by comparing the consistency of audio-text and visual-text feature similarities. The fine-tuning baseline incorporates lightweight temporal layers to encode temporal relations within the audio and visual modalities, using OV-AVEBench training data for model fine-tuning. We evaluate these baselines on the proposed OV-AVEBench dataset and discuss potential directions for future work in this new field.

• The paper introduces the OV-AVEL task that explicitly classifies unseen audio-visual events beyond generic 'unknown' labels.
• It develops the OV-AVEBench dataset with 24,800 videos across 67 scenes, offering a diverse benchmark for event localization.
• Baseline experiments using zero-shot and fine-tuning methods show an 11.2% performance boost, highlighting effective temporal modeling.

Toward Open-Vocabulary Audio-Visual Event Localization: A Summary and Analysis

The task of Audio-Visual Event Localization (AVEL) traditionally involves temporal localization and classification of events within videos where both audio and visual components are present. Conventional methodologies have predominantly focused on closed-set scenarios, relying heavily on training data to predetermine event categories. These approaches impose limitations when confronted with novel events not previously encountered during training. The paper "Towards Open-Vocabulary Audio-Visual Event Localization" by Zhou et al. introduces a novel problem termed Open-Vocabulary Audio-Visual Event Localization (OV-AVEL), addressing these limitations by expanding the scope to include unseen events and categorizing them beyond mere "unknown" labels.

Key Contributions

1. Introduction of the OV-AVEL Task: The authors propose the OV-AVEL task, a significant extension of conventional AVEL, which seeks to classify unseen test data explicitly rather than resorting to generic "unknown" categories. This enhances the practical applicability of AVEL models in real-world scenarios where event categories are diverse and evolving.
2. Development of the OV-AVEBench Dataset: To support the new task, the authors crafted the OV-AVEBench dataset comprising 24,800 videos grouped into 67 unique audio-visual scenes, with a manual segment-level annotation distinguishing seen from unseen categories (46 seen and 21 unseen). The dataset is a substantial contribution, offering six times more data variety compared to traditional datasets such as the AVE.
3. Baseline Methods and Evaluation: Two baseline approaches are explored: a training-free method leveraging zero-shot learning via pre-trained multimodal models, and a fine-tuning approach incorporating temporal modeling to enhance feature alignment across modalities. The effectiveness of these baselines is empirically demonstrated on the OV-AVEBench dataset, complemented by newly established evaluation metrics including segment-level, event-level F1-scores alongside accuracy.

Numerical Results and Analysis

The fine-tuning baseline showed an 11.2% improvement in overall performance average compared to the training-free approach, evidencing the efficacy of temporal modeling in discerning and aligning temporally-extended audio-visual features. Seen test data particularly benefit from this approach, though the increase in unseen event performance suggests that temporal relations captured during fine-tuning facilitate better feature generalization. The inclusion of a special category label 'other' also showed superior performance by improving the ability to handle out-of-label scenarios.

Implications and Future Developments

The OV-AVEL task and OV-AVEBench dataset foster broader investigations into audio-visual learning, challenging existing architectures to adapt to evolving environments without explicit prior exposure to all possible events. This task encourages the development of models that embody more flexible learning paradigms, such as dynamic category expansion and adaptive temporal modeling. The move toward open-vocabulary scenarios mirrors the demands of real-world applications, underscoring the importance of context-aware and dynamically adaptive models.

In summary, the introduction of OV-AVEL lays the groundwork for exciting advancements in the field of audio-visual event recognition, prompting further exploration into integrating comprehensive semantic understanding within multimodal models. The structural and methodological innovations proposed by Zhou et al. thus extend a crucial platform for future paper, likely spurring significant research endeavors aimed at enhancing multimodal AI's adaptability and intelligence.