Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)
Papers
Topics
Authors
Recent
Search
2000 character limit reached

AVROBUSTBENCH\texttt{AVROBUSTBENCH}: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time

Published 31 May 2025 in cs.SD, cs.AI, cs.LG, and eess.AS | (2506.00358v1)

Abstract: While recent audio-visual models have demonstrated impressive performance, their robustness to distributional shifts at test-time remains not fully understood. Existing robustness benchmarks mainly focus on single modalities, making them insufficient for thoroughly assessing the robustness of audio-visual models. Motivated by real-world scenarios where shifts can occur simultaneously\textit{simultaneously} in both audio and visual modalities, we introduce AVROBUSTBENCH\texttt{AVROBUSTBENCH}, a comprehensive benchmark designed to evaluate the test-time robustness of audio-visual recognition models. AVROBUSTBENCH\texttt{AVROBUSTBENCH} comprises four audio-visual benchmark datasets, AUDIOSET-2C\texttt{AUDIOSET-2C}, VGGSOUND-2C\texttt{VGGSOUND-2C}, KINETICS-2C\texttt{KINETICS-2C}, and EPICKITCHENS-2C\texttt{EPICKITCHENS-2C}, each incorporating 75 bimodal audio-visual corruptions that are co-occurring\textit{co-occurring} and correlated\textit{correlated}. Through extensive evaluations, we observe that state-of-the-art supervised and self-supervised audio-visual models exhibit declining robustness as corruption severity increases. Furthermore, online test-time adaptation (TTA) methods, on VGGSOUND-2C\texttt{VGGSOUND-2C} and KINETICS-2C\texttt{KINETICS-2C}, offer minimal improvements in performance under bimodal corruptions. We further propose AV2C\texttt{AV2C}, a simple TTA approach enabling on-the-fly cross-modal fusion by penalizing high-entropy samples, which achieves improvements on VGGSOUND-2C\texttt{VGGSOUND-2C}. We hope that AVROBUSTBENCH\texttt{AVROBUSTBENCH} will steer the development of more effective and robust audio-visual TTA approaches. Our code is available \href\href{https://github.com/sarthaxxxxx/AV-C-Robustness-Benchmark}{here}.

Citations (4)

Summary

No one has generated a summary of this paper yet.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Continue Learning

We haven't generated follow-up questions for this paper yet.

Tweets

Sign up for free to view the 1 tweet with 2 likes about this paper.