Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Siamese Vision Transformers are Scalable Audio-visual Learners (2403.19638v1)

Published 28 Mar 2024 in cs.CV, cs.SD, and eess.AS

Abstract: Traditional audio-visual methods rely on independent audio and visual backbones, which is costly and not scalable. In this work, we investigate using an audio-visual siamese network (AVSiam) for efficient and scalable audio-visual pretraining. Our framework uses a single shared vision transformer backbone to process audio and visual inputs, improving its parameter efficiency, reducing the GPU memory footprint, and allowing us to scale our method to larger datasets and model sizes. We pretrain our model using a contrastive audio-visual matching objective with a multi-ratio random masking scheme, which enables our model to process larger audio-visual instance batches, helpful for contrastive learning. Unlike prior audio-visual methods, our method can robustly handle audio, visual, and audio-visual inputs with a single shared ViT backbone. Furthermore, despite using the shared backbone for both modalities, AVSiam achieves competitive or even better results than prior methods on AudioSet and VGGSound for audio-visual classification and retrieval. Our code is available at https://github.com/GenjiB/AVSiam

Definition Search Book Streamline Icon: https://streamlinehq.com
References (82)
  1. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. In NeurIPS, 2021.
  2. Self-supervised multimodal versatile networks. In NeurIPS, 2020.
  3. Self-supervised learning by cross-modal audio-video clustering. In NeurIPS, 2020.
  4. Look, listen and learn. In ICCV, 2017.
  5. Objects that sound. In ECCV, 2018.
  6. Labelling unlabelled videos from scratch with multi-modal self-supervision. In NeurIPS, 2020.
  7. Soundnet: Learning sound representations from unlabeled video. In NeurIPS, 2016.
  8. Mae-ast: Masked autoencoding audio spectrogram transformer. In INTEERSPEECH, 2022.
  9. Multimae: Multi-modal multi-task masked autoencoders. In ECCV, 2022.
  10. Data2vec: A general framework for self-supervised learning in speech, vision and language. In ICML, 2022.
  11. Vggsound: A large-scale audio-visual dataset. In ICASSP, 2020.
  12. Pali: A jointly-scaled multilingual language-image model. In ICLR, 2023.
  13. Distilling audio-visual knowledge by compositional contrastive learning. In CVPR, 2021.
  14. Joint-modal label denoising for weakly-supervised audio-visual video parsing. In ECCV, 2022.
  15. One model, multiple modalities: A sparsely activated approach for text, sound, image, video and code. arXiv Preprint, 2022.
  16. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
  17. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
  18. Large scale audiovisual learning of sounds with weakly labeled data. In IJCAI, 2020.
  19. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, 2017.
  20. Audiovisual masked autoencoders. In ICCV, 2023.
  21. Omnimae: Single model masked pretraining on images and videos. In CVPR, 2023.
  22. Omnivore: A single model for many visual modalities. In CVPR, 2022.
  23. AST: Audio Spectrogram Transformer. In INTEERSPEECH, 2021.
  24. Psla: Improving audio tagging with pretraining, sampling, labeling, and aggregation. TASLP, 2021.
  25. Uavm: A unified model for audio-visual learning. IEEE Signal Processing Letters, 2022.
  26. Contrastive audio-visual masked autoencoder. In ICLR, 2023.
  27. Masked autoencoders are scalable vision learners. In CVPR, 2022.
  28. Mavil: Masked audio-video learners. In NeurIPS, 2023.
  29. Perceiver io: A general architecture for structured inputs & outputs. In ICLR, 2022.
  30. Perceiver: General perception with iterative attention. In ICML, 2021.
  31. Audio-visual contrastive learning with temporal self-supervision. In AAAI, 2023.
  32. Cross-attentional audio-visual fusion for weakly-supervised action localization. In ICLR, 2021.
  33. Acav100m: Automatic curation of large-scale datasets for audio-visual video representation learning. In ICCV, 2021.
  34. Learning to answer questions in dynamic audio-visual scenarios. In CVPR, 2022.
  35. Scaling language-image pre-training via masking. In CVPR, 2023.
  36. Mst: Masked self-supervised transformer for visual representation. In NeurIPS, 2021.
  37. Polyvit: Co-training vision transformers on images, videos and audio. arXiv Preprint, 2021.
  38. Eclipse: Efficient long-range video retrieval using sight and sound. In ECCV, 2022.
  39. Vision transformers are parameter-efficient audio-visual learners. In CVPR, 2023.
  40. Exploring cross-video and cross-modality signals for weakly-supervised audio-visual video parsing. In NeurIPS, 2021.
  41. Audiovisual transformer with instance attention for audio-visual event localization. In ACCV, 2020.
  42. Exploring target representations for masked autoencoders. arXiv Preprint, 2022.
  43. Active contrastive learning of audio-visual video representations. In ICLR, 2021.
  44. Contrastive learning of global and local audio-visual representations. In NeurIPS, 2021.
  45. Ave-clip: Audioclip-based multi-window temporal transformer for audio visual event localization. In WACV, 2023.
  46. A unified audio-visual learning framework for localization, separation, and recognition. In ICML, 2023.
  47. Class-incremental grouping network for continual audio-visual learning. In ICCV, 2023.
  48. Multi-modal grouping network for weakly-supervised audio-visual video parsing. In NeurIPS, 2022.
  49. Robust audio-visual instance discrimination. In CVPR, 2021.
  50. Audio-visual instance discrimination with cross-modal agreement. In CVPR, 2021.
  51. Multimodal contrastive learning with limoe: the language-image mixture of experts. In NeurIPS, 2022.
  52. Attention bottlenecks for multimodal fusion. In NeurIPS, 2021.
  53. Representation learning with contrastive predictive coding. arXiv Preprint, 2018.
  54. Audio-visual scene analysis with self-supervised multisensory features. In ECCV, 2018.
  55. Ambient sound provides supervision for visual learning. In ECCV, 2016.
  56. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
  57. Janani Ramaswamy. What makes the sound?: A dual-modality interacting network for audio-visual event localization. In ICASSP, 2020.
  58. See the sound, hear the pixels. In WACV, 2020.
  59. Dual perspective network for audio-visual event localization. In ECCV, 2022.
  60. Zorro: the masked multimodal transformer. arXiv Preprint, 2023.
  61. Learning audio-visual speech representation by masked multimodal cluster prediction. In ICLR, 2022.
  62. Everything at once–multi-modal fusion transformer for video retrieval. In CVPR, 2022.
  63. Tvlt: Textless vision-language transformer. In NeurIPS, 2022.
  64. Evading the simplicity bias: Training a diverse set of models discovers solutions with superior ood generalization. In CVPR, 2022.
  65. Unified multisensory perception: Weakly-supervised audio-visual video parsing. In ECCV, 2020.
  66. Audio-visual event localization in unconstrained videos. In ECCV, 2018.
  67. Clippo: Image-and-language understanding from pixels only. In CVPR, 2023.
  68. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.
  69. Equivariance and invariance inductive bias for learning from insufficient data. In ECCV, 2022.
  70. One-peace: Exploring one general representation model toward unlimited modalities. arXiv Preprint, 2023.
  71. What makes training multi-modal classification networks hard? In CVPR, 2020.
  72. Masked feature prediction for self-supervised visual pre-training. In CVPR, 2022.
  73. Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In CVPR, 2021.
  74. Dual attention matching for audio-visual event localization. In ICCV, 2019.
  75. Cross-modal background suppression for audio-visual event localization. In CVPR, 2022.
  76. Masked autoencoders that listen. In NeurIPS, 2022.
  77. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.
  78. Cross-modal attention network for temporal inconsistent audio-visual event localization. In AAAI, 2020.
  79. Learning visual representation from modality-shared contrastive language-image pre-training. In ECCV, 2022.
  80. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022.
  81. Audio-adaptive activity recognition across video domains. In CVPR, 2022.
  82. Contrastive learning relies more on spatial inductive bias than supervised learning: An empirical study. In ICCV, 2023.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now