Multiscale Matching Driven by Cross-Modal Similarity Consistency for Audio-Text Retrieval (2403.10146v1)
Abstract: Audio-text retrieval (ATR), which retrieves a relevant caption given an audio clip (A2T) and vice versa (T2A), has recently attracted much research attention. Existing methods typically aggregate information from each modality into a single vector for matching, but this sacrifices local details and can hardly capture intricate relationships within and between modalities. Furthermore, current ATR datasets lack comprehensive alignment information, and simple binary contrastive learning labels overlook the measurement of fine-grained semantic differences between samples. To counter these challenges, we present a novel ATR framework that comprehensively captures the matching relationships of multimodal information from different perspectives and finer granularities. Specifically, a fine-grained alignment method is introduced, achieving a more detail-oriented matching through a multiscale process from local to global levels to capture meticulous cross-modal relationships. In addition, we pioneer the application of cross-modal similarity consistency, leveraging intra-modal similarity relationships as soft supervision to boost more intricate alignment. Extensive experiments validate the effectiveness of our approach, outperforming previous methods by significant margins of at least 3.9% (T2A) / 6.9% (A2T) R@1 on the AudioCaps dataset and 2.9% (T2A) / 5.4% (A2T) R@1 on the Clotho dataset.
- “Visual semantic reasoning for image-text matching,” in Proc, ICCV, 2019, pp. 4654–4662.
- “Learning transferable visual models from natural language supervision,” in Proc, ICML. PMLR, 2021, pp. 8748–8763.
- “LXMERT: Learning cross-modality encoder representations from transformers,” in Proc, EMNLP-IJCNLP, 2019, pp. 5100–5111.
- “Filip: Fine-grained interactive language-image pre-training,” arXiv preprint arXiv:2111.07783, 2021.
- “Universal vision-language dense retrieval: Learning a unified representation space for multi-modal retrieval,” in Proc, ICLRs, 2022.
- “On metric learning for audio-text cross-modal retrieval,” in Proc, Interspeech 2022, 2022.
- “Audio retrieval with wavtext5k and clap training,” arXiv preprint arXiv:2209.14275, 2022.
- “Audio-text retrieval in context,” in Proc, ICASSP. IEEE, 2022, pp. 4793–4797.
- “Clap learning audio concepts from natural language supervision,” in Proc,ICASSP. IEEE, 2023, pp. 1–5.
- “Stacked cross attention for image-text matching,” in Proc, ECCV, 2018, pp. 201–216.
- “WeaQA: Weak supervision via captions for visual question answering,” in Proc, ACL-IJCNLP, 2021, pp. 3420–3435.
- “ONE-PEACE: Exploring one general representation model toward unlimited modalities,” arXiv preprint arXiv:2305.11172, 2023.
- “Masked autoencoders that listen,” Proc, NeurIPS, vol. 35, pp. 28708–28720, 2022.
- “Multi-grained representation learning for cross-modal retrieval,” in Proc, SIGIR, 2023, pp. 2194–2198.
- “Improving text-audio retrieval by text-aware attention pooling and prior matrix revised loss,” in Proc, ICASSP. IEEE, 2023, pp. 1–5.
- “Audiocaps: Generating captions for audios in the wild,” in Proc, NAACL, 2019, pp. 119–132.
- “Clotho: An audio captioning dataset,” in Proc, ICASSP. IEEE, 2020, pp. 736–740.
- “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in Proc, ICASSP. IEEE, 2023, pp. 1–5.
- “Audio retrieval with natural language queries: A benchmark study,” IEEE Transactions on Multimedia, 2022.
- “HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection,” in Proc, ICASSP. IEEE, 2022, pp. 646–650.
- “Attention is all you need,” Proc, NeurIPS, vol. 30, 2017.
- Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proc, NAACL, 2019, pp. 4171–4186.
- “Deep sparse rectifier neural networks,” in Proc, AISTATS. JMLR Workshop and Conference Proceedings, 2011, pp. 315–323.
- “Colbert: Efficient and effective passage search via contextualized late interaction over bert,” in Proc, SIGIR, 2020, pp. 39–48.
- “A simple framework for contrastive learning of visual representations,” in Proc, ICML. PMLR, 2020, pp. 1597–1607.
- “Audio set: An ontology and human-labeled dataset for audio events,” in Proc, ICASSP. IEEE, 2017, pp. 776–780.
- “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
- Qian Wang (453 papers)
- Jia-Chen Gu (42 papers)
- Zhen-Hua Ling (114 papers)