On Class Separability Pitfalls In Audio-Text Contrastive Zero-Shot Learning

Abstract: Recent advances in audio-text cross-modal contrastive learning have shown its potential towards zero-shot learning. One possibility for this is by projecting item embeddings from pre-trained backbone neural networks into a cross-modal space in which item similarity can be calculated in either domain. This process relies on a strong unimodal pre-training of the backbone networks, and on a data-intensive training task for the projectors. These two processes can be biased by unintentional data leakage, which can arise from using supervised learning in pre-training or from inadvertently training the cross-modal projection using labels from the zero-shot learning evaluation. In this study, we show that a significant part of the measured zero-shot learning accuracy is due to strengths inherited from the audio and text backbones, that is, they are not learned in the cross-modal domain and are not transferred from one modality to another.

• The paper demonstrates that data leakage from pre-trained unimodal backbones inflates zero-shot accuracy in audio-text learning.
• The paper uses T-SNE visualizations and silhouette scores to differentiate genuine cross-modal learning from backbone-induced clustering.
• The paper underscores the need for rigorous dataset cleaning and improved projection methods to achieve true semantic alignment.

On Class Separability Pitfalls In Audio-Text Contrastive Zero-Shot Learning

Recent advancements in cross-modal contrastive learning, particularly between audio and text modalities, have demonstrated potential in zero-shot learning tasks. The paper "On Class Separability Pitfalls In Audio-Text Contrastive Zero-Shot Learning" offers a critical evaluation of these methodologies, particularly focusing on the pitfalls associated with class separability due to data leakage. The authors primarily investigate how substantial portions of the measured zero-shot learning accuracy can be mistakenly attributed to strengths inherited from pre-trained unimodal backbone networks rather than genuine cross-modal learning.

Introduction and Framework

Zero-shot learning aims to classify instances from classes that were not part of the training set. Cross-modal zero-shot learning involves creating a shared latent space where both modalities are semantically represented. In this domain, the paper evaluates the Contrastive Language-Audio Pretraining (CLAP) model, which relies on pre-trained backbones (e.g., CNN14 for audio and BERT for text) and MLP projectors to map the backbone outputs onto a cross-modal space.

The study underscores the significance of pre-training and acknowledges that data leakage can considerably skew results. By leveraging T-SNE visualizations, silhouette scores, and neighborhood-based measures for topological similarity, the authors attempt to isolate the genuine learning achieved by the cross-modal projections from the inherited biases of the pre-trained backbones.

Key Experiments and Findings

The authors present six different training configurations for CLAP, varying both the cleanliness of data (dirty vs. clean) and the state of pre-training (dirty, clean, or none). The key numerical results center on zero-shot accuracy in the ESC50 dataset and silhouette scores for evaluating clustering quality.

1. Zero-shot Accuracy: As Table 1 indicates, configurations trained with data leakage ("dirty" datasets) exhibit higher zero-shot accuracy compared to their "clean" counterparts. Interestingly, the configurations without pre-training demonstrate significantly lower accuracy, implying that pre-training heavily influences performance.
2. Cluster Quality: T-SNE projections and silhouette scores reveal that embeddings from "dirty" datasets form better-defined clusters even without cross-modal training. Specific results (e.g., silhouette scores of $0.34$ for dirty/dirty versus $0.14$ for clean/clean) suggest a strong correlation between pre-trained backbone quality and zero-shot accuracy.
3. Topological Structure: The comparative analysis of topological structure similarities between unimodal embeddings ($x_a$, $x_t$) and their projected counterparts ($E_a$, $E_t$) shows that the cross-modal transformation did not effectively align topologies. Text embeddings maintained semantic grouping, whereas audio embeddings reflected their aural similarities. For instance, even post-cross-modal training, embeddings for "cow" and "sheep" remained grouped together based on sound rather than text semantics.
4. Impact of Training on Cluster Separation: Cross-modal training did not significantly adjust the topology of either audio or text embeddings. Initial states of $x_a$ and $x_t$ without cross-modal training revealed sufficient clustering which remained mostly unchanged post projection, indicating minimal topological modifications due to cross-modal learning.

Implications and Future Directions

This study highlights critical considerations for future research in contrastive cross-modal zero-shot learning:

1. Data Integrity: The significant influence of data integrity on performance necessitates rigorous dataset preprocessing to minimize leakage.
2. Evaluation Metrics: Silhouette scores and topological similarity measures serve as robust indicators of true cross-modal learning efficacy, offering a more nuanced understanding than accuracy metrics alone.
3. Backbone Influence: The paper establishes that pre-trained backbones are vital in determining cluster quality and zero-shot accuracy. Future work should explore methods to decouple the impact of pre-trained backbones from the learning achieved by cross-modal networks.

Future developments in cross-modal zero-shot learning should focus on improving projection methodologies to genuinely integrate information from different modalities, enhancing the semantic alignment of the shared latent space, and extending the analysis beyond accuracy metrics to include structural evaluations of embedding spaces. This will ensure that the strides made in developing these models are grounded in actual cross-modal learning rather than artifacts of pre-training and data leakage.