COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddings

Abstract: Contrastive Language-Audio Pretraining (CLAP) models are widely used for audio understanding and support modality-agnostic condition swapping in many zero-shot applications. However, their performance is heavily affected by the modality gap between audio and text embeddings. Existing explanations mainly attribute this gap to the cone effect, treating it as a shift between mean embeddings, yet correcting the mean alone yields only limited improvements. Alternative hypotheses, such as information imbalance and dimensionality collapse, have also been proposed, but they remain insufficiently verified and have not been thoroughly studied in the audio domain. Meanwhile, several works attempt to decompose multimodal contrastive embeddings into interpretable concepts, but none explicitly analyze the modality gap from the perspective of concept decomposition. In this work, we introduce COMET (Concept space Organization and Modality gap Explanation with PLS-SVD Transformation), a novel partial least squares singular value decomposition (PLS-SVD) framework for CLAP that unveils a broader perspective of the modality gap. Our framework reveals that only a small, interpretable subset of axes, which captures shared concepts, contributes substantially to similarity computation, and that the mean component represents only partially the modality gap. Building on this insight, we propose a simple spectral truncation method that mitigates the modality gap in a training-free manner. The method enables zero-shot audio captioning with condition swapping to approach fully supervised performance, without requiring large auxiliary memory banks or expensive computation. At the same time, it achieves substantial embedding dimensionality reduction while preserving strong performance on retrieval and audio captioning tasks.

• The paper introduces COMET, a PLS-SVD based framework that decomposes audio-text embeddings to identify a low-dimensional shared semantic head.
• It reveals that the static mean and modality-private tail contribute noise, while the compact head drives effective retrieval and captioning.
• PLSHead, a training-free spectral truncation method, compresses 1024D embeddings to 100D with zero-shot performance improvements.

Concept Space Dissection of the Modality Gap in Audio-Text CLAP Embeddings

Background and Motivation

Contrastive Language-Audio Pretraining (CLAP) has become a cornerstone in multimodal learning for audio understanding, facilitating efficient condition swapping between modalities in zero-shot and supervised tasks without requiring expensive retraining. Despite CLAP's promise, cross-modal discrepancies—termed the modality gap—persist between audio and text embeddings, adversely affecting downstream performance where modality-agnostic representations are crucial. Previous explanations for this gap, predominantly the "cone effect" (mean offset between modality clusters), have proven insufficient; centering embeddings only partly addresses performance loss, while other hypotheses such as information imbalance and dimension collapse lack comprehensive verification for audio-centric models.

PLS-SVD Framework: Concept-Aware Decomposition

The paper introduces COMET, leveraging Partial Least Squares Singular Value Decomposition (PLS-SVD) to dissect the shared embedding space generated by CLAP into interpretable, rank-structured subspaces. This decomposition reveals three major components:

• Mean component: Encodes the static modality gap attributed to the cone effect.
• Shared semantic head: A relatively low-dimensional (∼100 axes for 1024D embeddings), highly aligned subspace capturing cross-modal semantic concepts.
• Modality-private tail: Contains significant residual energy in both modalities yet is poorly aligned and largely irrelevant to similarity computation.

A key insight is that only the compact shared head significantly contributes to multimodal retrieval and captioning, while mean correction and tail components act as secondary or even detrimental sources to alignment.

Figure 1: The singular value $\Sigma_{ii}$ distribution and UV alignment show rapid decay, indicating a pronounced low-dimensional shared semantic subspace in CLAP embeddings.

Figure 2: Covariance decomposition reveals that self-variance in both modalities decays slowly, but cross-modal correlation drops abruptly in the shared head, substantiating the head-tail dichotomy.

Figure 3: The absolute values of the UV matrix $|U^TV|$, visualized, demonstrate strong diagonal alignment in the shared head, weak elsewhere, confirming direct-effect dominance in similarity computation.

Empirical Dissection: Modality Gap Structure

Experiments conducted on Clotho and AudioCaps, using multiple CLAP models, validate the low-rank nature (∼100 active dimensions) of shared semantics, with the tail subspace capturing substantial but unaligned energy likely comprising modality-specific noise or content. Concept axes derived from the head correspond to interpretable semantic categories (e.g., "traffic+birds", "speech", "machinery"). Most similarity computation involves direct-effect terms on the diagonal of the UV matrix; cross-effects are negligible.

Modal Gap Mitigation: Spectral Truncation and Embedding Compression

Building on the structural findings, the authors propose PLSHead, a training-free spectral truncation method retaining only the top-$K$ (e.g., 100) concept axes and discarding the tail. This achieves several practical advantages:

• Bridging modality gap: substantially reduces discrepancies without training or memory banks.
• Compression: transforms embeddings from 1024D to 100D with no loss—and often improvement—on retrieval and captioning tasks.
• Zero-shot performance: audio captioning approaches the performance of fully supervised setups.

Empirical results demonstrate PLSHead outperforming or matching original embeddings and projection decoding (PD), which relies on large memory banks, while PCA truncation performs poorly due to its unimodal nature.

Figure 4: Retrieval performance in mAP@10 saturates rapidly as the number of singular values increases, with the head subspace covering nearly all useful signal.

Theoretical Analysis of Projection Decoding

A rigorous analysis establishes that projection decoding (PD), previously treated as an empirical solution, operates as a concept space truncation and imputation filter. PD preserves the shared head, reconstructs tails from memory bank neighbors, and swaps mean terms, corroborating the multi-source nature of the modality gap. Experimental measurements confirm the head is reliably preserved, while the reconstructed tail diverges from the original.

Figure 5: Visualization of $\hat{X^T\hat{X}$, the covariance matrix of text projection coefficients, shows rapid decay along the diagonal and supports the truncation-imputation theory.

Cross-Model and Cross-Domain Robustness

The structural findings and PLSHead's efficacy are consistent across several CLAP architectures and training sets, indicating broad applicability. Retrieval and captioning with PLSHead shows robust generalization in cross-domain evaluation, suggesting portability of the concept space structure.

Figure 6: PLS covariance decomposition and UV alignment for diverse CLAP models confirm shared head sizes and alignment patterns.

Implications and Future Directions

COMET advances theoretical understanding of cross-modal embedding spaces in contrastive learning, highlighting that the modality gap is multifactorial—arising from mean, head misalignment, and unaligned tails. Concept-based spectral truncation enables efficient, scalable embedding compression and gap mitigation. Practically, this approach avoids retraining or reliance on memory-intensive methods while achieving state-of-the-art performance in retrieval and captioning.

Theoretical implications include the importance of low-dimensional, interpretable concept axes in defining modalities' intersection, suggesting future models might further optimize or explicitly regularize alignment in this subspace. The modality-tail analysis points to potential benefits from more rigorous denoising or domain adaptation techniques targeted at the tail. Prospective work may combine concept-space truncation with adaptive alignment or leveraging higher-order semantic relationships.

Conclusion

COMET presents a unified framework for dissecting the modality gap in audio-text contrastive embeddings. PLS-SVD analysis uncovers a compact, shared concept space crucial for similarity computation; mean and tail terms contribute secondary, often detrimental effects to modality alignment. The proposed spectral truncation method achieves efficient, training-free mitigation of the gap, compressing embeddings and facilitating high-performance zero-shot audio captioning and retrieval. These findings enhance both theoretical understanding and practical deployment of multimodal contrastive models.