Pretrained Text-Audio Embeddings: CLAP & MuQ-MuLan

• Pretrained text-audio embeddings are robust multimodal representations that map audio signals and text into a unified space using contrastive pretraining.
• These models employ two-tower architectures with modality-specific encoders and common projection layers, achieving state-of-the-art zero-shot audio retrieval and classification.
• Applications include music retrieval, text-guided audio generation, and semantic analysis, while challenges remain in dataset bias and perceptual alignment.

Pretrained text-audio embeddings are multimodal representation spaces enabling joint modeling of audio and language. Two prominent frameworks—Contrastive Language-Audio Pretraining (CLAP) and MuQ-MuLan—have established state-of-the-art performance in zero-shot audio retrieval, classification, and semantic audio analysis by learning robust, general-purpose mappings of audio and text into a shared space via large-scale contrastive learning. These models are pivotal for downstream tasks including music similarity, perceptual sound analysis, text-based audio generation, and cross-modal information retrieval.

1. Model Architectures and Training Objectives

Both CLAP and MuQ-MuLan adopt a two-tower design with modality-specific encoders projected into a common embedding space. The training objective is a symmetric contrastive (InfoNCE or derivatives), which encourages paired audio and text samples to have higher cosine similarity than negatives.

• CLAP (LAION-CLAP, MS-CLAP):
  • Audio Encoder: Typically a CNN (e.g., CNN14 from PANN, or a Hierarchical Token-Semantic Audio Transformer (HTS-AT)), mapping waveform or spectrograms to dense vectors.
  • Text Encoder: BERT/CLIP/RoBERTa transformers; frequently frozen to leverage large-scale pretraining.
  • Projection: Linear or shallow MLPs to the final latent space (commonly 512 or 1024 dimensions).
  • Training Data: Paired audio–language samples spanning environmental, musical, and spoken domains; e.g., LAION-Audio-630k, FSD50K, Clotho, AudioCaps.
  • Loss: Symmetric InfoNCE; for a batch $\{(x_i, t_i)\}_{i=1}^N$, the loss is

  $$L = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp(\operatorname{sim}(\Phi_\text{audio}(x_i), \Phi_\text{text}(t_i)) / \tau)} {\sum_{j=1}^N \exp(\operatorname{sim}(\Phi_\text{audio}(x_i), \Phi_\text{text}(t_j)) / \tau)},$$

  where $\operatorname{sim}$ is cosine similarity and $\tau$ is learnable.

• MuQ-MuLan:

  • Audio Encoder: Based on MuQ, a conformer architecture pretrained via masked Mel-residual vector quantization prediction. Input spectrograms are quantized using a multi-level vector quantizer ("Mel-RVQ"), providing discrete pseudo-tokens.
  • Text Encoder: XLM-RoBERTa or equivalent, often with additional layers for domain adaptation.
  • Projection: Linear layers with LayerNorm to 512 dimensions.
  • Training Data: Exclusively music audio and descriptive metadata (tags, captions), sampled to promote rare concept diversity (e.g., Music4All, in-house datasets).
  • Loss: Decoupled contrastive (DCL) or InfoNCE; consistently uses L2 normalization across modalities.

Model	Audio Encoder	Text Encoder	Embedding Size	Pretraining Data	Primary Loss	Citation
LAION-CLAP	HTS-AT	RoBERTa	512	LAION-Audio-630k, music/speech/env	InfoNCE	(Deng et al., 16 Oct 2025)
MuQ-MuLan	MuQ (Conformer + Mel-RVQ)	XLM-RoBERTa-base	512	Music4All, ∼130k h music-text pairs	DCL/InfoNCE	(Zhu et al., 2 Jan 2025)

2. Embedding Extraction and Similarity Computation

At inference, a fixed-length audio segment is embedded via the audio encoder, and a text string embedded via the text encoder, producing L2-normalized vectors in the shared space. Cosine similarity is the default metric for downstream zero-shot tasks:

• For retrieval: $$\operatorname{score}(X, V) = \cos(\Phi(X), \Phi(V))$$.
• For ABX similarity: given reference $X$ and two candidates $A, B$, select $$V^* = \arg\max_{V \in \{A, B\}} \cos(\Phi(X), \Phi(V))$$ (Vohra et al., 27 Jan 2026).
• Performance is robust to segment padding (e.g., 5 s audio zero-padded to 10 s).

3. Perceptual and Semantic Evaluation

Text-audio embeddings are validated as zero-shot predictors of both semantic and perceptual similarity in diverse regimes:

• Music similarity and retrieval: On the Inst-Sim-ABX dataset (Slakh2100), zero-shot LAION-CLAP and MuQ-MuLan reach 71.9% and 72.4% agreement with human listeners (XAB; full mixes), rivaling or exceeding self-supervised metric learning baselines and specialist models (e.g., Cascade-PAFT 70.5–76.3%) (Vohra et al., 27 Jan 2026).
• Timbre semantics: LAION-CLAP exhibits the strongest alignment with human-perceived timbre on both instrument and DSP-effect axes, with positive descriptor-level/instance-level correlations and robust monotonicity trends for spectral descriptors (EQ, reverb). In contrast, MuQ-MuLan demonstrates mixed or erratic behavior for adjectives not strongly represented in music metadata (e.g., "vigorous") (Deng et al., 16 Oct 2025).
• Audio tagging and music analysis: MuQ-MuLan sets state-of-the-art on MagnaTagATune (zero-shot tagging), ROC-AUC 79.3 vs. LAION-CLAP 73.9 (Zhu et al., 2 Jan 2025).
• Generalization: Models show strong latent semantic coverage over both environmental and musical domains. Unsupervised models (e.g., CLAP) maintain high accuracy with limited pretraining data; MuQ-MuLan leverages massive web-scale corpora for further boost in large-vocabulary and fine-grained retrieval.

4. Interpretability, Adaptation, and Post-hoc Alignment

• Stem-wise perceptual alignment: By linearly combining stem (instrumental) similarities—enabled by source separation (Demucs, Slakh2100)—and learning weights to fit listener judgments, MuQ-MuLan embeddings yield interpretable, instrument-wise contributions to perceived musical similarity. For example, weighting vectors show listener preference for melodic/harmonic (“residuals,” “mix”) and rhythmic (drums) cues (Vohra et al., 27 Jan 2026).
• Concept-based sparsification: CLAP embeddings can be decomposed over large audio-specific vocabulary sets via non-negative sparse coding (Lasso), yielding human-interpretable, concept-level activations (e.g., "train," "metallic-clatter") without degrading performance on classification or retrieval (Zhang et al., 18 Apr 2025).
• Fine-tuning for human perception: Directly aligning embedding similarity to subjective listener ratings ("Human-CLAP") substantially increases correlation between machine and human relevance judgements (SRCC +0.25 over conventional CLAP), with simple regression and re-weighted contrastive loss objectives (Takano et al., 30 Jun 2025).
• Soft-target and temporal modeling: Recent variants such as SmoothCLAP introduce softened cross-modal targets using intra-modal similarity and paralinguistic kernels, yielding marked gains for emotion-aware alignment. T-CLAP integrates synthetic temporal negatives and a dedicated temporal loss to improve modeling of sequential audio events (Jing et al., 18 Jan 2026, Yuan et al., 2024).

5. Comparison and Empirical Performance

Model	AudioCaps Retrieval R@1	MagnaTagATune ROC-AUC	ABX Music Agreement	Perceptual Timbre Alignment
LAION-CLAP	34–36%	73.9	71.9%	Strongest
MuQ-MuLan	42.2%	79.3	72.4%	Mixed
Human-CLAP	N/A	N/A	N/A	Strong (with adaptation)

Empirical comparisons indicate that MuQ-MuLan excels in music-centric tasks and fine-grained audio-text retrieval where massive pretraining data are available, while LAION-CLAP is preferred for perceptual alignment tasks demanding fine-grained control over descriptors not typically present in music metadata. Both maintain strong cross-domain generalizability, but the inductive biases of their pretraining corpora are reflected in the latent dimensions learned (e.g., instrument identity vs. perceptual adjective sensitivity) (Deng et al., 16 Oct 2025, Zhu et al., 2 Jan 2025).

6. Limitations, Open Problems, and Future Directions

• Dataset bias: Corpora such as Slakh2100 (MIDI-rendered audio) and music-video metadata impose representational limitations, manifesting as model idiosyncrasies (e.g., heavy “residual” reliance, less sensitivity to nuanced adjectives).
• Perceptual misalignment: Raw embedding similarity (e.g., CLAPScore) often has weak correlation with subjective human evaluation, motivating additional supervised or semi-supervised adaptation (e.g., Human-CLAP) (Takano et al., 30 Jun 2025).
• Emergent alignment: Alignment with perceptual attributes such as timbre and emotion is emergent, not directly supervised; targeted finetuning or multi-task objectives may further bridge the gap for perceptually critical tasks (Deng et al., 16 Oct 2025, Jing et al., 18 Jan 2026).
• Multi-query and temporal modeling: Architectural innovations (multi-query attention in MuQ-MuLan, temporal negatives in T-CLAP) address cross-token alignment and order, but temporal causality is largely absent from baseline models (Yuan et al., 2024).
• Scaling and interpretability trade-offs: Increasing encoder capacity and dataset size yields gains for fine-grained, long-tail semantics, but interpretability and practical deployment may require concept-based reductions or token-level decomposition (Zhang et al., 18 Apr 2025, Vohra et al., 27 Jan 2026).

A plausible implication is that future work will focus on integrating explicit perceptual annotation, leveraging source separation and hierarchical modeling (stems, instrument-wise features), and scaling multi-task or curriculum-based pretraining covering both semantic and perceptual dimensions.

7. Applications and Practical Significance

• Music retrieval and production: Both CLAP and MuQ-MuLan facilitate zero-shot, interpretable music search and stem-based audio manipulation, with the potential for instrument-aware query-by-example systems (Vohra et al., 27 Jan 2026).
• Text-guided generation and captioning: Pretrained embeddings provide strong conditioning vectors for text-to-audio and audio-to-text generative models (e.g., AudioLDM), improving control over perceptual and temporal properties of output (Yuan et al., 2024, Deng et al., 16 Oct 2025).
• Automated evaluation: Embedding-based relevance scores are widely used for evaluation of generative models; perceptually-adapted scores (e.g., Human-CLAP) improve correlation with true user satisfaction (Takano et al., 30 Jun 2025).
• Semantic exploration: Sparse concept decompositions allow for interpretable probing and analysis, supporting auditing, explainability, and human-in-the-loop downstream pipelines (Zhang et al., 18 Apr 2025).

These applications underline the centrality of robust, pretrained text-audio embedding spaces as foundation models for multimodal audio research and production workflows.