CLAP Score: Audio-Text Semantic Metric

• CLAP Score is a metric that quantifies semantic alignment between audio and text by embedding both in a shared latent space using dual-encoder architectures.
• It supports reference-free audio captioning, text-to-audio generation, and language-queried source separation through techniques like SLIDE-CLAP and variant adjustments.
• Empirical benchmarks show it excels in large-scale evaluations while highlighting challenges in fine-grained discrimination and the need for hybrid metric integrations.

The CLAP Score is a widely adopted metric for evaluating the semantic alignment between audio and text, especially in reference-free audio captioning, text-to-audio generation, and language-queried source separation. This metric exploits contrastive language–audio pretraining (CLAP) dual-encoder architectures to embed audio and textual descriptions in a shared latent space, quantifying their match as a single scalar via normalized cosine similarity. Because it circumvents the need for human-annotated references, the CLAP Score has become central in modern audio-language benchmarks, model training, and automatic evaluation pipelines across various generative and discriminative settings.

1. Mathematical Definition and Variants

Let $E_a$ be the audio encoder and $E_t$ the text encoder from a jointly trained CLAP model. Given an audio clip $A$ and a candidate caption $T$, the encoders map inputs to $d$-dimensional embeddings, which are then L₂-normalized: $$\mathbf{f}_A = \mathrm{Norm}(E_a(A)),\qquad \mathbf{f}_T = \mathrm{Norm}(E_t(T))$$ The raw CLAP Score, also denoted as CLAP-Similarity, is simply their cosine similarity: $$\mathrm{CLAPScore}(A,T) = \mathbf{f}_A^\top\,\mathbf{f}_T \in [-1,1]$$ If multiple captions are to be compared for the same audio (as in pairwise ranking), the one with higher CLAPScore is preferred (Guo et al., 11 Dec 2025, Elizalde et al., 2022).

For variable-length inputs, the SLIDE-CLAP procedure is used: the audio is segmented into $N$ overlapping windows, each segment embedded, then averaged and normalized: $$\mathbf{f}_A^{\mathrm{SLIDE}} = \mathrm{Norm}\left( \frac{1}{N} \sum_{i=1}^{N} \mathbf{f}_{a_i} \right)$$ and the score is computed as above with the aggregated embedding (Guo et al., 11 Dec 2025).

In source separation evaluation, additional variants are defined:

• CLAPScore-i: Measures improvement over the input mixture, $$\mathrm{CLAPScore}_i = \mathrm{CLAPScore}(\hat{S}, c) - \mathrm{CLAPScore}(M, c)$$.
• RefCLAPScore: If ground-truth is available, the harmonic mean of separated and reference scores (Xiao et al., 2024).

2. CLAP Model Architectures and Training Objectives

CLAP models consist of transformer-based or hybrid encoders for audio and text, projecting both modalities into a shared embedding space via learned projection heads. Prominent architectures and training regimes include:

• MS-CLAP (2022/2023): AudioSet-based training, transformer audio encoders, and L₂-normalized projections (Guo et al., 11 Dec 2025).
• M2D-CLAP: Incorporates masked spectrogram modeling with contrastive learning (Guo et al., 11 Dec 2025).
• LAION-CLAP: Trained on large web-scraped corpora with shared transformer backbones (Guo et al., 11 Dec 2025).

All use large-scale symmetric InfoNCE-style contrastive losses: $E_t$0 where $E_t$1 is a learnable temperature (Elizalde et al., 2022, Takano et al., 30 Jun 2025).

Some implementations clamp negative similarities to zero, yielding scores in $E_t$2 (Takano et al., 30 Jun 2025).

3. Evaluation Protocols and Benchmarks

Evaluation of CLAPScore typically follows two axes: general semantic alignment (audio-caption matching) and fine-grained discrimination (quality, hallucinations, syntax).

• BRACE Benchmark: Designed to rigorously test reference-free audio caption evaluation metrics (ACEMs).
  • BRACE-Main: Fine-grained pairwise caption comparisons across HH (human/human), HM (human/machine), and MM (machine/machine) pairs, annotated by experts.
  • BRACE-Hallucination: Detects subtle noun replacements (“dog bark”→“cat bark”) via LLM-driven corruption (Guo et al., 11 Dec 2025).
• Standard Metrics: F1-score of model preference against human annotation, mean and variance across multiple runs (accounting for audio windowing randomness). Hardware specification for large-scale evaluations includes clusters such as 8×NVIDIA H100 (Guo et al., 11 Dec 2025).
• Reference-free Source Separation: In language-queried source separation, CLAPScore substitutes for reference-dependent SDR or SI-SDR, allowing semantic evaluation without requiring clean ground-truth stems (Xiao et al., 2024).
• In Text-to-Audio (TTA) and Caption Selection: Used for n-best rescoring (e.g., in CLAP-Refine (Chen et al., 2024)) and preference optimization to align model generation with user intent (Hung et al., 2024, Karchkhadze et al., 2024).

4. Empirical Performance and Limitations

Quantitative results from BRACE show LAION-CLAP with sliding windows achieves a maximum F1 of 70.01 on BRACE-Main. However, all tested models excel only when distinguishing divergent human/machine pairs (HM), not subtle distinctions between two human or two competitive machine captions (HH/MM). F1 on hallucination detection exceeds 90 for some models, but general fine-grained discrimination remains challenging (Guo et al., 11 Dec 2025).

Table: Representative F1-scores from BRACE (Guo et al., 11 Dec 2025).

Model	BRACE-Main F1 (All)	BRACE-Hallucination F1 (All)
MS-CLAP-2023	61.51	85.99
LAION-CLAP	70.01	84.37
M2D-CLAP	60.91	88.26

Observed limitations:

• Foreground bias: Tends to prioritize dominant sounds, missing subtle background cues.
• Syntactic insensitivity: Fails to penalize grammatical errors if semantic tokens match.
• Window-induced variability: Short, fixed encoder windows mean output may change with input slicing unless mitigated.
• Inability to separate close paraphrases: Very poor at discriminating two high-quality, semantically similar captions (HH, MM pairs).

Empirical studies further show only moderate correlation between vanilla CLAPScore and human subjective ratings (Spearman’s ρ ~ 0.26–0.29), with regression-trained Human-CLAP raising this to >0.50 (Takano et al., 30 Jun 2025).

5. Extensions and Integrations

To address the above limitations, new scoring approaches and hybrid metrics have been proposed:

• SLIDE-CLAP: Averages overlapping embeddings to reduce window randomness (Guo et al., 11 Dec 2025).
• CAF-Score: Linearly combines an “S-CLAP” maximum-over-windows score with a Large Audio-LLM (LALM) rating (“FLEUR” probability) for improved sensitivity to syntax and hallucination (Lee et al., 20 Mar 2026).
• Human-CLAP: Introduces regression and weighted contrastive losses calibrated on subjective human preference, substantially improving correlation with listener judgments (Takano et al., 30 Jun 2025).
• CLAP-Refine (SLAM-AAC): Employs n-best CLAP-based rescoring to select the caption best aligned to an audio sample, yielding state-of-the-art performance on Clotho V2 and AudioCaps (Chen et al., 2024).
• Latent CLAP Loss: Fine-tunes TTA models to align generated audio latents with CLAP embeddings, removing the need for sample-level post-filtering (Karchkhadze et al., 2024).

Alternative architectures—such as M2D-CLAP’s masked modeling or LAION-CLAP’s broader training distribution—address robustness to corruption and rare event detection (Guo et al., 11 Dec 2025). Some proposals suggest syntax-aware boosting, event graph extraction, and hybrid reference/retrieval scoring for future metrics (Guo et al., 11 Dec 2025).

6. Implementation, Usage, and Practical Considerations

The CLAPScore admits simple, efficient implementation, typically involving embedding extraction, vector normalization, and batched dot products. It is used for:

• Caption ranking: Selects the highest-semantic match in n-best lists or for TTA preference optimization (Hung et al., 2024).
• Dataset filtering: Automated filtering or augmentation, e.g., in TTA or audio captioning data curation (Hung et al., 2024).
• Model training losses: Latent CLAP losses are often included in recent TTA or Foley sound generation systems as regularizers (Karchkhadze et al., 2024).

However, several practical issues persist:

• Training set coverage: Unseen sound categories or rare words degrade performance.
• Floor and ceiling effects: Empirical CLAP scores for real samples cluster between 0 and 0.6 even for high-quality, matched samples (Hung et al., 2024).
• Computational efficiency versus discriminative power: Extremely fast (enables large-batch auto-labeling), but pure CLAP-based evaluation sometimes fails in fine-grained settings compared to hybrid systems (e.g., CAF-Score) (Lee et al., 20 Mar 2026).
• **In TTA, over-optimization can lead to content collapse if not combined with fidelity-driven regularization as in Latent CLAP Loss (Karchkhadze et al., 2024).

Sample implementation (PyTorch-style): $E_t$3 (Xiao et al., 2024)

7. Implications and Directions for Future Audio-Language Evaluation

Current research highlights that while CLAPScore provides a scalable, reference-free proxy for semantic alignment, it is insufficient in isolation for high-stakes or fine-grained audio-language evaluation. Integration of multi-scale acoustic features, syntax-aware objectives, event-graph alignment, hybrid reference- and retrieval-based approaches, and direct calibration with human perception are active areas for improving the metric’s granularity and reliability (Guo et al., 11 Dec 2025, Lee et al., 20 Mar 2026, Takano et al., 30 Jun 2025).

Systematic benchmarking on datasets like BRACE is critical for revealing and quantifying the true strengths and blind spots of CLAPScore and its descendants, ultimately guiding the development of more robust, human-aligned automatic audio-language evaluation metrics (Guo et al., 11 Dec 2025).