AudioBERTScore: Evaluation of TTA Systems

• AudioBERTScore is an embedding-based metric inspired by BERTScore, designed to assess environmental sound synthesis in text-to-audio systems.
• It leverages high-level audio embeddings from pretrained models and employs max-norm and p-norm aggregations to capture both localized and distributed sound features.
• The metric shows stronger correlation with human perceptual judgments than traditional metrics, making it ideal for benchmarking and model selection in audio synthesis research.

AudioBERTScore is an objective, embedding-based metric introduced for the evaluation of environmental sound synthesis systems, particularly text-to-audio (TTA) models. Expanding on the paradigm set by BERTScore in natural language processing, AudioBERTScore assesses the similarity between sequences of high-level audio embeddings obtained via large pretrained neural networks, aiming to provide an automatic metric whose correlation with human perceptual judgments is significantly superior to traditional signal-level metrics. Its key innovation is the use of both max-norm and generalized $p$-norm aggregation schemes to account for the non-local, temporally distributed nature of environmental sounds (2507.00475).

1. Conceptual Motivation and Distinction from Prior Metrics

AudioBERTScore was designed in response to the observed shortcomings of established reference-based audio evaluation metrics—such as mel-cepstral distortion (MCD) and WARP-Q—whose reliance on low-level signal statistics results in weak correlation with subjective assessments, particularly for complex or semantically rich sounds. Unlike these approaches, which operate on framewise or envelope distances, AudioBERTScore utilizes semantic representations extracted from pretrained audio foundation models (e.g., Audio Spectrogram Transformer, AST) to compare synthesized and reference audio at a contextual level. This methodology parallels BERTScore in machine translation and image captioning evaluation, but introduces design changes for the idiosyncrasies of environmental audio.

2. Methodology: Embedding-Based Similarity

Calculation of AudioBERTScore proceeds in several stages:

1. Feature Extraction: Synthesized audio $\bm{s}$ and reference audio $\bm{r}$ are independently passed through a pretrained encoder (AST or ATST-Frame), yielding sequences of embedding vectors

$$\tilde{\bm{s}}_n \in \mathbb{R}^D, \quad n=1,\ldots,L_\text{gen},\quad \tilde{\bm{r}}_m \in \mathbb{R}^D, \quad m=1,\ldots,L_\text{ref}$$

1. Cosine Similarity Matrix: Pairwise cosine similarities are computed,

$$M_{nm} = \mathrm{sim}(\tilde{\bm{s}}_n, \tilde{\bm{r}}_m) = \frac{\tilde{\bm{s}}_n \cdot \tilde{\bm{r}}_m}{\|\tilde{\bm{s}}_n\|\|\tilde{\bm{r}}_m\|}$$

constructing a similarity matrix $M$.

1. Aggregation and F1 Metric: Similarities are aggregated over the time axis using either the max-norm or a generalized $p$-norm across rows and columns (see details below), yielding precision, recall, and ultimately the $F_1$-score used as AudioBERTScore.

3. Max-Norm, $p$-Norm, and Non-Locality

AudioBERTScore generalizes the aggregation approach of standard BERTScore to reflect both localized and distributed similarity characteristics in audio:

• Max-Norm Aggregation ($p\rightarrow \infty$): Assumes high similarity is localized in time, best suited for discrete or instantaneous sounds (e.g., speech, beeps):

$$\mathrm{precision}_{\max} = \frac{1}{L_\text{gen}} \sum_{i=1}^{L_\text{gen}} \max_{j} M_{ij}$$

$$\mathrm{recall}_{\max} = \frac{1}{L_\text{ref}} \sum_{j=1}^{L_\text{ref}} \max_{i} M_{ij}$$

• $p$-Norm Aggregation: Addresses temporally extended, non-local sound events (e.g., flowing water, rain). For a chosen $p$,

$$\mathrm{precision}_p = \frac{1}{L_\text{gen}} \sum_{i=1}^{L_\text{gen}} \left( \frac{1}{L_\text{ref}} \sum_{j=1}^{L_\text{ref}} (M_{ij})^p \right)^{1/p}$$

• Interpolated Score: An interpolation parameter $\lambda$ allows blending between max-norm and $p$-norm results,

$$\mathrm{precision}_{\lambda, p} = \lambda\, \mathrm{precision}_{\max} + (1-\lambda)\, \mathrm{precision}_p$$

Analogous definitions apply for recall and $F_1$. This design captures the range from strictly local to entirely distributed similarity, with $\lambda$ and $p$ tuned to maximize correlation with human perceptual judgments for different sound classes.

4. Empirical Evaluation and Correlation with Subjective Scores

Experiments were conducted on the PAM test set (AudioCaps-derived), encompassing 100 reference sounds and 400 synthesized samples from models including MelDiffusion, AudioLDM, and AudioGen. AudioBERTScore was computed using various audio encoders (AST, ATST-Frame, BYOL-A):

• The optimal configuration used AST (13th layer) embeddings.
• Correlations between AudioBERTScore and human ratings (Mean Opinion Scores for Overall Quality (OVL) and Relevance (REL)) were significantly higher than for traditional metrics.

Metric	OVL-LCC	OVL-SRCC	REL-LCC	REL-SRCC
MCD	0.004	0.008	0.029	0.050
WARP-Q	0.241	0.228	0.202	0.213
AudioBERTScore (F1, max-norm)	0.368	0.397	0.490	0.515
AudioBERTScore ($p$-norm, $\lambda=-3.5, p=106$)	0.424	0.433	0.546	0.567
CLAPScore	0.337	0.323	0.487	0.475
PAM	0.595	0.604	0.529	0.556

AudioBERTScore provided the highest or near-highest correlation among all training-free, reference-based metrics.

5. Practical Use Cases and Implementation

AudioBERTScore is intended for:

• Automatic Benchmarking: Enables reliable model comparison and ablation studies in TTA without recourse to extensive human listening tests.
• Model Selection: Facilitates hyperparameter optimization and architecture search in contexts where subjective evaluation is impractical.
• General Research Tool: Offers a language-independent metric suitable for diverse audio domains, provided appropriate foundation models are available.

Implementation details include:

• Pretrained Model Requirement: Requires an audio foundation model for embedding extraction; effectiveness depends on the pretraining data and architectural depth (e.g., AST, ATST-Frame, BYOL-A).
• No Model Training Needed: AudioBERTScore is a reference-based, training-free metric.
• Computational Considerations: Embedding extraction is computationally heavier than traditional signal metrics, but does not require re-training or additional supervision.

6. Limitations and Interpretive Considerations

While AudioBERTScore marks a notable advance, several considerations are essential:

• Encoder Domain Generality: Performance may depend on the pretraining corpus and robustness of the embedding network; transfer to distinct domains (music, synthetic sounds) requires suitable foundation models.
• Temporal Alignment: Embedding sequence comparison implicitly assumes consistent temporal progression; for highly misaligned or time-stretched sounds, additional alignment may be needed.
• Correlation with Human Judgement: Correlation remains imperfect, and scores may not fully capture all perceptual nuances; further work could investigate integrating multiple semantic and perceptual metrics.

7. Summary Comparison with Alternative Metrics

Metric	Requires Training	Reference Required	Uses Pretrained Model
MCD	No	Yes (audio)	No
WARP-Q	No	Yes (audio)	No
FAD	No	Yes (audio)	Yes (audio)
AudioBERTScore	No	Yes (audio)	Yes (audio)
CLAPScore	No	Yes (text)	Yes (text/audio)
RELATE	Yes	Yes (text)	Yes (text+audio)
PAM	No	No	Yes (text/audio)

AudioBERTScore's use of context-rich, high-level audio representations positions it as a meaningful alternative or complement to traditional approaches, bridging an important gap between signal-based and perceptually-aligned model evaluation in environmental sound synthesis (2507.00475).