ELSA: Acoustic Event-Level Semantic Alignment for Fine-Grained Reference-Free Text-to-Audio Evaluation

Abstract: Text-to-audio (TTA) generation, synthesizing audio from natural language, has been widely studied for its ability to capture precise user intent. To effectively advance TTA models, it is essential to reliably evaluate generated audio without relying on costly human subjective ratings, motivating the development of automatic evaluation metrics that correlate well with human judgments. While recent CLAP-based metrics provide practical reference-free solutions, their coarse-grained text-audio similarity matching often correlates poorly with human ratings. To address this, we propose ELSA, a reference-free evaluation metric for fine-grained text-audio alignment. ELSA decomposes generated audio guided by distinct acoustic events derived from the text query and assesses event-level alignment. Experiments across four TTA benchmarks show that ELSA reveals a higher correlation with human subjective ratings than prior metrics, highlighting its effectiveness for reliable TTA evaluation.

• The paper presents ELSA, a reference-free TTA evaluation metric that decomposes user text into atomic acoustic events and aligns them with specific audio segments.
• It employs a three-stage methodology: text decomposition with a large language model, event-level audio segmentation using LASS, and hierarchical alignment scoring in a shared embedding space.
• Empirical results show ELSA significantly improves correlation with human ratings, outperforming CLAPScore baselines by up to 13.1 points on key benchmarks.

ELSA: Acoustic Event-Level Semantic Alignment for Fine-Grained Reference-Free Text-to-Audio Evaluation

Motivation and Problem Setting

Text-to-audio (TTA) generation is increasingly prevalent in applications spanning speech synthesis, sound effects, and music generation. The primary impediment to advancing TTA systems is the lack of robust, fine-grained, and reference-free evaluation metrics that reliably correlate with human judgments. Most TTA evaluation protocols either require reference audio or depend on coarse-grained, global text–audio similarity metrics, notably those based on CLAP variants, which demonstrate limited correlation with subjective ratings.

This paper introduces ELSA (Acoustic Event-Level Semantic Alignment), a reference-free, fine-grained automatic evaluation metric for TTA. ELSA conceptualizes TTA evaluation as the problem of event-level semantic alignment, decomposing user-provided text queries into atomic acoustic events and directly assessing their correspondence to localized segments in generated audio. This enables ELSA to capture nuanced failures at the event level, which are typically obscured in global similarity spaces.

Figure 1: ELSA enables acoustic event-wise, fine-grained text–audio alignment for reference-free TTA evaluation, yielding high correlation with human subjective ratings.

ELSA Framework and Methodology

The ELSA framework consists of three core stages:

1. Text Decomposition into Acoustic Events: The text input is parsed via a LLM (GPT-5.2), which extracts concise noun–verb phrase event queries.
2. Event-Level Audio Segmentation: These text snippets are utilized as queries for a language-queried audio source separation (LASS) model (SAM Audio), isolating candidate time-frequency audio segments in the generated waveform that plausibly correspond to the specified events.
3. Hierarchical Alignment Scoring: Both global and event-level text–audio representations are embedded in a shared Human-CLAP space. ELSA computes a coarse global similarity and fine-grained event-level F1-scores over all possible text–audio event pairings. The final metric is an adaptive combination of these two, weighted by the number of events parsed from the text. This hierarchical approach is analogous in spirit to hierarchical metrics in vision–language evaluation, but is novel in the audio–language domain.

   Figure 2: ELSA hierarchically evaluates global text–audio matching and fine-grained acoustic-event alignment by combining shared text–audio embeddings with event-level representations extracted via a text parser and a language-queried audio source separation (LASS) model.

Empirical Evaluation

ELSA is evaluated against both standard reference-based and reference-free TTA metrics, including SI-SDR, AudioBERTScore, PAM, and several CLAPScore variants. Benchmarks include AudioCaps, Clotho, MusicCaps, and RELATE. Across all datasets, human subjective ratings provide the ground truth for semantic relevance (REL), and for three datasets, overall audio quality (OVL) is also available.

Key claims substantiated by the results:

• Consistent, significant improvement in Spearman's $\rho$ and Kendall's $\tau$ correlations with human REL and OVL ratings compared to all baselines. For example, on AudioCaps, ELSA achieves a Kendall’s $\tau$ of 32.7 for REL, exceeding the strongest CLAPScore baseline by 13.1 points.
• Superior performance in compositional and order-sensitive evaluation, as established by RELATE and CompA benchmarks, indicating the efficacy of explicit event-level mapping in capturing compositional semantics.

Robustness and Sensitivity Analyses

An extensive sensitivity analysis demonstrates that ELSA’s advantage is robust to the complexity of user intent: the correlation between ELSA and human REL ratings remains stable across queries with varying numbers of distinct acoustic events, whereas CLAPScore variants degrade with increasing event count.

Figure 3: Sensitivity analysis of metric–REL correlation with respect to the number of acoustic events in the text query.

ELSA’s empirical robustness is further backed by component ablations. Substitution of the LASS model introduces only minor changes, whereas altering the embedding space (e.g., replacing Human-CLAP with MS-CLAP or LAION-CLAP) significantly impacts performance, indicating the primacy of a perceptually-aligned cross-modal embedding space.

Score Distribution Analyses

Analysis of the distributional properties of ELSA reveals that, while its absolute scoring scale is compressive relative to human ratings (i.e., lower mean values), its relative ranking closely tracks subjective judgments. These findings corroborate the metric's reliability for model comparison and optimization, but suggest an opportunity for future calibration to improve interpretability in absolute terms.

Figure 4: Relationship between REL and ELSA on AudioCaps. (a) Histogram of normalized score distributions. (b) Sankey diagram visualizing score correspondence with a bin width of 0.2.

Limitations and Future Directions

A notable limitation is that ELSA’s alignment operates at the event level but does not explicitly model temporal order or event duration. Although ELSA yields stronger performance on order-sensitive datasets than prior metrics, fine-grained modeling of event sequences and durations within the audio is an open problem. Integrating explicit temporal structure (e.g., through alignment or temporal embedding) and scale calibration are prominent avenues for further research.

Implications and Outlook

ELSA’s event-level reference-free approach offers immediate practical utility for system development and human-aligned benchmarking of TTA generation models, eliminating the need for reference audio corpora. Theoretically, it affirms the value of hierarchical, compositional evaluation—paralleling trends observed in the vision–language domain. Future AI models for generative audio will likely integrate and optimize against event-level, human-aligned metrics such as ELSA, facilitating rapid, scalable, and human-correlated progress in the TTA field.

Conclusion

ELSA sets a new standard for fine-grained, reference-free TTA evaluation by leveraging compositional semantic alignment between text and audio. Its empirical superiority across multiple benchmarks, sensitivity to event complexity, and robust alignment with human subjective ratings establish ELSA as a highly effective solution for automatic evaluation of TTA models. Further advancements are anticipated through explicit modeling of event sequences and improved output calibration.

(2606.17404)