MAD Speech: Measures of Acoustic Diversity of Speech

Abstract: Generative spoken LLMs produce speech in a wide range of voices, prosody, and recording conditions, seemingly approaching the diversity of natural speech. However, the extent to which generated speech is acoustically diverse remains unclear due to a lack of appropriate metrics. We address this gap by developing lightweight metrics of acoustic diversity, which we collectively refer to as MAD Speech. We focus on measuring five facets of acoustic diversity: voice, gender, emotion, accent, and background noise. We construct the metrics as a composition of specialized, per-facet embedding models and an aggregation function that measures diversity within the embedding space. Next, we build a series of datasets with a priori known diversity preferences for each facet. Using these datasets, we demonstrate that our proposed metrics achieve a stronger agreement with the ground-truth diversity than baselines. Finally, we showcase the applicability of our proposed metrics across several real-life evaluation scenarios. MAD Speech is made publicly accessible.

• The paper develops MAD Speech, a comprehensive metric suite to quantify acoustic diversity in speech generation models.
• It employs specialized projection models and analyzes diversity across dimensions like voice, gender, emotion, accent, and background noise.
• Results highlight that the SpeechSim model, combined with the Vendi Score, outperforms alternatives in correlating with controlled diversity levels.

Developing MAD Speech: Comprehensive Metrics for Acoustic Diversity in Speech Generation Models

Introduction

The paper introduces MAD Speech, a suite of metrics designed to evaluate acoustic diversity in generative spoken LLMs (GSLMs). These metrics address a critical gap in current evaluation practices which often overlook the acoustic variability of generated speech. MAD Speech specifically measures diversity across five dimensions: voice, gender, emotion, accent, and background noise. By providing a structured approach to quantifying diversity, MAD Speech aims to enhance our understanding of GSLM performance and guide the development of more nuanced speech synthesis technologies.

Metrics and Methodology

Representation Models Used

The paper explores various speech representation models as bases for constructing the diversity metrics. Particularly, models like HuBERT, Wav2Vec-BERT, and SoundStream are evaluated alongside a bespoke model termed SpeechSim. SpeechSim operates on a contrastive self-supervised learning principle, designed to amplify acoustic similarities and differences in speech data.

Building Metrics

The metrics are constructed through a two-step process:

1. Embedding Generation: Speech input is transformed into a dense vector space using pretrained speech models.
2. Diversity Estimation: The distribution of these embeddings is then analyzed using aggregation functions such as average pairwise cosine dissimilarity and the Vendi Score, to quantify the diversity.

Specialized Projection Models

To isolate the contributions of each facet of diversity, projection models tailored to specific acoustic properties are applied atop the general embeddings. This hierarchical approach allows for a more granular analysis of diversity along designated dimensions.

Evaluation and Results

The metrics are assessed using datasets engineered to have varying levels of known diversity. These datasets are constructed by systematically sampling and altering existing speech corpora to manipulate the degree of diversity along each dimension. The efficacy of MAD Speech metrics is then measured through their Spearman rank correlation with these controlled diversity levels.

Key Findings:

• Performance of Representation Models: SpeechSim, particularly when combined with specialized projection models, consistently shows high correlation with ground-truth diversity, outperforming other general-purpose models.
• Aggregation Function Insights: The Vendi Score generally results in higher correlation measurements compared to average pairwise cosine similarity, suggesting it may be a more sensitive indicator of diversity in certain contexts.

Implications and Future Directions

MAD Speech provides a robust framework for assessing and understanding the acoustic diversity in speech synthesis, which is vital for avoiding biases and ensuring generative models produce varied and realistic outputs. The suite’s ability to dissect contributions from different acoustic factors (e.g., voice, accent) separately can greatly aid in model diagnosis and improvement.

Looking ahead, the adaptability of MAD Speech metrics across different languages and acoustic conditions represents a fruitful area for further research. Enhancing the suite to accommodate a wider array of speech characteristics and diversities could potentially set a new standard in GSLM evaluation.

Conclusion

The development of MAD Speech addresses a vital need for comprehensive diversity metrics in the evaluation of GSLMs. By providing detailed insights into how facets of diversity are represented in synthetic speech, MAD Speech aids in advancing the field towards more nuanced and inclusive speech generation technologies. Future expansions to include more languages and diversity dimensions are anticipated to broaden the utility and impact of this innovative metric suite.