Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2306.15687v2)

Abstract: Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are not filtered or enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster. Audio samples can be found in \url{https://voicebox.metademolab.com}.

Overview of "Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale"

The paper focuses on the development and evaluation of Voicebox, a text-guided multilingual speech generation model capable of accomplishing multiple tasks at scale. Voicebox introduces a novel approach to speech synthesis and editing, drawing parallels to the generative capabilities seen in large-scale text and image models like GPT and DALL-E. It seeks to bridge the gap between existing speech generation models and their counterparts in text and vision by implementing an approach that allows for extensive linguistic and contextual adaptability.

Technical Approach

Voicebox is built on a non-autoregressive flow-matching model, trained specifically to perform speech infilling tasks. This training involves predicting masked speech segments using the surrounding audio context and text transcripts. The training dataset spans over 50,000 hours of speech data across multiple languages, a notable augmentation from the typically limited datasets used in the domain, thus enhancing Voicebox's ability to generalize across tasks. By utilizing a conditional flow-matching approach with optimal transport paths, Voicebox is able to efficiently model the distribution of masked speech, allowing for the generation of coherent and intelligible audio.

Task Versatility

The model's versatility is highlighted through its ability to engage in zero-shot text-to-speech (TTS) synthesis, cross-lingual synthesis, noise removal, content editing, and style conversion. These tasks are facilitated via in-context learning, similarly to LLMs, but with additional flexibility due to conditioning on future context. Empirical results highlight that Voicebox outperforms state-of-the-art models such as VALL-E in zero-shot TTS, achieving lower word error rates and improved audio similarity scores while being 20 times faster in inference. Furthermore, Voicebox extends its capabilities to cross-lingual TTS across six languages without relying on style labels or multilingual embeddings, unlike previous models that faced substantial performance loss in cross-lingual scenarios.

Evaluation Metrics

The paper employs a variety of metrics to assess Voicebox's performance, focusing on correctness and intelligibility via word error rates and audio similarity scores using established speaker embeddings. The evaluation also encompasses diversity and quality assessments akin to those used in image generation via Fréchet Inception Distance (FID) adapted for speech as Fréchet Speech Distance (FSD). This metric assesses how closely the distribution of generated samples approximates that of real speech, reflecting both diversity and quality.

Implications and Future Developments

Voicebox's capacity to understand and generate speech conditioned on both prior and subsequent context enhances its practical applications, ranging from real-time synthesis in varied linguistic environments to adaptive voice interfaces in technology. The results also suggest potential for further improvements through scaling of diverse multilinguistic datasets, which could overcome current limitations such as reduced performance in conversational or less-scripted speech scenarios.

The paper's advancements posit a potential shift in the approach to speech synthesis and editing, especially highlighting the capabilities inherent in leveraging large, diverse datasets and advanced flow-matching models. The authors further suggest that future work may focus on disentangling the controls for various stylistic attributes in audio, which would allow even finer manipulation and generation of speech, well beyond the current capabilities.

In summary, "Voicebox" represents a significant stride towards more generalized and effective speech generation models that could parallel the advancements seen in text and image processing, opening new avenues for speech technology applications and research.