PromptTTS: Controllable Text-to-Speech

• PromptTTS is a controllable text-to-speech system that uses natural language prompts to define voice identity, style, prosody, and emotion.
• It employs cross-modal prompt encoders and diffusion-based variation networks to address one-to-many mapping challenges and enhance voice diversity.
• Automated prompt generation leveraging SLU and LLMs enables scalable, high-quality creation of prompt–speech pairs without manual annotation.

PromptTTS is a class of controllable text-to-speech (TTS) systems in which natural-language prompts—rather than speech reference utterances—are used to steer speaker identity, style, prosody, emotion, and other speech characteristics at generation time. PromptTTS is motivated by usability (natural language offers an expressive, accessible interface) and flexibility (enabling style and voice control without requiring reference audio). Major technical advances in PromptTTS systems include cross-modal prompt encoders, variation and diffusion-based networks tackling one-to-many mappings, automated prompt generation via LLMs, and hierarchical token generation for precise, fine-grained control. PromptTTS approaches have been demonstrated at scale across a wide range of datasets and tasks, from zero-shot voice cloning to multimodal emotional synthesis.

1. Foundations and Motivation

PromptTTS systems formalize controllable TTS as a conditional generation problem: given a content input (text) and a text prompt (describing style, speaker, or emotion), the model synthesizes speech such that the output matches both linguistic content and prompt-described characteristics (Guo et al., 2022). Typical prompts can specify fine-grained style (“a woman speaks slowly and softly”), speaker identity (“a deep, old, raspy male voice”), or affective/emotional state.

Two main challenges underlie prompt-driven TTS: (1) one-to-many mapping—text prompts may omit key dimensions of voice variability (e.g., subtle timbral or prosodic nuances), making the mapping from prompt to speech intrinsically ambiguous, and (2) scarcity of large-scale, high-quality prompt-annotated datasets, since manual prompt-writing and annotation do not scale (Leng et al., 2023).

PromptTTS aims to overcome these limitations, democratizing high-quality expressive speech generation and enabling users to steer generation with intuitive, flexible language.

2. Prompt Encoding and Conditioning Architectures

PromptTTS models utilize a cross-modal prompt encoder to map natural-language descriptions into the model’s latent style or speaker space. The core pattern is to encode the text prompt via a pretrained or fine-tuned LLM (usually BERT or RoBERTa), producing a fixed-length prompt embedding (Guo et al., 2022, Leng et al., 2023, Shimizu et al., 2023). This embedding is used to condition the TTS backbone at multiple locations (e.g., via cross-attention, concatenation, conditional normalization, or flow-matching).

For instance, PromptTTS 2 utilizes a BERT-based encoder to produce prompt embeddings, followed by a diffusion-guided variation network that models latent style variation not captured by the prompt description (Leng et al., 2023). PromptTTS++ extends this by separating speaker prompts (identity) from style prompts (utterance-level prosody), encoding each via a BERT+MDN module (Shimizu et al., 2023).

Table: Prompt Encoding Variants in PromptTTS Models

Model	Prompt Encoder	Conditioning Mechanism
PromptTTS	BERT	Concat/Add to Transformer
PromptTTS 2	BERT + cross-attention	Diffusion variation net
InstructTTS	RoBERTa (SimCSE+Xmod)	Discrete VQ, DIffusion
MPE-TTS	CLIP/Emotion2Vec	Multi-modal projection
HD-PPT	RoBERTa + CLAP	Hierarchical token gen

Prompt embeddings may be fused with speaker embeddings (lookup or reference-encoder based) via squeeze-excitation blocks (Bott et al., 2024) or used as conditioning vectors for duration, pitch, and energy predictors (Guo et al., 2022). Key design goals are to ensure (1) that the style/speaker controls are disentangled from content and (2) that prompt conditioning is expressive enough to transfer voice or emotion even when the text and prompt are decoupled.

3. Modeling One-to-Many Mappings and Voice Diversity

Natural-language prompts are necessarily lossy; many different voices/styles can match a given prompt. Naive supervised mapping from prompt-to-speech produces mode collapse or overfitting. PromptTTS 2 addresses this with a dedicated diffusion-based variation network $F_{\text{var}}$, which learns to generate the residual style representation that would be present in reference speech but is absent from the prompt itself (Leng et al., 2023). The network is trained to reconstruct reference speech encodings from prompt-derived representations plus Gaussian noise.

During inference, the model samples from the learned variation network, allowing stochastic exploration of the “one-to-many” voice realization space for the same prompt. This sampling mechanism is essential for supporting natural variation and avoiding deterministic, bland output.

Empirical results confirm that only the variation network—controlled by diffusion sampling—produces the desired voice diversity, while changes to other system components (text random seed, TTS backbone randomness, wording variants) have minimal effect (Leng et al., 2023). Similar methodologies are adopted in InstructTTS with discrete diffusion for expressive style synthesis (Yang et al., 2023), and in HD-PPT via hierarchical token generation (Nie et al., 23 Sep 2025).

4. Automated Prompt Generation and Data Scaling

A critical bottleneck for text-prompt TTS has been the limited availability of high-quality, attribute-rich prompt–speech pairs. PromptTTS 2 introduces a fully automated pipeline that leverages speech language understanding (SLU) models (e.g., wav2vec 2.0) to recognize key voice attributes (gender, speed, volume, pitch) from speech, and a LLM (LLM, e.g., GPT-3.5 Turbo) to generate diverse, human-like prompt sentences from these attributes (Leng et al., 2023):

1. SLU extracts attribute classes from audio.
2. LLM constructs attribute-specific keyword sets and templates.
3. LLM generates prompt sentences (with diverse placeholders).
4. Final system instantiates prompts for each speech sample via keyword filling.

This pipeline enables scalable, zero-human-cost prompt dataset creation. Evaluations show that LLM-generated prompts achieve attribute-classification accuracy 96.38%, exceeding the 92.74% of human-written prompts (Leng et al., 2023).

Table: Prompt Generation Pipeline Attribute Accuracy (test set)

Prompt Source	Attribute Acc. (%)
LLM (PromptTTS 2)	96.38
Human (PromptSpeech)	92.74
TextrolSpeech	94.71

The pipeline setup is flexible and can incorporate new attributes (e.g., age, accent) by merely providing SLU tags and template instructions, obviating costly human labeling.

5. Evaluation, Empirical Performance, and Applications

PromptTTS systems are evaluated along three axes: prompt-consistency (how well does synthesized speech match intended attributes), naturalness (MOS), and sampling diversity. On a 44K hour English MLS corpus:

• Prompt-consistency across gender, speed, volume, and pitch: PromptTTS 2 achieves a mean of 93.33% (vs. 91.54% for PromptTTS 1, 91.47% for InstructTTS).
• MOS (5-point) for generated speech: PromptTTS 2 achieves 3.88 ± 0.08, representing state-of-the-art for prompt-based models (ground-truth: 4.38) (Leng et al., 2023).
• Sampling diversity: only variation-network sampling produces substantial voice variation (pairwise similarity 0.355), while baseline TTS models exhibit >0.76 similarity regardless of input permutation (Leng et al., 2023).

PromptTTS methods have broad practical applications, including zero-shot speaker/voice cloning, expressive and emotional TTS, accessibility for non-expert users, and scalable creation of synthetic corpora for other speech tasks.

6. Extensions, Limitations, and Future Directions

PromptTTS approaches have been successfully extended to:

• Control over speaker identity and fine-grained style through dual prompt channels (PromptTTS++) (Shimizu et al., 2023).
• Multimodal and emotional prompting using text, image, or speech descriptors (MPE-TTS, UMETTS) (Wu et al., 24 May 2025, Cheng et al., 2024).
• Robust zero-shot synthesis and fine-tuning in noisy, in-the-wild settings with enhanced prompt denoising and flow-matching (StyleTTS2, F5-TTS, Sidon pipeline) (Giraldo et al., 5 Feb 2026).
• Hierarchical decoding of content- and prompt-preference tokens for precision-guided, instruction-based TTS (HD-PPT) (Nie et al., 23 Sep 2025).

Open limitations include the relatively coarse granularity of current SLU-derived attributes in prompt generation, entanglement between style and identity in prompt representations, challenges in scaling annotation beyond limited word vocabularies, and the need for more generalizable, richer prompt representations (e.g., covering accent, detailed timbre, multilingual support) (Shimizu et al., 2023).

Future directions may involve cross-lingual prompt TTS, multimodal prompt fusion (e.g., simultaneous text+image), richer and dynamically weighted attribute control, and joint representation learning for prompt and speech. The applicability of PromptTTS to domains beyond speech, such as audio–visual face-to-voice synthesis, is an active research topic (Leng et al., 2023).

7. Representative Models and Landscape

Key PromptTTS and related models include:

• PromptTTS (first BERT-based style prompt TTS with PromptSpeech dataset) (Guo et al., 2022)
• PromptTTS 2 (variation network diffusion, LLM-based prompt pipeline) (Leng et al., 2023)
• PromptTTS++ (separates speaker vs. style prompt control, diffusion+MDN) (Shimizu et al., 2023)
• InstructTTS (cross-modal metric learning, MI minimization, discrete diffusion acoustic) (Yang et al., 2023)
• PromptStyle (cross-modal alignment, VITS backbone) (Liu et al., 2023)
• HD-PPT (hierarchical, content/prompt-preference tokens, instruction-based TTS) (Nie et al., 23 Sep 2025)
• MPE-TTS, UMETTS/MM-TTS (multimodal prompt TTS) (Wu et al., 24 May 2025, Cheng et al., 2024)
• Mega-TTS 2 (multi-sentence prompt, large-scale zero-shot with multi-reference encoding) (Jiang et al., 2023)

These systems define the leading edge of prompt-controlled speech synthesis, enabling flexible, accessible, and powerful TTS controllability through natural language.