Instruction-guided Text-to-Speech

• ITTS is a speech synthesis approach where text and natural language instructions jointly control content and vocal attributes such as emotion, prosody, and style.
• It employs techniques like codec language models, hierarchical tokenization, and diffusion frameworks to improve style adherence and intelligibility.
• Benchmarks like MINT-Bench and InstructTTSEval reveal challenges in dynamic control, bias mitigation, and achieving fine-grained acoustic fidelity.

Searching arXiv for recent and foundational ITTS papers to ground the article. Instruction-guided Text-to-Speech (ITTS) denotes TTS systems whose output is controlled not only by the linguistic content—“what is said”—but also by natural-language instructions describing “how it should be said,” including emotion, prosody, speaking style, scenario, persona, timbre, and related paralinguistic factors (Nie et al., 23 Sep 2025). In recent work, ITTS has expanded from prompt-based expressive TTS into a broader instruction-following paradigm in which a single prompt may jointly specify content and style, or even high-level, open-vocabulary performance directions, dialogue roles, and cross-modal acoustic intent (Zhou et al., 2024, Ren et al., 4 Jan 2026). The field now spans codec LLMs, diffusion and flow-matching systems, reasoning-driven large audio LLMs, multilingual benchmarks, and analyses of instruction-perception mismatch and social bias.

1. Definition and task scope

A common formalization treats ITTS as a mapping

$f:(x,i)\mapsto y,$

where $x$ is the text transcript, $i$ is a natural-language instruction describing style, emotion, age, emphasis, or other vocal attributes, and $y$ is generated speech (Lin et al., 17 Sep 2025). This differs from conventional TTS, which generally conditions on content alone or on fixed style labels, prosody tokens, or reference audio. In the ITTS setting, the instruction is intended to act as the main control interface.

Several papers broaden that formulation. VoxInstruct defines “human instruction-to-speech” as generating expressive, multilingual speech directly from a unified natural-language instruction that freely mixes a content part $x_{con}$ and a description part $x_{des}$, rather than requiring those to be provided separately (Zhou et al., 2024). InstructAudio likewise defines instruction-guided TTS as speech generation from text plus a free-form natural-language description of how the speech should sound, without any reference audio, and embeds that formulation inside a unified speech-and-music model (Qiang et al., 23 Nov 2025). Adjacent work on text-instruction-guided voice conversion frames a closely related problem: source speech plus a natural-language instruction are transformed into speech that preserves content while modifying prosody, emotion, and style, suggesting a continuum between ITTS and instruction-guided speech editing or conversion (Kuan et al., 2023).

The scope of control studied in current ITTS literature is broad. MINT-Bench organizes instruction-following TTS around ten atomic acoustic attributes—age, gender, pitch, texture, speed, volume, emotion, accent, personality, and tone—and extends evaluation to composites, dynamic attribute changes, layered affect, conflictive cues, scenario prompts, character prompts, disfluency, dysphonia, and non-verbal events (Chen et al., 20 Apr 2026). InstructTTSEval evaluates three tasks—Acoustic-Parameter Specification, Descriptive-Style Directive, and Role-Play—using a 12-feature taxonomy spanning physiological, linguistic, social, and psychological dimensions (Huang et al., 19 Jun 2025). E-VOC further isolates fine-grained control of adverbs of degree, graded emotion intensity, speaker age, and word-level emphasis, explicitly measuring the “instruction-perception gap” between what an instruction asks for and what listeners actually hear (Lin et al., 17 Sep 2025).

2. Architectural patterns and intermediate representations

One major line of work uses discrete audio representations and codec LLMs. VoxInstruct employs a unified multilingual codec language modeling framework with an MT5-base text encoder, HuBERT-derived speech semantic tokens as an intermediate representation, EnCodec acoustic tokens, a LLaMA-based AR+NAR codec LLM, and multiple Classifier-Free Guidance strategies (Zhou et al., 2024). Its central claim is that semantic tokens allow the model to automatically extract the content of synthesized speech from raw human instructions, while separate guidance on instructions and semantic tokens improves style adherence and intelligibility. In the English instruction-to-speech setting, the full model reports MOS-Q $4.22$, MOS-I $3.76$, WER $2.5$, and mean attribute accuracy $80.54\%$ (Zhou et al., 2024).

A second line makes the hierarchy of speech tokens explicit. HD-PPT argues that existing LLM-based ITTS suffers from a modality gap between single-level text instructions and the multi-level structure of speech, and addresses this with a codec that extracts content-preference tokens $x$0 and prompt-preference tokens $x$1, supervised by ASR and CLAP objectives, together with a hierarchical decoder that predicts semantic, then style, then full speech tokens (Nie et al., 23 Sep 2025). The codec objective is

$x$2

with $x$3 and $x$4, and the hierarchical model factorizes generation through $x$5, $x$6, and $x$7 (Nie et al., 23 Sep 2025). On TextrolSpeech plus EmoVoice-DB, HD-PPT reports MOS-N $x$8, MOS-S $x$9, DNSMOS $i$0, EMO-SIM $i$1, and WER $i$2, with ablations showing that hierarchical decoding and both preference token streams are important (Nie et al., 23 Sep 2025).

A third line introduces explicit reasoning as an intermediate layer. OV-InstructTTS proposes open-vocabulary InstructTTS and trains a reasoning-driven framework on OV-Speech, where each sample includes an instruction, a reasoning chain, an enhanced transcript with emotion and paralinguistic tags, and audio (Ren et al., 4 Jan 2026). The model follows a “think, then speak” design on top of Step-Audio-2-mini: it first generates a > block, then interleaved text and audio tokens. In the paper’s notation,

$i$3

with supervised fine-tuning over reasoning tokens, enhanced transcript tokens, and audio tokens (Ren et al., 4 Jan 2026). This architecture is intended to bridge high-level, open-vocabulary instructions and low-level acoustic realization.

A fourth line remains diffusion- or flow-based. Guided-TTS is a conceptually earlier guidance-based TTS formulation in mel-spectrogram space: an unconditional diffusion model is guided by a separately trained phoneme classifier, and the conditional score is written as

$i$4

(Kim et al., 2021). Although its control signal is phonemic rather than natural-language style, it establishes a general pattern in which generation and control are separated and recombined through guidance. InstructAudio extends flow matching in a different direction: it uses a latent audio codec, a multimodal diffusion transformer with joint and single diffusion layers, and a standardized instruction-phoneme input format, trained jointly on speech and music (Qiang et al., 23 Nov 2025).

3. Data construction, supervision, and training regimes

Current ITTS systems rely heavily on synthetic or semi-synthetic instruction data. OV-Speech is built on ContextSpeech and contains 316,807 training utterances from 83 novels, each paired with three open-vocabulary instructions, plus a 1,500-utterance test set from three held-out novels (Ren et al., 4 Jan 2026). Instructions are grounded in a $i$5-word context window, structured with Qwen3-32B into environmental descriptions, current event, speaker personality, interlocutor state, and speaker intent, and then rewritten as “a director’s note to a voice actor” (Ren et al., 4 Jan 2026). Samples are filtered by an LLM-as-a-judge pipeline, and transcripts are enriched with 18 paralinguistic tags using a paralinguistic-aware annotation pipeline whose selected PC-PTI strategy achieves S-CER $i$6 on OV-Speech (Ren et al., 4 Jan 2026).

DeepDubbing shows a different large-scale strategy for audiobook production. It uses an internal multi-participant audiobook corpus of more than 4,000 hours; an LLM extracts characters and generates timbre descriptions using a fixed “Gender $i$7 Age $i$8 Personality $i$9 Identity” template; a Text-to-Timbre model then generates speaker embeddings; and a Context-Aware Instruct-TTS model synthesizes speech from text, contextual emotion-scene instructions, and those timbre embeddings (Dai et al., 19 Sep 2025). The system reports more than 300K timbre descriptions and more than 2M emotion-scene instructions covering more than 44 fine-grained emotion categories (Dai et al., 19 Sep 2025). This suggests a practical route for domain-specific ITTS in narrative media.

ISSE is a large instruction-guided speech style editing dataset and benchmark with nearly 400 hours of speech and over 100,000 source-target pairs, each aligned with detailed editing instructions (Chen et al., 29 Sep 2025). Its pipeline combines expressive TTS, voice conversion, ASR filtering with WER $y$0, style similarity filtering via emotion2vec, and speaker similarity filtering via ECAPA-TDNN (Chen et al., 29 Sep 2025). The final instructions are generated by Qwen3-8B from source and target style descriptions, producing examples such as “Convert the source speech to a whispering style with a high-pitched voice” and “Convert the source speech to a medium-pitched, raspy monotone that conveys awe” (Chen et al., 29 Sep 2025). Although aimed at editing, the resource directly operationalizes instruction-conditioned control of pitch, phonation, and emotion.

Synthetic speech instruction data for downstream speech-LLMs has also become a research topic in its own right. A query rewriting framework with multi-LLM knowledge fusion rewrites instruction text into TTS-friendly forms, synthesizes candidate speech, validates it with multiple ASR and embedding models, and reports an increase in data usability from $y$1 to $y$2 (Hei et al., 11 Jul 2025). For low-resource languages, Speechless bypasses waveform synthesis entirely: it trains a residual vector quantizer aligned to Whisper, a 1B decoder-only model that maps text to semantic tokens, and then instruction-tunes an LLM on those tokens, thereby avoiding the requirement for a high-quality TTS model in the target language (Dao et al., 23 May 2025). Taken together, these works suggest that the data problem in ITTS is now addressed not only through curated prompt-speech corpora but also through instruction rewriting, semantic tokenization, and LLM-mediated annotation.

4. Evaluation, benchmarking, and empirical performance

The evaluation literature now treats instruction following as a distinct axis, separate from intelligibility and naturalness. InstructTTSEval provides a benchmark of 6,000 test cases—three tasks, two languages, 1,000 items per task-language pair—and uses Gemini as an automatic judge of whether synthesized speech aligns with the instruction (Huang et al., 19 Jun 2025). Human–Gemini agreement reaches $y$3 on APS, $y$4 on DSD, $y$5 on RP, and $y$6 overall (Huang et al., 19 Jun 2025). In the English subset, Gemini-Flash TTS reports APS $y$7, DSD $y$8, RP $y$9, average $x_{con}$0; by contrast, leading open-source systems are far lower, with VoxInstruct at average $x_{con}$1 and Parler-TTS variants around the mid-40s (Huang et al., 19 Jun 2025). The benchmark also documents difficult failure modes, including sighs, screams, laughter, extreme emotions, and rapid transitions (Huang et al., 19 Jun 2025).

MINT-Bench generalizes this evaluative program to a comprehensive multilingual benchmark over ten languages, with a hierarchical taxonomy spanning easy, hard, and special control regimes, and a hybrid protocol that measures content consistency, instruction following, perceptual quality, and timbre diversity (Chen et al., 20 Apr 2026). Its instruction-following score is scaled by a content-consistency coefficient $x_{con}$2, and its perceptual score adds bonuses for naturalness and expressiveness only when instruction following is already correct (Chen et al., 20 Apr 2026). The benchmark reports that frontier commercial systems lead overall, while Qwen3TTS-12Hz-1.7B-VD is the strongest open-source model and can outperform commercial systems in Chinese (Chen et al., 20 Apr 2026). MINT-Bench further states that harder compositional and paralinguistic controls remain major bottlenecks, including dynamic control, layered affect, conflictive cues, disfluency, dysphonia, and non-verbal events (Chen et al., 20 Apr 2026).

E-VOC isolates the instruction-perception problem through large-scale human ratings on adverbs of degree, graded emotion intensity, speaker age, and word-level emphasis (Lin et al., 17 Sep 2025). Across five evaluated ITTS systems, the paper concludes that gpt-4o-mini-tts is the most reliable model for alignment between instructions and generated utterances across acoustic dimensions, but also shows that all five systems tend to generate Adult voices even when prompts request Child or Elderly voices (Lin et al., 17 Sep 2025). Age accuracy remains low across systems, and word-level emphasis remains difficult: gpt-4o-mini-tts achieves the best emphasis accuracy at $x_{con}$3, while age control is much weaker and heavily biased toward adult-sounding outputs (Lin et al., 17 Sep 2025). This benchmark is important because it distinguishes coarse controllability from truly fine-grained perceptual control.

Representative system-level results reinforce the same pattern. HD-PPT improves both naturalness and instruction adherence relative to CosyVoice2, including higher MOS-N, MOS-S, and EMO-SIM with a lower WER (Nie et al., 23 Sep 2025). OV-InstructTTS-TEP reaches Gemini Score $x_{con}$4, SIM $x_{con}$5, CER $x_{con}$6, MOS $x_{con}$7, and ICMOS $x_{con}$8, outperforming GPT4o, Higgs Audio V2, and Step-Audio-2-mini on instruction-following metrics in the reported setup (Ren et al., 4 Jan 2026). InstructAudio, evaluated against MaskGCT, E2-TTS, F5-TTS, ZipVoice, CosyVoice1, and CosyVoice2, reports WER $x_{con}$9 for English and $x_{des}$0 for Chinese on Seed-TTS, while also providing text control over gender, age, emotion, style, accent, and dialogue, which several baselines do not support (Qiang et al., 23 Nov 2025).

5. Applications and neighboring problem settings

ITTS is increasingly treated as a general acoustic control interface rather than a narrow style-prompting add-on. DeepDubbing illustrates this in audiobook production: an LLM extracts characters, generates role descriptions, a Text-to-Timbre model synthesizes speaker embeddings from those descriptions, and a Context-Aware Instruct-TTS system uses contextual emotion-scene instructions to produce multi-participant expressive dubbing (Dai et al., 19 Sep 2025). The system improves emotion MOS from $x_{des}$1 to $x_{des}$2 relative to a text-only CA-TTS baseline while maintaining similar intelligibility, showing that instruction conditioning can be practically integrated into long-form narration workflows (Dai et al., 19 Sep 2025).

Voice conversion and speech style editing are closely related neighboring tasks. The 2023 text-instruction-guided voice conversion work formulates a system in which source speech plus instructions such as “articulate slowly with a deep tone” or “speak in a cheerful boyish voice” are processed by a neural codec LLM, replacing the usual reliance on reference utterances (Kuan et al., 2023). ISSE similarly focuses on modifying stylistic properties while preserving linguistic content and speaker identity, but does so with over 100,000 source-target pairs and detailed editing instructions, making it a useful proxy task for ITTS-style control over prosody and timbre (Chen et al., 29 Sep 2025). A plausible implication is that editing and synthesis are converging toward a shared abstraction: natural-language instructions operating over discrete acoustic representations.

The same control logic now extends beyond speech alone. InstructAudio is explicitly presented as the first instruction-controlled framework unifying speech and music generation, using a standardized instruction-phoneme format and a shared flow-matching MM-DiT trained on 50K hours of speech and 20K hours of music (Qiang et al., 23 Nov 2025). It supports expressive speech, music, and dialogue generation in English and Chinese, and reports $x_{des}$3 classification accuracy for gender and accent, $x_{des}$4 for age and style, $x_{des}$5 for emotion, and $x_{des}$6 for dialogue control in the reported evaluation (Qiang et al., 23 Nov 2025). This suggests that ITTS may increasingly be viewed as one instance of a broader instruction-guided acoustic generation problem.

Downstream speech-language applications also depend on ITTS-adjacent capabilities. Synthetic spoken instruction corpora constructed through query rewriting improve training data for end-to-end large speech LLMs (Hei et al., 11 Jul 2025), while Speechless shows that semantic speech tokens can replace waveforms during instruction tuning for low-resource languages (Dao et al., 23 May 2025). These lines of work do not generate expressive speech directly, but they reinforce the same principle: instruction following in speech systems depends critically on how natural-language control is represented, normalized, and aligned to acoustic or semantic speech spaces.

6. Limitations, bias, and open research problems

A recurring technical issue is the mismatch between the single-level symbolic form of text instructions and the inherently multi-level structure of speech. HD-PPT describes this as a modality gap between single-level text instructions and multilevel speech tokens, and its ablations show that monolithic token prediction is inferior to structured hierarchical decoding (Nie et al., 23 Sep 2025). OV-InstructTTS argues similarly that prior InstructTTS methods rely too heavily on direct combinations of audio-related labels or their rephrasings, making them rigid and weak at handling flexible, high-level instructions (Ren et al., 4 Jan 2026). This suggests that the future of ITTS depends on better factorization of content, style, speaker, and paralinguistic events.

Another limitation is ambiguity and generalization. Open-vocabulary instructions are desirable precisely because they are not fixed tags, but they are also vague, compositional, and often context-dependent. OV-InstructTTS introduces explicit reasoning chains to infer emotional, acoustic, and paralinguistic attributes from narrative context (Ren et al., 4 Jan 2026); InstructTTSEval and MINT-Bench both show that role-play, dynamic control, layered affect, and conflictive cues remain substantially harder than simpler attribute prompts (Huang et al., 19 Jun 2025, Chen et al., 20 Apr 2026). The E-VOC results sharpen this diagnosis by showing that many systems flatten fine-grained distinctions such as “slightly” versus “very” and default to adult voices even under child or elderly prompts (Lin et al., 17 Sep 2025).

Bias is now a central concern rather than a peripheral one. A dedicated study of occupation prompts such as “Act like a nurse” and “Act like a fisherman” finds systematic gender bias across multiple Parler-TTS variants, with strong female bias for occupations like nurse, nanny, receptionist, secretary, midwife, and strong male bias for occupations like fisherman, electrician, plumber, carpenter, and soldier (Kuan et al., 8 Feb 2025). Prompt-based mitigation attempts such as appending “paying attention to diversity” or “recognizing that this occupation can be male or female” are reported as inconsistent and sometimes counterproductive (Kuan et al., 8 Feb 2025). The later “Binding Effect” study broadens the analysis to combinations of social status, career stereotypes, and persona descriptors, formalizing non-additive interaction terms in log-odds space and concluding that generic diversity prompting is insufficient to override entrenched patterns, which track both text-encoder priors and skewed training distributions (Chen et al., 21 Mar 2026).

Evaluation itself remains unsettled. MINT-Bench provides a diagnostically rich multilingual framework, InstructTTSEval provides task-specific automated judging, and E-VOC provides fine-grained human perception data, but each exposes a different failure mode (Chen et al., 20 Apr 2026, Huang et al., 19 Jun 2025, Lin et al., 17 Sep 2025). This suggests that no single metric presently captures ITTS performance. A plausible implication is that future work will need layered evaluation: ASR-based content checks, attribute- or judge-based instruction-following scores, perceptual quality measures, and bias audits under both univariate and compositional prompts.

The research trajectory described across these papers points toward several concrete directions. One is richer representation learning: more disentangled or hierarchical tokenizations, explicit reasoning stages, and multimodal conditioning that combines instructions with optional reference audio or dialogue context (Nie et al., 23 Sep 2025, Ren et al., 4 Jan 2026). Another is broader and more carefully curated data: open-vocabulary instruction corpora, paralinguistic annotations, multilingual coverage, and better balancing across age, gender, and occupational or social descriptors (Qiang et al., 23 Nov 2025, Dai et al., 19 Sep 2025, Kuan et al., 8 Feb 2025). A third is encoder-aware debiasing, since the Binding Effect paper argues that acoustic gender biases are strongly associated with the semantic priors of pre-trained text encoders such as BERT, Flan-T5, and mT5 (Chen et al., 21 Mar 2026). Across all of these, the central open problem remains the same: to translate natural-language instructions into controllable, perceptually faithful, and socially robust acoustic behavior.