HD-PPT: Hierarchical TTS Framework

• HD-PPT is a hierarchical framework for instruction-based TTS that decouples semantic content from style by reformulating speech synthesis as a sequential decoding problem.
• It employs a supervised speech-token codec with dual quantization for content-preference and prompt-preference tokens to enforce precise disentanglement and improve acoustic fidelity.
• The hierarchical decoder, augmented by a large language model and auxiliary objectives, sequentially generates semantic, style, and acoustic tokens, leading to superior MOS and WER metrics.

HD-PPT is a framework for instruction-based text-to-speech that reformulates speech synthesis as a structured hierarchical generation problem. It is designed to address a limitation identified in prior Instruct-TTS systems: single-level natural-language instructions do not align cleanly with multilevel speech-token representations, which constrains fine-grained control at inference time. The framework combines a supervised speech-token codec, a LLM augmented with a lightweight hierarchical decoder, and a standard neural vocoder. Its central design goal is to disentangle “what to say” from “how to say it,” and then to reconstruct a complete acoustic representation through ordered decoding of semantic, style, and acoustic tokens (Nie et al., 23 Sep 2025).

1. Conceptual framing and problem formulation

HD-PPT, short for Hierarchical Decoding of Content- and Prompt-Preference Tokens for Instruction-based TTS, treats instruction-based speech synthesis as a three-stage hierarchical generation problem. The framework consists of three core components: a speech-token codec that decomposes pretrained acoustic tokens into content-preference and prompt-preference streams, a Qwen2.5-0.5B-based LLM with a lightweight hierarchical decoder that sequentially emits semantic, style, and full acoustic tokens, and a pre-trained neural vocoder that renders the final waveform (Nie et al., 23 Sep 2025).

The motivating claim is that existing instruction-based TTS models still lack fine-grained control because they attempt to map single-level text instructions directly onto multilevel speech tokens. HD-PPT addresses this by explicitly decomposing speech into semantically oriented and style-oriented latent streams and by imposing a decoding order that mirrors this decomposition. In the paper’s formulation, the distinction between content and prompt preference is not merely descriptive; it is operationalized through separate quantization paths and separate supervisory signals.

This suggests a broader interpretation of HD-PPT as a factorized latent-variable approach to controllable TTS. A plausible implication is that the framework’s controllability derives less from a larger generative backbone than from the imposition of intermediate structure between text instruction and acoustic realization.

2. Speech-token codec and latent decomposition

The codec begins with a sequence of discrete speech tokens $T_{s,1\ldots J}$ produced by a pretrained CosyVoice2 tokenizer at 25 Hz. A 5-layer Conformer encoder maps these tokens to continuous frame-level vectors $Z_{1\ldots J}\in\mathbb{R}^d$. Two independent Finite Scalar Quantization (FSQ) modules then discretize each frame-level vector $Z_j$ into a content-preference index $c_j$ and a prompt-preference index $p_j$ using separate learned codebooks of size 1 296 and 64, respectively (Nie et al., 23 Sep 2025).

The quantization is defined as

$$c_j = \arg\min_{k}\lVert Z_j - e^{(c)}_k\rVert,\qquad p_j = \arg\min_{\ell}\lVert Z_j - e^{(p)}_\ell\rVert,$$

where $\{e^{(c)}_k\}$ and $\{e^{(p)}_\ell\}$ are the learned codebooks. A 4-layer causal Transformer combiner reconstructs the original speech tokens $T_s$ from the paired discrete streams $(c_j,p_j)$ in an auto-regressive manner. The reconstruction objective is

$Z_{1\ldots J}\in\mathbb{R}^d$0

The codec’s role is therefore not only compression or token transformation; it is a supervised disentanglement mechanism. By replacing a single speech-token sequence with two lower-level discrete streams whose intended semantics differ, HD-PPT establishes the internal variables later consumed by the hierarchical decoder. The distinction between “content-preference” and “prompt-preference” is a specific design choice of the framework, and the paper’s results treat it as foundational to instruction adherence and style control.

3. Supervised disentanglement of semantics and style

To enforce separation between semantic content and style information, HD-PPT applies two auxiliary objectives to the codec. The ASR loss is imposed on content-preference tokens. A Whisper-Small decoder consumes $Z_{1\ldots J}\in\mathbb{R}^d$1 and predicts text: $Z_{1\ldots J}\in\mathbb{R}^d$2

The CLAP contrastive loss is imposed on prompt-preference tokens. After pooling $Z_{1\ldots J}\in\mathbb{R}^d$3 into a fixed embedding $Z_{1\ldots J}\in\mathbb{R}^d$4 by cross-attention, the model maximizes cosine similarity to the RoBERTa embedding $Z_{1\ldots J}\in\mathbb{R}^d$5 of the style prompt and minimizes similarity to $Z_{1\ldots J}\in\mathbb{R}^d$6 negatives: $Z_{1\ldots J}\in\mathbb{R}^d$7

The total codec loss is

$Z_{1\ldots J}\in\mathbb{R}^d$8

with $Z_{1\ldots J}\in\mathbb{R}^d$9 and $Z_j$0 (Nie et al., 23 Sep 2025).

These objectives define the intended semantics of the two token streams. The ASR pathway forces content-preference tokens toward linguistic recoverability, while the CLAP pathway ties prompt-preference tokens to natural-language style descriptions. A common misconception would be to regard the two streams as arbitrary learned partitions of acoustic latent space. In the paper’s construction, they are instead supervised partitions, each anchored by an external objective aligned with either textual content or prompt-level style semantics.

4. Hierarchical decoding with the LLM

HD-PPT conditions on a natural-language text instruction $Z_j$1. Its Qwen2.5-0.5B backbone produces hidden states $Z_j$2 auto-regressively, while an auxiliary 2-layer hierarchical decoder emits $Z_j$3, $Z_j$4, and finally $Z_j$5. At each step $Z_j$6, the decoding process follows three ordered stages (Nie et al., 23 Sep 2025):

1. Semantic foundation

$Z_j$7

2. Style rendering

$Z_j$8

3. Acoustic completion

$Z_j$9

The joint probability factorizes as

$c_j$0

Training includes regularization by randomly masking $c_j$1 or $c_j$2, along with an auxiliary linear projection from $c_j$3 directly to $c_j$4 so that the LLM remains acoustically grounded.

The ordering is the defining feature of the framework. Rather than predicting complete acoustic tokens in a single step or predicting multiple latent streams in parallel, HD-PPT imposes a dependency chain in which semantic structure precedes style realization and both precede acoustic completion. This suggests that the model’s control properties arise partly from causal ordering: style is generated conditionally on semantic tokens, and full acoustics are generated conditionally on both.

5. Implementation and training configuration

The speech-token codec uses a 5-layer Conformer extractor and a 4-layer causal Transformer combiner. The FSQ codebooks have sizes 1 296 for content and 64 for prompt, and the frame rate is 25 Hz. Training uses AdamW with learning rate $c_j$5 for 50 epochs on 4× NVIDIA 4090 GPUs (Nie et al., 23 Sep 2025).

The hierarchical LLM uses Qwen2.5-0.5B as backbone and a 2-layer auto-regressive Transformer as hierarchical decoder with receptive length = 3. It is optimized with AdamW at learning rate $c_j$6 and trained for 16 epochs on the same hardware. The vocoder is pre-trained CosyVoice2 Flow-matching + HiFi-GAN.

Component	Configuration	Training
Speech-token codec	5-layer Conformer + 4-layer causal Transformer; FSQ codebooks 1 296 / 64; 25 Hz	AdamW, $c_j$7, 50 epochs, 4× NVIDIA 4090 GPUs
Hierarchical LLM	Qwen2.5-0.5B + 2-layer auto-regressive Transformer; receptive length = 3	AdamW, $c_j$8, 16 epochs, same hardware
Vocoder	pre-trained CosyVoice2 Flow-matching + HiFi-GAN	Pre-trained

The implementation profile indicates that HD-PPT adds structural complexity primarily through token decomposition and staged decoding rather than through a larger backbone model. A plausible implication is that the framework is intended to trade some inference simplicity for improved controllability and instruction fidelity.

6. Empirical performance, ablations, and limitations

On the combined TextrolSpeech and EmoVoice-DB test sets, HD-PPT is reported to outperform both explicit style-encoding systems and previous LLM-driven Instruct-TTS systems. The reported metrics are MOS-N, MOS-S, DNSMOS, EMO-SIM, and WER (Nie et al., 23 Sep 2025).

Model	MOS-N	MOS-S
CosyVoice2	3.920±0.112	3.885±0.116
EmoVoice-PP	3.694±0.123	3.594±0.128
HD-PPT	4.108±0.105	4.167±0.103

Model	DNSMOS	EMO-SIM	WER
CosyVoice2	3.83	0.714	5.71 %
EmoVoice-PP	3.87	0.613	8.56 %
HD-PPT	3.84	0.753	5.18 %

Relative to CosyVoice2, HD-PPT reports +0.19 MOS-N and +0.28 MOS-S, the highest EMO-SIM among all, and the second-lowest WER. Ablation studies attribute these gains to both latent disentanglement and hierarchical decoding. Removing content-preference tokens raises WER to 8.04 %. Removing prompt-preference tokens drops EMO-SIM to 0.728. Bypassing both yields WER = 10.10 % and EMO-SIM = 0.716. Alternative parallel and single-step decoders also underperform the hierarchical strategy, with EMO-SIM 0.736 / 0.713 vs. 0.753.

The reported limitations are also explicit. HD-PPT adds approximately 34 % inference overhead relative to a single-step LLM, although it is stated to remain real-time for 24 kHz audio generation. Future directions proposed for the framework include extending the hierarchical paradigm to singing voice or cross-lingual TTS, introducing finer granularity such as separate prosody versus speaker attributes, adapting the codec to continuous prosody contours or explicit emotional dimensions, and jointly training the vocoder in the loop for end-to-end instruction fidelity.

Taken together, the results support the paper’s central claim that HD-PPT’s effectiveness arises from a two-stage disentanglement: first at the codec level, where semantics and style are split into supervised token streams, and then at the decoder level, where these streams are consumed in a layered order. This suggests that the framework’s contribution lies in aligning representation structure, supervision, and generation order within instruction-based TTS rather than in any single architectural component alone.