--- title: 'HD-PPT: Hierarchical TTS Framework' url: https://www.emergentmind.com/topics/hd-ppt type: topic --- # HD-PPT: Hierarchical TTS Framework HD-PPT is a framework for instruction-based text-to-speech that reformulates speech synthesis as a structured hierarchical generation problem. It is designed to address a limitation identified in prior Instruct-TTS systems: single-level natural-language instructions do not align cleanly with multilevel speech-token representations, which constrains fine-grained control at inference time. The framework combines a supervised speech-token codec, a Large Language Model augmented with a lightweight hierarchical decoder, and a standard neural vocoder. Its central design goal is to disentangle “what to say” from “how to say it,” and then to reconstruct a complete acoustic representation through ordered decoding of semantic, style, and acoustic tokens [2509.19001]. ## 1. Conceptual framing and problem formulation HD-PPT, short for **Hierarchical Decoding of Content- and Prompt-Preference Tokens for Instruction-based TTS**, treats instruction-based speech synthesis as a three-stage hierarchical generation problem. The framework consists of three core components: a speech-token codec that decomposes pretrained acoustic tokens into **content-preference** and **prompt-preference** streams, a Qwen2.5-0.5B-based LLM with a lightweight hierarchical decoder that sequentially emits semantic, style, and full acoustic tokens, and a pre-trained neural vocoder that renders the final waveform [2509.19001]. The motivating claim is that existing instruction-based TTS models still lack fine-grained control because they attempt to map single-level text instructions directly onto multilevel speech tokens. HD-PPT addresses this by explicitly decomposing speech into semantically oriented and style-oriented latent streams and by imposing a decoding order that mirrors this decomposition. In the paper’s formulation, the distinction between content and prompt preference is not merely descriptive; it is operationalized through separate quantization paths and separate supervisory signals. This suggests a broader interpretation of HD-PPT as a factorized latent-variable approach to controllable TTS. A plausible implication is that the framework’s controllability derives less from a larger generative backbone than from the imposition of intermediate structure between text instruction and acoustic realization. ## 2. Speech-token codec and latent decomposition The codec begins with a sequence of discrete speech tokens \(T_{s,1\ldots J}\) produced by a pretrained CosyVoice2 tokenizer at **25 Hz**. A **5-layer Conformer** encoder maps these tokens to continuous frame-level vectors \(Z_{1\ldots J}\in\mathbb{R}^d\). Two independent **Finite Scalar Quantization (FSQ)** modules then discretize each frame-level vector \(Z_j\) into a content-preference index \(c_j\) and a prompt-preference index \(p_j\) using separate learned codebooks of size **1 296** and **64**, respectively [2509.19001]. The quantization is defined as \[ c_j = \arg\min_{k}\lVert Z_j - e^{(c)}_k\rVert,\qquad p_j = \arg\min_{\ell}\lVert Z_j - e^{(p)}_\ell\rVert, \] where \(\{e^{(c)}_k\}\) and \(\{e^{(p)}_\ell\}\) are the learned codebooks. A **4-layer causal Transformer** combiner reconstructs the original speech tokens \(T_s\) from the paired discrete streams \((c_j,p_j)\) in an auto-regressive manner. The reconstruction objective is \[ \mathcal{L}_{rec} = -\sum_{j=1}^J\log p\bigl(T_{s,j}\,\big|\,c_{1\ldots j},\,p_{1\ldots j}\bigr)\,. \] The codec’s role is therefore not only compression or token transformation; it is a supervised disentanglement mechanism. By replacing a single speech-token sequence with two lower-level discrete streams whose intended semantics differ, HD-PPT establishes the internal variables later consumed by the hierarchical decoder. The distinction between “content-preference” and “prompt-preference” is a specific design choice of the framework, and the paper’s results treat it as foundational to instruction adherence and style control. ## 3. Supervised disentanglement of semantics and style To enforce separation between semantic content and style information, HD-PPT applies two auxiliary objectives to the codec. The **ASR loss** is imposed on content-preference tokens. A **Whisper-Small** decoder consumes \(\{c_j\}\) and predicts text: \[ \mathcal{L}_{asr} = -\sum_{t}\log p\bigl(w_t \,\big|\, c_{1\ldots J}\bigr)\,. \] The **CLAP contrastive loss** is imposed on prompt-preference tokens. After pooling \(\{p_j\}\) into a fixed embedding \(v_p\) by cross-attention, the model maximizes cosine similarity to the **RoBERTa** embedding \(v_{text}\) of the style prompt and minimizes similarity to \(N\) negatives: \[ \mathcal{L}_{clap} = -\log\frac{\exp\bigl(\cos(v_p,v_{text})/\tau\bigr)} {\sum_{n=0}^N\exp\bigl(\cos(v_p,v_{neg(n)})/\tau\bigr)}. \] The total codec loss is \[ \mathcal{L}_{total} = \mathcal{L}_{rec} + \lambda_{asr}\,\mathcal{L}_{asr} + \lambda_{clap}\,\mathcal{L}_{clap}, \] with \(\lambda_{asr}=2.0\) and \(\lambda_{clap}=0.8\) [2509.19001]. These objectives define the intended semantics of the two token streams. The ASR pathway forces content-preference tokens toward linguistic recoverability, while the CLAP pathway ties prompt-preference tokens to natural-language style descriptions. A common misconception would be to regard the two streams as arbitrary learned partitions of acoustic latent space. In the paper’s construction, they are instead supervised partitions, each anchored by an external objective aligned with either textual content or prompt-level style semantics. ## 4. Hierarchical decoding with the LLM HD-PPT conditions on a natural-language text instruction \(T_t\). Its **Qwen2.5-0.5B** backbone produces hidden states \(\{h_{1\ldots J}\}\) auto-regressively, while an auxiliary **2-layer hierarchical decoder** emits \(c_j\), \(p_j\), and finally \(s_j\). At each step \(j\), the decoding process follows three ordered stages [2509.19001]: 1. **Semantic foundation** \[ p(h_j\mid T_t,T_{s,