Single-Codebook TTS LLMs

• Single-Codebook TTS LLMs are specialized speech synthesis models that use a single discrete codebook to tokenize continuous audio, simplifying integration with transformer architectures.
• They enable unified autoregressive decoding by mapping the entire speech waveform into a univariate token sequence, ensuring efficient inference and prospects for large-scale deployment.
• Recent advancements address challenges in semantic fidelity and token alignment through hybrid training strategies, reinforcement learning, and postprocessing techniques to mitigate decoding errors.

Single-codebook Text-to-Speech (TTS) LLMs form a distinct paradigm within LLM-based speech synthesis, defined by the use of a single discrete codebook for audio tokenization and generation. In these models, the continuous speech waveform is mapped to a univariate sequence of discrete tokens by a neural audio codec, streamlining autoregressive modeling and ensuring full compatibility with established transformer-based LLM architectures. This approach offers architectural simplicity, seamless integration with LLM scaling laws, and efficiency at inference time, all while addressing complex requirements for semantic and acoustic fidelity, prosodic control, and scalability.

1. Architectural Fundamentals

Single-codebook TTS LLMs operate by first encoding input speech into a sequence of discrete tokens using vector quantization (VQ) methods. The primary innovation is the elimination of multi-codebook token hierarchies—prevalent in residual vector quantization (RVQ) and group VQ—in favor of a single codebook. The standard pipeline is as follows:

• Codec (Tokenizer): Operates via either direct or grouped VQ (e.g., FSQ, SimVQ, VQ-EMA) over continuous representations (often fused semantic/acoustic encodings). For example, LLaSA adopts FSQ with a 65,536-entry codebook at 50 Hz token rate (Ye et al., 6 Feb 2025); Single-Codec uses an 8,192-entry codebook coupled with BLSTM context modules (Li et al., 11 Jun 2024).
• Transformer Decoding: The LLM (e.g., LLaMA, Qwen2.5) is directly expanded to include the audio token vocabulary. No auxiliary AR+NAR handoff or diffusion backends are required, as in prior systems.
• Decoding: The autoregressive LM produces a single stream of audio tokens conditioned on textual (and optional audio) prompts, which are then passed to the codec decoder for waveform synthesis.

This design subsumes both zero-shot TTS (by conditioning on reference audio) and fully controllable, attribute-aware synthesis. It is particularly conducive to large-scale data and model scaling, enabling direct application of the scaling laws that drive recent LLM progress.

2. Single-Codebook Codec Design and Optimization

The codec architecture determines the quality, semantic richness, and controllability of synthesized speech. Distinct innovations include:

• Disentangled Factorization: Architectures like Single-Codec (Li et al., 11 Jun 2024) and BiCodec (Spark-TTS, (Wang et al., 3 Mar 2025)) explicitly separate time-invariant speaker/global tokens from temporally local content tokens, or compress all content into a single token sequence while carrying global information as embeddings.
• Domain Partitioning: UniCodec (Jiang et al., 27 Feb 2025) partitions a single 16,384-entry codebook into contiguous, domain-adaptive subspaces, supporting speech, music, and sound within a unified TTS-LM pipeline.
• Vector Quantization and Embedding: All systems employ nearest-neighbor assignment:

$k^*(t) = \arg\min_j \|h_t - e_j\|_2^2,$

where $h_t$ is the input frame and $e_j$ are the codebook embeddings. Modern codebooks leverage enhanced utilization (up to 100% in DistilCodec (Wang et al., 23 May 2025)), and large embedding dimensions (e.g., 3,584 in DistilCodec to match LLM word-piece embeddings).

• Regularization and Semantic Enrichment:
  • UniCodec employs a two-stage training regime, first optimizing for acoustic reconstruction before introducing self-supervised semantic mask prediction akin to Wav2Vec 2.0.
  • SpeechAccentLLM (Zhuangfei et al., 2 Jul 2025) integrates CTC within the VQ-encoder to produce tokens with controllable locality and robust alignment to phonetic ground truth.
  • Some systems (e.g., CaT-TTS) have advanced beyond single-codebook by distilling explicit ASR-derived semantic content into their principal codebooks, highlighting a persistent tension between compact univariate token streams and guaranteed linguistic structure (Cao et al., 26 Sep 2025).

3. Alignment with LLM Architectures and Decoding Strategies

The integration of single codebooks into LLMs for TTS is characterized by:

• Vocabulary Expansion and Autoregressive Decoding: The token vocabulary of the underlying LLM is directly expanded to include all discrete codec tokens. Decoding is 1D autoregressive with standard text-LM sampling and search strategies (top-k, nucleus, temperature).
• Prompt and Conditioning Flexibility:
  • Models such as UniTTS (Wang et al., 23 May 2025) support mixed-modality (interleaved text/audio) prompts and can autoregressively generate either modality.
  • Attribute-controllable generation, exemplified by Spark-TTS (Wang et al., 3 Mar 2025), is realized via explicit semantic and global tokens and chain-of-thought (CoT) reasoning steps.
• Inference and Verifier-Driven Search: LLaSA (Ye et al., 6 Feb 2025) demonstrates the value of scaling inference-time compute by integrating external speech verifiers, i.e., WavLM-based speaker verification and Whisper-Large-v3 for WER, and adopting hybrid best-of-N and beam search strategies targeting specific verifier preferences.
• Postprocessing: SpeechRestorer (Zhuangfei et al., 2 Jul 2025) introduces a lightweight transformer to denoise and regularize AR decoding outputs, mitigating erratic token errors and increasing prosodic stability.

4. Training, Supervisory Signals, and Reinforcement Optimization

Recent advances in model stability and output quality are driven by sophisticated training objectives:

• Multi-Reward Reinforcement Learning: The Multi-Reward GRPO framework (Zhong et al., 26 Nov 2025) extends beyond standard supervised next-token prediction by directly optimizing sampling policy using a composite reward,

$$R(s_t, a_t) = \alpha_{intl} R_{intl} + \alpha_{sim} R_{sim} + \alpha_{len} R_{len} + \alpha_{ent} R_{ent} + \alpha_{pro} R_{pro},$$

incorporating Whisper-based intelligibility, WavLM speaker similarity, pause-aware prosody alignment (via reasoning LLMs), entropy regularization, and length penalties. Policy updates normalize reward advantages within small groups (GRPO), avoiding instability from reward scale disparities.

• Alignment and Preference Optimization:
  • UniTTS adopts a three-stage pretrain/SFT/alignment process, closing with Linear Preference Optimization (LPO, a DPO variant) targeting prosodic and stability defects (Wang et al., 23 May 2025).
  • Prosody alignment is enforced by mapping LLM-annotated pause structures to timestamped outputs, using external LLMs such as DeepSeek-R1 as oracles for robust rhythm supervision (Zhong et al., 26 Nov 2025).
• Data Regimes and Massive Scale: Systems are now regularly trained on 100k–250k hours of diverse audio, leveraging both labeled and unlabeled data, with codebooks tuned for maximal utilization (e.g., ≥99%).

5. Empirical Results, Comparative Metrics, and Scaling Laws

The performance of single-codebook TTS LLMs is benchmarked across standard speech synthesis and reconstruction metrics; SOTA models consistently outperform multi-codebook or diffusion-based pipelines conditional on sufficient scale.

System	zh CER↓	zh SIM↑	en WER↓	en SIM↑	MOS↑
Seed-TTS	1.12	0.796	2.62	0.714	–
Spark-TTS	1.20	0.672	1.98	0.584	4.01
CosyVoice3	1.12	0.781	2.21	0.720	4.07
LLaSA+SFT	1.51	0.688	2.89	0.582	3.76
LLaSA+RL	1.10	0.758	2.12	0.672	4.12
LLaSA+RL+FM	1.08	0.790	2.08	0.733	4.21

Key findings include:

• Scale-Driven Quality: WER, CER, and SIM improve monotonically with both model and data scale in LLaSA and UniTTS, reflecting direct translation of LLM scaling laws to TTS (Ye et al., 6 Feb 2025, Wang et al., 23 May 2025).
• Codebook Efficiency and Utilization: DistilCodec achieves near-100% code usage and high perplexity (up to 2.7e4), minimizing quantization loss (Wang et al., 23 May 2025).
• Fine Controllability and Attribute Alignment: Spark-TTS achieves 99.77% gender classification and >90% pitch/speed alignment, demonstrating attribute control (Wang et al., 3 Mar 2025).
• Domain-Generality: UniCodec supports multi-domain audio generation (speech, music, environment) within one codebook, outperforming competing unified codecs and matching domain-specific models (Jiang et al., 27 Feb 2025).

6. Limitations and Advances Beyond the Single-Codebook Model

While the single-codebook paradigm excels in simplicity and AR-LM compatibility, certain structural limitations persist:

• Information Loss and Semantic Ambiguity: Single-codebook systems face a trade-off between bandwidth and preservation of fine phonetic/acoustic detail. S3Codec’s hybrid approach demonstrates that semantic codebooks distilled from ASR models yield more linguistically structured tokens, substantially improving WER and speaker similarity at similar bitrates (Cao et al., 26 Sep 2025).
• Error Accumulation in Decoding: AR processes are prone to drift and compounding errors. Techniques such as MAPI (parallel masked inference) (Cao et al., 26 Sep 2025) and downstream correctors like SpeechRestorer (Zhuangfei et al., 2 Jul 2025) are effective mitigations.
• Token Locality and Faithfulness: Explicit regularization (e.g., CTC in SpeechCodeVAE (Zhuangfei et al., 2 Jul 2025)) is required to ensure locality of token-to-frame alignment and reduce temporal artifacts.

A plausible implication is the emergence of semi-structured codebooks and hybrid AR/NAR pipelines, combining the causal single-sequence simplicity of pure single-codebook approaches with modular semantic grounding and parallel error correction.

7. Reference Implementations and Open-Source Ecosystem

Most leading models provide code and checkpoints (LLaSA (Ye et al., 6 Feb 2025), UniTTS and DistilCodec (Wang et al., 23 May 2025), Spark-TTS (Wang et al., 3 Mar 2025)), supported by massive open datasets (VoxBox: 100k h, Emilia+LibriHeavy). Empirical reproducibility encourages continued benchmarking and ablation-driven progress. Experimental setups routinely employ 1–8 B parameter models, 1M training examples, and large-scale reinforcement or preference-based fine-tuning for robust deployment across domains, speakers, and languages.

---

In summary, single-codebook TTS LLMs define a rapidly maturing class of end-to-end, streamable, and highly scalable text-to-speech architectures. They have shifted the domain toward unified token modeling, efficient AR inference, and direct alignment with LLM advances, while ongoing research targets information-theoretic limits, semantic structure integration, and prosody/expressiveness at ultra-low bitrates (Zhong et al., 26 Nov 2025, Wang et al., 23 May 2025, Li et al., 11 Jun 2024, Ye et al., 6 Feb 2025, Wang et al., 3 Mar 2025, Jiang et al., 27 Feb 2025, Cao et al., 26 Sep 2025, Zhuangfei et al., 2 Jul 2025).