- The paper presents a novel end-to-end TTS architecture that bypasses traditional text normalization to achieve high-fidelity Thai speech synthesis with zero-shot voice cloning.
- It integrates components like TSLM, FSQ, RALM, and LocDiT to effectively handle code-switched Thai-English inputs and numeric text while ensuring robust performance.
- Empirical evaluations demonstrate substantially lower Character Error Rates, high speaker similarity, and over 13x faster processing compared to leading baselines.
JaiTTS: A State-of-the-Art Thai Voice Cloning Model
Introduction
This essay provides a detailed analysis of "JaiTTS: A Thai Voice Cloning Model" (2604.27607), emphasizing its technical advancements in Thai text-to-speech (TTS) synthesis, architectural innovations, experimental setup, results, and broader implications for TTS research in low-resource languages. JaiTTS-v1.0 introduces a novel, end-to-end architecture built on VoxCPM, targeting high-fidelity zero-shot voice cloning and robust handling of code-switching between Thai and English as well as numeric text inputs—capabilities largely absent in extant systems.
Model Architecture and Methodology
JaiTTS-v1.0 is derived from the VoxCPM architecture, which eschews the use of external neural audio codecs in favor of direct continuous latent speech modeling. The generation process is architected as follows:
- Text-Semantic LLM (TSLM): A decoder-only Transformer initialized from MiniCPM-4. The TSLM encodes the semantic intent and prosodic plan from both BPE-tokenized input text and historical acoustic embeddings, producing a continuous semantic-prosodic representation.
- Finite Scalar Quantization (FSQ): The hidden state from TSLM is discretized into a semi-discrete skeleton via per-dimension scalar quantization. FSQ acts as a regularizer, maintaining differentiability for end-to-end optimization.
- Residual Acoustic LLM (RALM): Another decoder-only Transformer that supplements the FSQ skeleton with speaker-specific acoustic detail, leveraging both TSLM output and historical auditory context.
- Local Diffusion Transformer (LocDiT): Operates bidirectionally to decode the synthesized latent patches, using a flow-matching denoising process conditioned on both the FSQ and RALM representations.
- Stop Predictor: A lightweight head to determine sequence termination.
All components are optimized jointly, enabling direct synthesis from raw, code-switched text and numerals, circumventing the explicit text normalization stages traditionally required for Thai TTS. The system enables classifier-free guidance via random dropout of LLM conditioning during training.
Dataset and Experimental Setup
JaiTTS was trained on a comprehensive Thai-centric corpus exceeding 10,000 hours, featuring diverse genre distributions and domain-specific subsets to ensure stylistic and lexical breadth. The training data were scrupulously curated, including both clean studio and in-the-wild crowd-sourced recordings, with rigorous multi-stage transcript validation.
Two distinct evaluation protocols were established:
- Short-duration benchmark: 1-15s utterances filtered from Thai Common Voice using DNSMOS Pro for high-fidelity evaluation.
- Long-duration benchmark: 16-30s utterances from YouTube, manually verified for transcription accuracy, designed to test long-term prosodic consistency and synthesis stability.
The model's performance was benchmarked against Qwen3-TTS (0.6B and 1.7B) and ThonburianTTS, as well as human ground truth.
Quantitative Results
Character Error Rate (CER) and Speaker Similarity
JaiTTS-v1.0 achieves notable advancements across both short and long utterances:
- Short-duration (1-15s): JaiTTS-v1.0 achieves a CER of 1.94%, outperforming the human ground truth (1.98%) and all baselines. Speaker similarity (SIM) matches or exceeds all compared systems.
- Long-duration (16-30s): JaiTTS-v1.0 maintains a low CER of 2.55%, closely paralleling the human reference at 2.47%. Competing models Qwen3-TTS-1.7B and Qwen3-TTS-0.6B lag behind at 3.64% and 6.10% CER, respectively. ThonburianTTS fails on longer forms.
These results demonstrate state-of-the-art Thai TTS intelligibility and stability, with synthesized audio occasionally surpassing human reference intelligibility likely due to cleaner synthesized signal.
Computational Efficiency
JaiTTS achieves a Real-Time Factor (RTF) of 0.1136, making it over 13x faster than the Qwen3-TTS baselines and faster even than non-autoregressive approaches like ThonburianTTS, making it highly practical for interactive and real-time deployment scenarios.
Human Preference Evaluation
Native-speaker blind A/B evaluations revealed substantial human preference for JaiTTS-v1.0: across 400 trials, it was preferred in 283 cases versus commercial flagships from ElevenLabs (eleven_v3) and MiniMax Speech (speech-2.8-hd), and lost only 58, with 59 ties. Evaluators judged on both naturalness and speaker identity matching, with JaiTTS excelling in natural prosody and accuracy for raw code-switched and numeric-rich texts.
Theoretical and Practical Implications
The elimination of explicit text normalization in Thai TTS workflows offered by JaiTTS represents a methodological shift. The ability to directly synthesize mixed Thai-English code-switched and numerically laden text—common in actual communications—is facilitated by robust pretraining, the architectural split between TSLM (semantic/prosodic planning) and RALM (acoustic realization), and hierarchical diffusion decoding.
From a theoretical perspective, the direct continuous latent modeling (eschewing discrete tokenizers/codecs) and semi-discrete skeleton stabilization via FSQ present a promising alternative to prevailing codec-dependent, token-based approaches—addressing observed shortfalls in modeling underrepresented phonetic contrasts and prosodic structures in Thai.
Practically, the increase in real-time factor and intelligibility suggests that this architecture is not only suitable for high-fidelity batch synthesis, but also for real-world interactive voice applications with stringent latency requirements.
Directions for Future Research
This work lays the groundwork for further domain adaptation in TTS for languages with scarce data and complex text input patterns (e.g., pervasive code-switching, numeral handling). The joint optimization paradigm over continuous and semi-discrete representations, and explicit modeling divide between semantics/prosody and acoustics/speaker features, is broadly applicable beyond Thai.
Future research could probe adaptivity to dialectal Thai, extension to other low-resource tonal languages, improved speaker adaptation with even fewer reference samples, and integrating joint ASR-TTS pretraining for further robustness in noisy, spontaneous speech scenarios.
Conclusion
JaiTTS-v1.0 significantly advances the state of Thai voice cloning by combining architectural innovation with language-specific data curation and evaluation. It demonstrates superior intelligibility, human preference, and computational efficiency when compared to both open-source baselines and leading commercial systems. The architecture's capacity to process unnormalized text containing numerals and English code-switching simplifies real-world adoption of high-quality TTS systems in Thai and provides a reproducible framework for other low-resource and typologically complex languages.
