UtterTune: Lightweight LLM-TTS Adaptation

• UtterTune is a lightweight adaptation strategy for multilingual TTS that employs LoRA modules for explicit phoneme and pitch-accent control, specifically addressing Japanese pronunciation challenges.
• It integrates special control tokens and low-rank matrices to correct segmental and prosodic errors, significantly improving naturalness and accent correctness with less than 0.5% additional parameters.
• The system enables rapid adaptation with approximately one hour of training on a single GPU, ensuring accurate pronunciation adjustments without degrading overall synthesis quality.

UtterTune is a lightweight parameter-efficient adaptation strategy for multilingual LLM-based text-to-speech (TTS) systems, providing precise, phoneme-level control over pronunciation and pitch accent in Japanese, while preserving the synthesis quality and language coverage of the underlying model. By employing low-rank adaptation (LoRA) modules and special control tokens, UtterTune enables explicit correction of segmental and prosodic errors, particularly targeting systematic Japanese G2P and accentuation failures that arise in pretrained models lacking explicit front-end modules.

1. Base Multilingual LLM-TTS Foundation

UtterTune adapts CosyVoice 2, a Qwen2.5-0.5B-based, autoregressive Transformer LLM serving as a multilingual TTS backbone. CosyVoice 2 employs a minimal, uniform byte-pair encoding (BPE) text front end. The model receives text tokens (kanji, kana, Latin letters, etc.) and autoregressively predicts discrete, semantic audio tokens generated by a vector-quantized codec; these tokens are subsequently waveform-synthesized by a separate vocoder. The training objective is standard cross-entropy prediction of the next token over the concatenated [text tokens; speech tokens] sequence.

The BPE front end—deliberately omitting explicit grapheme-to-phoneme (G2P) modules—facilitates scaling across dozens of languages but presents particular challenges for Japanese, where homographic kanji often have multiple context-dependent readings and pitch accent is non-trivial. This yields systematic errors in pronunciation and prosody, motivating the need for post hoc segmental and pitch control mechanisms (Kato, 13 Aug 2025).

2. Low-Rank Adaptation Architecture

UtterTune integrates LoRA into every self-attention projection (Q, K, V, O) of the Transformer, introducing trainable, low-rank matrices while freezing the original model weights. The effective transformation applied at training and inference is:

$W' = W + \alpha B C,$

where $W \in \mathbb{R}^{d \times k}$ is the frozen pre-trained weight matrix, $B \in \mathbb{R}^{d \times r}$ and $C \in \mathbb{R}^{r \times k}$ are trainable low-rank matrices, $r$ is the rank ($r=16$), and $\alpha$ is a scaling parameter ($\alpha=64$). LoRA dropout is set to $0.05$. Initializations ensure that $\alpha BC$ has variance comparable to $W \in \mathbb{R}^{d \times k}$0.

Training is restricted solely to all LoRA parameters $W \in \mathbb{R}^{d \times k}$1, along with two new embedding vectors for special control tokens (see Section 3); all backbone weights remain frozen. This architecture results in less than $W \in \mathbb{R}^{d \times k}$2 additional parameters and supports adaptation with approximately one hour of training on a single RTX 4090 GPU (Kato, 13 Aug 2025).

3. Explicit Phoneme and Pitch-Accent Control

UtterTune introduces two BPE-special tokens, <PHON_START> and <PHON_END>, which delineate spans of text to be interpreted as explicit phoneme and accent-level control specifications. Within these boundaries, pronunciation is provided in katakana (standard Tokyo dialect), with pitch-accent nuclei denoted by apostrophes (’), and accent phrase boundaries by slashes (/). An example:

$B \in \mathbb{R}^{d \times r}$4 corresponds to the kanji 魑魅魍魎, with the high–fall accent nucleus on the first mora. When activated, the model is trained to force-match the exact sequence of phonemes and pitch accent, overriding the default G2P and prosody inference, while leaving behavior outside tagged spans and for other languages unchanged.

4. Training Regimen and Data Preparation

Fine-tuning is performed on a composite of Japanese speech datasets: JSUT “basic5000”, JVS “parallel100”, and “voiceactress100”, yielding $W \in \mathbb{R}^{d \times k}$3 utterance pairs ($W \in \mathbb{R}^{d \times k}$4 for validation). For each transcript, one noun is automatically transcribed into phoneme+accent strings using pyopenjtalk, then manually corrected by a native speaker. The base checkpoint is CosyVoice 2-0.5B.

Key training hyperparameters:

• Optimizer: AdamW, learning rate $W \in \mathbb{R}^{d \times k}$5, $W \in \mathbb{R}^{d \times k}$6 warm-up, cosine annealing schedule
• Batch size: $W \in \mathbb{R}^{d \times k}$7
• Training steps: $W \in \mathbb{R}^{d \times k}$8
• Loss: cross-entropy on next-speech-token prediction only
• Trainable parameters: LoRA matrices and <PHON_START>/<PHON_END> token embeddings (Kato, 13 Aug 2025)

5. Evaluation Protocols and Metrics

Two evaluation regimes are used:

• Test Set 1 (General Japanese): 50 sentences of 20–40 characters, synthesized for 42 speakers from ReazonSpeech v2.0 (2,100 stimuli per system).
• Test Set 2 (Controlled word stress test): 50 sentences, each containing a difficult-to-read word (2–6 morae, variable accent nucleus), with the target word bracketed by phoneme-control tags.

Metrics employed:

1. UTMOSv2: An automatic, non-intrusive MOS predictor; samples with UTMOSv2<2.0 are dropped.
2. Human naturalness MOS: 5-point scale, rated by 16 native listeners, 4 ratings per sample.
3. Speaker similarity (SS): Cosine similarity between ERes2Net embeddings of prompt and synthesized audio.
4. Character error rate (CER): Measured by Whisper large-v3 ASR forced to katakana, with symbol stripping.
5. Accent correctness: Manual evaluation of accent pattern for four speakers (Test 2 only).

Objective and subjective results are summarized below:

Test Set	System	UTMOSv2↑	MOS↑	SS↑	CER↓	Accent correctness↑
1	CosyVoice 2-0.5B (base)	3.25	3.44	0.693	0.0557	–
1	UtterTune	3.28	3.88	0.696	0.0639	–
2	CosyVoice 2-0.5B (base)	3.27	–	0.695	0.1005	–
2	CosyVoice 2 (kana)	3.28	–	0.692	0.0595	0.472
2	UtterTune	3.30	–	0.697	0.0626	0.975

Notably, naturalness MOS improves significantly ($W \in \mathbb{R}^{d \times k}$9, $B \in \mathbb{R}^{d \times r}$0) on general text. UtterTune achieves accent correctness of $B \in \mathbb{R}^{d \times r}$1 on stress-test samples with negligible impact on CER and without loss of speaker similarity. Control tests show that phoneme-mode interventions do not affect untagged words (Kato, 13 Aug 2025).

6. Application, Limitations, and Future Extensions

In practical usage, UtterTune enables direct correction of problematic kanji or pitch accent by annotating the input with phoneme-accent tags. For example, inputting

$B \in \mathbb{R}^{d \times r}$5 results in accurate synthesis of "魑魅魍魎" with proper segmental and prosodic realization.

Parameter overhead is kept below $B \in \mathbb{R}^{d \times r}$2, permitting rapid adaptation (≈$B \in \mathbb{R}^{d \times r}$3 GPU-hour). The mechanism is language-scoped: LoRA modules are omitted entirely for non-Japanese text, ensuring no degradation in other languages.

Limitations include a dependence on manual G2P correction for training instances and a binary (high/low) pitch-accent representation that does not capture finer prosodic nuances. Future directions include automating G2P and accent annotation, extending to additional Japanese dialects, and adapting the recipe to other languages requiring fine-grained pronunciation control (Kato, 13 Aug 2025).