- The paper shows that LoRA fine-tuning outcomes depend strongly on acoustic diversity: energy variation above roughly 13 dB predicts DNS-MOS gains, while homogeneous data can reduce quality despite steadily improving validation loss.
- The paper finds that speaker similarity improves across all tested speakers, but noisy reference recordings amplify artifacts, making perceptual evaluation and data-quality screening more reliable than loss-based checkpoint selection.
- The paper demonstrates that mixed multi-speaker training improves zero-shot DNS-MOS by 0.293 on unseen speakers, while GGUF 8-bit quantization delivers 4.5–6.9× faster generation with near-preserved quality.
Overview and motivation
This paper studies parameter-efficient fine-tuning (PEFT) of the LLM (LM) backbone in LLM-based text-to-speech systems, using NeuTTS with a Qwen2.5-0.5B backbone (Qwen et al., 2024). The authors observe that existing LoRA-based TTS literature adapts synthesis-side components—acoustic decoders, speaker embeddings, style modules—while keeping the semantic backbone frozen, and that only UtterTune (Kato, 13 Aug 2025) injects LoRA into LM layers, and only for pronunciation and pitch control. The paper addresses four gaps: the interaction of LoRA adaptation with the pretrained acoustic prior of speech-trained LMs; the reliability of validation loss as a proxy for perceptual quality; the role of training-data characteristics (acoustic variability, energy spread) in adaptation success or failure; and the joint optimization of decoding hyperparameters with adapted backbones.
Methodology
The authors fine-tune the Qwen-0.5B LM backbone that predicts acoustic tokens decoded by a neural codec, for per-speaker voice cloning. Two regimes are compared: full fine-tuning (batch size 2 on a 24 GB L4 GPU, 5 epochs) and LoRA (rank 8, alpha 16, applied to the q-, k-, and v-projection attention layers, batch size 4 with gradient accumulation 2). Six single-speaker datasets are used: three speakers from HiFi-TTS (3.5–3.9 hours each, short clips averaging 2.5–2.8 s) and three from Libriheavy-HQ (13.7–18.5 hours, longer clips averaging ~14–15 s). Evaluation uses DNS-MOS (OVRL) as the primary perceptual metric, chosen after a comparison showing UTMOSv2 is inconsistent across runs, while WVMOS and TorchAudio-Squim exhibit length bias; speaker similarity via WeSpeaker embedding cosine similarity; and blind SNR via WADA-SNR.
Perceptual outcomes of LoRA fine-tuning
The central empirical finding is a strong dependence of fine-tuning outcome on training-data characteristics. Across the six speakers, LoRA at 1000 training steps yields DNS-MOS gains of up to +0.424 (Speaker 2, HiFi-TTS), while Libriheavy speakers show losses of up to −0.414 (Speaker 1212) at the same checkpoint. Speaker similarity, by contrast, improves for all six speakers (e.g., +24.6% for Speaker 1401), indicating that LoRA adapts speaker identity reliably regardless of data quality. SNR improves substantially for speakers with clean references (up to +33.8% over base for Speaker 1) but degrades sharply for speakers with noisy reference audio (−39.2% for Speaker 1212), demonstrating that LoRA faithfully amplifies both desirable speaker characteristics and recording artifacts present in narrow training distributions. Full fine-tuning is competitive with LoRA on MOS (e.g., 4.077 vs. 3.861 for Speaker 1) but at higher memory cost and with less stable loss trajectories.
Loss–quality divergence
A key diagnostic result is that training and validation loss converge smoothly and monotonically for all speakers, while DNS-MOS behaves non-monotonically: it improves for high-variability speakers within the first 1000 steps but degrades markedly for low-variability speakers, partially recovering by five epochs. The authors attribute the recovery to the frozen backbone's pretrained acoustic prior reasserting itself, with norm-bounded low-rank updates acting as implicit regularization. The practical implication is stated directly: checkpoint selection in LLM-based TTS cannot rely on loss convergence and must be guided by perceptual evaluation.
Training-data variability as a predictor
Analysis of energy statistics reveals that mean energy has no consistent relationship with fine-tuning outcome, whereas energy standard deviation is a strong predictor: speakers with energy std above ~13 dB achieve the largest DNS-MOS gains, while those below ~10 dB suffer degradation or stagnation. DNS-MOS dispersion in the training data correlates similarly. The authors conclude that distributional diversity, not absolute signal statistics, governs adaptation success, and that enforcing minimum energy-variability thresholds during data curation is more effective than energy normalization alone. This is the paper's most actionable claim, though it rests on only six speakers.
Decoding-time control and reference audio length
Constrained decoding (T=0.8, k=40) partially mitigates degradation for low-quality speakers—Speaker 1212 improves from 3.233 to 3.461—while slightly reducing quality for high-quality speakers (Speaker 2: 4.141 to 4.048), consistent with expressive nuances occupying lower-probability regions of the learned distribution. Experiments on reference (cloning) audio length produced inconsistent, speaker-dependent results; the authors note that concatenated or sliced reference audio introduced prosodic discontinuities that confounded the length–quality relationship, and recommend single continuous recordings with SNR above 25 dB and DNS-MOS above 3.5 for cloning.
Mixed and multi-speaker training
Three multi-speaker configurations are compared against speaker-specific fine-tuning. Models trained exclusively on HiFi-TTS speakers generalize zero-shot to unseen Libriheavy speakers: the 2+2+2 configuration achieves an average DNS-MOS of 3.806 versus 3.513 for speaker-specific fine-tuning (+0.293) on speakers never encountered in training, with substantially lower MOS variance (0.008 vs. 0.052). The mechanism is that the multi-speaker model avoids overfitting the narrow acoustic manifold of low-variability Libriheavy data. The cost is a fidelity–generalization trade-off: similarity on training speakers drops modestly (0.774 to 0.738), and zero-shot similarity on unseen speakers is low (0.425). Mix FT, trained on all six speakers with only 11–22% of per-speaker data, achieves similarity within 5–9% of dedicated single-speaker models and outperforms the zero-shot model on Libriheavy similarity (0.506 vs. 0.425), supporting a single shared adapter as a scalable alternative to per-voice weights.
Latency and quantization
GGUF 8-bit quantization reduces non-streaming generation time from 24.4–25.7 s (F32) to 4.4–6.6 s, a 4.5–6.9× speedup, with LoRA Q8 within 4% of Base Q8 generation time for Speaker 1. First-chunk latency with the NeuCodec ONNX runtime is 0.31–0.42 s, with near-identical MOS, similarity, and SNR between CPU and GPU codec execution, indicating the codec is not the deployment bottleneck.
Limitations and open questions
The paper is explicit about several constraints. All experiments use a 0.5B-parameter backbone, and the authors state the findings may not transfer to larger LLMs or to non-TTS tasks. Only the first 5000 samples per HiFi-TTS speaker were used, leaving the effect of the full data volume unexplored. Data analysis was limited to frequency and energy statistics rather than spectrogram-level analysis. The variability thresholds (13 dB / 10 dB energy std) are derived from six speakers and two audiobook-domain corpora; their generality across domains, languages, and model scales remains an open question. The zero-shot generalization result is also confounded with domain: HiFi-TTS-trained models may simply transfer better within the audiobook domain rather than across speakers generally.
Conclusion
This work demonstrates that LoRA adaptation of an LLM-based TTS backbone is governed primarily by training-data acoustic diversity rather than by optimization dynamics: energy variability above ~13 dB reliably predicts perceptual gains, while homogeneous data causes loss–quality divergence in which improving validation loss masks degrading DNS-MOS. Speaker similarity improves universally, but SNR gains require clean references, and multi-speaker mixed training with as little as 1–2 hours per speaker yields both zero-shot MOS gains on unseen speakers and near-parity similarity at a fraction of the per-speaker data. Combined with GGUF quantization delivering sub-second first-chunk latency, the results support LoRA-adapted compact LM backbones as a practical mechanism for speaker-level TTS adaptation, contingent on data-diversity screening during curation.