---
title: 'LibriTTS-VI: Voice Impression-Controlled TTS'
url: https://www.emergentmind.com/topics/libritts-vi
type: topic
---

# LibriTTS-VI: Voice Impression-Controlled TTS

LibriTTS-VI is a public corpus and methodological framework for voice-impression-controlled text-to-speech, introduced to make fine-grained, human-interpretable control reproducible in zero-shot multi-speaker synthesis. In this setting, a *voice impression* (VI) is a perceptual, human-understandable quality such as calmness, brightness, warmth, firmness, or youthfulness, represented along named semantic dimensions rather than only through low-level acoustic variables such as pitch, energy, or duration. The corpus is built by annotating LibriTTS-R and releasing both manual VI annotations and estimated VI values for the full corpus. The associated modeling work addresses *impression leakage*, in which synthesized speech inherits undesired impressions from the reference utterance used for speaker identity, even when a separate target impression has been specified. The paper proposes two leakage-mitigation strategies—VIC-sep and VIC-rfg—and reports that the best method reduces the mean squared error of 11-dimensional voice impression vectors from 0.61 to 0.41 objectively and from 1.15 to 0.92 subjectively, while maintaining high fidelity [2509.15626].

## 1. Voice impression as a control target

Voice impression control is motivated by the need for coarse-to-fine, semantically legible control over synthesized speech. The central premise is that application domains such as assistants, accessibility, entertainment, and advertising require nuanced control that is both precise and interpretable. In this formulation, users or designers can target named perceptual dimensions—for example, asking for speech that is calmer or brighter—rather than manually tuning low-level acoustics. This distinguishes VI control from acoustic feature control, fixed emotional labels, reference-audio control, and natural-language prompting, each of which offers a different trade-off between usability and precision [2509.15626].

A key technical obstacle is *impression leakage*. Leakage occurs when the synthesized output remains biased by the impression carried by the reference utterance used for speaker encoding, instead of obeying the separately specified target VI. The paper characterizes this failure mode explicitly: one may request a brighter voice yet obtain a darker one because the reference audio itself conveys a dark impression. Prior work had attempted to suppress such leakage with adversarial devices such as dropout and gradient reversal and with dedicated control modules, but when identity and impression are both extracted from a single utterance, disentanglement remained insufficient.

A common misconception is to treat VI as a direct substitute for low-level acoustic control. The formulation in LibriTTS-VI instead positions VI as a perceptual layer: it is human-interpretable, numerically targetable, and intended to complement rather than simply replace direct acoustic parameter manipulation. The dataset and methods are therefore framed around zero-shot TTS with explicit semantic conditioning rather than around a purely signal-level parameterization.

## 2. Corpus construction and the 11-dimensional taxonomy

LibriTTS-VI is defined as the first public voice impression dataset with clear annotation standards, created by annotating the LibriTTS-R multi-speaker TTS corpus. It inherits LibriTTS-R speech audio, text transcripts, speaker IDs, predefined splits, and the sampling rate and file formats of LibriTTS-R. No audio reprocessing beyond LibriTTS-R’s restored quality pipeline is introduced; the added contribution is an annotation layer targeted at robust and practical VI control. The release includes manual VI annotations and estimated VI values for the entire LibriTTS-R generated by a trained Voice Impression Estimator (VIE). The repository is hosted at `https://github.com/sony/LibriTTS-VI`, with licensing, versioning, documentation, and updates handled there [2509.15626].

The dataset defines an 11-dimensional VI taxonomy. Ten dimensions are subjective and rated on a 7-point Likert-type scale with explicit annotation standards, including a neutral point and semantics; the eleventh dimension, speaking rate, is derived objectively from timestamped ASR and then rescaled to 1–7 across the dataset.

| Code | Semantic pair | Measurement |
|---|---|---|
| A | Low–High | 7-point subjective scale |
| B | Masculine–Feminine | 7-point subjective scale |
| C | Clear–Hoarse | 7-point subjective scale |
| D | Calm–Restless | 7-point subjective scale |
| E | Powerful–Weak | 7-point subjective scale |
| F | Youthful–Aged | 7-point subjective scale |
| G | Thick–Thin | 7-point subjective scale |
| H | Firm–Relaxed | 7-point subjective scale |
| I | Dark–Bright | 7-point subjective scale |
| J | Cold–Warm | 7-point subjective scale |
| K | Slow–Fast | ASR-derived, rescaled to 1–7 |

The semantic interpretations are specified dimension by dimension. A corresponds to perceived pitch height or overall register; B to perceived gendered quality; C to vocal clarity versus hoarseness, related to breathiness and roughness; D to tranquility versus agitation; E to strength versus softness; F to perceived age quality; G to vocal weight or body versus lightness; H to articulation firmness versus laxness; I to spectral shading, contrasting dark or warm with bright or crisp; J to affective temperature, contrasting cool or distant with warm or approachable; and K to speaking rate. The inclusion of Masculine–Feminine is explicitly marked as sensitive, but retained for alignment with prior VIC work.

Relative to prior resources, LibriTTS-VI is positioned not merely as another expressive-speech dataset but as a corpus with transparent standards for non-expert voice impression annotation. This suggests a shift from ad hoc or in-house VI studies toward benchmarkable and reproducible numeric control across semantically named dimensions.

## 3. Annotation protocol, reliability, and Voice Impression Estimation

Manual annotation covers 100 utterances drawn by randomly selecting 100 speakers from LibriTTS-R’s training set and then taking one utterance per speaker. Four annotators with professional experience in speech synthesis evaluation rated the ten subjective VI dimensions, using clear written guidelines and a neutral reference audio sample for each dimension. Each of the ten subjective VIs was rated on a 7-point scale by all four annotators, yielding 400 independent annotations in total. The Slow–Fast dimension was not manually rated; it was computed from ASR timings and rescaled [2509.15626].

Per-utterance ratings were aggregated, for example by the mean over annotators, and agreement was quantified with Krippendorff’s alpha. The reported values are 0.715 for Low–High, 0.875 for Masculine–Feminine, 0.408 for Clear–Hoarse, 0.253 for Calm–Restless, 0.342 for Powerful–Weak, 0.562 for Youthful–Aged, 0.447 for Thick–Thin, 0.418 for Firm–Relaxed, 0.429 for Dark–Bright, and 0.194 for Cold–Warm; Slow–Fast is not applicable because it is objective. The average $\alpha$ is 0.464. Many dimensions are therefore below the conventional reliability threshold of $\alpha \ge 0.667$, but the paper states that these values are comparable or better than those found in other subjective audio perception tasks, including emotion and singing voice. Inter-scale correlations are broadly consistent with prior VIC work, with some dataset-related discrepancies such as a correlation inversion between C and F, which the paper attributes likely to audiobook characteristics in LibriTTS-R.

The VIE is trained to predict VI vectors from audio, using the mean squared error between manual VI and predicted VI:
$\mathcal{L}_{\text{VIE}} = || \mathbf{v_{\text{manual} - \hat{\mathbf{v} ||_2^2$,
with prediction defined from the speaker-encoder intermediate representation. The training challenge is that prior VIC had assumed constant VI per speaker, an assumption described as unsuitable for audiobooks because narration and expressive passages can vary substantially within speaker. To address this, the paper augments the 100 manually annotated utterances by retrieving up to 100 acoustically similar utterances from the same speaker using L1 norms of pitch and energy differences together with cosine similarity of WavLM embeddings. The top 100 are assigned the manual VI vector for VIE training, after which the trained VIE produces VI estimates for the entire LibriTTS-R.

## 4. Leakage mitigation methods

The baseline system, denoted VIC-base, inserts a Control Module (CM) into the speaker encoder before the style token layer (STL). It takes both the encoded reference vector and the target VI vector, attempts to suppress VI information in the reference path through dropout and gradient reversal, concatenates the result with the VI path, and projects back to the original representation. In preliminary experiments, however, leakage persisted: the synthesized voice remained biased by the impression of the reference audio [2509.15626].

The first proposed remedy is VIC-sep, which performs disentanglement by using separate utterances. During fine-tuning, two utterances from the same speaker are used. The original utterance $r$ provides the ground-truth audio target and its target VI vector, while a separate utterance $r'$ from the same speaker is fed to the identity path of the speaker encoder in place of $r$. Identity conditioning is derived from $r'$ via the HuBERT plus BiLSTM-with-attention encoder, then passed through the CM and STL. Impression conditioning comes from the target VI vector associated with $r$, whether manual or estimated. The explicit rationale is that dropout and gradient reversal are insufficient when identity and impression are entangled in a single utterance, whereas using $r$ and $r'$ cleanly separates impression and identity sources without modifying the backbone.

The second remedy is VIC-rfg, a reference-free generation model. Here the source of leakage is removed architecturally by eliminating reference audio from synthesis entirely. The CM conditions only on the target 11-dimensional VI vector, while the usual reference-encoded vector $\mathbf{x}$ is replaced with 32-dimensional Gaussian noise during fine-tuning and inference. In this setup the CM maps the VI vector into a representation suitable for STL-based global conditioning of the TTS backbone. Because speaking rate becomes unstable in the fully reference-free configuration, the stochastic duration predictor (SDP) is jointly fine-tuned. The reported benefits are twofold: improved robustness against leakage by design, since there is no reference audio to leak impression information, and the convenience of reference-free zero-shot generation.

These methods embody two different structural responses to leakage. VIC-sep decouples the sources of identity and impression while retaining a reference utterance; VIC-rfg removes the reference signal altogether. A plausible implication is that the paper treats leakage not merely as a training-regularization issue but as an architectural conditioning problem.

## 5. Backbone architecture, training regime, and empirical results

The TTS backbone is a VITS-based zero-shot multi-speaker model. Its text encoder is Conformer-based, with an auxiliary CTC loss and an adaptive layer-norm zero Transformer enhancement. The speaker encoder uses a pre-trained HuBERT front end followed by a Bi-LSTM with attention, then the Control Module, then the style token layer, with the resulting speaker representation used for global conditioning of VITS modules. The intermediate vector $\mathbf{x}$ and speaker embedding $\mathbf{g}$ are 256-dimensional, while $\mathbf{x}$ and the VI vector $\mathbf{v}$ are each projected to 32 dimensions inside the CM. The baseline CM structure is given as
$\mathbf{x'} = \text{Linear}(\text{concat}[\text{Linear}(\mathbf{v}) ; \text{Linear}(\text{Dropout}(\mathbf{x}))])$ [2509.15626].

Training proceeds in two stages. First, the base VITS model is trained on LibriTTS-R for 600k steps. Fine-tuning of the CM then lasts 60k steps for VIC-base and VIC-sep with the backbone frozen, while VIC-rfg also fine-tunes the SDP jointly to stabilize speaking rate. Optimization uses AdamW with initial learning rate $2 \times 10^{-4}$, batch size 20, on eight NVIDIA A100 GPUs. The paper states that CM fine-tuning uses the standard TTS training losses, whereas the VIE uses the MSE objective above.

Evaluation is conducted in zero-shot mode on LibriTTS-R test-clean with 39 unseen speakers. Intelligibility is measured by CER and WER using Whisper large-v3, quality by UTMOS, speaker similarity by SECS, and controllability by VI-MSE, RVI-MSE, and the leakage gap $\Delta_V = \text{RVI-MSE} - \text{VI-MSE}$. VI-MSE compares the target VI of the reference utterance with the VIE estimate from synthesized speech using that same target; RVI-MSE is the harder condition in which the target VI is sourced from a random unrelated utterance.

Across systems, intelligibility and quality remain high and comparable, with CER around 3%, WER around 8%, and UTMOS around 4.23–4.26. SECS is approximately 0.84 for VIC-base, 0.82 for VIC-sep, and 0.76 for VIC-rfg, with the paper describing the drop for VIC-rfg as an expected trade-off of reference-free identity reconstruction. On VI-MSE, the three systems score about 0.39, 0.37, and 0.36 respectively. On the more revealing RVI-MSE metric, VIC-base is about 0.61, VIC-sep about 0.51, and VIC-rfg about 0.41. The leakage gap $\Delta_V$ drops from about 0.22 in VIC-base to about 0.14 in VIC-sep and about 0.05 in VIC-rfg. The headline result is therefore an objective reduction in RVI-MSE from 0.61 to 0.41.

The modulation experiment probes whether predicted VI changes monotonically as targets are swept. For each of the 39 unseen speakers, each VI dimension is modulated from $-3$ to $+3$, and the slope of target versus predicted VI is computed using the VIE. Average slope rises from about 0.096 in VIC-base to about 0.135 in VIC-sep and about 0.177 in VIC-rfg, indicating increasingly responsive control. The paper notes that Powerful–Weak is not learned effectively by any system.

Subjective evaluation uses human VI ratings and MOS. In the multiple-VI modulation condition, controllability measured as MSE between target VI and human-rated VI improves from about 1.15 for VIC-base to about 1.04 for VIC-sep and about 0.92 for VIC-rfg. For single-VI modulation on four representative dimensions—A, B, F, and I—the proposed methods generally outperform the baseline, although results vary by dimension and modulation direction. For fidelity and naturalness, 30 native or near-native English listeners provide MOS on a 5-point scale with 95% confidence intervals and significance testing at $p < 0.05$. Across single-VI modulation there is no consistent pattern of degradation relative to baseline; VIC-sep shows roughly balanced significant improvements and degradations, and VIC-rfg is similarly balanced. In multiple-VI modulation, audio quality remains comparable to baseline, indicating improved controllability without a consistent naturalness penalty.

## 6. Reproducibility, limitations, and ethical considerations

The release is designed to support reproducible VI research. The repository provides the dataset and demo, and the paper describes a concrete reproduction path: train the VITS backbone on LibriTTS-R for 600k steps; train the VIE using manual annotations and augmentation with up to 100 similar utterances per annotated item based on pitch and energy L1 differences and WavLM cosine similarity; fine-tune the CM as VIC-base, VIC-sep, or VIC-rfg; evaluate on test-clean with 39 unseen speakers using Whisper large-v3 for CER and WER, UTMOS, SECS, VI-MSE, RVI-MSE, $\Delta_V$, human VI ratings, and MOS. For preprocessing, Slow–Fast is derived from ASR timings via `whisper-timestamped` and then rescaled to 1–7. The release may also include trained VIE models and estimated VI values for the entire LibriTTS-R, along with code and documentation for the CM variants when available in the repository [2509.15626].

Several limitations are explicit. Annotation reliability is moderate, with average Krippendorff’s alpha around 0.464, so downstream models inherit uncertainty from subjective perception. Manual annotations cover only 100 utterances from 100 speakers, even though VIE estimates are released for the full corpus. Domain effects are also salient: LibriTTS-VI is based on audiobook speech, and the paper notes dataset-related correlation discrepancies, implying that controllability characteristics may differ in conversational or studio domains.

The modeling limitations are equally specific. Powerful–Weak remains difficult to learn across all systems. Reference-free generation minimizes leakage most strongly but sacrifices some speaker similarity, as reflected in the lower SECS for VIC-rfg. Cross-lingual generalization is not studied, and future work is described as needing to test whether the proposed dimensions and scales remain valid across languages and cultures.

Ethical considerations arise from both annotation semantics and deployment. The Masculine–Feminine axis is described as potentially sensitive, and the paper recommends attention to ethical implications and regional or cultural variation in perception. More broadly, impression manipulation can be used to influence persuasion or mimic personal styles; the stated mitigations are disclosure, usage policies, and consent when simulating identifiable voices. Future directions listed in the paper include expanding manual annotations, refining dimension definitions, exploring hybridization with expert-informed voice quality descriptors, extending language and domain coverage, and developing stronger identity-preserving but leakage-resistant architectures such as contrastive disentanglement, orthogonality constraints, or multi-branch decoders.

Source: https://www.emergentmind.com/topics/libritts-vi