---
title: 'TACA-VITS: Expressive Audiobook TTS'
url: https://www.emergentmind.com/topics/taca-vits
type: topic
---

# TACA-VITS: Expressive Audiobook TTS

TACA-VITS is a neural text-to-speech (TTS) architecture for expressive audiobook synthesis that integrates a text-aware style embedding and a context encoder into the VITS 2.0 backbone. It is designed to capture a broad range of expressive speaking styles, as found in professional audiobook narration, directly from text inputs—without requiring manually labeled style annotations or reference speech at inference. TACA-VITS leverages cross-modal contrastive learning to align text and speech in a shared style space, and employs cross-sentence textual context to yield long-form, coherent, and highly expressive synthetic speech [2406.05672].

## 1. Architectural Foundations

TACA-VITS extends the single-stage VITS 2.0 architecture, which unifies variational inference, adversarial training, and normalizing flows for end-to-end TTS. In the vanilla VITS pipeline, a text encoder generates hidden representations $h$ from input phonemes $x$. The prior $p_\theta(z|h)$ parameterizes a normalizing-flow prior for the latent acoustic variable $z$, the posterior $q_\phi(z|h,y)$ infers $z$ from ground-truth speech $y$, and a generator $G_\theta(z)$—paired with a multi-period discriminator $D_\psi$—maps $z$ to the waveform.

TACA-VITS introduces two core augmentations:
- A text-aware style embedding $s_{text}$, inferred purely from the input text and trained to inhabit a speech-derived “style space” via cross-modal alignment.
- A context encoder that processes not only the sentence in focus but also its neighboring sentence embeddings and the quantized style code, producing context-aware representations $c$ that augment or replace the canonical $h$ in all generative steps.

This architectural adjustment allows TACA-VITS models to handle text-predicted expressive styles and model prosody over multiple sentences, which is central to the audiobook domain where stylistic consistency and long-range prosodic coherence are required.

## 2. Text-Aware Style Space Learning

### 2.1 Speech-Style Encoder

A speech-style embedding $s_{speech}$ is extracted using an off-the-shelf SRL speech encoder initialized from HuBERT (layer 6). For each speech-text pair $(S_i, T_i)$, $E_{speech}(S_i) \in \mathbb{R}^{384}$ is computed. This embedding is spread out via semi-supervised contrastive loss, encouraging the formation of a diverse expressive latent manifold absent explicit labels.

### 2.2 Cross-modal Contrastive Training

A text encoder $E_{text}$, initialized from Chinese T5, maps $T_i$ to $s_{text}$, which is trained to align with its speech counterpart and to preserve the structural geometry of speech-based style relations. The contrastive objective samples positive and negative text-text pairs based on the cosine similarity of their associated speech-style embeddings:
- Positive if $\operatorname{sim}(s_{speech,i},s_{speech,j}) \geq \beta$
- Negative if $\operatorname{sim}(s_{speech,i},s_{speech,j}) \leq \alpha$
- Pairs with $\alpha < \operatorname{sim} < \beta$ are ignored.

The text-style loss combines an InfoNCE objective $L_{con}$ and direct cosine alignment $L_{cos}$:
$$
L_{style} = L_{con} + \lambda_{cos} L_{cos},\quad \lambda_{cos}=1.
$$
Only the T5 encoder is optimized in this stage; the speech encoder remains frozen.

A vector quantization (VQ) module discretizes $s_{text}$ to a codebook index $\hat{s}$, promoting the reuse of a finite number of prototypical expressive styles and stabilizing generation.

## 3. Context Encoder Integration

The context encoder $CE(\cdot)$ combines:
- Phoneme embeddings from a three-sentence window (previous, current, next)
- The corresponding BERT embeddings
- The quantized style code $\hat{s}$

The network architecture consists of two 1D convolutional layers over concatenated phoneme-BERT inputs, a stack of four Transformer layers (dimension 768, 8 heads), injection of the style code via addition at each frame, and a final projection to 512 dimensions (VITS-compatible). During training, the context encoder is fine-tuned on top of a pre-trained VITS model—initializing from the official Bert-VITS 2.0 checkpoint and replacing its BERT block.

A two-phase procedure is used:
1. Pre-train VITS on standard text input.
2. Fine-tune with the full context encoder, sampling style codes from either speech or text encoders to promote generalization.

## 4. Training Regimen and Objective Functions

### 4.1 Loss Formulation

During fine-tuning, the model minimizes:
$$
L_{total} = L_{TTS} + \lambda_{style} L_{style},\quad \lambda_{style}=0.1
$$
$L_{TTS}$ is the sum of VITS losses:
- KL divergence: $L_{KL} = \mathbb{E}_{(c, y)}[ D_{KL}( q_\phi(z|c, y) \,||\, p_\theta(z|c) ) ]$
- Flow matching: $L_{flow} = \mathbb{E}[ \| f_\theta(\epsilon; c) - z \|_1 ],\, \epsilon \sim \mathcal{N}(0, I) $
- Mel-spectrogram L1 loss: $L_{mel} = \mathbb{E}[ \| \text{Mel}(y) - \text{Mel}(G_\theta(z))\|_1 ]$
- Adversarial loss: $L_{adv} = \mathbb{E}_y [ \log D_\psi(y) ] + \mathbb{E}_{z, c} [ \log(1 - D_\psi(G_\theta(z))) ]$

Loss weights are fixed as $\gamma_1=4.0$, $\gamma_2=45.0$, $\gamma_3=1.0$.

During this phase, $L_{style}$ is only back-propagated through the context encoder's style pathway; the text encoder weights are frozen.

### 4.2 Data and Optimization

The model is trained and fine-tuned on:
- 100H-Multi-Style (66 hours manually labeled) for the speech-style encoder
- 6kH unlabeled audiobook speech (auto-transcribed) for text-style contrastive learning
- 20H-Audiobook-HQ (utterance-ordered) for full-system fine-tuning

Optimizers are AdamW (T5, $3\times 10^{-5}$; speech encoder, $10^{-4}$) and Adam ($10^{-3}$) for the TACA-VITS full system. VQ codebook has $64\times32$ entries.

## 5. Empirical Evaluation

### 5.1 Style Alignment

TACA-VITS’s text encoder achieves strong alignment with the speech-style encoder: cosine similarity $\operatorname{sim}(s_{speech}, s_{text}) = 0.93$ for the T5-based encoder versus $0.82$ for a BERT-based baseline. t-SNE visualization of 450 sentences across 9 chapters shows that the learned text-style vectors cluster tightly by chapter but remain well-dispersed within clusters, reflecting both stylistic consistency and intra-chapter variability.

### 5.2 Objective and Subjective Metrics

Objective TTS performance is reported via Mel Cepstral Distortion (MCD, dB) and Character Error Rate (CER, %):

| TTS Model   | MCD ↓   | CER(%) ↓ |
|-------------|---------|----------|
| VITS (base) | 4.23    | 5.80     |
| TACA-VITS   | 4.21    | 5.99     |
| LM (base)   | 8.92    | 13.9     |
| TACA-LM     | 8.15    | 13.1     |

TACA-VITS maintains parity with baseline VITS in both metrics. For LM-based models, the TACA variant yields measurable improvements.

Subjective naturalness and expressiveness are assessed using MOS (1–5 scale, 20 native listeners):

| Model       | NMOS↑   | EMOS↑    |
|-------------|---------|----------|
| VITS (base) | 3.84±0.11 | 3.61±0.14 |
| TACA-VITS   | 3.90±0.10 | 3.93±0.11 |
| LM (base)   | 2.91±0.09 | 3.80±0.11 |
| TACA-LM     | 3.22±0.10 | 4.05±0.10 |

TACA-VITS yields a +0.06 gain in NMOS and +0.32 in EMOS over VITS. The improvements for TACA-LM over LM (0.31 NMOS, 0.25 EMOS) further confirm the effectiveness of the TACA methodology for both VITS- and LM-based TTS.

## 6. Analytical Perspectives

- Cross-modal contrastive learning taps large-scale unannotated audiobook data to induce a nuanced, high-dimensional style manifold from text. This facilitates fine-grained control of style and prosody without style labels or reference audio.
- Vector quantization in the style space promotes re-use of prototypical style embeddings, which contributes to synthesis stability and coverage of diverse expressive modes.
- The context encoder, with its access to past and future sentences, enables coherent long-form prosody—addressing a well-known shortcoming in prior sentence-level TTS systems.
- Text-based style prediction allows fully text-driven inference, removing the need for external style references and supporting scalable deployment.
- Both the text-style modeling and context encoder modules are modular and can be integrated into other single-stage TTS systems.

## 7. Significance and Future Directions

TACA-VITS advances the state of expressive TTS in the audiobook domain. By combining a robust text-predicted style embedding, context modeling over sentence windows, and a rigorously balanced VITS-style training objective, TACA-VITS achieves high-fidelity, expressive, and contextually coherent speech synthesis without explicit reference data or style annotation requirements. A plausible implication is that these mechanisms could generalize well across languages and genres, especially for long-form, multi-style synthesizer deployments [2406.05672].

Source: https://www.emergentmind.com/topics/taca-vits