MELA-TTS: End-to-End Continuous TTS System

• MELA-TTS is an end-to-end text-to-speech system that directly generates continuous mel-spectrogram frames using a joint transformer-diffusion approach, eliminating multi-stage pipelines.
• Its representation alignment module integrates semantic embeddings from a pretrained ASR encoder, thereby accelerating training convergence and enhancing cross-modal coherence.
• The system supports unified offline and streaming synthesis with robust zero-shot voice cloning, achieving competitive WER and speaker similarity metrics.

MELA-TTS is an end-to-end text-to-speech system that directly autoregressively generates continuous mel-spectrogram frames from linguistic and speaker conditions through a joint transformer-diffusion framework, thereby eliminating speech tokenization and multi-stage processing pipelines. Its central technical distinction is a representation alignment module that aligns transformer-decoder outputs with semantic embeddings from a pretrained ASR encoder during training, with the stated effects of speeding up training convergence and enhancing cross-modal coherence between the textual and acoustic domains. The model is presented as a continuous-feature alternative to discrete-token TTS, while also supporting robust zero-shot voice cloning in both offline and streaming synthesis modes (An et al., 18 Sep 2025).

1. Conceptual position and motivation

MELA-TTS was introduced against two established design lines in modern TTS. The first is discrete-token-based TTS, exemplified in the paper by VALL-E, CosyVoice, and Seed-TTS, which typically follows a two-stage discrete pipeline: a tokenizer converts speech into discrete tokens, an autoregressive LLM predicts the next token, and a decoder reconstructs continuous audio. In the formulation given for MELA-TTS, these systems benefit from strong naturalness and zero-shot voice cloning, but are also associated with information loss, multi-stage complexity, and error cascading because speech tokenization coarsens continuous audio into discrete codes and requires pretrained intermediate components (An et al., 18 Sep 2025).

The second line is continuous end-to-end TTS, represented in the MELA-TTS discussion by Melle, DiTAR, and CLEAR. These systems aim to generate continuous mel-spectrograms or other continuous acoustic representations directly. The motivation for MELA-TTS is that such continuous pipelines preserve acoustic detail and simplify the pipeline, but remain difficult to optimize, converge slowly, and show a performance gap to the best discrete models, especially in content consistency as measured by WER or CER (An et al., 18 Sep 2025).

Within that setting, MELA-TTS is defined by three design goals. It combines a transformer decoder for autoregressive low-rate generation of continuous hidden vectors, a diffusion transformer for mel-spectrogram chunk generation, and a representation alignment mechanism that injects ASR-derived semantic supervision into the continuous pipeline. The model is also designed as a unified offline and streaming TTS system rather than separate architectures for the two modes (An et al., 18 Sep 2025).

2. Core architecture and signal representations

MELA-TTS operates on 24 kHz waveform audio and uses an 80-dimensional mel-spectrogram at 50 Hz, with window length 1920 and hop 480. The mel sequence is partitioned into chunks of size $N = 8$, so each chunk corresponds to 160 ms. At this granularity the transformer operates at a chunk rate of $50/8 = 6.25$ Hz, producing one hidden vector $h_i$ per chunk (An et al., 18 Sep 2025).

Text is tokenized by Qwen2 BPE and embedded with the Qwen2 text embedding layer. Conditioning is augmented by two speech-derived vectors. The speaker embedding $\mathbf{v}$ is extracted by a pretrained 3D-Speaker encoder, and the utterance embedding $\mathbf{u}$ is produced by a jointly trained transformer encoder applied to a random speech segment and pooled to a single vector. The utterance embedding is intended to encode utterance-level acoustic characteristics, including prosody and fine-grained speaker information (An et al., 18 Sep 2025).

The autoregressive backbone is a causal decoder initialized from Qwen2-0.5B. Previously generated mel chunks are downsampled by a strided convolution and projected into a single transformer-space vector, which is then consumed along with text tokens and conditioning vectors. The decoder outputs a hidden sequence $\mathbf{h} = [h_1, h_2, \dots]$, where each $h_i \in \mathbb{R}^{D_{\text{trans}}}$ conditions the acoustic generation for chunk $i$. Sequence termination is not modeled with EOS text tokens; instead, MELA-TTS uses a stop prediction module trained with binary cross-entropy over the hidden sequence (An et al., 18 Sep 2025).

The acoustic generator is a 22-layer diffusion transformer with hidden dimension 1024 and 16 attention heads. For chunk $i$, the condition set is

$$\Psi_i = \{[h_{i-1}, h_i], \mathbf{v}, \mathbf{u}\},$$

and the diffusion transformer denoises the current chunk while also receiving the previous clean chunk as prefix context:

$50/8 = 6.25$0

The paper emphasizes that the DiT operates directly on mel-spectrograms rather than latent codec tokens (An et al., 18 Sep 2025).

3. Representation alignment and semantic supervision

The representation alignment module is the defining training mechanism of MELA-TTS. Its stated motivation is that continuous end-to-end systems lack an explicit semantic intermediate comparable to supervised semantic tokens in discrete-token systems such as CosyVoice. Without such a constraint, transformer hidden states may be adequate as diffusion conditions yet remain weakly aligned to textual semantics, which can degrade content consistency and slow convergence (An et al., 18 Sep 2025).

To address this, MELA-TTS aligns the transformer hidden sequence $50/8 = 6.25$1 to semantic representations from the pretrained SenseVoice-Large ASR encoder. The ASR side uses 16 kHz audio and a 128-dimensional mel representation at 100 Hz, then downsamples by a factor of 4 to produce semantic embeddings $50/8 = 6.25$2 at 25 Hz. Because the transformer hidden sequence is generated at 6.25 Hz, MELA-TTS inserts a time alignment module that upsamples $50/8 = 6.25$3 by a factor of 4 to 25 Hz through a linear layer followed by reshape operations (An et al., 18 Sep 2025).

The alignment loss is expressed as a cosine-similarity term between the temporally aligned transformer states and ASR semantic embeddings:

$50/8 = 6.25$4

This is combined with the diffusion reconstruction loss and stop-prediction loss:

$50/8 = 6.25$5

The paper reports that aligning to mel-spectrograms directly was tested and performed substantially worse than aligning to ASR semantic embeddings, indicating that semantic targets are more effective supervision signals for the transformer hidden space (An et al., 18 Sep 2025).

The empirical ablation is unusually explicit. On LibriTTS, adding representation alignment alone improved WER from 6.3 to 5.3 and accelerated training by about $50/8 = 6.25$6, reaching similar WER in fewer than 30 epochs instead of 100. When utterance embedding was added together with representation alignment, WER improved further to 5.2 and speaker-similarity metrics increased to SS1 = 0.48 and SS2 = 0.58. By contrast, replacing ASR-semantic alignment with mel alignment degraded performance to WER = 6.7, SS1 = 0.41, and SS2 = 0.48 (An et al., 18 Sep 2025).

4. Training procedure, diffusion process, and streaming operation

MELA-TTS was trained in two regimes. LibriTTS, comprising 585 hours of English speech, is used for ablation studies, while a 170,000-hour in-house multilingual corpus is used for scaling experiments. The latter consists of 130k hours of Chinese, 30k hours of English, and 10k hours of other languages (An et al., 18 Sep 2025).

The diffusion process uses a variance-preserving schedule. For clean chunk $50/8 = 6.25$7, the forward process is

$50/8 = 6.25$8

with $50/8 = 6.25$9. Training minimizes an $h_i$0 reconstruction loss over mel chunks, and sampling uses DDIM with 10 function evaluations. The model also applies classifier-free guidance:

$h_i$1

with $h_i$2 (An et al., 18 Sep 2025).

Offline inference proceeds by embedding the full text, extracting $h_i$3 and $h_i$4 from prompt speech, autoregressively generating hidden vectors $h_i$5, and invoking the diffusion transformer chunk by chunk until the stop module predicts termination. The concatenated mel-spectrogram is then converted to waveform by a neural vocoder, although the vocoder implementation is not specified in the provided description (An et al., 18 Sep 2025).

Streaming synthesis reuses the same model rather than a separate streaming-specific network. The key mechanism is interleaving text tokens and mel tokens in an $h_i$6 ratio, with default $h_i$7. For every 4 text tokens received, the system generates 3 mel chunks, corresponding to 480 ms of speech. Training includes both non-interleaved sequences for offline behavior and interleaved sequences for streaming behavior. Two special tokens are introduced: a turn-of-speech token marking the end of text input, and filling tokens used as positional placeholders but excluded from prediction targets and loss computation. The reported result is that streaming WER and speaker similarity are almost identical to offline mode (An et al., 18 Sep 2025).

5. Evaluation results and comparative performance

MELA-TTS is evaluated with content-consistency metrics and speaker-similarity metrics. Chinese content is measured by CER using Paraformer, English content by WER using Whisper-large V3, and speaker similarity by SS1 and SS2, defined as cosine similarities from WavLM-TDNN and ERes2Net speaker verification systems, respectively (An et al., 18 Sep 2025).

On LibriTTS test-en, the baseline without streaming, utterance embedding, or representation alignment achieved WER = 6.3, SS1 = 0.46, and SS2 = 0.55. Adding utterance embedding alone improved speaker similarity to SS1 = 0.47 and SS2 = 0.57 while giving WER = 6.0. Adding representation alignment alone reduced WER to 5.3 with essentially unchanged SS1 and SS2. The combination of utterance embedding and representation alignment produced the best offline configuration, with WER = 5.2, SS1 = 0.48, and SS2 = 0.58. In streaming mode, the full system reached WER = 5.0 with SS1 and SS2 almost identical to offline values (An et al., 18 Sep 2025).

On the 170k-hour seed-tts-eval benchmark, the aligned offline model reported test-zh CER = 0.9, SS1 = 0.72, SS2 = 0.77; test-en WER = 2.4, SS1 = 0.59, SS2 = 0.68; and test-hard CER = 7.6, SS1 = 0.71, SS2 = 0.76. Removing representation alignment degraded the same model to test-zh CER = 1.2, test-en WER = 4.0, and test-hard CER = 10.9, which the paper summarizes as about 25–40% relative CER/WER reduction from the alignment mechanism. Streaming with representation alignment remained very close to offline, reporting test-zh CER = 0.9, SS1 = 0.72, SS2 = 0.78; test-en WER = 2.5; and test-hard CER = 7.7, SS1 = 0.71, SS2 = 0.77 (An et al., 18 Sep 2025).

In the baseline comparison, MELA-TTS attained the best listed Chinese CER on test-zh, outperforming Human at 1.3, CosyVoice 3.0-0.5B at 1.3, CosyVoice 2.0 at 1.5, F5-TTS at 1.6, and MaskGCT at 2.3. On test-en, DiTAR reported a lower WER of 1.7 versus MELA-TTS at 2.4, while F5-TTS reported 1.8. On test-hard, MELA-TTS was reported as similar to CosyVoice and non-autoregressive baselines and clearly improved over its own no-alignment variant. The paper also notes that speaker similarity remains somewhat below CosyVoice on English, and attributes this to the diffusion module using only local context—current and previous chunks—whereas CosyVoice’s decoder can use the entire token sequence plus prompt speech globally (An et al., 18 Sep 2025).

6. Relation to adjacent TTS paradigms

MELA-TTS belongs to the family of mel-spectrogram-centered TTS systems, but its specific configuration differs from several neighboring paradigms. FELLE, for example, is an autoregressive continuous-mel model that integrates language modeling with token-wise flow matching, uses a previous-frame-centered dynamic prior, and introduces a coarse-to-fine hierarchy for mel-frame generation. On LibriSpeech continuation and cross-sentence tasks, FELLE preserved MELLE-level intelligibility while improving similarity metrics, illustrating a different route to continuous mel modeling through token-wise flow matching rather than a joint transformer-diffusion chunk generator (Wang et al., 16 Feb 2025).

M3-TTS occupies a non-autoregressive mel-latent design space. It uses a multi-modal diffusion transformer for text-speech alignment, a Mel-VAE codec for approximately $h_i$8 temporal and $h_i$9 channel compression, and reports a $\mathbf{v}$0 training acceleration from the codec. Its alignment mechanism is implicit in joint text-speech attention, without duration models or pseudo-alignment, whereas MELA-TTS retains an autoregressive transformer and adds explicit representation alignment to ASR semantics (Wang et al., 4 Dec 2025).

Other nearby systems differ in their treatment of discreteness. MELD introduces discrete latent variables over mel spectrograms and jointly optimizes the mel encoder and speech LLM; dMel discretizes mel-filterbank channels into intensity bins through a training-free representation; and R-MelNet predicts low-resolution mel features that are interpolated and decoded by a WaveRNN-style backend. These systems show that mel representations can support TTS without neural codec tokenizers, but they do so through discrete latent variables, training-free discretization, or heavily reduced mel resolution rather than the continuous chunk-level diffusion formulation of MELA-TTS (Yeh et al., 28 May 2026, Bai et al., 2024, Kastner et al., 2022).

A separate line, Meta-TTS, addresses few-shot speaker personalization through Model-Agnostic Meta-Learning on a multi-speaker FastSpeech 2 architecture. That approach is centered on rapid speaker adaptation with 10–20 gradient steps, rather than direct continuous mel autoregression with zero-shot prompt conditioning. This suggests that MELA-TTS and meta-learning-based personalization address related but distinct problems: one emphasizes end-to-end continuous feature generation and semantic alignment, the other fast few-shot adaptation for unseen speakers (Huang et al., 2021).

7. Limitations and future directions

The limitations stated for MELA-TTS are primarily about similarity, compute, and dependence on external supervision. Speaker similarity is described as slightly inferior to the best discrete-token systems on some English benchmarks. Computationally, the combination of a 0.5B transformer decoder, a 22-layer DiT, and DDIM sampling with 10 function evaluations remains substantial at inference time. The model also depends on a pretrained ASR encoder, SenseVoice-Large, for its alignment mechanism, so the quality and language coverage of that ASR encoder influence TTS training (An et al., 18 Sep 2025).

The paper further notes that robustness to extremely long or complex utterances remains an open question, despite evaluation on a hard benchmark split. The authors identify several directions for future work: improving voice cloning by leveraging more global context inside the diffusion component, extending the framework to broader audio domains such as music and sound effects, and further optimizing latency and efficiency through fewer diffusion steps, lighter DiT variants, or hybrid architectures (An et al., 18 Sep 2025).

Taken together, these features position MELA-TTS as a continuous end-to-end TTS architecture that attempts to recover the semantic discipline of token-based systems without adopting speech tokenization itself. Its defining claim is not merely direct mel generation, but the combination of chunk-level autoregression, diffusion-based acoustic realization, and ASR-guided representation alignment, with empirical evidence that this combination materially improves content consistency while preserving zero-shot voice cloning and unified offline-streaming operation (An et al., 18 Sep 2025).