MahaTTS-v2: Multilingual TTS for Indic Languages

• MahaTTS-v2 is a multilingual text-to-speech system that uses a two-stage modular design for high-quality, expressive speech synthesis.
• The system features a text-to-semantics model paired with a conditional flow-based semantics-to-mel converter and BigVGAN vocoder for 24 kHz waveform generation.
• Trained on over 20K hours in 22 languages, it outperforms traditional models in intelligibility, especially for Indic languages.

Searching arXiv for the specified paper and closely related work to ground the article. MahaTTS-v2 is a multilingual multi-speaker text-to-speech system introduced in "MahaTTS: A Unified Framework for Multilingual Text-to-Speech Synthesis" (Singh et al., 5 Aug 2025). It is presented as a response to a multilingual TTS landscape in which most systems traditionally focus on English and European languages, thereby limiting broader access to information. The system is described as having excellent multilingual expressive capabilities in Indic languages and is trained on around 20K hours of data specifically focused on Indian languages. Architecturally, it is a two-stage modular pipeline: a text-to-semantics model maps Unicode text and language identity to discrete semantic tokens, and a semantics-to-mel model followed by a BigVGAN vocoder produces 24 kHz speech waveforms (Singh et al., 5 Aug 2025).

1. Problem setting and system scope

MahaTTS-v2 is framed as a multilingual TTS system designed to address undercoverage of Indian and other non-English languages in existing speech synthesis models (Singh et al., 5 Aug 2025). The core claim is not merely multilinguality in the abstract, but multilingual expressive synthesis with explicit emphasis on Indic languages and multi-speaker generation.

The system is modular rather than end-to-end joint-trained. Stage M1 performs text-to-semantics mapping, while Stage M2 performs semantics-to-mel conversion and then waveform generation. No joint training between M1 and M2 is required; they can be trained and released independently. At inference time, the stated pipeline is text $\rightarrow$ M1 $\rightarrow$ semantic tokens $\rightarrow$ M2-Flow $\rightarrow$ mel $\rightarrow$ BigVGAN $\rightarrow$ waveform (Singh et al., 5 Aug 2025).

This decomposition is central to how the paper positions MahaTTS-v2. The semantic interface isolates linguistic content in a discrete token space, while speaker information is supplied through embeddings derived from reference audio. A plausible implication is that the architecture separates content modeling from acoustic realization in a way that is convenient for multilingual and speaker-adaptive deployment, although the paper’s concrete claim is the independence of training and release for the two stages.

2. Modular architecture and parameterization

The overall system comprises a decoder-only LLM for text-to-semantics generation, a Conditional Flow Model for semantic-to-mel synthesis, and a pretrained BigVGAN vocoder for waveform generation (Singh et al., 5 Aug 2025).

Component	Function	Size
M1 “Gemma” decoder	Text $\rightarrow$ semantic tokens	$\approx 0.5$ B parameters
M2-Flow	Semantic tokens $\rightarrow$ mel-spectrogram	$\approx 300$ M parameters
BigVGAN vocoder	Mel-spectrogram $\rightarrow$0 24 kHz waveform	$\rightarrow$1 M parameters

The M1 model is a decoder-only “Gemma” transformer with alternating self-attn and FFN layers and Rotary positional encodings (RoPE). Its inputs are UTF-encoded text tokens, learned language embeddings, and learned speaker embeddings. The text vocabulary is reported as $\rightarrow$2 characters. The output space is a joint vocabulary of size

$\rightarrow$3

where the additional semantic symbols come from W2V2-kmeans plus an end-of-semantic marker (Singh et al., 5 Aug 2025).

The M2 component is a conditional continuous normalizing-flow matching model with alternating layers of 1D convs and multi-head attention with residual connections. It is conditioned on the semantic token embedding sequence and a 1024-D speaker embedding. The paper states that the total system size excluding BigVGAN is approximately 800 M parameters, and that M1+M2 inference is real-time or faster on a single A100, while vocoder speed depends on BigVGAN at approximately $\rightarrow$4–$\rightarrow$5 real-time (Singh et al., 5 Aug 2025).

3. Semantic representation and text-to-semantics modeling

Semantic extraction uses a pretrained XLSR-1B Wav2Vec2.0 encoder trained on 436 K h of unlabeled speech across 128 languages (Singh et al., 5 Aug 2025). Frame-level continuous embeddings are extracted and then quantized through k-means clustering with $\rightarrow$6. The algorithmic steps are stated explicitly:

1. Given raw audio $\rightarrow$7, extract frame embeddings $\rightarrow$8.
2. Fit k-means on all collected $\rightarrow$9, yielding centroids $\rightarrow$0.
3. Quantize each frame by

$\rightarrow$1

1. Collapse or downsample to approximately 25 tokens per second.

The resulting token inventory is described as a vocabulary of 10,000 semantic units that represent purely linguistic content, specifically phonetic and intonational information, divorced from speaker identity (Singh et al., 5 Aug 2025).

The text-to-semantics stage treats synthesis as an autoregressive language-modeling problem with two classification heads and a weighted cross-entropy loss. The paper gives the objective as

$\rightarrow$2

with $\rightarrow$3 and $\rightarrow$4 (Singh et al., 5 Aug 2025). A compact version provided in the implementation summary is

$\rightarrow$5

Speaker embeddings are 1024-D and are computed by averaging a small set of reference mel spectrograms passed through a speaker encoder. At inference, text and optional reference audio clips are provided, the speaker embedding is computed, and semantic tokens are generated by greedy or nucleus sampling until an end-token $\rightarrow$6 is produced (Singh et al., 5 Aug 2025).

4. Conditional flow matching and waveform synthesis

The second stage maps semantic tokens to mel-spectrograms by means of a Conditional Flow Model. The paper defines the training setup in terms of a real mel-spectrogram sample $\rightarrow$7, a noise sample $\rightarrow$8, and a linear interpolation path

$\rightarrow$9

The associated vector field is

$\rightarrow$0

The model predicts a conditioned vector field $\rightarrow$1, where conditioning is on the semantic token sequence $\rightarrow$2 and a reference speaker embedding. The flow-matching loss is

$\rightarrow$3

The text explains this as learning to steer noise $\rightarrow$4 to real data $\rightarrow$5 along straight geodesics in mel-space, conditioned on semantics and speaker identity (Singh et al., 5 Aug 2025).

At inference, the model samples $\rightarrow$6 and integrates the learned ODE

$\rightarrow$7

from $\rightarrow$8 to $\rightarrow$9, yielding a synthetic mel-spectrogram approximating the real data distribution. The M2 model is trained on 10 s segments but is stated to generalize up to 30 s via relative positional biases (Singh et al., 5 Aug 2025).

Waveform synthesis is performed by a pretrained BigVGAN vocoder with approximately 281 M parameters and multi-resolution spectrogram discriminators. BigVGAN is frozen during training and invoked only at inference time. Its role is strictly post-acoustic: after M2-Flow produces $\rightarrow$0, BigVGAN converts it to a 24 kHz waveform (Singh et al., 5 Aug 2025).

5. Data, preprocessing, and optimization

The training corpus totals approximately 20,630 h of speech in 22 languages (Singh et al., 5 Aug 2025). The major language shares reported in the paper are English 11 K h (58%), Hindi 2.3 K h (11%), German 1.5 K h (7%), and French 1 K h (5%). The paper also mentions an additional 100 h proprietary set in 15 languages, and states that future work has 150 K h in 33 languages.

The preprocessing pipeline is specified as a six-step sequence:

1. VAD segments audio into 1–30 s clips and normalizes the distribution via recursive splitting.
2. A speech-quality classifier discards noisy or multi-speaker samples.
3. A dual-ASR transcription consistency check keeps only utterances where ASR$\rightarrow$1 and ASR$\rightarrow$2 transcripts differ by less than a threshold.
4. Text normalization applies language-specific rules, including examples such as “nuktas” in Hindi, along with punctuation and whitespace processing.
5. Wav2Vec2.0 followed by k-means generates semantic tokens.
6. Text tokenization maps text into a character set of size 4,893 (Singh et al., 5 Aug 2025).

The M1 model is trained on data augmented from 20 K h to 50 K h to cover up to 30 s context. The reported hardware is 8$\rightarrow$3A100 40 GB with bfloat16 precision, batch size 1,024 tokens using mixed lengths via bucketing, AdamW with learning rate $\rightarrow$4 and weight decay $\rightarrow$5, over 7 epochs (Singh et al., 5 Aug 2025). For speaker adaptation, the paper states that best stability was achieved by freezing both classification heads when adapting to a new speaker.

The M2-Flow model is trained on a 1 K h subset across all languages, using 4$\rightarrow$6A5000, in 32-bit or bf16, batch size 1,024 on 10 s segments, AdamW with learning rate $\rightarrow$7 and weight decay $\rightarrow$8, for approximately 700 K iterations (Singh et al., 5 Aug 2025). Fine-tuning on a new speaker is reported to require approximately 10 K steps with batch size 64.

6. Evaluation, reported performance, and implementation notes

Evaluation uses Word-Error Rate on 10 sentences per language, with a commercial ASR transcribing synthesized speech; lower WER is better (Singh et al., 5 Aug 2025). The paper reports that across 22 languages, MahaTTS-v2 significantly outperforms existing multilingual TTS systems in intelligibility. Selected results from Table 2 are reported as follows.

Language	Reported WER comparison
Assamese	MahaTTS-v2 = 42% WER vs Indic-Parler-TTS = 80%
Bengali	7% vs 39%
English	2% vs 2% (tie)
German	15% vs 75%
Odia	73% vs 40% vs other baselines

These figures are important for interpreting the system’s claims. The aggregate statement is that MahaTTS-v2 significantly outperforms existing multilingual TTS systems in intelligibility across 22 languages, but the sample results also show that performance is not uniformly better on every listed language, with the Odia example reported as 73% versus 40% for Indic-Parler-TTS (Singh et al., 5 Aug 2025). Subjective listening tests, although not detailed numerically in the paper, are said to confirm high cross-lingual expressiveness and naturalness.

The implementation notes state that the code and models are open-source and permit research and commercial use (Singh et al., 5 Aug 2025). Two repository strings appear in the supplied paper description: the abstract gives https://github.com/dubverse-ai/MahaTTSv2, while the implementation details list https://github.com/dubverse-ai/MahaTTSU2. The paper description does not resolve this discrepancy.