Inworld TTS-1: Unified Transformer TTS Models

• Inworld TTS-1 is a family of two autoregressive Transformer-based TTS models (1.6B and 8.8B parameters) designed for real-time and high-quality voice synthesis.
• It employs a unified codec–SpeechLM architecture that merges text and audio tokens, enabling zero-shot voice cloning and precise expressive control.
• The training pipeline integrates pre-training, supervised fine-tuning, and RL alignment to achieve state-of-the-art multilingual intelligibility and low-latency streaming.

Inworld TTS-1 is a family of two Transformer-based autoregressive text-to-speech models centered on a speech-LLM, or SpeechLM, that operates over a joint text-and-audio token space. The family comprises TTS-1, a 1.6B-parameter model positioned for real-time speech synthesis and on-device use cases, and TTS-1-Max, an 8.8B-parameter model positioned for maximum quality and expressiveness. Both models synthesize 48 kHz speech with low latency, support 11 languages, perform zero-shot voice cloning through in-context learning from short reference audio and transcript pairs, and expose fine-grained emotional and non-verbal control through textual audio markups (Atamanenko et al., 22 Jul 2025).

1. Model family and design objectives

The technical report defines two variants with a common architectural template but different scale. TTS-1 uses a LLaMA-3.2-1B SpeechLM plus codec components for a total of 1.6B parameters, while TTS-1-Max uses a LLaMA-3.1-8B SpeechLM plus codec components for a total of 8.8B parameters (Atamanenko et al., 22 Jul 2025).

Model	Parameters	Stated role
TTS-1	1.6B	real-time speech synthesis and on-device use cases
TTS-1-Max	8.8B	utmost quality and expressiveness in demanding applications

The report frames these models against several limitations attributed to contemporary TTS systems: lack of 48 kHz high-fidelity output, weak multilingual support and cross-lingual voice cloning, artifacts and instability in streaming or low-latency generation, limited expressive control, and reliance on explicit speaker embedding models or long voice-enrollment procedures. Its stated goals are correspondingly broad: high quality and naturalness, fine-grained emotional and non-verbal control, robust multilingual and cross-lingual behavior, low latency and streaming, in-context voice learning without a dedicated speaker encoder, and an explicit quality-versus-efficiency scaling path between TTS-1 and TTS-1-Max (Atamanenko et al., 22 Jul 2025).

Within the 2025 TTS landscape, this places Inworld TTS-1 in a distinct niche. FireRedTTS-1S exemplifies a streaming foundation TTS system based on text-to-semantic and semantic-to-acoustic decoding with latency below 150 ms (Guo et al., 26 Mar 2025). RWKVTTS instead argues for replacing the transformer language-model backbone of codec-based TTS with RWKV-7 to obtain recurrent-state efficiency while retaining near-transformer quality (Yueyu et al., 4 Apr 2025). Compact accessibility-oriented neural TTS demonstrates a much smaller three-stage on-device stack with latency on the order of 15 ms and low disk footprint (Jain et al., 28 Jan 2025). Inworld TTS-1 differs from all three by combining a large autoregressive SpeechLM, a unified codec token space, RL alignment, and markup-conditioned expressivity in a single production-oriented system (Atamanenko et al., 22 Jul 2025).

2. Unified codec–SpeechLM architecture

The report describes a three-part pipeline: an audio codec encoder/decoder, a SpeechLM, and a prompt-and-generation flow. Reference audio is first converted into discrete codec tokens. Text tokens, reference-audio tokens, and special control tokens are then concatenated into a single sequence for the SpeechLM, which autoregressively generates new audio tokens; these are finally decoded back into waveform by the codec decoder (Atamanenko et al., 22 Jul 2025).

A central architectural decision is the use of a unified codec rather than a separate semantic-plus-acoustic token hierarchy. The codec is based on X-codec2 with substantial modifications for 48 kHz synthesis. It uses a single discrete codebook of size 65,536 at 50 tokens per second, and the token stream merges acoustic and semantic information into one code stream. The encoder operates on 16 kHz input with hop length 320 and has approximately 55M parameters. The decoder is built from a ResNet-plus-Transformer backbone predicting acoustic features followed by iSTFT, and it is extended with a super-resolution module formed by strided 1D transposed convolutions interleaved with ResNet blocks. The 48 kHz decoder variant uses strides $(3, 2)$, hop length 160, and approximately 193.43M parameters. On the English subset of Seed-TTS evaluation, the reported DNSMOS values are 4.110 for 16 kHz, 4.178 for 24 kHz, and 4.195 for 48 kHz, with the 48 kHz model described as providing better high-frequency detail (Atamanenko et al., 22 Jul 2025).

The SpeechLM extends the original LLaMA vocabulary of 128,256 tokens to 193,856 tokens by adding 65,536 audio tokens and 29 special tokens, then padding to a multiple of 32 for Tensor Core efficiency. New embeddings are initialized by sampling from a Gaussian with the mean and covariance of the existing embedding matrix. Both model variants are trained with sequence length 2048 and use FlashAttention-2 (Atamanenko et al., 22 Jul 2025).

The generation interface is built around in-context continuation rather than explicit speaker embeddings. The model sees a sequence of reference text, a speech-start marker, reference audio tokens, a speech-end marker, new text to be synthesized, and a final speech-start marker. When decoding reaches that final marker, it generates new audio tokens that continue the prompt voice, prosody, and style. The report emphasizes that there is no explicit speaker encoder: speaker timbre and style are implicitly encoded in the prompt codec tokens, and the SpeechLM learns to continue them autoregressively (Atamanenko et al., 22 Jul 2025).

This unified-token design contrasts with multi-stage semantic-token systems such as FireRedTTS-1S and with transducer-based semantic-token prediction systems such as "Transduce and Speak" (Guo et al., 26 Mar 2025, Kim et al., 2023). A plausible implication is that Inworld TTS-1 trades some architectural decomposition for a closer fit to LLM training, alignment, and serving infrastructure.

3. Pre-training, supervised fine-tuning, and RL alignment

The training recipe is explicitly sequential: pre-training, supervised fine-tuning, and RL alignment. The pre-training set contains approximately 1,000,000 hours of raw speech audio, approximately 30,000 hours of non-speech audio, an initial approximately 15,000-hour high-quality fine-tuning subset to bootstrap text-speech alignment, and approximately 20B text tokens from filtered RedPajama-v2 and LAION OIG instructions. The supervised fine-tuning set contains approximately 200,000 hours of high-quality audio-text pairs, filtered by DNSMOS, characters-per-second per language, and text heuristics (Atamanenko et al., 22 Jul 2025).

SpeechLM pre-training is cast as pure next-token prediction over mixed speech-and-text sequences. Raw audio is segmented into clips of at most 40 seconds and wrapped by <|#1|>{speech_start} and <|#1|>{speech_end} tokens. The optimization target is the standard language-model objective

$$\min \; \mathbb{E}\left[-\sum_t \log P(z_t \mid z_{<t})\right],$$

where $z_t$ ranges over both text and audio tokens (Atamanenko et al., 22 Jul 2025).

Supervised fine-tuning then specializes the model to text-to-speech by teacher forcing on audio tokens. With text tokens $x_1,\ldots,x_T$ and audio tokens $y_1,\ldots,y_S$, the report gives

$$\mathcal{L}_{\text{SFT}} = - \sum_{s=1}^{S} \log P(y_s \mid x_1, \ldots, x_T, y_1, \ldots, y_{s-1}).$$

The input sequence for each example is

$\big[ \texttt{<|#1|>\{begin\_of\_text\}}, x_1, \ldots, x_T, \texttt{<|#1|>\{speech\_start\}}, y_1, \ldots, y_S, \texttt{<|#1|>\{speech\_end\}} \big].$

The report notes that adding text-only instruction data during SFT degraded synthesis quality, with the model sometimes stopping speech generation, so the final recipe keeps instruction tuning in pre-training and uses speech-focused SFT (Atamanenko et al., 22 Jul 2025).

The final stage applies Group Relative Policy Optimization (GRPO) to align the SpeechLM with intelligibility, similarity, and perceptual-quality preferences. For a query $q$, a group of $G$ candidate responses $\{o_1,\dots,o_G\}$ is sampled; if reward values are $$\min \; \mathbb{E}\left[-\sum_t \log P(z_t \mid z_{<t})\right],$$0, the advantage is mean-centered rather than variance-scaled:

$$\min \; \mathbb{E}\left[-\sum_t \log P(z_t \mid z_{<t})\right],$$1

The total reward is

$$\min \; \mathbb{E}\left[-\sum_t \log P(z_t \mid z_{<t})\right],$$2

with all components normalized to $$\min \; \mathbb{E}\left[-\sum_t \log P(z_t \mid z_{<t})\right],$$3. The component definitions are:

$$\min \; \mathbb{E}\left[-\sum_t \log P(z_t \mid z_{<t})\right],$$4

$$\min \; \mathbb{E}\left[-\sum_t \log P(z_t \mid z_{<t})\right],$$5

$$\min \; \mathbb{E}\left[-\sum_t \log P(z_t \mid z_{<t})\right],$$6

On a 1,000-hour English fine-tuning subset, the report states that single-signal GRPO improves its target metric strongly but may compromise others, whereas the combined reward model yields balanced improvements across all three metrics and outperforms the SFT-only baseline (Atamanenko et al., 22 Jul 2025).

The hardware profile is also documented. Pre-training and SFT required approximately two months on four nodes of eight H100 GPUs each, while codec training, RL, style LoRA, and evaluations required approximately three additional months on two nodes of eight A100 GPUs (Atamanenko et al., 22 Jul 2025).

4. In-context voice learning, multilinguality, and markup-conditioned expressivity

Inworld TTS-1 treats voice cloning as an in-context continuation problem rather than a speaker-classification or speaker-embedding problem. A short reference audio clip is encoded into codec tokens, its transcript is concatenated in the prompt, and the model continues with new audio tokens for new text. The report describes this as zero-shot “instant voice cloning” from a short reference audio and transcript, and it explicitly contrasts the method with systems that require a dedicated speaker encoder. For production settings, it also describes optional professional voice cloning through LoRA fine-tuning of the SpeechLM for a specific speaker (Atamanenko et al., 22 Jul 2025).

Multilingual support is built into both training and evaluation. The models support 11 languages, are pre-trained on multilingual speech and text, and share a single SpeechLM across languages with the same audio-token space. The report states that TTS-1-Max consistently outperforms TTS-1 on both WER/CER and speaker similarity across all languages. Cross-lingual voice transfer is described as implicit: the same speaker prompt in one language can be used to synthesize another language, because speaker identity is represented in audio tokens while language is carried by text tokens (Atamanenko et al., 22 Jul 2025).

Expressive control is mediated by audio markups, textual tags that become ordinary conditioning tokens for the SpeechLM. The style-tag inventory is [angry], [disgusted], [fearful], [happy], [laughing], [sad], [surprised], and [whispering]. The non-verbal tag inventory is [breathe], [clear_throat], [cough], [cry], [laugh], [sigh], and [yawn]. These tags are inserted directly into the text stream (Atamanenko et al., 22 Jul 2025).

The report emphasizes that simple prepending of a style tag to a single utterance did not work because the codec entangles timbre and style. To make style controllable, the authors constructed a paired neutral-plus-stylized dataset from the same speaker by concatenating transcripts as neutral_transcript [style_tag] stylized_transcript and concatenating the corresponding audio with 0.5 to 1.5 seconds of silence. The style dataset contains approximately 100,000 English examples, approximately 180 hours, and approximately 340 unique speakers; approximately 20% of samples include non-verbals, and approximately 30% remain pure neutral samples to preserve baseline quality (Atamanenko et al., 22 Jul 2025).

Style learning is implemented with LoRA rather than full-model fine-tuning. For TTS-1, LoRA uses rank 16, alpha 32, dropout 0.2, and is applied to Q, K, V, O, and MLP weights. For TTS-1-Max, the same LoRA hyperparameters are used but the adapters are applied to Q, K, V, and O only. The report states that LoRA produced better generalization, measured by lower validation loss, than full-model fine-tuning. Human evaluations were used to select checkpoints, and the authors report clear audible tag obedience, some cross-lingual generalization with reduced fidelity, and useful multi-tag combinations such as [sad] [whispering] (Atamanenko et al., 22 Jul 2025).

This markup-centered control mechanism differs from the cue-specific duration intervention proposed for L2-tailored clear TTS, where the control variable is a per-phoneme duration multiplier rather than a discrete style tag (Tuttösí et al., 29 Jun 2025). A plausible implication is that Inworld TTS-1 is optimized for broad expressive coverage rather than for narrowly targeted intelligibility interventions of that kind.

5. Streaming synthesis, serving stack, and deployment

Both TTS-1 and TTS-1-Max support streaming generation. The report describes a four-step streaming loop: the SpeechLM generates audio tokens autoregressively, tokens accumulate in a buffer, the buffer is decoded into waveform segments, and the resulting segments are concatenated and streamed to the listener. Three engineering problems are emphasized: seamless concatenation without pops or clicks, volume consistency across segments, and preservation of speaker identity in early segments (Atamanenko et al., 22 Jul 2025).

To reduce boundary artifacts, the decoder searches for a non-voicing region in each decoded segment $$\min \; \mathbb{E}\left[-\sum_t \log P(z_t \mid z_{<t})\right],$$7 around a candidate cut point $$\min \; \mathbb{E}\left[-\sum_t \log P(z_t \mid z_{<t})\right],$$8 and radius $$\min \; \mathbb{E}\left[-\sum_t \log P(z_t \mid z_{<t})\right],$$9:

$z_t$0

If such a region exists, audio up to $z_t$1 is emitted and the tail is deferred; if not, the entire segment is deferred to the next chunk. To stabilize loudness, the decoder is given an extended token context corresponding to $z_t$2, after which the prefix context is discarded. The report states that audio decoding is much faster than token generation, so this extended-context decoding has negligible latency cost but significantly improves loudness consistency (Atamanenko et al., 22 Jul 2025).

The decoder can also condition on prompt audio context. On Seed-TTS evaluation, decoding generated tokens without prompt context yields speaker-similarity score $z_t$3, while decoding with prompt context yields $z_t$4. The stated interpretation is that the decoder better matches prompt timbre when it sees prompt waveform context alongside generated tokens (Atamanenko et al., 22 Jul 2025).

Serving efficiency is improved through several system-level optimizations developed with Modular: a multi-step scheduler that keeps token generation on the GPU without CPU round-trips, a batched audio decoder with padded batched inputs and custom kernels, GPU-native penalty sampling implemented with Mojo kernels, and a graph-compiler pipeline that performs kernel fusion and memory planning. The report states that the P90 latency for the first 2-second audio chunk is approximately 70% lower than a standard vLLM-based implementation (Atamanenko et al., 22 Jul 2025).

Deployment trade-offs remain scale-dependent. TTS-1 is the smaller model and is described as better suited for real-time and on-device deployment; TTS-1-Max provides higher quality but lower serving efficiency. The report also notes several operational constraints: caching prompt tokens can cause style bleeding, short reference audio limits reliable maximum generation length, and temperature and top-$z_t$5 choices mediate a familiar trade-off between speaker similarity and expressiveness (Atamanenko et al., 22 Jul 2025).

The codebase is open-sourced under an MIT license at https://github.com/inworld-ai/tts, covering training, modeling, benchmarking, and examples. At the same time, model weights are not publicly released due to voice-cloning risk. Production deployments apply inaudible watermarking, and users must confirm that they have rights to any voice they clone (Atamanenko et al., 22 Jul 2025).

6. Benchmarks, limitations, and position within current TTS research

Evaluation in the report spans multilingual intelligibility and similarity, input-length robustness, internal human preference tests, and human assessment of emotional and non-verbal realization. On the multilingual benchmark of 100 sentences per language for 11 languages, both models achieve high speaker similarity, and TTS-1-Max consistently outperforms TTS-1 on both WER/CER and speaker similarity (Atamanenko et al., 22 Jul 2025).

On English input-length robustness, the report uses three corpora: short (208 sentences, average 7 characters), medium (150 sentences, average 87 characters), and long (517 sentences, average 153 characters). For TTS-1, the SFT-only model yields WER values of 11.3, 4.0, and 8.5, with 7.9 total, while the RL-aligned model yields 9.6, 2.3, and 7.0, with 6.3 total. For TTS-1-Max, the SFT-only model yields 10.1, 3.3, and 7.3, with 6.9 total, while the RL-aligned model yields 8.2, 1.9, and 5.2, with 5.1 total. These numbers are used in the report to argue that RL alignment improves intelligibility across input lengths for both scales (Atamanenko et al., 22 Jul 2025).

In the internal TTS Arena, approximately 20 annotators produced more than 400 pairwise votes across Inworld TTS-1, Inworld TTS-1-Max, 11labs Multilingual V2, Cartesia Sonic 2, and OpenAI TTS-1-HD. The reported head-to-head win rates are 56.0% for Inworld TTS-1 versus 11labs, 63.2% versus Cartesia, and 56.1% versus OpenAI TTS-1-HD. For TTS-1-Max, the corresponding win rates are 59.1%, 60.9%, and 60.7%, and TTS-1-Max is preferred over TTS-1 at 55.3% (Atamanenko et al., 22 Jul 2025).

The report nevertheless documents several limitations. Prompt style bleed can occur when prompt tokens are cached for latency reduction. Short reference audio does not fully constrain long continuations. Higher temperature increases expressiveness but also variance and artifacts. Language imbalance affects quality across languages, and style tags are less effective in underrepresented languages (Atamanenko et al., 22 Jul 2025).

Relative to adjacent TTS research, Inworld TTS-1 occupies a particular operating point. FireRedTTS-1S reaches streaming generation with latency below 150 ms by separating text-to-semantic and semantic-to-acoustic decoding (Guo et al., 26 Mar 2025). RWKVTTS proposes a drop-in RWKV-7 replacement for the transformer LM inside a CosyVoice-style codec-token pipeline, arguing for constant-memory recurrent inference (Yueyu et al., 4 Apr 2025). SupertonicTTS pursues a radically smaller 44M-parameter, flow-matching latent architecture with character-level text and no G2P or external aligners (Kim et al., 29 Mar 2025). Compact neural TTS for accessibility prioritizes 13 ms end-to-end latency and an 18 MB footprint on recent mobile hardware (Jain et al., 28 Jan 2025). Inworld TTS-1 instead concentrates on large-scale autoregressive modeling, unified codec tokens, RL alignment, and markup-conditioned expressivity at 1.6B and 8.8B scales (Atamanenko et al., 22 Jul 2025).

This comparison suggests a broader taxonomy of current TTS systems. Some systems optimize for extreme serving efficiency or on-device compactness; some decompose generation into semantic and acoustic stages for streamability; some replace transformer attention with recurrent-state models; and some, including Inworld TTS-1, import large-language-model training and alignment methods directly into TTS. In that sense, Inworld TTS-1 is best understood as a codec-language-model TTS system that extends the LLM paradigm into high-resolution, multilingual, markup-controllable, and streaming speech synthesis (Atamanenko et al., 22 Jul 2025).