---
title: Units-to-Speech (U2S) Overview
url: https://www.emergentmind.com/topics/units-to-speech-u2s
type: topic
---

# Units-to-Speech (U2S) Overview

A Units-to-Speech (U2S) system converts symbolic, pseudo-linguistic or acoustic “units” into an intelligible, natural-sounding speech waveform. In contemporary research, these units are typically discrete symbols—obtained without text supervision—encoding phonetic, subword, or mid-level acoustic information derived from raw speech or, in specialized settings, from alternative sensory modalities. U2S is a critical component in textless speech-to-speech and multimodal generative systems, enabling modalities such as direct speech-to-speech translation, silent speech interfaces, and TTS without intermediate text. The following sections survey major classes of U2S system, their mathematical underpinnings, representation choices, generator/vocoder architectures, evaluation regimes, and the principal technical and empirical insights motivating this research direction.

## 1. Discrete Unit Representations and Extraction

Most state-of-the-art U2S systems leverage discrete representations derived from self-supervised learning (SSL) models such as HuBERT, Wav2Vec2, mHuBERT, or XLS-R. These models project raw audio to high-dimensional latent features, from which discrete units are extracted by clustering or vector quantization:

- **K-means clustering on SSL features**: Frame-level hidden vectors $h_t$ (sampled every 20 ms) are clustered into $K$ centroids, yielding hard cluster assignments $u_t \in \{1,...,K\}$. Typical $K$ values range from 200 to 1024; optimal $K$ balances phonetic granularity and robustness to noise artifacts [2407.18332][2308.01831].
- **VQ-VAE and related quantizers**: Vector-Quantized Autoencoders and their variants directly learn codebooks and quantization assignments end-to-end. This is often used in zero-resource and unsupervised TTS pipelines [1904.11469].
- **Run-length encoding and deduplication**: To ensure compactness and temporal alignment, consecutive repeated unit indices are collapsed, optionally producing a parallel duration or boundary sequence [2211.06535][2312.15197].
- **Non-acoustic alternatives**: For silent-speech and multimodal pipelines, alternative unit sources include ultrasound tongue images (Ultra2Speech: mapping US images to formant trajectories) [2006.16367], or segmental units from visual grounding models [2012.15454].

These procedures yield a temporally aligned sequence of discrete units $u_1, \ldots, u_N$, which serves as the symbolic input to the U2S generator.

## 2. Architectural Patterns for Units-to-Speech Synthesis

U2S decoders fall broadly into three categories:

**a. Neural Vocoder Architectures**
- **Unit-based HiFi-GAN**: The dominant approach is to embed unit indices via a learned lookup $E(u_t)$ and feed these embeddings through a stack of upsampling (transposed-convolutional) blocks and multi-rate residual stacks, culminating with waveform synthesis. Discriminators (multi-period, multi-scale) enforce adversarial and perceptual alignment [2309.07478][2308.01831][2511.12690][2406.02733]. The generator loss combines adversarial, feature-matching, and STFT or mel-spectrogram reconstruction terms.
- **Tacotron and variants**: Some systems utilize attention-based sequence-to-sequence models (e.g., Tacotron-2), predicting mel-spectrograms from unit sequences before inversion via WaveGlow or HiFi-GAN. This pipeline is frequently used when explicit duration modeling or expressive prosody control is required [2012.15454][2211.06535].
- **Diffusion-based decoders**: Diffusion probabilistic models invert noise to spectrograms conditioned on unit encodings and speaker embeddings. UnitSpeech, for example, integrates a diffusion TTS backbone with a unit encoder, enabling adaptation and rapid voice conversion [2306.16083].

**b. Prosody and Expressivity Modeling**
- **Explicit duration, pitch, and energy prediction**: Several U2S systems use cascaded or parallel networks for estimating duration, F0, energy, and voicing for each unit, yielding richer prosodic control and better transferability of rhythm and style [2211.06535][2312.15197][2406.02733].
- **Expressivity encoders and style transfer**: FastSpeech2-style models, ECAPA-TDNN embeddings, and Feature-wise Linear Modulation (FiLM) layers enable conditioning on speaker or vocal style, even under domain-heterogeneous or noisy conditions [2406.02733].

**c. Specialized and Multimodal Decoding**
- **Formant-based synthesis**: For articulatory-to-acoustic mapping, systems such as Ultra2Speech predict vocal tract formant trajectories from non-acoustic input (e.g., ultrasound) and synthesize speech via classical formant-based synthesizers [2006.16367].
- **Audio-visual speech synthesis**: TransFace’s Unit2Lip module jointly generates both speech and synchronized lip movement frames from unit sequences, incorporating a bounded duration predictor and synchronization loss against lip-sync encodings [2312.15197].

## 3. Training, Objective Functions, and Data Regimes

U2S model training typically proceeds in distinct stages and employs specialized objective functions:

- **Adversarial and feature-matching losses**: HiFi-GAN-based decoders are trained with a weighted sum of adversarial, feature-matching, and STFT/mel losses, ensuring fidelity and naturalness across a range of speakers and languages [2309.07478][2407.18332][2511.12690].
- **Spectrogram and duration losses**: Tacotron-style pipelines are optimized with mean-squared error (MSE) on mel-spectrograms, cross-entropy for stop tokens, and duration prediction MSE [2012.15454][2211.06535].
- **Diffusion and alignment losses**: Diffusion models minimize direct score-matching objectives for noisy spectrogram restoration, with additional encoder-decoder alignment and monotonicity losses [2306.16083].
- **Prosody and expressivity**: Additional BCE and cross-entropy terms model pitch bin distributions, voicing, and energy, allowing style-transfer and robust rhythmic alignment [2211.06535][2312.15197][2406.02733].
- **Self-supervised distillation**: DINO-style cross-entropy distillation losses align noisy and clean expressivity embeddings via teacher-student architectures, improving robustness [2406.02733].

Transfer and fine-tuning protocols differ: some systems pre-train on large multi-speaker corpora and fine-tune for a new speaker or style with a single $<$unit, speech$>$ pair, whereas others employ synthetic parallel data to augment low-resource settings [2306.16083][2511.12690].

## 4. Evaluation Metrics and Empirical Findings

U2S quality is assessed via several complementary metrics:

| Metric              | Subject/Objective | Purpose                                      |
|---------------------|------------------|----------------------------------------------|
| Mean Opinion Score (MOS)    | Subjective         | Speech naturalness, fluidity                 |
| Character Error Rate (CER)  | Objective         | Intelligibility, ASR transcribability        |
| BLEU (ASR-bleu)             | Objective         | Translation/tts intelligibility between ASR hypothesis and reference |
| Speaker similarity (SMOS/SES/SIM) | Mixed           | Voice fidelity                              |
| Prosody metrics (AutoPCP, LSE-*)   | Mixed           | Rhythm, pitch, style transfer, audio-visual sync |

Empirical findings include:
- HiFi-GAN models conditioned on units achieve MOS up to 3.8 on reconstruction and 4.2+ in speaker conversion [2407.18332][2306.16083].
- For speech translation, optimal unit choices for U2S (maximizing MOS) often diverge from those optimizing BLEU or CER; thus, S2ST system design must trade off linguistic abstraction and acoustic fidelity [2407.18332].
- Diffusion decoders and explicit prosody networks have enabled radical improvements in voice similarity and style transfer with minimal adaptation data [2306.16083][2211.06535].
- Self-supervised distillation confers dramatic robustness to channel and environmental noise in expressive S2ST scenarios, outperforming conventional PRETSSEL under SNR degradation [2406.02733].
- Cascade and direct pipelines perform comparably in high-resource settings, but unit-based models often close the gap in low-resource regimes by decoupling the vocoder from the translation model [2309.07478][2511.12690].
- In cross-modal and multimodal settings (e.g., silent-speech, textless image-to-speech), learned units enable U2S synthesis without text or phonetic supervision at all [2012.15454][2006.16367].

## 5. U2S in Multilingual, Textless, and Multimodal Systems

U2S models are a foundational component in textless and multimodal generative pipelines:

- **Textless S2ST and TTST**: U2S enables direct mapping from speech or text in any input language to a discretized intermediate (pseudo-text), from which a single universal vocoder generates waveforms, greatly simplifying many-to-many translation [2308.01831][2309.07478][2505.15333].
- **Silent and visual speech interface**: Non-acoustic “units” such as tongue ultrasound (Ultra2Speech) or visual-codebook units (ResDAVEnet-VQ) extend U2S technology to domains where text or audio is absent [2006.16367][2012.15454].
- **Audio-visual generation**: Joint synthesis models such as TransFace’s Unit2Lip module perform perfectly isochronous length-matching for combined audio/lip-stream generation [2312.15197].
- **Prosody and voice conversion**: Unified pipelines integrate unit-based conditioning with one-shot adaptivity in prosody, speaker, and emotion, allowing rapid transfer across speaker, language, and style [2306.16083][2211.06535].
- **Expressivity under noise**: DINO-PRETSSEL demonstrates significant gains in noisy and real-world conditions, maintaining voice identity and expressivity even under severe degradation [2406.02733].

## 6. Methodological Insights, Limitations, and Tradeoffs

Key technical and empirical insights include:

- **Unit design is central**: The performance ceiling of U2S is set as much by unit extraction (choice of SSL model, clustering granularity, layer selection) as by the subsequent generator [2407.18332][2308.01831].
- **Decoupling translation and synthesis**: By using discrete units as a pivot representation, the same vocoder can serve direct speech-to-speech, text-to-speech, and even multimodal pipelines, supporting modularity and efficient adaptation [2309.07478][2505.15333].
- **Acoustic–linguistic tradeoff**: More detailed units (higher $K$, lower-layer features) typically improve resynthesis but may impair translation or linguistic abstraction; the converse holds for higher-layer, more abstract units [2407.18332].
- **Prosody modeling remains an open challenge**: While explicit modeling of duration, F0, and energy improves naturalness and expressivity, automatic transfer of prosody, especially under minimal supervision or in unseen styles, is still imperfect [2211.06535][2306.16083].
- **Synthetic data and pretraining regimes**: In low-resource language scenarios, pipeline augmentation using synthetic parallel data, pretrained multilingual encoders, and self-supervised adaptation substantially closes performance gaps to high-resource settings [2511.12690][2309.07478].
- **No universal optimum**: Empirical optimization of both acoustic and translation metrics is required; no single set of unit extraction parameters is optimal for all downstream purposes [2407.18332].

## 7. Historical Perspectives and Applications

U2S technology has evolved from early unsupervised TTS (e.g., ZR19: clustering plus Merlin, Tacotron, or WORLD/Griffin-Lim vocoders) [1904.11469] to present large-scale, multilingual S2ST systems using robust SSL encoders, adversarial/generative vocoders, and explicit prosody adversarial training [2308.01831][2306.16083][2211.06535][2406.02733]. Applications now include S2ST, expressive/voice-transfer TTS, textless image-to-speech, silent-speech and medical interfaces, and multimodal talking-head translation.

Comprehensive empirical studies and shared benchmark datasets (e.g., Zero Resource Challenge, CVSS, mExpresso, VoxPopuli, LJSpeech, VCTK) provide objective and subjective comparisons along MOS, CER, BLEU, SIM, AutoPCP, and lip-sync metrics.

Ongoing research addresses open challenges in optimizing unit selection, robust cross-lingual transfer, multimodal synthesis synchrony, and the joint design of unit extraction and U2S generative models. The U2S paradigm remains at the core of textless and modular speech generation architectures for both research and applied domains.

Source: https://www.emergentmind.com/topics/units-to-speech-u2s