---
title: Speech to Speech Synthesis Network (STSSN)
url: https://www.emergentmind.com/topics/speech-to-speech-synthesis-network-stssn
type: topic
---

# Speech to Speech Synthesis Network (STSSN)

Speech to Speech Synthesis Network (STSSN) denotes a class of systems that take speech-derived input and produce speech output, with the mapping designed to preserve some attributes of the input while altering or conditioning others. In the narrow formulation introduced for voice impersonation, STSSN receives a **source utterance** carrying linguistic content and a **target utterance** carrying speaker style, then generates a new utterance that preserves the source content while adopting the target’s voice, style, and mannerisms [2602.16721]. In broader usage suggested by related work, the same design pattern encompasses text-mediated cascades such as ASR $\rightarrow$ TTS, direct speech-to-speech voice conversion, and even articulatory-to-acoustic mappings in which the “source” representation is not acoustic but articulatory, as in ultrasound tongue imaging-based silent speech interfaces [2305.19130].

## 1. Definitions and task formulations

The central problem addressed by STSSN is **speech-to-speech style transfer**. In the formulation explicitly named STSSN, the task is to preserve, approximately, the **linguistic content** of a source utterance while adopting the **voice / style / mannerisms** of a target speaker [2602.16721]. In that work, “style transfer” primarily means **speaker identity** and gross voice characteristics such as timbre, pitch range, and accent or manner of articulation, rather than fine-grained prosody control [2602.16721].

A broader interpretation follows from related systems that implement speech-conditioned synthesis through intermediate representations. One important variant is the **text-mediated** pipeline: speech is transcribed by ASR, target identity is encoded separately, and a multispeaker TTS model reconstructs speech from text plus speaker representation [2602.16721]. Another variant uses a **continuous acoustic or articulatory front-end** instead of text. The ultrasound tongue imaging silent speech interface described as an articulatory-to-acoustic mapping learns
\[
f_{\theta}: \text{UTI image(s)} \rightarrow \mathbf{y} \in \mathbb{R}^{80},
\]
where $\mathbf{y}$ is an 80-dimensional mel-spectrogram frame, followed by WaveGlow waveform generation; this is described as analogous to an STSSN in which the usual source acoustic domain is replaced by an articulatory domain [2305.19130].

These formulations imply a spectrum of STSSN designs. At one end are systems that pass through explicit symbolic content, usually characters or phonemes. At the other end are direct mappings over acoustic, articulatory, or latent representations. This suggests that STSSN is best understood not as a single architecture but as a family of speech-conditioned generation systems organized around the decomposition of **content**, **speaker/style**, and **waveform synthesis**.

## 2. Canonical architectural patterns

A canonical STSSN architecture in the voice-impersonation setting is a cascade of three components: a **content encoder / ASR**, a **target speaker encoder**, and a **text-to-speech decoder** [2602.16721]. In that implementation, the ASR is a modified DeepSpeech-like model trained with CTC, the speaker encoder is an LSTM-based network producing a fixed 256-dimensional speaker/style vector, and the decoder is Tacotron2 conditioned on both text and speaker embedding, with a vocoder producing waveform from a 128-bin mel-spectrogram [2602.16721]. The resulting factorization is
\[
x_{\text{src}}(t)\xrightarrow{\text{ASR}} c,\qquad
x_{\text{tgt}}(t)\xrightarrow{\text{spk-enc}} s,\qquad
(c,s)\xrightarrow{\text{Tacotron2}} \hat{Y}\xrightarrow{\text{vocoder}} \hat{x}(t),
\]
where $c$ is a character sequence, $s\in\mathbb{R}^{256}$ is a speaker embedding, and $\hat{Y}\in\mathbb{R}^{F\times T}$ is a mel-spectrogram with $F=128$ [2602.16721].

A closely related architecture is the voice-cloning stack consisting of a **speaker verification system**, a **synthesizer**, a **vocoder**, and **noise reduction** [2401.11771]. In that system, a text-independent LSTM-based speaker encoder trained with **Generalized End-to-End (GE2E) loss** produces a speaker embedding or “d-vector,” a Tacotron-like text-to-mel synthesizer conditions on that embedding, a WaveRNN vocoder converts mel-spectrograms to waveform, and a denoising stage based on the `noisereduce` package is applied after synthesis [2401.11771]. The paper explicitly notes that an STSSN can be built on top of this architecture by adding a **speech $\rightarrow$ linguistic content** step such as ASR in front of the existing TTS pipeline [2401.11771].

A different architectural direction is represented by **STTATTS**, a unified encoder-decoder Transformer trained jointly for ASR and TTS via multi-task learning [2410.18607]. STTATTS uses the same encoder and decoder for all tasks, with modality-specific pre-nets and post-nets and a small **task fusion** module driven by a 128-dimensional task embedding [2410.18607]. The design supports ASR, TTS, and voice conversion within one parameter-sharing backbone. The paper states that voice conversion is “just another task” in this framework: speech enters through the speech encoder pre-net, task fusion conditions the shared latent sequence, and the speech decoder post-net predicts 80-dimensional log-Mel features for vocoding [2410.18607]. This places STSSN within a broader trend toward unified speech-text generative models.

The relation among these architectures is straightforward. Cascade systems explicitly separate recognition, speaker representation, and synthesis; unified models collapse much of this decomposition into a shared encoder-decoder; articulatory-front-end systems replace the source acoustic representation with an articulatory one while preserving the downstream mel-to-waveform structure [2305.19130; 2410.18607].

## 3. Internal representations and synthesis mechanisms

The internal representation chosen by an STSSN determines which factors of variation are preserved, discarded, or made controllable. In text-mediated systems, the source signal is collapsed to a **character sequence** or related symbolic transcription. In the DeepSpeech-like STSSN for voice impersonation, the ASR outputs 29 characters, corresponding to 26 letters plus space, apostrophe, and blank, using a CTC objective
\[
\mathcal{L}_{\text{CTC}} = - \log \sum_{\pi \in \mathcal{A}(c)} \prod_{t=1}^T p_t(\pi_t),
\]
and Tacotron2 then reconstructs a 128-dimensional mel-spectrogram conditioned on the recognized text and the target speaker embedding [2602.16721]. This formulation yields strong control over lexical content and speaker identity but introduces a **text bottleneck**, since non-textual detail is not preserved [2602.16721].

Speaker identity is commonly encoded as a fixed-dimensional embedding. In the voice-cloning architecture, the speaker encoder maps variable-length log-mel input to an L2-normalized embedding
\[
e_{ji} = \frac{f(x_{ji}; W)}{\|f(x_{ji}; W)\|_2},
\]
with GE2E training driven by cosine similarities to speaker centroids
\[
C_k = \frac{1}{M}\sum_{m=1}^M e_{km}, \qquad
S_{ji,k} = w \cdot \cos(e_{ji}, C_k) + b
\]
[2401.11771]. This design is explicitly intended to generalize to unseen speakers and is used for both seen-speaker library synthesis and voice cloning from a reference utterance [2401.11771].

Waveform generation is usually delegated to a vocoder conditioned on predicted mel features, but not all STSSNs use the same vocoder class. One synthesis route uses **WaveGlow**, as in the ultrasound tongue imaging pipeline, where 80-dimensional mel-spectrogram frames compatible with WaveGlow are generated from articulatory input [2305.19130]. Another uses **WaveRNN**, as in the voice-cloning package, trained with MSE on waveform samples and teacher forcing [2401.11771]. A third route uses **differentiable digital signal processing (DDSP)**, which is especially relevant when controllability is a design goal [2010.15084].

The DDSP vocoder decomposes waveform generation into a **harmonic sinusoidal oscillator** and a **time-varying filtered-noise generator**, controlled by explicit trajectories: fundamental frequency $f_1(n)$, amplitude envelope $A(n)$, harmonic distribution $c_k(n)$, and time-varying noise filter coefficients [2010.15084]. The harmonic component is
\[
y_{\text{harm}}(n) = \sum_{k=1}^{H} A_k(n)\sin(\phi_k(n)),
\qquad
A_k(n)=A(n)c_k(n),
\]
and the noise component is produced through frequency-domain filtering followed by inverse STFT with overlap-add [2010.15084]. The paper argues that this vocoder can be used in an STSSN by driving it with controls extracted from a source speech signal and then swapping, modifying, or re-mapping these controls to match a target speaker [2010.15084]. This exposes pitch, loudness, timbre, and aspects of rhythm as explicit variables rather than leaving them implicit in spectrogram inversion.

A common misconception is that all STSSNs are direct waveform-to-waveform models. The literature represented here does not support that view. Several prominent designs are explicitly mediated by text, mel-spectrograms, or structured control trajectories, and the choice of intermediate representation is consequential for both controllability and information loss [2010.15084; 2602.16721].

## 4. Adaptation, conditioning, and domain mismatch

A major technical issue in STSSN is adaptation across speakers, sessions, accents, and recording conditions. The articulatory SSI work provides a particularly explicit treatment of this problem by inserting a **Spatial Transformer Network (STN)** before the regression network [2305.19130]. The STN performs a 2D affine transformation of UTI input images using parameters $\theta\in\mathbb{R}^6$:
\[
\begin{bmatrix}
x^s\\
y^s
\end{bmatrix}
=
\begin{bmatrix}
\theta_{11} & \theta_{12} & \theta_{13}\\
\theta_{21} & \theta_{22} & \theta_{23}
\end{bmatrix}
\begin{bmatrix}
x^t\\
y^t\\
1
\end{bmatrix},
\]
with translation, scaling, rotation, shearing, and cropping included in the affine family [2305.19130]. The module is composed of a localization network, grid generator, and sampler, and is placed before the first convolutional layer of the acoustic regressor [2305.19130].

The empirical observation is that the STN occupies only about **10% of the free parameters** of the full network, while the output layer has about **1% of the weights**; adapting only the STN, or the STN plus output layer, captures a substantial fraction of full retraining performance [2305.19130]. For cross-speaker adaptation with a 2D-CNN, adapting only the STN closes on average about **76%** of the performance gap relative to no adaptation, while adapting **STN + out** closes **88%**; for cross-session adaptation, the corresponding value for **STN + out** is **92%** [2305.19130]. The paper explicitly notes that the effect is stronger for session adaptation than for cross-speaker adaptation, which fits the geometric nature of the affine STN [2305.19130].

This result is relevant to STSSN because it formalizes the idea of a **small, learnable transformation module in front of the encoder/regressor**, trained to compensate for domain mismatch while leaving the main mapping network fixed [2305.19130]. The DDSP paper generalizes the same pattern conceptually to **time-frequency spectrogram STNs**, **articulatory parameter STNs**, and **latent space transforms**
\[
\mathbf{z}'=\mathbf{A}\mathbf{z}+\mathbf{b},
\]
as possible mechanisms for adapting input representations or latent states to the distribution expected by a fixed backbone [2010.15084].

Speaker and accent conditioning in voice-cloning systems use a different but related mechanism: rather than geometrically transforming input features, they encode a reference utterance into a speaker embedding and condition the synthesizer on that vector [2401.11771]. The speaker encoder is frozen at inference, which enables zero-shot or few-shot cloning without per-speaker fine-tuning of the TTS model [2401.11771]. In the voice-impersonation STSSN, the same principle appears as a 256-dimensional target embedding concatenated with Tacotron2 encoder outputs at every time step [2602.16721].

These adaptation mechanisms reflect distinct assumptions about variability. STNs target systematic geometric mismatch in image-like articulatory inputs; d-vectors and x-vectors target speaker identity; task embeddings in unified models target functional mode selection such as ASR, TTS, or voice conversion [2305.19130; 2401.11771; 2410.18607].

## 5. Evaluation protocols and empirical findings

Evaluation practice in STSSN depends strongly on the chosen architecture. Text-mediated and voice-cloning systems typically combine **subjective** and **objective** measures, while articulatory-mapping systems often emphasize regression fidelity in acoustic feature space.

In the voice-cloning system for Indian and Western accents, the reported metrics are **Mean Opinion Score (MOS)**, **Gross Pitch Error (GPE)**, and **Spectral Distortion (SD)** [2401.11771]. MOS is computed as
\[
\text{MOS} = \frac{1}{N}\sum_{n=1}^N r_n,
\]
with listener judgments of speaker similarity ranging from “Not at all similar” to “Very similar” [2401.11771]. Reported MOS values for several unseen speakers are above 4, including 4.47 for Bangla, 4.26 for Telugu, and 4.17 for Malayalam, while Western accents are reported around 4.5–4.8 in the proposed work [2401.11771]. GPE is described as a measure of disparity between input and synthesized pitch, and SD measures spectral difference between original and synthesized speech [2401.11771].

The DDSP vocoder paper evaluates waveform quality by comparison with WaveNet, WaveGlow, and Griffin–Lim using a MUSHRA listening test with 29 reliable listeners [2010.15084]. The reported medians are approximately **40/100** for the DDSP vocoder, approximately **90** for WaveNet, approximately **84** for WaveGlow, and a similar score for Griffin–Lim with different artifacts [2010.15084]. The same paper emphasizes that the key benefit of the DDSP approach is explicit control over pitch, loudness, and timbre rather than top-tier vocoder naturalness [2010.15084]. It also reports that speech is fully intelligible, while consonants sound somewhat artificial and the noise generator leaves slight residual noise during vowels [2010.15084].

The voice-impersonation STSSN paper primarily relies on qualitative listening and spectrogram inspection rather than detailed numeric MOS tables [2602.16721]. It states that STSSN “succeeds in generating realistic audio samples despite a number of drawbacks in its capacity” and produces “much more realistic speech” than the CycleGAN baseline, while the CycleGAN results are described as “ok but not optimal” and prone to overfitting and noise sensitivity [2602.16721]. The baseline CycleGAN operates on encoded spectral envelopes with adversarial and cycle-consistency losses, whereas STSSN uses supervised ASR, speaker encoding, and TTS components with no adversarial training [2602.16721].

Unified ASR/TTS models are evaluated with both recognition and synthesis metrics. STTATTS reports English ASR results on LibriSpeech test-clean with WER/CER of **4.84/1.63** for the small configuration, **3.47/1.07** for the medium configuration, and **2.99/0.90** for the large configuration [2410.18607]. For TTS, it reports CER of Whisper transcription, human MOS for naturalness and intelligibility, and WV-MOS [2410.18607]. In English, the large configuration reports **CER = 2.10**, **naturalness MOS = 3.00**, **intelligibility MOS = 4.38**, and **WV-MOS = 4.26**; in Arabic, the model reports **WER = 10.22**, **CER = 2.63** for ASR, and **TTS CER = 6.22**, **naturalness MOS = 3.28**, **intelligibility MOS = 2.78**, **WV-MOS = 3.69** [2410.18607]. The same work emphasizes a roughly **50% reduction** in total parameters compared with separate ASR and TTS models, with about **155M** parameters for the joint model versus about **296M** for a separate SpeechT5 ASR and TTS pair [2410.18607].

A recurring methodological point is that objective metrics vary in what they actually measure. MSE on mel frames in articulatory mapping quantifies spectral regression fidelity, MOS and MUSHRA quantify human perception, and WER or CER quantify recognizability or intelligibility through transcription [2305.19130; 2010.15084; 2410.18607]. None of these is a complete proxy for the full STSSN objective of content preservation plus target-style realization.

## 6. Limitations, controversies, and future directions

The most persistent limitation across STSSN designs is the **text bottleneck**. In the ASR $\rightarrow$ TTS formulation, the source speech is reduced to a character sequence, which discards detailed prosody, exact durations, micro-prosody, and other paralinguistic cues [2602.16721]. As a result, the output is more accurately described as “generic Tacotron prosody in the target voice” than as a faithful acoustic rendering of the source utterance’s expressive structure [2602.16721]. The accent- and voice-cloning pipeline has the same structural limitation when adapted to speech-to-speech via an external ASR front-end [2401.11771].

A second limitation concerns the trade-off between **controllability** and **naturalness**. DDSP exposes pitch, loudness, timbre, and noise-shaping controls explicitly, but the reported quality remains below WaveNet and WaveGlow in listening tests [2010.15084]. The paper also notes residual entanglement of pitch and timbre, indicating that explicit factorization is not perfect even when pitch is supplied as a separate control signal [2010.15084]. This counters the misconception that interpretable control spaces automatically imply clean disentanglement.

A third issue is **data and domain dependence**. The STN-based articulatory system is evaluated on only four speakers for cross-speaker adaptation and one speaker for cross-session analysis, with no perceptual evaluation and no explicit quantification of the minimum adaptation data required [2305.19130]. The voice-impersonation STSSN depends on a pretrained ASR, a pretrained speaker encoder, and a pretrained Tacotron2, and it does not perform end-to-end optimization of the whole speech-to-speech path [2602.16721]. The multi-accent voice-cloning system depends on substantial speaker-encoder training data, reported as **600 hours** across approximately **3,000 speakers**, with **480 hours** of Indian-accented data and **120 hours** of Western-accented data [2401.11771]. STTATTS demonstrates joint training viability in both English and Arabic, but it remains monolingual per model and does not yet incorporate direct speech-to-speech loss or speech translation [2410.18607].

The literature also identifies distinct future directions. The voice-impersonation STSSN proposes moving toward a single end-to-end model operating directly on spectrogram embeddings and avoiding the text bottleneck, with content and style consistency losses over learned embeddings [2602.16721]. The STN-based articulatory work suggests **3D localization networks** and truly **3D transforms** for spatio-temporal distortions, alongside reduced adaptation-material requirements and integration with multi-speaker training and speaker embeddings [2305.19130]. The DDSP vocoder work highlights stronger factor disentanglement, different synthesis operators such as wavetable oscillators for faster CPU inference, and the use of statistical models over disentangled controls [2010.15084]. STTATTS points toward adding explicit speech-to-speech tasks, speaker-identity objectives, multilingual extensions, and semi-supervised learning with unpaired speech and text [2410.18607].

Ethical concerns are explicit in the voice-impersonation literature. Applications include identity protection, accessibility, entertainment, and dubbing, but the same systems can be used for telemarketing scams, falsified recordings, and other malicious deepfake use cases [2602.16721]. The proposed safeguards include detection tools, watermarking or traceability of synthetic audio, and consent- and policy-based constraints on voice use [2602.16721].

Taken together, these works suggest that STSSN is evolving along two partially competing trajectories. One trajectory favors modularity, explicit intermediate representations, and controllability through ASR, speaker embeddings, and structured vocoders. The other favors unified or end-to-end models with shared representations and fewer parameters. A plausible implication is that future STSSNs will combine both tendencies: unified backbones for efficiency and transfer, plus explicitly parameterized control or adaptation modules where interpretability and personalization are required [2010.15084; 2410.18607].

Source: https://www.emergentmind.com/topics/speech-to-speech-synthesis-network-stssn