Speech to Speech Synthesis Network (STSSN)

• STSSN is a family of systems that map source speech to output speech by preserving linguistic content and transferring target speaker style.
• Canonical architectures use cascaded components like ASR, speaker encoders, and TTS decoders or unified encoder-decoder models for efficient synthesis.
• Key challenges include overcoming the text bottleneck, controlling prosody and style fidelity, and adapting to speaker and domain mismatches.

Speech to Speech Synthesis Network (STSSN) denotes a class of systems that take speech-derived input and produce speech output, with the mapping designed to preserve some attributes of the input while altering or conditioning others. In the narrow formulation introduced for voice impersonation, STSSN receives a source utterance carrying linguistic content and a target utterance carrying speaker style, then generates a new utterance that preserves the source content while adopting the target’s voice, style, and mannerisms (Johnson et al., 13 Feb 2026). In broader usage suggested by related work, the same design pattern encompasses text-mediated cascades such as ASR $\rightarrow$ TTS, direct speech-to-speech voice conversion, and even articulatory-to-acoustic mappings in which the “source” representation is not acoustic but articulatory, as in ultrasound tongue imaging-based silent speech interfaces (Tóth et al., 2023).

1. Definitions and task formulations

The central problem addressed by STSSN is speech-to-speech style transfer. In the formulation explicitly named STSSN, the task is to preserve, approximately, the linguistic content of a source utterance while adopting the voice / style / mannerisms of a target speaker (Johnson et al., 13 Feb 2026). In that work, “style transfer” primarily means speaker identity and gross voice characteristics such as timbre, pitch range, and accent or manner of articulation, rather than fine-grained prosody control (Johnson et al., 13 Feb 2026).

A broader interpretation follows from related systems that implement speech-conditioned synthesis through intermediate representations. One important variant is the text-mediated pipeline: speech is transcribed by ASR, target identity is encoded separately, and a multispeaker TTS model reconstructs speech from text plus speaker representation (Johnson et al., 13 Feb 2026). Another variant uses a continuous acoustic or articulatory front-end instead of text. The ultrasound tongue imaging silent speech interface described as an articulatory-to-acoustic mapping learns

$$f_{\theta}: \text{UTI image(s)} \rightarrow \mathbf{y} \in \mathbb{R}^{80},$$

where $\mathbf{y}$ is an 80-dimensional mel-spectrogram frame, followed by WaveGlow waveform generation; this is described as analogous to an STSSN in which the usual source acoustic domain is replaced by an articulatory domain (Tóth et al., 2023).

These formulations imply a spectrum of STSSN designs. At one end are systems that pass through explicit symbolic content, usually characters or phonemes. At the other end are direct mappings over acoustic, articulatory, or latent representations. This suggests that STSSN is best understood not as a single architecture but as a family of speech-conditioned generation systems organized around the decomposition of content, speaker/style, and waveform synthesis.

2. Canonical architectural patterns

A canonical STSSN architecture in the voice-impersonation setting is a cascade of three components: a content encoder / ASR, a target speaker encoder, and a text-to-speech decoder (Johnson et al., 13 Feb 2026). In that implementation, the ASR is a modified DeepSpeech-like model trained with CTC, the speaker encoder is an LSTM-based network producing a fixed 256-dimensional speaker/style vector, and the decoder is Tacotron2 conditioned on both text and speaker embedding, with a vocoder producing waveform from a 128-bin mel-spectrogram (Johnson et al., 13 Feb 2026). The resulting factorization is

$$x_{\text{src}}(t)\xrightarrow{\text{ASR}} c,\qquad x_{\text{tgt}}(t)\xrightarrow{\text{spk-enc}} s,\qquad (c,s)\xrightarrow{\text{Tacotron2}} \hat{Y}\xrightarrow{\text{vocoder}} \hat{x}(t),$$

where $c$ is a character sequence, $s\in\mathbb{R}^{256}$ is a speaker embedding, and $\hat{Y}\in\mathbb{R}^{F\times T}$ is a mel-spectrogram with $F=128$ (Johnson et al., 13 Feb 2026).

A closely related architecture is the voice-cloning stack consisting of a speaker verification system, a synthesizer, a vocoder, and noise reduction (R et al., 2024). In that system, a text-independent LSTM-based speaker encoder trained with Generalized End-to-End (GE2E) loss produces a speaker embedding or “d-vector,” a Tacotron-like text-to-mel synthesizer conditions on that embedding, a WaveRNN vocoder converts mel-spectrograms to waveform, and a denoising stage based on the noisereduce package is applied after synthesis (R et al., 2024). The paper explicitly notes that an STSSN can be built on top of this architecture by adding a speech $\rightarrow$ linguistic content step such as ASR in front of the existing TTS pipeline (R et al., 2024).

A different architectural direction is represented by STTATTS, a unified encoder-decoder Transformer trained jointly for ASR and TTS via multi-task learning (Toyin et al., 2024). STTATTS uses the same encoder and decoder for all tasks, with modality-specific pre-nets and post-nets and a small task fusion module driven by a 128-dimensional task embedding (Toyin et al., 2024). The design supports ASR, TTS, and voice conversion within one parameter-sharing backbone. The paper states that voice conversion is “just another task” in this framework: speech enters through the speech encoder pre-net, task fusion conditions the shared latent sequence, and the speech decoder post-net predicts 80-dimensional log-Mel features for vocoding (Toyin et al., 2024). This places STSSN within a broader trend toward unified speech-text generative models.

The relation among these architectures is straightforward. Cascade systems explicitly separate recognition, speaker representation, and synthesis; unified models collapse much of this decomposition into a shared encoder-decoder; articulatory-front-end systems replace the source acoustic representation with an articulatory one while preserving the downstream mel-to-waveform structure (Tóth et al., 2023, Toyin et al., 2024).

3. Internal representations and synthesis mechanisms

The internal representation chosen by an STSSN determines which factors of variation are preserved, discarded, or made controllable. In text-mediated systems, the source signal is collapsed to a character sequence or related symbolic transcription. In the DeepSpeech-like STSSN for voice impersonation, the ASR outputs 29 characters, corresponding to 26 letters plus space, apostrophe, and blank, using a CTC objective

$$\mathcal{L}_{\text{CTC}} = - \log \sum_{\pi \in \mathcal{A}(c)} \prod_{t=1}^T p_t(\pi_t),$$

and Tacotron2 then reconstructs a 128-dimensional mel-spectrogram conditioned on the recognized text and the target speaker embedding (Johnson et al., 13 Feb 2026). This formulation yields strong control over lexical content and speaker identity but introduces a text bottleneck, since non-textual detail is not preserved (Johnson et al., 13 Feb 2026).

Speaker identity is commonly encoded as a fixed-dimensional embedding. In the voice-cloning architecture, the speaker encoder maps variable-length log-mel input to an L2-normalized embedding

$$f_{\theta}: \text{UTI image(s)} \rightarrow \mathbf{y} \in \mathbb{R}^{80},$$0

with GE2E training driven by cosine similarities to speaker centroids

$$f_{\theta}: \text{UTI image(s)} \rightarrow \mathbf{y} \in \mathbb{R}^{80},$$1

(R et al., 2024). This design is explicitly intended to generalize to unseen speakers and is used for both seen-speaker library synthesis and voice cloning from a reference utterance (R et al., 2024).

Waveform generation is usually delegated to a vocoder conditioned on predicted mel features, but not all STSSNs use the same vocoder class. One synthesis route uses WaveGlow, as in the ultrasound tongue imaging pipeline, where 80-dimensional mel-spectrogram frames compatible with WaveGlow are generated from articulatory input (Tóth et al., 2023). Another uses WaveRNN, as in the voice-cloning package, trained with MSE on waveform samples and teacher forcing (R et al., 2024). A third route uses differentiable digital signal processing (DDSP), which is especially relevant when controllability is a design goal (Fabbro et al., 2020).

The DDSP vocoder decomposes waveform generation into a harmonic sinusoidal oscillator and a time-varying filtered-noise generator, controlled by explicit trajectories: fundamental frequency $$f_{\theta}: \text{UTI image(s)} \rightarrow \mathbf{y} \in \mathbb{R}^{80},$$2, amplitude envelope $$f_{\theta}: \text{UTI image(s)} \rightarrow \mathbf{y} \in \mathbb{R}^{80},$$3, harmonic distribution $$f_{\theta}: \text{UTI image(s)} \rightarrow \mathbf{y} \in \mathbb{R}^{80},$$4, and time-varying noise filter coefficients (Fabbro et al., 2020). The harmonic component is

$$f_{\theta}: \text{UTI image(s)} \rightarrow \mathbf{y} \in \mathbb{R}^{80},$$5

and the noise component is produced through frequency-domain filtering followed by inverse STFT with overlap-add (Fabbro et al., 2020). The paper argues that this vocoder can be used in an STSSN by driving it with controls extracted from a source speech signal and then swapping, modifying, or re-mapping these controls to match a target speaker (Fabbro et al., 2020). This exposes pitch, loudness, timbre, and aspects of rhythm as explicit variables rather than leaving them implicit in spectrogram inversion.

A common misconception is that all STSSNs are direct waveform-to-waveform models. The literature represented here does not support that view. Several prominent designs are explicitly mediated by text, mel-spectrograms, or structured control trajectories, and the choice of intermediate representation is consequential for both controllability and information loss (Fabbro et al., 2020, Johnson et al., 13 Feb 2026).

4. Adaptation, conditioning, and domain mismatch

A major technical issue in STSSN is adaptation across speakers, sessions, accents, and recording conditions. The articulatory SSI work provides a particularly explicit treatment of this problem by inserting a Spatial Transformer Network (STN) before the regression network (Tóth et al., 2023). The STN performs a 2D affine transformation of UTI input images using parameters $$f_{\theta}: \text{UTI image(s)} \rightarrow \mathbf{y} \in \mathbb{R}^{80},$$6: $$f_{\theta}: \text{UTI image(s)} \rightarrow \mathbf{y} \in \mathbb{R}^{80},$$7 with translation, scaling, rotation, shearing, and cropping included in the affine family (Tóth et al., 2023). The module is composed of a localization network, grid generator, and sampler, and is placed before the first convolutional layer of the acoustic regressor (Tóth et al., 2023).

The empirical observation is that the STN occupies only about 10% of the free parameters of the full network, while the output layer has about 1% of the weights; adapting only the STN, or the STN plus output layer, captures a substantial fraction of full retraining performance (Tóth et al., 2023). For cross-speaker adaptation with a 2D-CNN, adapting only the STN closes on average about 76% of the performance gap relative to no adaptation, while adapting STN + out closes 88%; for cross-session adaptation, the corresponding value for STN + out is 92% (Tóth et al., 2023). The paper explicitly notes that the effect is stronger for session adaptation than for cross-speaker adaptation, which fits the geometric nature of the affine STN (Tóth et al., 2023).

This result is relevant to STSSN because it formalizes the idea of a small, learnable transformation module in front of the encoder/regressor, trained to compensate for domain mismatch while leaving the main mapping network fixed (Tóth et al., 2023). The DDSP paper generalizes the same pattern conceptually to time-frequency spectrogram STNs, articulatory parameter STNs, and latent space transforms

$$f_{\theta}: \text{UTI image(s)} \rightarrow \mathbf{y} \in \mathbb{R}^{80},$$8

as possible mechanisms for adapting input representations or latent states to the distribution expected by a fixed backbone (Fabbro et al., 2020).

Speaker and accent conditioning in voice-cloning systems use a different but related mechanism: rather than geometrically transforming input features, they encode a reference utterance into a speaker embedding and condition the synthesizer on that vector (R et al., 2024). The speaker encoder is frozen at inference, which enables zero-shot or few-shot cloning without per-speaker fine-tuning of the TTS model (R et al., 2024). In the voice-impersonation STSSN, the same principle appears as a 256-dimensional target embedding concatenated with Tacotron2 encoder outputs at every time step (Johnson et al., 13 Feb 2026).

These adaptation mechanisms reflect distinct assumptions about variability. STNs target systematic geometric mismatch in image-like articulatory inputs; d-vectors and x-vectors target speaker identity; task embeddings in unified models target functional mode selection such as ASR, TTS, or voice conversion (Tóth et al., 2023, R et al., 2024, Toyin et al., 2024).

5. Evaluation protocols and empirical findings

Evaluation practice in STSSN depends strongly on the chosen architecture. Text-mediated and voice-cloning systems typically combine subjective and objective measures, while articulatory-mapping systems often emphasize regression fidelity in acoustic feature space.

In the voice-cloning system for Indian and Western accents, the reported metrics are Mean Opinion Score (MOS), Gross Pitch Error (GPE), and Spectral Distortion (SD) (R et al., 2024). MOS is computed as

$$f_{\theta}: \text{UTI image(s)} \rightarrow \mathbf{y} \in \mathbb{R}^{80},$$9

with listener judgments of speaker similarity ranging from “Not at all similar” to “Very similar” (R et al., 2024). Reported MOS values for several unseen speakers are above 4, including 4.47 for Bangla, 4.26 for Telugu, and 4.17 for Malayalam, while Western accents are reported around 4.5–4.8 in the proposed work (R et al., 2024). GPE is described as a measure of disparity between input and synthesized pitch, and SD measures spectral difference between original and synthesized speech (R et al., 2024).

The DDSP vocoder paper evaluates waveform quality by comparison with WaveNet, WaveGlow, and Griffin–Lim using a MUSHRA listening test with 29 reliable listeners (Fabbro et al., 2020). The reported medians are approximately 40/100 for the DDSP vocoder, approximately 90 for WaveNet, approximately 84 for WaveGlow, and a similar score for Griffin–Lim with different artifacts (Fabbro et al., 2020). The same paper emphasizes that the key benefit of the DDSP approach is explicit control over pitch, loudness, and timbre rather than top-tier vocoder naturalness (Fabbro et al., 2020). It also reports that speech is fully intelligible, while consonants sound somewhat artificial and the noise generator leaves slight residual noise during vowels (Fabbro et al., 2020).

The voice-impersonation STSSN paper primarily relies on qualitative listening and spectrogram inspection rather than detailed numeric MOS tables (Johnson et al., 13 Feb 2026). It states that STSSN “succeeds in generating realistic audio samples despite a number of drawbacks in its capacity” and produces “much more realistic speech” than the CycleGAN baseline, while the CycleGAN results are described as “ok but not optimal” and prone to overfitting and noise sensitivity (Johnson et al., 13 Feb 2026). The baseline CycleGAN operates on encoded spectral envelopes with adversarial and cycle-consistency losses, whereas STSSN uses supervised ASR, speaker encoding, and TTS components with no adversarial training (Johnson et al., 13 Feb 2026).

Unified ASR/TTS models are evaluated with both recognition and synthesis metrics. STTATTS reports English ASR results on LibriSpeech test-clean with WER/CER of 4.84/1.63 for the small configuration, 3.47/1.07 for the medium configuration, and 2.99/0.90 for the large configuration (Toyin et al., 2024). For TTS, it reports CER of Whisper transcription, human MOS for naturalness and intelligibility, and WV-MOS (Toyin et al., 2024). In English, the large configuration reports CER = 2.10, naturalness MOS = 3.00, intelligibility MOS = 4.38, and WV-MOS = 4.26; in Arabic, the model reports WER = 10.22, CER = 2.63 for ASR, and TTS CER = 6.22, naturalness MOS = 3.28, intelligibility MOS = 2.78, WV-MOS = 3.69 (Toyin et al., 2024). The same work emphasizes a roughly 50% reduction in total parameters compared with separate ASR and TTS models, with about 155M parameters for the joint model versus about 296M for a separate SpeechT5 ASR and TTS pair (Toyin et al., 2024).

A recurring methodological point is that objective metrics vary in what they actually measure. MSE on mel frames in articulatory mapping quantifies spectral regression fidelity, MOS and MUSHRA quantify human perception, and WER or CER quantify recognizability or intelligibility through transcription (Tóth et al., 2023, Fabbro et al., 2020, Toyin et al., 2024). None of these is a complete proxy for the full STSSN objective of content preservation plus target-style realization.

6. Limitations, controversies, and future directions

The most persistent limitation across STSSN designs is the text bottleneck. In the ASR $\mathbf{y}$0 TTS formulation, the source speech is reduced to a character sequence, which discards detailed prosody, exact durations, micro-prosody, and other paralinguistic cues (Johnson et al., 13 Feb 2026). As a result, the output is more accurately described as “generic Tacotron prosody in the target voice” than as a faithful acoustic rendering of the source utterance’s expressive structure (Johnson et al., 13 Feb 2026). The accent- and voice-cloning pipeline has the same structural limitation when adapted to speech-to-speech via an external ASR front-end (R et al., 2024).

A second limitation concerns the trade-off between controllability and naturalness. DDSP exposes pitch, loudness, timbre, and noise-shaping controls explicitly, but the reported quality remains below WaveNet and WaveGlow in listening tests (Fabbro et al., 2020). The paper also notes residual entanglement of pitch and timbre, indicating that explicit factorization is not perfect even when pitch is supplied as a separate control signal (Fabbro et al., 2020). This counters the misconception that interpretable control spaces automatically imply clean disentanglement.

A third issue is data and domain dependence. The STN-based articulatory system is evaluated on only four speakers for cross-speaker adaptation and one speaker for cross-session analysis, with no perceptual evaluation and no explicit quantification of the minimum adaptation data required (Tóth et al., 2023). The voice-impersonation STSSN depends on a pretrained ASR, a pretrained speaker encoder, and a pretrained Tacotron2, and it does not perform end-to-end optimization of the whole speech-to-speech path (Johnson et al., 13 Feb 2026). The multi-accent voice-cloning system depends on substantial speaker-encoder training data, reported as 600 hours across approximately 3,000 speakers, with 480 hours of Indian-accented data and 120 hours of Western-accented data (R et al., 2024). STTATTS demonstrates joint training viability in both English and Arabic, but it remains monolingual per model and does not yet incorporate direct speech-to-speech loss or speech translation (Toyin et al., 2024).

The literature also identifies distinct future directions. The voice-impersonation STSSN proposes moving toward a single end-to-end model operating directly on spectrogram embeddings and avoiding the text bottleneck, with content and style consistency losses over learned embeddings (Johnson et al., 13 Feb 2026). The STN-based articulatory work suggests 3D localization networks and truly 3D transforms for spatio-temporal distortions, alongside reduced adaptation-material requirements and integration with multi-speaker training and speaker embeddings (Tóth et al., 2023). The DDSP vocoder work highlights stronger factor disentanglement, different synthesis operators such as wavetable oscillators for faster CPU inference, and the use of statistical models over disentangled controls (Fabbro et al., 2020). STTATTS points toward adding explicit speech-to-speech tasks, speaker-identity objectives, multilingual extensions, and semi-supervised learning with unpaired speech and text (Toyin et al., 2024).

Ethical concerns are explicit in the voice-impersonation literature. Applications include identity protection, accessibility, entertainment, and dubbing, but the same systems can be used for telemarketing scams, falsified recordings, and other malicious deepfake use cases (Johnson et al., 13 Feb 2026). The proposed safeguards include detection tools, watermarking or traceability of synthetic audio, and consent- and policy-based constraints on voice use (Johnson et al., 13 Feb 2026).

Taken together, these works suggest that STSSN is evolving along two partially competing trajectories. One trajectory favors modularity, explicit intermediate representations, and controllability through ASR, speaker embeddings, and structured vocoders. The other favors unified or end-to-end models with shared representations and fewer parameters. A plausible implication is that future STSSNs will combine both tendencies: unified backbones for efficiency and transfer, plus explicitly parameterized control or adaptation modules where interpretability and personalization are required (Fabbro et al., 2020, Toyin et al., 2024).