SwanVoice: Expressive Zero-Shot TTS & AAC Workflow

Updated 4 July 2026

SwanVoice is a suite of expressive voice systems that generate long-form zero-shot speech synthesis for both monologue and dialogue with maintained conversational continuity.
It employs innovative techniques including a variational autoencoder, flow-matching formulation, and multi-stage training for robust speaker control and expressive output.
In AAC applications, SwanVoice facilitates guided voice elicitation to create culturally and linguistically aligned synthetic voices that support user identity and agency.

SwanVoice is a research name applied to multiple voice and speech systems, most prominently a 2026 zero-shot text-to-speech model for expressive long-form monologue and dialogue, and also a guided voice-elicitation and voice-generation workflow for augmentative and alternative communication (AAC) users. Across these uses, the term is associated with systems that treat voice not merely as acoustic output, but as a structured object involving identity, speaker control, contextual coherence, and cultural or conversational continuity (Li et al., 29 May 2026, Weinberg et al., 23 May 2026).

1. Terminological scope and disambiguation

The name does not denote a single universally shared architecture. In the recent literature, it is used for at least two distinct systems, and it also appears in adjacent descriptions of singing voice generation and conversion research.

Usage of the name	Associated paper	Characterization
SwanVoice	"SwanVoice: Expressive Long-Form Zero-Shot Speech Synthesis for Both Monologue and Dialogue" (Li et al., 29 May 2026)	Zero-shot TTS for 1–4 speakers with monologue and dialogue support
SwanVoice	"Me, Myself, and My Voice: Exploring Cultural and Linguistic Identity in AAC AI-generated Voices" (Weinberg et al., 23 May 2026)	Guided elicitation and voice-generation workflow for AAC users
SwanVoice label applied in description	"Unsupervised Cross-Domain Singing Voice Conversion" (Polyak et al., 2020)	Wav-to-wav singing voice conversion from unlabeled audio

A further source of confusion is the spatial-audio system "SwanSphere," whose actual system name is not SwanVoice, despite superficial overlap in surrounding summaries (Lei et al., 29 May 2026). For technical precision, the title-form use of SwanVoice refers directly to the long-form zero-shot TTS model introduced in 2026, whereas the AAC paper uses the term for a tool-centric workflow rather than a single generative backbone (Li et al., 29 May 2026, Weinberg et al., 23 May 2026).

2. SwanVoice as expressive long-form zero-shot speech synthesis

In "SwanVoice: Expressive Long-Form Zero-Shot Speech Synthesis for Both Monologue and Dialogue," the central problem is long-form expressive synthesis under full-scene constraints rather than isolated utterance generation. The motivating claim is that synthesizing each dialogue turn independently with a monologue TTS model and stitching the outputs together often breaks acoustic consistency, conversational coherence, and affective continuity across turns. SwanVoice addresses this by treating dialogue as a single structured generation problem with shared acoustic scene, turn structure, and emotional continuity, and by using a non-autoregressive flow-matching formulation so that the entire text and speaker-turn structure can be conditioned jointly (Li et al., 29 May 2026).

The model is designed for 1–4 speakers and aims to preserve monologue quality while also supporting dialogue. This positioning is important because prior dialogue TTS systems are described as struggling to keep expressive coherence, controllable speaker switching, and monologue quality at the same time. SwanVoice therefore sits at the intersection of long-context TTS, zero-shot speaker conditioning, and dialogue-aware acoustic modeling, with its main technical emphasis on expressive continuity over long spans rather than turnwise local quality alone (Li et al., 29 May 2026).

A major enabling component is SwanData-Speech, a corpus-construction pipeline for monologue and dialogue from in-the-wild audio. The raw source is about 2.59 million hours in total, with roughly 2.24 million hours Chinese and 0.35 million hours English. The pipeline uses vocal separation, diarization with the 3D-Speaker toolkit, utterance-like chunking with VAD, speaker embeddings and clustering, and explicit dialogue construction by greedily merging consecutive multi-speaker regions up to 120 seconds while requiring 2–4 speakers and no silence interval longer than 2 seconds. This data-centric design reflects the paper’s broader claim that long-form expressive synthesis depends as much on alignment and segmentation quality as on the generator itself (Li et al., 29 May 2026).

3. Architecture, conditioning, and latent formulation

SwanVoice is built around a 25 Hz variational autoencoder and a flow-matching DiT. The encoder compresses waveform into a latent sequence $z$ , and a HiFi-GAN-style decoder reconstructs speech. The VAE objective combines spectrogram reconstruction, KL regularization, and adversarial losses; in the paper’s notation, the reconstruction term is $\mathcal{L}_{\mathrm{rec}}=\|\Phi(s)-\Phi(\hat{s})\|_2^2$ , together with $\mathcal{L}_{\mathrm{KL}}$ and an LSGAN-style $\mathcal{L}_{\mathrm{Adv}}$ from multi-period, multi-scale, and multi-resolution discriminators (Li et al., 29 May 2026).

On the text side, the model uses raw text directly through a CosyVoice-style BPE tokenizer rather than a separate G2P pipeline. Pause modeling is made explicit through the insertion of the special token <|sp|>. For Chinese, the vocabulary is augmented with 1,549 pinyin syllable combinations, and during training a random subset of Chinese characters is substituted with pinyin extracted by pypinyin. The stated purposes are pronunciation regularization and inference-time override for difficult pronunciations, especially polyphonic characters and dialect-sensitive forms. Dialogue conditioning is represented with <S{id}> ... </S{id}> wrappers, from which a speaker-turn label sequence aligned to text tokens is constructed (Li et al., 29 May 2026).

The generative backbone uses a conditioning strategy that avoids naive early concatenation of all conditions with the speech latent. Padded text tokens and turn embeddings are first processed by a lightweight Transformer so that text-side and turn-side features can interact before fusion with the speech latent. The latent input is split into a reference prefix and a target suffix; for dialogue, the reference segment must contain at least a short span from every speaker so that all voices are observed before target-region generation. The flow objective is

$\mathbf{z}_t=(1-t)\epsilon+t\mathbf{z}^\star,\quad t\sim \mathcal{U}(0,1),$

with loss

$\mathcal{L}_{\mathrm{flow}} = \mathbb{E}\left[ \left\| \mathbf{u}_{\theta}(\mathbf{z}_{t}, t, \mathbf{c}) - (\mathbf{z}^{\star} - \epsilon) \right\|_2^2 \right].$

RMSNorm is used throughout, and AdaLN global adapters are included to stabilize long-form consistency in timbre and recording conditions (Li et al., 29 May 2026).

The same paper also describes a pronunciation-hard synthetic subset, RobustMegaTTS3, built from GCIDE English word lists, official Chinese character lists, LLM-generated example sentences, and hard-case sets including 20K Chinese cases, 20K English cases, and 100K Chinese-English code-switching texts across 13 scenarios. These cover polyphonic Chinese characters, tone sandhi, erhua, onomatopoeia, homographs, stress shifts, irregular spelling, and code-switching. The design goal is explicit coverage of pronunciation phenomena that naturally occurring speech alone underserves (Li et al., 29 May 2026).

4. Training curriculum, post-training, inference, and empirical profile

Training is staged. The first stage is monologue pretraining on about 2 million hours of monologue speech plus pronunciation-hard and code-switching synthetic data. The second stage mixes monologue with concatenated 2–4-speaker dialogue data and oversamples dialogue so turn switching is frequent while monologue remains in the mix. The third stage is supervised fine-tuning on monologue plus real 2–4-speaker dialogue from movies, TV dramas, and podcasts. The stated rationale is that dialogue quality improves if noisy real conversations are not introduced before the model has learned monologue alignment and basic speaker control (Li et al., 29 May 2026).

After supervised training, SwanVoice is post-trained with DiffusionNFT using online reinforcement learning over sampled outputs. The reward averages a phone-level consistency term and a speaker-similarity term. The phone reward is defined from phone-tone token WER as $r_{\mathrm{phone}}=\exp(-\mathrm{WER})$ , while speaker similarity is $r_{\mathrm{sim}}(\hat{x},x^{\mathrm{ref}})=\cos(f_{\mathrm{spk}}(\hat{x}),f_{\mathrm{spk}}(x^{\mathrm{ref}}))$ . The final reward is $r=\frac{1}{2}(r_{\mathrm{phone}}+r_{\mathrm{sim}})$ , with within-prompt baselines $A_i=r_i-\bar r$ . DiffusionNFT then combines positive and implicit-negative denoising branches together with a reference-policy regularizer, written as $\mathcal{L}_{\mathrm{rec}}=\|\Phi(s)-\Phi(\hat{s})\|_2^2$ 0 (Li et al., 29 May 2026).

At inference, the model takes a reference speech segment and target text, transcribes the reference to obtain speaker-specific prompting text, estimates target duration with a speaking-rate heuristic, and uses sway sampling. It also uses a staircase classifier-free guidance rule that separates text guidance from reference guidance:

$\mathcal{L}_{\mathrm{rec}}=\|\Phi(s)-\Phi(\hat{s})\|_2^2$ 1

This is presented as a practical mechanism for controlling the trade-off between content fidelity and speaker/style similarity in long-form synthesis (Li et al., 29 May 2026).

Evaluation is conducted on SwanBench-Speech. Acoustic metrics include timbre consistency, reverb consistency, and sound fidelity; semantic metrics include content error rate and prosodic coherence; expressive metrics use an MLLM-as-judge protocol with Gemini-3-Pro for richness and hierarchy. Reported results place SwanVoice at 3.81 richness and 3.62 hierarchy on monologue, and 3.62 richness and 3.71 hierarchy on dialogue. Timbre consistency is reported as 0.93 for monologue and 0.92 for dialogue, while sound fidelity is 3.60 and 3.77 respectively. The paper explicitly identifies content accuracy as the main remaining weakness, and also notes failures in speaker switching when speakers are acoustically close or prompts are short (Li et al., 29 May 2026).

5. SwanVoice as an AAC voice-elicitation and generation workflow

In AAC research, SwanVoice denotes a guided workflow intended to help users articulate what an identity-aligned voice should sound like. The underlying problem is framed not as a generic speech-synthesis task but as an identity-alignment problem: for AAC users, the device voice is the public voice others associate with them, and when that voice is generic, binary-gendered, or culturally mismatched, it can undermine belonging, self-recognition, and agency (Weinberg et al., 23 May 2026).

The first empirical component is a survey of 53 validated AAC users drawn from 692 raw responses, distributed in English and Spanish and validated because of high risk of fraudulent or AI-generated submissions. The validated sample represented eight countries of birth and was heavily US-based, with 81.1% born in the United States. Mean ratings were 3.42/5 for satisfaction, 3.49 for linguistic representation, 3.66 for cultural representation, 3.70 for comfort, and 3.96 for the importance of cultural reflection, while ease of personalization was notably weak at 2.71 and 57.1% rated it negatively. Non-binary respondents and respondents born outside the US rated alignment consistently lower, with free-text responses emphasizing mispronounced names, limited prosody, effective monolingualism, and accent dominance such as Spain Spanish over Latin American Spanish (Weinberg et al., 23 May 2026).

The system’s second component is a guided prompting tool described as a “cultural probing tool” or Cultural Voice Explorer. It is explicitly two-stage. First, the participant goes through 10 dynamically generated questions about cultural and linguistic background, produced by GPT-4o-mini under a system prompt that requests concise, warm, non-leading questions and coverage of age, gender, heritage, region, multilingualism, accent, code-switching, preferred expressions, tone, and avoid-list items. Second, another GPT-4o-mini uses the transcript to generate a voice prompt, which the participant can review and edit before sending it to ElevenLabs to synthesize three candidate voices. The selected voice is then used to read four short everyday sentences for contextual judgment (Weinberg et al., 23 May 2026).

Implementation is described as a three-layer composition: GPT-4o-mini for dynamic question generation, GPT-4o-mini for prompt synthesis from the transcript, and ElevenLabs for text-to-speech generation. The web interface includes a question pane, progress indicator, controls to submit, skip, restart, or reload, and a voice-generation pane showing the synthesized prompt and three unlabeled audio candidates. The interviewer agent produces structured JSON output to drive the interface, and iteration is treated as central rather than optional (Weinberg et al., 23 May 2026).

The qualitative interview study involved six adult AAC users using speech-generating devices as a primary or significant communication mode. Four themes emerged: voice as iterative construction, voice as a private self, voice as temporal identity, and voice as cultural negotiation. Tool-generated voices generally outperformed manually authored prompts on cultural match and reduced stereotyping, but participants also encountered representational ceilings: non-binary voices, neurodivergent prosody, and some racialized or multilingual combinations often collapsed back toward binary gender, generic American English, or a dominant accent. The paper is explicit that these studies are small and not generalizable across all AAC users or cultures, and it raises accessibility, stereotyping, update-consent, and model-change risks as central design concerns (Weinberg et al., 23 May 2026).

6. Relation to singing voice research and broader significance

The SwanVoice label also sits near singing voice research, although the nomenclature is less stable there. In the provided literature, the label is applied to "Unsupervised Cross-Domain Singing Voice Conversion," a wav-to-wav system that converts a source singing recording into the vocal style of a target identity without requiring lyrics, musical notes, parallel samples, or other manual supervision. That system uses Wav2Letter-derived acoustic features, CREPE-derived melody features transformed into a sine-excited representation, loudness conditioning, a non-autoregressive non-causal WaveNet-like generator, and GAN, multi-resolution spectral, and perceptual losses. It is described as fully convolutional, real-time capable, and reported as 10.14× faster than real-time on a single Tesla V100 GPU (Polyak et al., 2020).

Adjacent work sharpens the distinction between canonical SwanVoice and neighboring voice-generation paradigms. "Adapting Speech LLM to Singing Voice Synthesis" adapts a 1.7B TTS-pretrained SLM to singing using ESPnet-SpeechLM, ACE-Opencpop, multi-stream token prediction, conditional flow matching, and HiFi-GAN, achieving results comparable to dedicated discrete-token SVS systems (Zhao et al., 16 Dec 2025). "Everyone-Can-Sing: Zero-Shot Singing Voice Synthesis and Conversion with Speech Reference" develops a unified zero-shot SVS/SVC framework with speech-reference timbre control, style tokens, score-conditioned performance control, Resemblyzer speaker embeddings, and a diffusion-based mel generator, thereby showing a separate line of work on voice identity transfer across speech and singing (Dai et al., 23 Jan 2025).

A plausible implication is that the name SwanVoice has come to index a family of problems rather than a single technical recipe: zero-shot voice conditioning, identity-bearing voice design, cross-speaker or cross-domain transfer, and long-context coherence. In its title-form 2026 usage, however, the term most precisely denotes a long-form expressive TTS system for monologue and dialogue (Li et al., 29 May 2026). In the AAC literature, it denotes a workflow for discovering and specifying culturally and linguistically aligned synthetic voices (Weinberg et al., 23 May 2026). The common thread is that voice is treated not as a static timbre label, but as a structured representation tied to identity, context, and control.