ScenA: Multi-Speaker Audio Scene Generation

• ScenA is a reference-driven framework for multi-speaker audio scene generation that synthesizes integrated dialogue scenes with overlapping speech, ambient sounds, and acoustics.
• It uses a modified rectified flow training with a Beta+Uniform noise schedule to prevent acoustic shortcutting and enforce prompt-mediated speaker binding.
• The architecture leverages a diffusion transformer with identity-aware positional encodings, adversarial reference injection, and slot-shuffle augmentation to enhance controllability and realism.

Searching arXiv for the primary ScenA paper and directly related multi-speaker audio scene generation work. ScenA is a reference-driven framework for multi-speaker audio scene generation that conditions a text-to-audio flow-matching foundation model on multiple reference voices and a single free-form natural language prompt describing an entire conversational scene. Unlike prior multi-speaker dialogue systems that bind speakers to utterances through structured supervision such as per-turn tags, multi-stream transcriptions, or learnable speaker embeddings, ScenA performs speaker assignment through prompt-level references such as “reference 1” and “reference 2,” while jointly generating overlapping dialogue, ambient sound, room acoustics, and paralinguistic events in a single waveform (Finkelson et al., 17 Jun 2026). The system is designed to inherit the acoustic realism of a large-scale in-the-wild pretrained model while adding multi-speaker controllability without introducing per-turn structure.

1. Definition and problem formulation

ScenA addresses the task of synthesizing a single coherent audio track in which multiple speakers converse within a shared acoustic environment. The target output is not a stack of clean speech streams but an integrated scene that can include overlapping speech, breaths, sighs, laughter, background noise, and room-dependent reverberation. This distinguishes the problem from speech-only dialogue synthesis, which typically produces clean vocal sequences and relies on structured interfaces for speaker binding (Finkelson et al., 17 Jun 2026).

The framework is explicitly reference-driven. Given up to $K$ reference clips, each corresponding to a speaker identity, and a free-form prompt describing the scene, the model must determine which reference speaks which content and in what manner. The prompt describes the entire scene rather than individual turns. It may specify overlap, affect, non-speech events, and ambient context, and it uses indexed mentions such as “reference 1” and “reference 2” to associate semantic roles with reference voices (Finkelson et al., 17 Jun 2026).

This design removes the need for per-turn tags, multi-stream transcripts, and identity encoders. A plausible implication is that the interface is substantially less structured than conventional dialogue-generation pipelines, but the effectiveness of that simplification depends on whether the model can reliably bind prompt-level mentions to the correct references.

2. Architectural formulation

ScenA fine-tunes the audio stream of LTX-2.3, described as an audio-video diffusion transformer, into a standalone audio diffusion transformer. The system reuses a variational autoencoder to encode and decode audio into latent token sequences, a frozen text encoder whose embeddings condition the transformer through cross-attention, and the backbone’s prompt-embedding adapter (Finkelson et al., 17 Jun 2026).

A target audio clip is encoded into a latent sequence $\mathbf{z}_0 \in \mathbb{R}^{N \times D}$, where $N$ is the token count and $D$ is latent dimensionality. The model supports up to 20 seconds per audio output. Each reference clip is encoded by the same VAE into a latent sequence $\mathbf{r}_k \in \mathbb{R}^{N_k \times D}$. ScenA constructs the self-attention input by concatenating the noised target with the clean reference latents:

$$\mathbf{z}_\text{input} = [\mathbf{z}_t;\; \mathbf{r}_1;\; \ldots;\; \mathbf{r}_K].$$

Only the target is noised; all tokens participate in self-attention (Finkelson et al., 17 Jun 2026).

To distinguish the references from one another and from the target, ScenA uses identity-aware positional encodings. The default mechanism is additive slot embeddings, where each reference slot $k$ has a learned embedding $e_k \in \mathbb{R}^D$ that is broadcast across the tokens of $\mathbf{r}_k$:

$$\mathbf{r}_k \leftarrow \mathbf{r}_k + e_k, \qquad k = 1, \ldots, K_\text{max}.$$

Target tokens receive no additive embedding. This introduces only $\mathbf{z}_0 \in \mathbb{R}^{N \times D}$0 vectors and does not modify the backbone architecture. ScenA also evaluates a RoPE-based alternative in which positional encoding composes temporal and reference components,

$\mathbf{z}_0 \in \mathbb{R}^{N \times D}$1

with a brief warmup on general audio before reference fine-tuning (Finkelson et al., 17 Jun 2026).

The prompt enters through cross-attention and remains frozen. Because the prompt describes the entire audio scene, the same conditioning pathway controls speaker assignment, overlap structure, affective description, background texture, and other acoustic properties. In experiments, the system uses $\mathbf{z}_0 \in \mathbb{R}^{N \times D}$2 references.

3. Training objective and the “Reference Shortcut”

ScenA uses rectified flow training. The noised target latent at timestep $\mathbf{z}_0 \in \mathbb{R}^{N \times D}$3 is

$\mathbf{z}_0 \in \mathbb{R}^{N \times D}$4

and the corresponding velocity field is

$\mathbf{z}_0 \in \mathbb{R}^{N \times D}$5

The training objective is mean squared error between the predicted and true velocities:

$\mathbf{z}_0 \in \mathbb{R}^{N \times D}$6

Standard practice often samples $\mathbf{z}_0 \in \mathbb{R}^{N \times D}$7 from a logit-normal distribution, $\mathbf{z}_0 \in \mathbb{R}^{N \times D}$8, which concentrates probability mass at intermediate noise levels (Finkelson et al., 17 Jun 2026).

A central contribution of ScenA is the diagnosis of the “Reference Shortcut.” Speaker binding is intended to proceed through two attention paths: prompt mentions such as “reference $\mathbf{z}_0 \in \mathbb{R}^{N \times D}$9” must associate through cross-attention with the slot embedding $N$0, and target tokens must then use self-attention to route toward the correct reference tokens. However, when the noise level is moderate, $N$1 retains enough acoustic information that target tokens can identify the matching reference by direct acoustic similarity. In that regime, self-attention can bypass the text path entirely. This reduces training loss but fails at inference, because generation begins near $N$2, where the target is close to pure noise and the shortcut is unavailable (Finkelson et al., 17 Jun 2026).

ScenA probes this effect with a binary classifier operating on frozen backbone features. Given $N$3 and two candidate references, one matching and one non-matching, the probe predicts the match. Accuracy remains at least $N$4 for $N$5, at least $N$6 through approximately $N$7, is still $N$8 at $N$9, and collapses only near $D$0. This shows that standard schedules place substantial training mass in a regime where acoustic similarity is sufficient for speaker binding, thereby incentivizing the shortcut rather than text-grounded assignment (Finkelson et al., 17 Jun 2026).

4. High-noise-biased training and auxiliary augmentations

To prevent the shortcut, ScenA replaces the logit-normal noise schedule with a Beta+Uniform mixture:

$D$1

Because $D$2 concentrates mass toward large $D$3, this schedule biases training toward high-noise timesteps where the target is acoustically uninformative and the model must rely on the prompt to infer speaker assignment. The uniform component preserves coverage across the full interval. In the noise-schedule ablation, all samplers share a 10% uniform mix, while ScenA’s mixture further concentrates mass toward high $D$4 (Finkelson et al., 17 Jun 2026).

The loss remains the rectified-flow velocity-prediction objective, but the timestep distribution changes the information pathway used during training. This suggests that ScenA treats speaker binding not primarily as an architectural problem, but as a training-dynamics problem arising from the interaction between denoising schedules and latent acoustic similarity.

Two additional augmentations are used without adding extra loss terms. The first is adversarial reference injection, in which unused reference slots are filled with distractor references that do not appear in the prompt. This forces the model to resolve assignment from textual mentions rather than by indiscriminate copying. The second is slot-shuffle augmentation, in which the order of reference latents is permuted and the prompt mentions are rewritten accordingly. A curriculum delays shuffling for the first 10,000 steps, then enables it afterward; always shuffling from step 0 causes binding to collapse (Finkelson et al., 17 Jun 2026).

Training uses 20,000 steps, a global batch size of 128, AdamW with $D$5, $D$6, $D$7, and weight decay 0.01, a peak learning rate of $D$8 with 1,000-step linear warmup, and EMA decay 0.9999. The reported setup uses 16 NVIDIA GB200 GPUs and approximately 24 hours of training time. The VAE and text encoder remain frozen, references and targets share the same VAE, and video-to-audio cross-attention is removed from the backbone (Finkelson et al., 17 Jun 2026).

5. Inference, controllability, and generation behavior

At inference time, ScenA takes a free-form prompt describing the entire scene and up to $D$9 indexed references. It concatenates clean reference latents with the current target latent state, injects slot embeddings for the reference tokens, and runs standard rectified-flow denoising from $\mathbf{r}_k \in \mathbb{R}^{N_k \times D}$0 toward data using the backbone’s sampler (Finkelson et al., 17 Jun 2026).

The intended binding mechanism is temporally structured across denoising. At high noise, the high-noise-biased training forces the model to rely on the text path for speaker assignment. As denoising proceeds, self-attention can then refine acoustic detail while preserving those earlier assignment decisions. Overlap and turn-taking are controlled directly in the prompt, with examples such as interruptions, talk-overs, and unison reading. Ambient textures and paralinguistics are likewise described inline and synthesized jointly with speech (Finkelson et al., 17 Jun 2026).

The framework exposes several controllability dimensions through prompt design alone. Speaker assignment is performed through explicit indexed mentions rather than dialogue markup. Overlap and background conditions can be specified in ordinary language. Emotion and affect are encoded by prompt attributes such as “says softly” or “hum in agreement.” The model also supports multiple references bound to a single speaker if desired (Finkelson et al., 17 Jun 2026).

A common misconception in multi-speaker generation is that accurate binding requires structured supervision or explicit speaker encoders. ScenA contests that assumption by showing that prompt-mediated assignment can be effective if the model is trained to avoid acoustic shortcutting. Another common misconception is that speech realism and environmental realism must be handled by separate systems. ScenA instead uses a general-purpose in-the-wild foundation model so that room acoustics, noise, overlap, and non-speech behaviors are synthesized in the same generative process (Finkelson et al., 17 Jun 2026).

6. Evaluation, ablations, and limitations

ScenA is evaluated on CoVoMix2-Dialogue-20s, a subset of the public CoVoMix2 dialog test set pairing 1,000 DailyDialog transcripts with LibriSpeech test-clean references, restricted to the 291 dialogs that fit within 20 seconds, and on CoVoMix2-Dialogue-WildRef, a harder benchmark of 100 examples formed by re-pairing 50 dialogs with 30 in-the-wild English clips containing crowd noise, music, street ambience, wind, and cartoon voices (Finkelson et al., 17 Jun 2026).

The evaluation uses WER, UTMOS, SQUIM, cpWER, SIM-O, cpSIM, and ACC. On CoVoMix2-Dialogue-20s, ScenA achieves cpWER 0.145, cpSIM 0.567, ACC 0.866, WER 0.020, SIM-O 0.451, UTMOS 3.44, and SQUIM 4.32. On the same benchmark, MOSS-TTSD reports cpWER 0.232, cpSIM 0.547, ACC 0.855, WER 0.109, SIM-O 0.443, UTMOS 3.76, and SQUIM 4.28; VibeVoice-7B reports cpWER 0.206, cpSIM 0.527, ACC 0.821, WER 0.044, SIM-O 0.451, UTMOS 3.58, and SQUIM 4.28; ZipVoice-Dialog reports cpWER 0.176, cpSIM 0.538, ACC 0.847, WER 0.032, SIM-O 0.446, UTMOS 3.57, and SQUIM 4.34; and Dia reports cpWER 0.303, cpSIM 0.339, ACC 0.757, WER 0.133, SIM-O 0.312, UTMOS 2.69, and SQUIM 4.09 (Finkelson et al., 17 Jun 2026).

On CoVoMix2-Dialogue-WildRef, ScenA retains best cpSIM 0.424, best SIM-O 0.348, best WER 0.022, and best SQUIM 4.28, while obtaining cpWER 0.167 and ACC 0.819 close to the leader. The paper reports that all baselines’ cpSIM drops by approximately 0.15 absolute, falling below 0.40, whereas ScenA remains above 0.42 (Finkelson et al., 17 Jun 2026). This indicates particular robustness to reference conditions outside the relatively clean reference distribution used by conventional systems.

Human A/B testing, mixing items from both benchmark subsets, shows ScenA preferred over ZipVoice-Dialog at 84.6% with $\mathbf{r}_k \in \mathbb{R}^{N_k \times D}$1, over Dia at 74.2% with $\mathbf{r}_k \in \mathbb{R}^{N_k \times D}$2, over VibeVoice-7B at 68.3% with $\mathbf{r}_k \in \mathbb{R}^{N_k \times D}$3, and over MOSS-TTSD at 59.8% with $\mathbf{r}_k \in \mathbb{R}^{N_k \times D}$4 (Finkelson et al., 17 Jun 2026).

Ablation results support the core design claims. In the noise-schedule ablation, progressively shifting logit-normal mass toward higher noise improves binding metrics, and the Beta+Uniform schedule yields the best cpWER, cpSIM, and ACC. For identity-aware positional encodings, additive slot embeddings outperform the RoPE-based identity dimension, while removing explicit slot signaling produces ACC 0.513, described as near chance for two speakers. Removing adversarial references reduces cpSIM from 0.567 to 0.467 and SIM-O from 0.451 to 0.368 without materially changing WER or naturalness. Always shuffling slots from step 0 collapses binding with ACC 0.502, while never shuffling yields competitive cpWER and ACC but weaker strict speaker fidelity than the default training recipe (Finkelson et al., 17 Jun 2026).

The reported limitations are also explicit. The duration cap is 20 seconds, inherited from the backbone. The system supports at most three references, and self-attention cost grows linearly with the number of references. Flow-matching requires the generation duration to be specified before sampling. UTMOS can be conservative because the backbone’s in-the-wild acoustic profile differs from cleaner speech synthesis systems, and the paper notes that SQUIM better reflects perceived naturalness (Finkelson et al., 17 Jun 2026).

These constraints delimit the current scope of ScenA. A plausible implication is that extensions to longer conversations or larger speaker sets are primarily constrained by backbone capacity and token budget rather than by the prompt-mediated binding mechanism itself.