Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors

Abstract: Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision: per-turn tags, multi-stream transcriptions, or learnable speaker embeddings. These systems operate within speech-only pipelines that produce clean vocal sequences without the ambient texture of real conversations. We take a different approach. Our method, ScenA, conditions a text-to-audio flow-matching foundation model, pretrained on large-scale in-the-wild data, directly on multiple reference voices and a free-form natural language prompt that describes an entire multi-speaker audio scene. Leveraging such a foundational model allows us to inherit its capacity for natural, non-studio audio: background noise, room acoustics, overlapping dialogue, and spontaneous paralinguistic events, while adding multi-speaker control without any per-turn structure. Concretely, reference latents are concatenated into the model's token sequence and distinguished by lightweight identity-aware positional encodings. However, we identify a critical obstacle to this approach: the \textit{Reference Shortcut}. During training under standard noise schedules, the model can identify the matching reference by acoustic similarity to the noisy target, bypassing the text prompt entirely. We address this with a high-noise-biased timestep distribution that forces the model to rely on the text prompt for speaker assignment. We evaluate ScenA on the CoVoMix2-Dialogue benchmark, showing that it outperforms existing multi-speaker systems on speaker-binding metrics while generating rich conversational audio with overlapping speech, emotional vocalizations, and ambient sound. Our results demonstrate the advantage of using a general-purpose audio model conditioned on a free-form scene description, rather than passing structured dialog scripts through a speech-only pipeline.

• The paper introduces ScenA, a framework that synthesizes realistic multi-speaker conversational scenes without requiring structured dialog annotations.
• It employs a flow-matching diffusion transformer with high-noise timestep sampling to enforce text-mediated speaker binding and avoid reference shortcuts.
• Empirical results show ScenA outperforms baselines on metrics like cpWER, cpSIM, and ACC, validating its robustness even under noisy reference conditions.

Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors: An Expert Analysis

Overview

This paper introduces ScenA, a reference-driven multi-speaker audio scene generation framework leveraging foundation models trained on in-the-wild audio. ScenA is distinct in its ability to synthesize full conversational scenes involving multiple speakers, paralinguistic events, ambient sounds, and overlapping speech, conditioned on a free-form natural language prompt and several reference voices. The framework outperforms dialog-aware TTS baselines by removing the need for structured dialog annotations, relying exclusively on text, and applying flow-matching training with a novel high-noise-biased timestep distribution to address speaker binding.

Figure 1: The ScenA framework transforms free-form natural language prompts and reference voices into rich, multi-speaker conversational scenes, using only natural language for speaker binding and complex audio events.

Methodological Contributions

The central technical proposition is to inject voice references as latent tokens concatenated to the noised target sequence, distinguished by lightweight index-aware positional encodings. Conditioning is accomplished by cross-attending to the text prompt. The model backbone is a flow-matching based diffusion transformer (adapted from LTX-2) operating in latent audio space.

A primary innovation is the identification and mitigation of the "reference shortcut," a failure mode in reference-conditioned flow-matching architectures. Under conventional training noise schedules, the model trivially binds speakers by acoustic similarity between the noised target and clean references, bypassing the text pathway meant for binding. At inference, with initial noise near maximum, this shortcut fails, resulting in catastrophic speaker confusion.

To enforce learning of text-mediated binding, ScenA employs a Beta+Uniform mixture for timestep selection during training. This concentrates sampling on high-noise regions where acoustic information is suppressed, forcing the model to rely on the text prompt for reference binding. Two further augmentations—adversarial reference injection and slot shuffling—are utilized to increase binding robustness under adversarial and varying reference order scenarios.

Figure 2: Diagram showing DiT-based synthesis in ScenA, reference latent concatenation, identity-aware positional codes, and text cross-attention under the flow-matching framework, enabling joint multi-speaker scene synthesis.

Reference Shortcut Analysis

A thorough probe-based analysis quantifies the regime under which the shortcut is available. Using a classifier attached to the feature backbone, the authors demonstrate that acoustic similarity enables correct speaker selection at all but the highest noise levels, precisely where standard logit-normal noise schedules focus training. Only by shifting substantial probability mass toward $t\simeq 1$ does the shortcut become unavailable, thus requiring reliance on textual speaker labels, which is substantiated by monotonic binding metric improvement as the schedule shifts to higher noise.

Figure 3: Binary probe accuracy for shortcut availability as a function of noise level $t$, confirming high discriminability except near pure noise.

Figure 4: Noise schedule ablation showing stronger binding performance for high-noise-weighted regimes, highlighting the effect of timestep distribution on the model's reference-text binding capability.

Empirical Results

Evaluations are conducted on the CoVoMix2-Dialogue-20s and a synthesized "in-the-wild" reference robustness set. ScenA achieves state-of-the-art results across binding-aware metrics (cpWER, cpSIM, ACC), outperforming baselines including MOSS-TTSD, VibeVoice, ZipVoice-Dialog, and Dia. On the noisy reference scenarios, baseline binding collapses while ScenA's cpSIM and SQUIM remain stable, evidencing strong robustness to non-ideal reference conditions.

Human preference evaluations report ScenA as significantly preferred in pairwise dialog quality and speaker fidelity comparisons, confirming quantitative findings.

Notable Numerical Results

• On CoVoMix2-Dialogue-20s:
  • cpWER: 0.145 (best),
  • cpSIM: 0.567 (best),
  • ACC: 0.866 (best),
  • WER: 0.020 (best),
  • SQUIM: 4.32 (tied best),
• On WildRef, cpSIM remains >0.42 for ScenA, whereas all baselines fall <0.40.

Qualitative Scene Generation

ScenA generalizes beyond linear dialog and produces scenes containing overlapping utterances, spontaneous laughter, ambient sound interruptions, and dynamic speaker interactions solely from textual descriptions. This is illustrated in the provided scenarios, where mixed acoustic events—such as speech interleaved with non-verbal events or ambient disruptions—are generated seamlessly and attributed correctly.

Figure 5: Audio scene spectrogram showing rapid-fire quiz turns and a non-speech buzzer conclusion.

Figure 6: Scene with mid-utterance ambient interruption and inline paralinguistic reactions, showcasing ScenA’s prompt-driven compositionality.

Implications and Theoretical Significance

This research challenges the prevailing dogma that structured supervision (per-turn tags, dialog scripts, identity encoders) is required for robust multi-speaker scene synthesis. Instead, it demonstrates that, with rigorous training schema targeting shortcut elimination (i.e., appropriate noise scheduling), minimal architectural additions suffice. The analysis and solutions are not modality-specific and can be extrapolated to flow-matching models with reference conditioning in other domains, including imagery and video, where similar shortcut dynamics might exist.

Practically, the system’s design, interface simplicity (prompt + references), and demonstrated robustness position it as a backbone for open-ended scene-aware dialog modeling and content creation in real-world, multi-modal environments.

Limitations and Future Directions

Current limits are inherited from the backbone: 20s generation cap and $K_{\text{max}}=3$ references. Both are expandable with sequence length scaling and reference compression. Another fundamental constraint is fixed generation duration under flow-matching, requiring additional user or heuristic input.

Extension to longer scenes, more speakers, and adaptation of the high-noise training principle to visual or multi-modal settings constitutes promising directions. Cross-modal validation of shortcut and training regimen transferability should become a focus for expanding reference-driven generative modeling.

Conclusion

ScenA establishes a new paradigm for open-ended, multi-speaker audio scene generation using natural language and in-the-wild references. By leveraging the strengths of foundation audio models and overcoming shortcut pitfalls in flow-matching training, it achieves superior speaker binding, ambient realism, and interactional nuance without any structured dialog markup. The theoretical diagnosis and practical fix for the "reference shortcut" extends beyond audio, underscoring its broader impact on the design of reference-conditioned diffusion models.

(2606.19325)