DialogueSidon: Recovering Full-Duplex Dialogue Tracks from In-the-Wild Dialogue Audio

Abstract: Full-duplex dialogue audio, in which each speaker is recorded on a separate track, is an important resource for spoken dialogue research, but is difficult to collect at scale. Most in-the-wild two-speaker dialogue is available only as degraded monaural mixtures, making it unsuitable for systems requiring clean speaker-wise signals. We propose DialogueSidon, a model for joint restoration and separation of degraded monaural two-speaker dialogue audio. DialogueSidon combines a variational autoencoder (VAE) operates on the speech self-supervised learning (SSL) model feature, which compresses SSL model features into a compact latent space, with a diffusion-based latent predictor that recovers speaker-wise latent representations from the degraded mixture. Experiments on English, multilingual, and in-the-wild dialogue datasets show that DialogueSidon substantially improves intelligibility and separation quality over a baseline, while also achieving much faster inference.

• The paper introduces DialogueSidon, which unifies restoration and speaker separation to recover full-duplex dialogue from degraded monaural recordings.
• It employs an SSL-VAE for compact latent speech representations and a diffusion transformer to refine speaker-specific latent vectors, achieving significant improvements in WER and MOS.
• Experiments across multilingual and in-the-wild datasets demonstrate a 60x speedup over baselines, underscoring its practical scalability for large-scale dialogue resource creation.

DialogueSidon: Joint Restoration and Separation for Full-Duplex Dialogue Recovery

Motivation and Problem Setting

The development of robust spoken dialogue systems capable of handling backchannels, overlapped speech, and flexible turn-taking is hindered by a scarcity of full-duplex conversational audio—that is, collections in which each speaker is recorded on an independent track. In contrast, the majority of internet-sourced, in-the-wild dialogue data is obtainable only in degraded monaural mixtures, confounded by environmental noise, compression artifacts, limited bandwidth, and overlapping speakers. Existing pipelines reliant on telephone or controlled environments are cost-prohibitive and ill-suited to capturing naturalistic phenomena (e.g., spontaneous overlaps, realistic turn exchanges). The synthesis of dialogue via TTS systems often fails to faithfully model these complex behaviors.

This work introduces DialogueSidon, a generative model explicitly architected for the joint restoration (denoising and enhancement) and speaker separation of degraded two-speaker monaural mixtures into clean, speaker-annotated full-duplex tracks.

Figure 1: Comparison of current full-duplex dialogue audio acquisition methods, which are costly and limited, versus DialogueSidon's approach to recovery from noisy monaural mixtures.

Model Architecture

DialogueSidon extends the Sidon speech restoration pipeline to the dialogue domain, unifying latent speech enhancement and separation. The backbone consists of an SSL-VAE (self-supervised learning-based variational autoencoder), which constructs a compact, speech-relevant latent representation; and a diffusion-based latent predictor that, conditioned on features from the degraded input, reconstructs a set of speaker-wise latent vectors from which clean dialogue tracks are decoded.

Key architectural choices include:

• Latent representations—extracted via a VAE from w2v-BERT 2.0 high-level features—are both low-dimensional and highly expressive for downstream diffusion-based generative modeling;
• A diffusion transformer operating in latent space refines auxiliary (coarse) speaker latents, resolving permutation ambiguity via permutation-invariant training;
• Conditioning is performed using deep SSL encoders adapted with LoRA, tailored to the domain of degraded mixtures.

Inference proceeds by extracting conditioning features from the mixture, producing auxiliary latent predictions, resolving speaker permutation, and running the diffusion reverse process to obtain fine-grained latents, which are then decoded to waveforms.

Figure 2: Overview of DialogueSidon training. Stage 1: SSL-VAE learns to encode/decode clean tracks. Stage 2: Diffusion-based latent predictor is trained to estimate speaker-wise latents from mixtures.

Experimental Methodology

Evaluation is structured around three corpora:

• Switchboard (SWB): English telephone dialogue, representing in-domain performance.
• CallFriend: Multilingual telephone dialogue (German, French, Japanese, Spanish, Mandarin), assessing generalization.
• OpenDialog: In-the-wild internet dialogue mixtures, representing the target scenario with unconstrained degradations.

The training corpus consists of 8,900+ hours of artificially degraded mixtures, with degradations sampled from real-world distributions (reverberation, environmental noise, channel reduction, quantization, packet loss, and more). Gold target tracks are produced using Sidon-enhanced speech as clean references.

Comprehensive evaluation uses both objective (WER/p-CER, DNSMOS, NISQA, speaker similarity, VAD accuracy) and subjective (MOS) metrics, with Whisper-v3 for intelligibility and Silero-based VAD.

GENESES (flow-matching-based separation and restoration (Asai et al., 26 Jan 2026)), Sidon (Nakata et al., 21 Sep 2025), and Noisy (unrestored input) serve as primary baselines. The effect of SSL-VAE latent dimensionality is also ablated.

Results

English (SWB): DialogueSidon achieves a minimum WER of 14.4% with $D=32$, compared with 33.5% (GENESES) and 57.5% (Sidon, single-channel restoration without separation). Notably, while GENESES achieves comparable or higher NISQA/DNSMOS, it exhibits significant content loss under intelligibility metrics—substantiating the importance of content-preserving separation. Subjective MOS is highest for DialogueSidon (3.90), indicating improved perceptual quality and speaker consistency.

Multilingual (CallFriend): Across all tested languages, DialogueSidon yields consistently lower p-CER and higher speaker similarity compared to GENESES, supporting strong language transfer and robustness to phonotactic diversity. For especially challenging scenarios (e.g., Mandarin), absolute performance drops, revealing potential future work for adaptation to heavily overlapped or highly non-stationary multilingual mixtures.

In-the-Wild (OpenDialog): On genuinely unconstrained internet mixtures, DialogueSidon delivers a WER of 13.9%, with MOS scores of 3.71, outperforming retrained and original GENESES in both axes. DialogueSidon achieves a 60x real-time factor (RTF) speedup over GENESES (0.01 RTF vs. 0.60 RTF), enabling practical deployment for large-scale corpus construction.

Technical Strengths and Contradictory Findings

• Joint Restoration and Separation: DialogueSidon's tight integration, as opposed to pipeline approaches (restoration then separation or vice versa), outperforms cascaded strategies, which struggle with overlap artifacts and content hallucination.
• Latent Space Design: Dimension ablation shows a sweet spot around $D=32$; larger capacity ($D=128$) degrades performance (possibly by overfitting/underconstrained latent mapping), contradicting expectations from single-speaker enhancement literature.
• Subjective/Objective Quality Trade-offs: DialogueSidon decouples perceptual quality from intelligibility and speaker assignment, directly addressing the requirements for resource construction rather than optimizing for standard speech enhancement metrics alone.

Implications and Future Directions

DialogueSidon provides a scalable method for recovering full-duplex, speaker-annotated dialogue datasets from existing internet-scale monaural mixed sources. Practically, this enables the creation of resources critical for next-generation conversational AI modeling—systems robust to overlap, rapid turn exchanges, and noisy real-world input. Theoretically, the diffusion-latent framework here demonstrates that fine-grained separation and restoration can be addressed in a unified, permutation-aware manner, challenging prior assumptions about necessary separation between denoising and speaker disentanglement.

Future research directions include:

• Generalizing to more than two simultaneous speakers, with modeling innovations for scaling permutation-invariant training and speaker assignment.
• Extending coverage to extremely low-resource and highly code-switched dialogues, leveraging advances in SSL-based universal feature extraction.
• Integration with multi-modal conversational modeling (audio-visual data) and alignment for dialogue agent self-training.

Conclusion

DialogueSidon constitutes a highly effective and efficient pipeline for full-duplex dialogue recovery from in-the-wild monaural mixtures. The combination of SSL-driven VAE latent representations and diffusion-based separation/refinement delivers domain-agnostic content preservation, speaker disentanglement, and practical runtimes suitable for large-scale deployment. The results position DialogueSidon as a foundational tool for constructing the next generation of conversational resources, accelerating progress in robust dialogue agent development (2604.09344).