ZipL-Dialog: Memory-Efficient Long-Form Spoken Dialog Synthesis via Latent Flow Matching

Abstract: Zero-shot dialog TTS benefits from flow-matching, but minute-scale generation on dense mel-spectrograms causes severe memory bottlenecks, often forcing unnatural chunked synthesis. We propose ZipL-Dialog, which shifts conditional flow-matching into a 4x time-compressed (25 Hz) latent space. To preserve acoustic fidelity under compression, we employ a deterministic mel autoencoder with auxiliary mel-domain supervision and optimize the ZipFormer's hierarchical downsampling schedule. Experiments show that ZipL-Dialog reduces maximum peak GPU memory by 11.22x and accelerates inference by 2.23x over the baseline, substantially lowering the memory footprint of single-pass multi-minute dialog synthesis while maintaining perceptual naturalness.

• The paper introduces a latent-space conditional flow matching approach with a deterministic autoencoder that significantly reduces memory usage for long-dialog synthesis.
• It employs a custom ZipFormer downsampling schedule and an auxiliary mel reconstruction loss to maintain high fidelity and naturalness despite aggressive compression.
• Experimental results demonstrate up to 11.22× memory reduction and improved perceptual quality, enabling efficient multi-minute dialog generation on commodity hardware.

ZipL-Dialog: Memory-Efficient Long-Form Spoken Dialog Synthesis via Latent Flow Matching

Introduction

ZipL-Dialog introduces a latent-space conditional flow-matching (CFM) approach tailored for memory- and compute-efficient synthesis of long-form, multi-turn spoken dialogue. The primary technical innovation is the displacement of flow-matching dynamics from the conventional dense mel-spectrogram domain (e.g., 100 Hz, high-dimensional acoustic sequences) into a compressed latent space operating at 25 Hz. This architectural shift alleviates the severe memory scaling associated with handling multi-minute dialogs, especially when compared with direct mel-domain synthesis paradigms.

Maintaining naturalness and fidelity in such a compressed domain is nontrivial due to inherent risks of phonetic and speaker information loss. ZipL-Dialog solves this via a deterministic mel autoencoder, reinforced by an auxiliary reconstruction loss in the mel domain and a customized ZipFormer downsampling hierarchy optimized for compressed representations.

Framework and Methodology

ZipL-Dialog’s synthesis pipeline comprises three stages: mel-to-latent encoding, latent-space CFM-based generation, and latent-to-mel decoding.

Figure 1: Training overview of ZipL-Dialog, depicting ground-truth mel-spectrogram encoding, ZipFormer-based CFM prediction on a masked latent region, and joint optimization with latent velocity and mel-domain reconstruction losses.

Deterministic Mel Autoencoder and Latent Compression

Unlike common VAE-based compressive paradigms, ZipL-Dialog employs a deterministic autoencoder. An input mel-spectrogram $\mathbf{Y} \in \mathbb{R}^{T \times F}$ is encoded via a patch-wise convolutional stem followed by Transformer blocks and a bottleneck projection, yielding a compressed latent sequence $\mathbf{Z} \in \mathbb{R}^{T' \times D}$ with $T'=T/4$ rate and $D=100$ dimensions. The decoder restores the temporal resolution through transposed convolutional upsampling and ConvNeXt-style refinement.

This deterministic approach was empirically validated to preserve phonetic and speaker details more robustly than variational approaches, which tend to induce oversmoothing of local detail under strong compression.

Latent Conditional Flow Matching with Masked Velocity Objective

In the latent domain, ZipL-Dialog forms a CFM training scenario where the observed context and target regions are explicitly separated by a learned binary mask. During training, only the unobserved (target) span is interpolated between Gaussian latent noise and the ground truth; velocity estimates are optimized exclusively with respect to this region.

A critical auxiliary signal, the mel-reconstruction loss, is applied to decoded outputs reconstructed from the denoised latent predictions. This term is essential for mitigating high-frequency degradation induced by latent compression and further enforces fidelity at the audio signal level.

ZipFormer Downsampling Schedule for Latent Contexts

Directly porting ZipFormer’s original hierarchical attention design to the compressed latent sequence misaligns the balance between local and global modeling, as aggressive downsampling in a low-rate space drastically enlarges each token’s temporal coverage, destroying fine-grained acoustic structure. ZipL-Dialog adopts a milder downsampling schedule ([1,1,2,1,1]), which, alongside temporal compression, secures both a broad receptive field for global dialog context and sufficient resolution for high-quality speech reconstruction.

Experimental Results

Memory and Runtime Efficiency

Across CoVoMix2 and OpenDialog benchmarks, ZipL-Dialog obtains up to 11.22$\times$ lower maximum peak GPU memory than the direct mel-domain ZipVoice-Dialog baseline, reducing maximum memory load from 36.21 GB to 3.23 GB on the most challenging samples. End-to-end inference is accelerated by 2.23$\times$ (vs. ZipVoice-Dialog) and >43$\times$ over the large-scale AR baseline VibeVoice 1.5B (Peng et al., 26 Aug 2025). Thus, ZipL-Dialog enables multi-minute dialog generation in a single pass on moderate hardware, addressing the central bottleneck of current Transformer-based sequence models in long-form speech synthesis.

Quality and Perceptual Trade-offs

While some objective degradation in ASR-derived WER and speaker similarity (cpSIM) is observed compared to uncompressed baselines, UTMOS—an established perceptual quality metric—remains preserved or even improved. For instance, on CoVoMix2, ZipL-Dialog matches VibeVoice 1.5B’s UTMOS (3.523) and marginally exceeds ZipVoice-Dialog, despite its substantial resource reduction. On OpenDialog, VibeVoice’s WER (12.979%) is worse than ZipL-Dialog (5.362%) and ZipVoice-Dialog (3.550%), emphasizing the difficulty AR models face in dialog settings.

Ablation Analyses

Ablations highlight that:

• A deterministic AE with auxiliary mel reconstruction loss is indispensable for intelligible, natural output; VAE-based latents halve intelligibility at comparable compression.
• The optimized ZipFormer downsampling ([1,1,2,1,1]) vastly outperforms the original aggressive schedule for latent-space modeling, validating the need for architectural adaptation in compressed regimes.

Theoretical and Practical Implications

ZipL-Dialog evidences that latent-space CFM is both viable and highly advantageous for long-duration dialog synthesis. The architectural paradigm moves past the conventional memory-bound regime, without resorting to unnatural chunked or truncated synthesis. The study clarifies that deterministic autoencoder bottlenecks with carefully calibrated auxiliary objectives are critical for the compressed setting, potentially influencing future neural codec designs (Langman et al., 2024, Wang et al., 4 Dec 2025).

Practically, the approach enables interactive and large-scale deployment of conversational TTS services on commodity hardware and in edge environments. Furthermore, the general modeling principles may be extensible to other generative sequence domains (music, multimodal), provided similar fidelity–compression trade-offs can be achieved.

Conclusion

ZipL-Dialog establishes a new operating point in the quality–efficiency spectrum for large-context spoken dialog generation by integrating deterministic latent compression, auxiliary reconstructive supervision, and a custom-tailored hierarchical backbone. The framework decouples GPU memory growth from sequence length in a way not previously realized in dialog TTS, without severe compromise in naturalness or dialog-level continuity. Remaining limitations around fine-grained phonetic detail invite continued exploration of improved latent representations and denoising objectives, with implications likely to inform both speech and broader generative modeling communities.