WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling

Abstract: Integrating speech understanding and generation is a pivotal step toward building unified speech models. However, the different representations required for these two tasks currently pose significant compatibility challenges. Typically, semantics-oriented features are learned from self-supervised learning (SSL), and acoustic-oriented features from reconstruction. Such fragmented representations hinder the realization of truly unified speech systems. We present WavCube, a compact continuous latent derived from an SSL speech encoder that simultaneously supports speech understanding, reconstruction, and generation. WavCube employs a two-stage training scheme. Stage 1 trains a semantic bottleneck to filter off-manifold redundancy that makes raw SSL features intractable for diffusion. Stage 2 injects fine-grained acoustic details via end-to-end reconstruction, while a semantic anchoring loss ensures the representation remains grounded within its original semantic manifold. Comprehensive experiments show that WavCube closely approaches WavLM performance on SUPERB despite an 8x dimensional compression, attains reconstruction quality on par with existing acoustic representations, delivers state-of-the-art zero-shot TTS performance with markedly faster training convergence, and excels in speech enhancement, separation, and voice conversion tasks on the SUPERB-SG benchmark. Systematic ablations reveal that WavCube's two-stage recipe resolves two intrinsic flaws of SSL features for generative modeling, paving the way for future unified speech systems. Codes and checkpoints are available at https://github.com/yanghaha0908/WavCube.

• The paper presents a two-stage compress-then-enrich approach that compresses high-dimensional SSL features into a compact 128-dim latent while preserving acoustic details.
• It achieves superior performance in speech reconstruction, understanding, and zero-shot TTS with notable metrics such as STOI of 0.97 and WER of 4.20%.
• It demonstrates that unified semantic-acoustic modeling eliminates dual-tower pipelines, paving the way for simpler, scalable, and efficient speech systems.

Unifying Speech Understanding and Generation: An In-Depth Analysis of WavCube

Motivation and Background

Conventional speech systems rely on disjoint representations for understanding and generation: self-supervised learning (SSL) features, such as those from WavLM or HuBERT, dominate understanding due to their high-level semantic compositionality, while acoustic-oriented features like Mel-spectrograms or VAE latents are preferred for generative modeling because of their fine-grained signal fidelity. This dichotomy enforces a dual-tower pipeline, inhibits joint modeling, and hinders progress toward unified speech systems—a gap the vision community has begun bridging via representation-centric generative models. WavCube (2605.06407) directly addresses this fundamental challenge by constructing a single, compact continuous latent that unifies speech understanding, reconstruction, and generation via semantic-acoustic joint modeling.

WavCube Architecture and Training Paradigm

WavCube is predicated on a two-stage "compress-then-enrich" approach leveraging a frozen SSL encoder (WavLM-Large) as the semantic backbone. This architecture, shown in (Figure 1), systematically resolves the high-dimensional redundancy and acoustic fidelity gaps that frustrate naive SSL repurposing for generation.

Figure 1: The WavCube architecture: Stage 1 compresses SSL features into a 128-dimensional latent while separately pre-warming the acoustic decoder. Stage 2 jointly injects acoustic detail via reconstruction, anchored to the semantic manifold to prevent drift.

Stage 1: Semantic Feature Compression

A symmetric autoencoder distills the 1024-dim SSL features into a diffusion-friendly 128-dim bottleneck ($z$), using a compressor and complementary restorer trained to minimize MSE and cosine distance to the original features. In parallel, an acoustic decoder is warmed-up on the detached latent, preventing semantic interference. This process aggressively reduces redundant, off-manifold ambient structure, yielding a latent suitable for generative modeling.

Stage 2: Semantic-Acoustic Joint Enrichment

The pipeline is then optimized end-to-end, unfreezing the SSL encoder and conditioning both the latent and encoder for high-fidelity acoustic reconstruction. Two semantic anchoring losses—on adapted encoder features and reconstructed features—strictly align both with the original SSL manifold, ensuring semantic structure is not compromised during enrichment.

Representation Analysis: Semantic Structure and Discriminability

WavCube's latent space balances semantic separability and compactness. t-SNE visualization on ESC-50 (Figure 2) demonstrates that traditional acoustic latents form overlapping, poorly discriminated clusters, while WavCube exhibits compact intra-class grouping and pronounced inter-class boundaries, mirroring the strong discriminative organization of SSL features.

Figure 2: 2D t-SNE projections of various representations on ESC-50; WavCube achieves superior intra-class compactness and inter-class separation versus acoustic and VAE-based baselines.

Empirical Evaluation

Speech Reconstruction

Despite $8 \times$ compression, WavCube matches VAE-based and Mel-spectrogram baselines in reconstruction tasks: on LibriSpeech test-clean, it achieves STOI of 0.97, UTMOS of 4.04, speaker similarity (SIM) of 0.94, and WER of 4.20%, rivaling much higher-dimensional or exclusively reconstruction-trained alternatives.

Speech Understanding

On the SUPERB benchmark, WavCube approaches the SSL upper bound with a 128-dim latent: it substantially outperforms acoustic baselines (Fbank, VAE, Semantic-VAE) across all content, speaker, and paralinguistic understanding tasks, while incurring only a minor performance drop relative to the original 1024-dim WavLM-Large—showing robust preservation of semantic topologies even after aggressive compression and acoustic enrichment.

Speech Generation

In zero-shot TTS, WavCube yields state-of-the-art results, outperforming VAE, Semantic-VAE, and Mel-spectrogram baselines on both LibriTTS and the large-scale Emilia corpus. Notably, it enables significantly faster DiT convergence and better speaker similarity (e.g., WER 1.86% and SIM 0.678 on LibriTTS). Against large-scale models (e.g., F5-TTS, CosyVoice, FireRedTTS), WavCube achieves superior or comparable WER and speaker similarity at similar or lower computational scales.

Additionally, on SUPERB-SG generative tasks (Speech Enhancement, Separation, Voice Conversion), WavCube latents provide strong signal fidelity and identity preservation, closely tracking the performance of the original SSL encoder and consistently outperforming acoustic alternatives.

Ablation Study and Architectural Insights

Directly using high-dimensional SSL features for generation results in severe failure due to redundant noise and loss of decodable acoustic detail. Even massive diffusion transformer scaling (from 339M to 753M parameters) fails to bridge this gap, resulting in unreadable output and poor speaker similarity. The WavCube pipeline demonstrates through systematic ablation that the two-stage paradigm is essential:

• Stage 1: Filters redundancy and improves diffusion tractability at the cost of some acoustic detail.
• Stage 2: Recoups fidelity via acoustic enrichment while semantic anchoring prevents degeneration to a purely acoustic latent.

The architectural choices (autoencoder bottleneck, 50 Hz frame rate, 128-dim bottleneck, last SSL layer) are validated to best trade off intelligibility and fidelity.

Theoretical and Practical Implications

WavCube establishes that semantic compressive representation from SSL encoders, when enriched by acoustic fine-tuning under semantic anchoring, can simultaneously unify understanding and generative modeling in speech. This design harmonizes three previously competing desiderata: semantic discriminability, acoustic fidelity, and diffusion-friendliness. Practically, this enables simpler, more efficient, and scalable deployment of speech systems for TTS, speech enhancement, separation, and other generative or discriminative tasks. Theoretically, it closes a major representational gap and aligns the speech domain with recent paradigmatic shifts in vision and language.

Future Directions

WavCube provides a concrete substrate for fully unified speech architectures, facilitating native joint modeling across understanding and generation. Anticipated future developments include:

• Construction of large-scale, natively unified speech LMs on WavCube latents.
• Extension to multilingual, multimodal, and interactive speech settings.
• Deeper integration into cross-modal reasoning and conversational agents, eliminating format translation bottlenecks.

Conclusion

WavCube (2605.06407) systematically unifies speech understanding, reconstruction, and generation via a compact, semantically anchored, and acoustically enriched continuous latent. Comprehensive empirical validation affirms that strong semantic and generative performance can coexist in a low-dimensional embedding. The two-stage compress-then-enrich paradigm defined by WavCube offers a principled blueprint for eliminating fragmented dual-tower speech architectures, catalyzing the next phase of unified speech systems.