SARA: A Dual-Stream VAE for High-Fidelity Speech Generation via Integrating Semantic and Acoustic Representations

Abstract: Zero-shot text-to-speech (TTS) relies on robust speech representations. However, current speech tokenizers face a fundamental trade-off: acoustic codecs preserve high-fidelity audio but lack linguistic constraints, causing content errors during generation, whereas semantic tokens from self-supervised learning (SSL) models ensure precise text alignment but discard some acoustic information. To bridge this gap, we propose SARA, a dual-stream VAE that directly fuses a frozen SSL semantic anchor with a dedicated residual acoustic encoder. This effectively mitigates the dilemma, creating an efficient and compact latent space without relying on complex regularizers. SARA achieves superior reconstruction quality over strong baselines. Furthermore, in downstream zero-shot TTS tasks, it yields highly natural and expressive synthesis quality, and maintains robust generation performance even under accelerated inference, offering a favorable trade-off between synthesis speed and computational cost.

• The paper presents a dual-stream VAE architecture that directly integrates a frozen SSL semantic encoder with a trainable residual acoustic encoder for enhanced TTS quality.
• It achieves state-of-the-art speech reconstruction with a compressed 64-dimensional latent space at 50 Hz, significantly improving PESQ, STOI, and UTMOS metrics.
• The method sets new benchmarks in zero-shot TTS, offering robust speaker similarity and acceleration-friendly inference suitable for real-world applications.

SARA: Integrating Semantic and Acoustic Representations for High-Fidelity Speech Generation

Motivation and Problem Formulation

Advancements in zero-shot TTS have underscored the criticality of speech tokenization for text- and speaker-controllable natural speech synthesis. However, existing tokenization paradigms manifest a fundamental trade-off: acoustic codecs facilitate high-level reconstruction fidelity, preserving signal detail, but are agnostic to linguistic structure, resulting in inferior alignment and elevated content error rates. Conversely, semantic representations derived from SSL or ASR models impose strong alignment, supporting robust content preservation, but systematically eliminate information critical to vocal timbre, speaker identity, prosody, and emotional content. Efforts that introduce auxiliary regularization losses within VAE frameworks—such as semantic alignment objectives—have seen partial improvement but remain indirect and challenging to tune. There is a persistent need for an architectural solution that can simultaneously and explicitly capture complementary semantic and acoustic information in a compact and effective latent space.

The SARA Architecture

SARA (Semantic-Acoustic Residual Autoencoder) is presented as a dual-stream VAE specifically designed to overcome the aforementioned trade-off by directly integrating complementary semantic and acoustic latent factors.

The overall architecture fuses two heterogeneous encoding streams:

• A frozen, pretrained W2v-BERT 2.0 encoder as a semantic branch provides temporally aligned, content-rich representations, acting as an invariant anchor for linguistic information.
• A learnable residual acoustic encoder, based on deep, strided CNNs and LSTM modules (inspired by BigCodec), captures information orthogonal to the semantic anchor—most prominently, speaker-specific and fine-grained waveform details.

At the output of both encoders, framewise concatenation is performed, followed by channel reduction via linear projection to obtain a 64-dimensional latent space at 50 Hz. This joint latent is then decoded using a HiFi-GAN-based architecture, which enables high-fidelity waveform reconstruction. Adversarial and feature matching losses are employed, together with an ELBO objective, to regularize the latent manifold and maximize perceptual quality.

Figure 1: SARA’s dual-stream VAE architecture—frozen SSL encoder for semantic anchoring, residual CNN/LSTM for complementary acoustic encoding, followed by latent fusion and HiFi-GAN decoding.

Training Paradigm and Optimization

The VAE backbone is trained on a heterogeneous corpus of over 50,000 hours spanning LibriTTS and LibriHeavy, with all data upsampled to 24 kHz. Waveform segments are subjected to multi-scale mel-spectrogram alignment for reconstruction loss, adversarial discrimination, and L1 feature matching. The two-branch encoder design is directly optimized without explicit semantic regularizers—representations are allowed to disentangle via architectural separation, with the semantic branch frozen to provide a consistent, content-anchored reference point. The dual-stream integration ensures high-fidelity detail is archived even as overall latent dimensionality is aggressively compressed.

Empirical Results

Speech Reconstruction

SARA establishes new state-of-the-art performance on the LibriSpeech test-clean set in both objective and subjective reconstruction metrics. At only 50 frames/s and a 64-dimensional latent bottleneck, SARA achieves a PESQ of 4.389 and STOI of 0.993, outperforming high-bandwidth Vocos and prior Semantic-VAE baselines. The competitive UTMOS of 4.100 exceeds ground truth, highlighting strong perceptual fidelity. Notably, ablative removal of the residual acoustic encoder leads to pronounced drops in both speaker similarity and acoustic metrics, evidencing that the frozen SSL anchor alone is insufficient for full-fidelity speech.

Zero-Shot TTS

Integrated with an F5-TTS backbone, SARA sets a new benchmark in zero-shot synthesis, yielding a WER of 1.79 (F5-TTS-Small) and 1.74 (F5-TTS-Base), which is a substantial improvement over existing parameter-heavy systems (e.g., Cosyvoice, E2-TTS, and vanilla F5-TTS). Speaker similarity and naturalness metrics confirm SARA’s ability to generalize robustly across both content and timbre dimensions.

Additionally, SARA maintains synthesis quality under accelerated inference (low flow-matching steps), demonstrating the regularization strength of its dual-branch latent structure. A negligible degradation in SIM and WER with reduced NFE further positions SARA as a practical candidate for real-time high-fidelity TTS applications.

Implications and Future Directions

SARA’s architecture demonstrates that direct architectural fusion of SSL-based semantic anchors and trainable acoustic encoders is a superior strategy to regularized, monolithic VAEs. Unlike previous methods that rely on explicit loss-based alignment, SARA natively exploits temporal synchrony between branches to enable concise, information-rich latent representations.

The implications are considerable for both practical and theoretical research in speech generation:

• Practical: Provides a drop-in, high-fidelity representation mechanism for zero-shot and few-shot TTS with strong robustness to content and speaker variation; supports acceleration-friendly inference without forfeiting output quality.
• Theoretical: Validates a strong inductive bias—namely, explicit separation and fusion of high-level semantic and low-level acoustic encoders—over simple regularization or naive feature mixing. This architectural prior can inform future designs for encoder-decoder models in speech and sequence generation more broadly.

Future research is likely to explore extensions to multi-lingual tokenization, multi-modal representation, and integration with AR and non-AR generative backbones. The separation principle underlying SARA’s dual-stream design may stimulate analogous innovations in other generative domains where content and style disentanglement is paramount.

Conclusion

SARA provides compelling empirical and architectural evidence that dual-stream VAEs, fusing a frozen SSL semantic anchor and a dedicated residual acoustic encoder, constitute an effective solution to the entrenched semantic-acoustic representation trade-off in high-fidelity TTS. Its robust performance across both objective and subjective evaluations, combined with architectural simplicity and strong inference efficiency, positions SARA as a new reference model for the next generation of neural speech generation frameworks.