ProsoCodec: Prosody-Oriented Speech Codec for Voice Conversion

Abstract: Neural speech codecs efficiently compress speech and have become a foundation for speech generation, but they are typically learned as holistic representations that intertwine linguistic content, speaker identity, and prosody. While this design is effective for zero-shot voice cloning, it hinders downstream tasks that require prosody preservation or transfer, such as voice conversion. To address this, we introduce ProsoCodec, a prosody-oriented speech codec that models prosody as a conditional residual rather than as a disentangled stream. Specifically, by conditioning both the encoder and decoder on text and speaker embeddings as prefix tokens, the discrete bottleneck is encouraged to capture prosodic variation not explained by content and speaker. To further preserve prosody, we use the low-frequency mel band and train the model on paired same-speaker utterances. Experiments on voice conversion show improved prosody preservation and reduced source-timbre leakage.

• The paper presents ProsoCodec, a conditional residual modeling approach that isolates prosody cues to achieve precise voice conversion.
• It employs a Transformer-based encoder with BSQ quantization and a diffusion-based decoder conditioned on text and speaker priors for effective prosody preservation.
• Experimental results on LibriTTS and VCTK show low WER, minimal source leakage, and superior prosody transfer compared to previous methods.

ProsoCodec: Prosody-Oriented Speech Codec for Voice Conversion

Motivation and Background

Neural speech codecs have become fundamental components in state-of-the-art speech generation, reliably compressing speech signals into compact discrete representations that facilitate high-fidelity reconstruction and text-driven synthesis. However, their holistic representation intertwines content, timbre, and prosody, hampering fine-grained attribute manipulation and downstream tasks demanding precise prosody preservation or transfer. Voice conversion is a prototypical scenario where the source prosody must be strictly retained, while the timbre is mapped to a target speaker. Previous efforts based on attribute decomposition or explicit disentanglement either hinge on adversarial training or suffer from residual entanglement, limiting expressiveness and failing to capture contextual prosodic nuances.

Architecture and Residual Prosody Modeling

ProsoCodec addresses the above limitations through a conditional residual modeling paradigm. By conditioning both the encoder and diffusion-based decoder on robust text and speaker priors, the information bottleneck is compelled to capture only prosodic variation unexplained by content and speaker identity. This approach avoids brittle disentanglement mechanisms and leverages pretrained ASR and SV embeddings for high-fidelity prior guidance.

The architecture consists of a Transformer-based encoder and a Diffusion Transformer (DiT) decoder. During training, source and reference utterances are sampled from the same speaker, while at inference they can be from different speakers. The encoder ingests low-frequency mel-spectrograms (prioritizing prosodic cues) concatenated with text and speaker embeddings as prefix tokens, thereby biasing the representation toward prosodic style (Figure 1).

Figure 1: Model architecture of ProsoCodec—conditional residual prosody modeling via prefix-conditioned encoder and diffusion-based decoder.

The encoder output is quantized via binary spherical quantization (BSQ), enforcing a strict information bottleneck and further mitigating redundant content or timbre capture. The decoder reconstructs the mel-spectrogram by combining the quantized source tokens (holding source prosody) with the full-band reference mel-spectrogram and robust speaker/text embeddings, conditioned through adaptive normalization with diffusion timesteps. Training utilizes dual-utterance style alternation and random span masking for effective leakage suppression and prosody preservation.

Experimental Evaluation

ProsoCodec is trained on LibriTTS and evaluated on LibriTTS test-clean, test-other, and VCTK datasets for out-of-domain generalization. Evaluation metrics encompass objective (WER, cosine speaker similarity, $f_0$ RMSE) and subjective (MOS for speaker similarity, prosody similarity, naturalness) criteria. Robust ASR (Whisper-large-v3) and SV (WavLM-Large) models isolate evaluation from potential bias.

Notably, ProsoCodec achieves a WER of 4.451, significantly outperforming previous codecs in content preservation. SIM$_\text{r}$ of 0.565 and SIM$_\text{s}$ of 0.167 indicate superior target timbre adoption and minimal source leakage. Prosody similarity MOS scores (3.852) and $f_0$ RMSE (0.428) demonstrate fine-grained prosody transfer with minimal distortion. Comparative ablation studies verify the impact of each architectural element: removal of text conditioning results in catastrophic WER degradation, absence of speaker conditioning increases prosody leakage, and disabling dual-utterance training impairs prosody transfer fidelity.

The pitch contours in Figure 2 empirically demonstrate the ability of ProsoCodec to precisely resynthesize the source $f_0$ trajectory, outperforming zero-shot TTS baselines.

Figure 2: Pitch contour comparison—ProsoCodec preserves source prosody more faithfully than zero-shot TTS.

Bottleneck configuration ablations confirm that lowering bitrate and focusing on low-frequency bands reduce timbre leakage and optimize the trade-off between resynthesis accuracy and conversion quality.

Implications and Future Prospects

ProsoCodec establishes conditional residual modeling as an effective paradigm for prosody-preserving voice conversion, sidestepping adversarial disentanglement and enabling token-efficient, context-sensitive prosody control. The explicit conditioning on pretrained feature embeddings leverages advances in ASR and SV to drive robust conditional representation learning.

Practically, ProsoCodec can be deployed for expressive TTS, voice conversion, spoken dialogue systems, and any task requiring precise prosodic manipulation across domains or languages. Theoretical implications extend to the design of selective neural codecs—conditional bottlenecks tailored for downstream attribute control, rather than maximal holistic reconstruction. Future directions may include hierarchical residual modeling, multimodal conditioning (style or emotion tokens), cross-lingual prosody transfer, and integration with LLM-driven speech generation frameworks. The dual-utterance training regime could be further generalized for sequence-level control in other generative modalities.

Conclusion

ProsoCodec pioneers prosody-oriented speech tokenization through explicit conditional residual modeling, delivering robust zero-shot voice conversion with superior prosody and timbre preservation. The architecture, annihilating prompt-style leakage and leveraging low-frequency prosody cues, sets a strong precedent for flexible prosodic control in generative speech systems. Its methodological innovations and empirical performance suggest broad applicability and motivate further exploration of conditional residual codecs for attribute-controllable speech generation (2606.21888).