ProsoCodec: Prosody-Centric Speech Codec

• ProsoCodec is a prosody-oriented speech codec that models prosodic nuances as a conditional residual using text and speaker embeddings.
• It employs a prefix-conditioned residual architecture with binary spherical quantization on low-frequency mel features to efficiently capture pitch, energy, and rhythm.
• Empirical evaluations show that ProsoCodec achieves balanced performance in ASR, speaker similarity, and prosody retention, outperforming traditional voice conversion systems.

Searching arXiv for ProsoCodec and closely related codec papers to ground the article in recent literature. ProsoCodec denotes a prosody-oriented line of speech codec research concerned with representing pitch, energy, rhythm, and related temporal variation in a form that is controllable, compact, and minimally contaminated by speaker timbre or linguistic content. In its most specific sense, ProsoCodec is a prosody-oriented speech codec for voice conversion that models prosody as a conditional residual rather than as a disentangled stream, conditioning both encoder and decoder on text and speaker embeddings as prefix tokens so that the discrete bottleneck is encouraged to encode residual prosodic variation (Choi et al., 20 Jun 2026). In a broader research sense, the term also intersects with prosody-first codec design principles developed in disentangled codecs, prosody-aware codec encoders, and multi-stream residual codecs for zero-shot TTS and voice conversion (Li et al., 15 Dec 2025).

1. Problem formulation and conceptual basis

Standard neural speech codecs are typically optimized for holistic reconstruction under a tight bitrate, so their discrete tokens encode linguistic content, speaker identity, and prosody jointly. This is useful for zero-shot voice cloning, where reproducing the prompt speech holistically is desirable, but it is problematic for voice conversion and controllable synthesis, where the objective is to preserve or transfer prosody without carrying over source timbre. The literature summarized here describes the resulting failure modes as source-timbre leakage, prosody contamination, and limited independent control (Choi et al., 20 Jun 2026).

A central conceptual divide concerns whether prosody should be represented as an explicitly disentangled stream or as a residual conditioned on content and speaker. DisCodec, FreeCodec, AugCodec, and MSR-Codec all implement explicit factorization of speech attributes into separate pathways or streams, with dedicated prosody latents or tokens (Li et al., 15 Dec 2025). ProsoCodec instead argues that speaker- and content-conditioned prosodic nuances are not truly independent of content and speaker, and therefore models prosody as a conditional residual. In this view, the codec should not attempt to strip away all interaction between prosody, text, and timbre; rather, it should provide text and speaker priors explicitly so that the bottleneck need only encode what remains unexplained.

This distinction matters operationally. In explicit disentanglement systems, the design problem is to reduce mutual information or empirical leakage between prosody and other factors. In ProsoCodec, the design problem is to bias the discrete bottleneck toward prosodic residuals by supplying text and speaker context directly to both encoder and decoder, by restricting encoder input to the low-frequency mel band, and by using paired same-speaker utterances during training (Choi et al., 20 Jun 2026).

2. Prefix-conditioned residual architecture

ProsoCodec is implemented as a diffusion autoencoder with a discrete bottleneck based on binary spherical quantization. The codec operates on mel features rather than raw waveform inside the codec. Its encoder is a Transformer with 8 layers, hidden size 1024, intermediate size 4096, and 16 attention heads; its decoder is a 16-layer Diffusion Transformer with the same hidden sizes, with timestep embedding injected via adaptive layer normalization (Choi et al., 20 Jun 2026).

The conditioning sources are explicit. Text is obtained with Qwen3-ASR, then embedded through a lookup table and an MLP-based text encoder to yield $e_{txt} \in \mathbb{R}^{L \times D}$. Speaker identity is obtained from CAM++, then mapped through an MLP-based speaker encoder to $e_{spk} \in \mathbb{R}^{1 \times D}$. Both are injected as prefix tokens on the time axis for encoder and decoder. The encoder sequence is

$$e = [e_{spk}; e_{txt}; e_{mel}] \in \mathbb{R}^{(1 + L + T) \times D},$$

with the corresponding hidden states

$h = Encoder(e), \quad h' = h_{1+L : 1+L+T}.$

The quantizer applies binary spherical quantization after interpolation:

$$h_q = Quantizer(Interp(h')), \quad h_q \in \{-1, 1\}^{T' \times d}.$$

The decoder receives the same prefix priors together with the diffusion-state input and quantized codes:

$$e = [e_{spk}; e_{txt}; e_{noisy} + e_q], \quad h = Decoder(e), \quad h' = h_{1+L : 1+L+T}.$$

The conditional-residual interpretation is stated explicitly:

$$z \approx r_{prosody}, \quad x \approx Dec(r_{prosody}; [e_{text}, e_{spk}]).$$

This formulation pushes the discrete bottleneck toward prosodic variability because content and speaker are already available as explicit priors. The encoder is additionally biased toward prosody by seeing only a low-frequency subset of 128-bin mel-spectrograms at 24 kHz, with 80 ms window and 20 ms hop, while the decoder prompt uses the full-band mel to anchor speaker timbre. The paper does not fix an exact low-frequency cutoff, but it emphasizes the biasing strategy (Choi et al., 20 Jun 2026).

Bitrate is controlled through token frame rate and bottleneck dimensionality. The chosen operating point is 12.5 Hz with codebook size 4096, corresponding to approximately 150 bps. Other reported settings are 25 Hz with codebook size 4096 and 300 bps, 12.5 Hz with codebook size 16384 and 175 bps, 12.5 Hz with codebook size 1024 and 125 bps, and 6.25 Hz with codebook size 4096 and 75 bps (Choi et al., 20 Jun 2026).

3. Objectives, training protocol, and inference workflow

ProsoCodec is trained with conditional flow matching rather than adversarial or explicit VQ commitment losses. Given clean mel $x \in \mathbb{R}^{T \times 128}$, noise $\epsilon \sim N(0, I)$, and $t \sim U(0, 1)$, the interpolation is

$e_{spk} \in \mathbb{R}^{1 \times D}$0

With a random span mask $e_{spk} \in \mathbb{R}^{1 \times D}$1, the partially noised input becomes

$e_{spk} \in \mathbb{R}^{1 \times D}$2

The Diffusion Transformer predicts the velocity field, and the training loss is

$e_{spk} \in \mathbb{R}^{1 \times D}$3

where $e_{spk} \in \mathbb{R}^{1 \times D}$4 corresponds to the linear interpolation schedule and is effectively proportional to $e_{spk} \in \mathbb{R}^{1 \times D}$5 in this parameterization (Choi et al., 20 Jun 2026).

Training uses a dual-utterance strategy designed specifically for prosody preservation. Two utterances from the same speaker are used per sample: one as the prompt and the other as the source to be regenerated. This is alternated with random span masking. The stated purpose is to prevent prompt-style leakage, so that the decoder learns to rely on source tokens for utterance-specific prosody while using the prompt primarily for speaker timbre. Model initialization is taken from TaDiCodec. Optimization uses AdamW with learning rate $e_{spk} \in \mathbb{R}^{1 \times D}$6, $e_{spk} \in \mathbb{R}^{1 \times D}$7, $e_{spk} \in \mathbb{R}^{1 \times D}$8, weight decay $e_{spk} \in \mathbb{R}^{1 \times D}$9, 10k-step linear warmup, and 150k updates total, with a global batch size of 160 seconds of audio (Choi et al., 20 Jun 2026).

The training corpus is LibriTTS, 585 hours, 24 kHz, 2,456 speakers. At inference, the voice conversion pipeline takes a source waveform $$e = [e_{spk}; e_{txt}; e_{mel}] \in \mathbb{R}^{(1 + L + T) \times D},$$0 and a target/reference waveform $$e = [e_{spk}; e_{txt}; e_{mel}] \in \mathbb{R}^{(1 + L + T) \times D},$$1. The source transcript is obtained from ASR if needed, then embedded as text conditioning; the reference utterance provides the speaker embedding. Source prosody tokens are extracted from the low-frequency mel band, quantized with BSQ, and decoded with the target speaker prefix. The decoder uses 32 ODE sampling steps with an Euler solver, and Vocos synthesizes the 24 kHz waveform (Choi et al., 20 Jun 2026).

The same paper reports a zero-shot TTS ablation in which the decoder runs without codec tokens. In that setting, the model behaves like a prompt-driven TTS or voice cloning system: it follows prompt timbre well, but prosody drifts toward the prompt or toward generic text-driven prosody. The codec tokens therefore function as the vehicle for source-prosody retention rather than merely as an auxiliary conditioning channel (Choi et al., 20 Jun 2026).

4. Empirical performance in voice conversion

Evaluation is reported on a merged LibriTTS test-clean, test-other, and VCTK setup for utterances of duration 2–8 s. Objective metrics are ASR WER with Whisper-large-v3, speaker similarity to the reference speaker $$e = [e_{spk}; e_{txt}; e_{mel}] \in \mathbb{R}^{(1 + L + T) \times D},$$2 using WavLM-Large, source leakage $$e = [e_{spk}; e_{txt}; e_{mel}] \in \mathbb{R}^{(1 + L + T) \times D},$$3, log-F0 RMSE using Harvest F0, and UTMOS. Subjective evaluation uses 15 listeners per subset and reports S-MOS, P-MOS, and N-MOS on a 5-point scale (Choi et al., 20 Jun 2026).

On the merged evaluation, ProsoCodec reports WER 4.451, $$e = [e_{spk}; e_{txt}; e_{mel}] \in \mathbb{R}^{(1 + L + T) \times D},$$4 0.565, $$e = [e_{spk}; e_{txt}; e_{mel}] \in \mathbb{R}^{(1 + L + T) \times D},$$5 0.167, S-MOS $$e = [e_{spk}; e_{txt}; e_{mel}] \in \mathbb{R}^{(1 + L + T) \times D},$$6, RMSE 0.428, P-MOS $$e = [e_{spk}; e_{txt}; e_{mel}] \in \mathbb{R}^{(1 + L + T) \times D},$$7, UTMOS 3.853, and N-MOS $$e = [e_{spk}; e_{txt}; e_{mel}] \in \mathbb{R}^{(1 + L + T) \times D},$$8 (Choi et al., 20 Jun 2026). The paper summarizes these outcomes as follows: best WER versus Vevo 4.826 and Seed-VC 5.078; highest $$e = [e_{spk}; e_{txt}; e_{mel}] \in \mathbb{R}^{(1 + L + T) \times D},$$9 0.565 and lowest $h = Encoder(e), \quad h' = h_{1+L : 1+L+T}.$0 0.167; best log-F0 RMSE 0.428 and best P-MOS $h = Encoder(e), \quad h' = h_{1+L : 1+L+T}.$1; and naturalness that is comparable to Vevo’s N-MOS $h = Encoder(e), \quad h' = h_{1+L : 1+L+T}.$2.

Ablation results on LibriTTS test-clean isolate the contribution of each design choice. The full model yields WER 4.684, $h = Encoder(e), \quad h' = h_{1+L : 1+L+T}.$3 0.608, $h = Encoder(e), \quad h' = h_{1+L : 1+L+T}.$4 0.168, RMSE 0.376, and UTMOS 3.978. Removing the low-frequency mel bias gives WER 4.810, $h = Encoder(e), \quad h' = h_{1+L : 1+L+T}.$5 0.585, $h = Encoder(e), \quad h' = h_{1+L : 1+L+T}.$6 0.192, RMSE 0.386, and UTMOS 4.004, which the paper interprets as worse prosody retention and increased source-timbre leakage. Removing dual-utterance training gives WER 5.630, $h = Encoder(e), \quad h' = h_{1+L : 1+L+T}.$7 0.634, $h = Encoder(e), \quad h' = h_{1+L : 1+L+T}.$8 0.166, RMSE 0.421, and UTMOS 3.884, indicating increased prosody error. Removing speaker conditioning gives WER 6.793, $h = Encoder(e), \quad h' = h_{1+L : 1+L+T}.$9 0.600, $$h_q = Quantizer(Interp(h')), \quad h_q \in \{-1, 1\}^{T' \times d}.$$0 0.186, RMSE 0.397, and UTMOS 3.891. Removing text conditioning gives WER 86.601, $$h_q = Quantizer(Interp(h')), \quad h_q \in \{-1, 1\}^{T' \times d}.$$1 0.569, $$h_q = Quantizer(Interp(h')), \quad h_q \in \{-1, 1\}^{T' \times d}.$$2 0.108, RMSE 0.434, and UTMOS 3.569, establishing that explicit text is essential to the conditional-residual design (Choi et al., 20 Jun 2026).

The bottleneck trade-off is also quantified. At 25 Hz, codebook size 4096, and 300 bps, VC performance is WER 4.49, $$h_q = Quantizer(Interp(h')), \quad h_q \in \{-1, 1\}^{T' \times d}.$$3 0.56, $$h_q = Quantizer(Interp(h')), \quad h_q \in \{-1, 1\}^{T' \times d}.$$4 0.20, RMSE 0.36. At 12.5 Hz, codebook size 4096, and 150 bps, VC performance is WER 4.68, $$h_q = Quantizer(Interp(h')), \quad h_q \in \{-1, 1\}^{T' \times d}.$$5 0.61, $$h_q = Quantizer(Interp(h')), \quad h_q \in \{-1, 1\}^{T' \times d}.$$6 0.17, RMSE 0.38, while resynthesis gives WER 4.30, SIM 0.71, RMSE 0.22. At 6.25 Hz, codebook size 4096, and 75 bps, VC performance becomes WER 6.17, $$h_q = Quantizer(Interp(h')), \quad h_q \in \{-1, 1\}^{T' \times d}.$$7 0.64, $$h_q = Quantizer(Interp(h')), \quad h_q \in \{-1, 1\}^{T' \times d}.$$8 0.14, RMSE 0.41. The reported trend is that higher bitrate improves resynthesis but increases source timbre leakage, whereas the moderate 12.5 Hz, 4096-code, 150 bps setting gives the best overall VC balance (Choi et al., 20 Jun 2026).

5. Relation to adjacent prosody-centric codec formulations

Related work provides several alternative answers to the same problem: how to preserve, transfer, or control prosody while preventing it from collapsing into speaker identity or linguistic content. PACE integrates a prosody-aware audio codec encoder with VALLE-X and injects explicit F0 and voicing/unvoiced features into codec embeddings after CLUB-based mutual-information minimization (Zhao et al., 21 May 2025). FreeCodec uses a global timbre vector, a 50 Hz content encoder, and a long-stride prosody encoder operating at roughly 7 Hz, using low-band mel input and stride-8 pooling to remove speaker and content leakage (Zheng et al., 2024). SoCodec compresses speech into an ordered multi-stream semantic sequence with delayed language-model generation, emphasizing time-variant semantic content while leaving prosodic variables implicit rather than separately named (Guo et al., 2024). DisCo-Speech with DisCodec adopts explicit tri-factor disentanglement into content, prosody, and timbre subspaces, with pitch-first residual FSQ, soft orthogonality constraints, GRL-based de-timbrization, and LM-friendly fusion into a unified content-prosody token stream (Li et al., 15 Dec 2025). MSR-Codec uses a cascaded residual design with semantic, timbre, prosody, and residual streams, defining the prosody stream as a 12.5 Hz residual correction supervised by F0 and energy prediction (Li et al., 16 Sep 2025). AugCodec factorizes speech into semantic, speaker, and prosody tokens and relies on data augmentation, including voice conversion and low-frequency STFT masking, to isolate attributes while operating at 12.5 Hz semantic rate and 6.25 Hz prosody rate (Wang et al., 20 Jun 2026).

System	Prosody representation	Distinctive mechanism
PACE	F0 and uv in codec embeddings	CLUB MI minimization + VALLE-X ICL
FreeCodec	$$h_q = Quantizer(Interp(h')), \quad h_q \in \{-1, 1\}^{T' \times d}.$$9 Hz prosody stream	low-band mel + stride-8 pooling
SoCodec	time-variant semantic sequence	OPQ + delayed multi-stream LM
DisCodec	explicit $$e = [e_{spk}; e_{txt}; e_{noisy} + e_q], \quad h = Decoder(e), \quad h' = h_{1+L : 1+L+T}.$$0 with $$e = [e_{spk}; e_{txt}; e_{noisy} + e_q], \quad h = Decoder(e), \quad h' = h_{1+L : 1+L+T}.$$1 and $$e = [e_{spk}; e_{txt}; e_{noisy} + e_q], \quad h = Decoder(e), \quad h' = h_{1+L : 1+L+T}.$$2	soft orthogonality + GRL + $$e = [e_{spk}; e_{txt}; e_{noisy} + e_q], \quad h = Decoder(e), \quad h' = h_{1+L : 1+L+T}.$$3 fusion
MSR-Codec	12.5 Hz prosody residual	F0 and energy auxiliary prediction
AugCodec	6.25 Hz prosody tokens	low-frequency STFT + augmentation loss
ProsoCodec	conditional residual in BSQ bottleneck	prefix-conditioned text and speaker priors

These systems outline two major design lineages. One lineage makes prosody an explicit latent or token stream and then attempts to minimize leakage through architecture, supervision, or augmentation. The other lineage, represented most directly by ProsoCodec, conditions on text and speaker so aggressively that the residual bottleneck is implicitly prosodic. This suggests a methodological contrast rather than a simple performance ranking: explicit-factorization systems optimize separability, while conditional-residual systems optimize prosody retention under explicit content and timbre priors.

6. Limitations, misconceptions, and future directions

A recurring misconception in this literature is that prosody can always be cleanly isolated as a fully independent factor. ProsoCodec explicitly rejects that premise for voice conversion, arguing that speaker- and content-conditioned prosodic nuances are not truly independent of content and speaker (Choi et al., 20 Jun 2026). Conversely, explicit-disentanglement work continues to treat low mutual information or controlled residualization as an achievable engineering target, but often acknowledges residual overlap or trade-offs. DisCo-Speech notes residual entanglement under extreme emotions and states that stronger disentanglement can reduce fine acoustic details (Li et al., 15 Dec 2025). AugCodec notes that some residual overlap between semantic and prosody inputs may persist even with low-frequency STFT for prosody, and it does not report explicit prosody metrics such as F0 RMSE, correlation, energy statistics, or duration and rhythm measures (Wang et al., 20 Jun 2026).

ProsoCodec itself has clear limitations. It relies on accurate transcripts, and removing text conditioning causes catastrophic WER. It is trained on English 24 kHz read speech from LibriTTS, so generalization to spontaneous speech, other languages, and noisy conditions remains open. Prosody transfer can still be challenged by extreme speaking styles or rapid expressive variations, and some residual timbre information may leak at higher bitrates or with full-band encoder inputs (Choi et al., 20 Jun 2026).

Other systems expose complementary limitations. PACE currently models only F0 and uv, leaving energy, duration, and explicit pause structure for future work (Zhao et al., 21 May 2025). FreeCodec learns prosody implicitly through reconstruction and adversarial objectives and does not introduce explicit F0 or energy supervision (Zheng et al., 2024). MSR-Codec validates disentanglement through controlled manipulation and metrics rather than adversarial probing or MI-based analysis (Li et al., 16 Sep 2025).

Future directions are correspondingly diverse. Reported proposals include integrating energy and duration features, explicit pause modeling, improved disentanglement techniques, stronger mutual-information estimators, multi-timescale prosody representations, longer-context LLMs, codebook optimization, and explicit control knobs for F0 or tempo (Zhao et al., 21 May 2025). Taken together, these directions indicate that ProsoCodec is best understood not as a single settled architecture, but as a research program centered on how prosody should be encoded: as an independent stream, as a supervised residual, or as a conditional remainder after text and timbre are made explicit.