Sparse Prosody Transmission for Ultra-Low Bitrate Speech

• Sparse prosody transmission is a compression method that encodes key prosodic features—pitch, energy, and speaking rate—into low-frequency keyframes for bandwidth-efficient communication.
• It employs sparse sampling, delta encoding, and dead-zone quantization to achieve ultra-low bitrate transmission while preserving perceptual quality comparable to higher bitrate systems.
• The STCTS framework uses cubic-spline interpolation for TTS prosody reconstruction, balancing bitrate reduction with natural and expressive speech synthesis.

Sparse prosody transmission is a compression paradigm that encodes prosodic information—pitch, energy, and speaking rate—using highly efficient, low-frequency keyframes, optimized for bandwidth-constrained voice communication. In the STCTS framework, prosody is disentangled from linguistic content and timbre, allowing for modular, ultra-low bitrate transmission while preserving expressive and natural speech. The approach enables practical deployment in environments where traditional speech codecs fail below 1 kbps, using as little as 0.7–13.9 bps for prosody and achieving perceptual quality scores comparable to much higher-bitrate baselines (Wang et al., 29 Nov 2025).

1. Mathematical Formulation of Prosody Features

Prosodic features are extracted at a fixed frame rate of $f_p=100\,$Hz (10 ms hop), encompassing fundamental frequency ($F_0$), short-term energy ($E$), and instantaneous speaking rate ($R$):

• Pitch ($F_0[t]$): Extracted via the YIN algorithm. Normalization occurs in the log-domain using speaker-dependent mean and standard deviation:

$$\hat F_0[t] = \begin{cases} (\log F_0[t] - \mu_{F_0}) / \sigma_{F_0}, & F_0[t] > 0 \ 0, & F_0[t] = 0 \end{cases}$$

• Short-term energy ($E[t]$): Computed over a 40 ms window, followed by log normalization:

$$\hat E[t] = \frac{\log(E[t] + \epsilon) - \log E_{\min}}{\log E_{\max} - \log E_{\min}},\quad \epsilon=10^{-6}$$

• Speaking rate ($R[t]$): Derived by counting syllable nuclei in a 1 s window and normalizing by speaker baseline:

$$R[t] = \frac{1}{T_w}\sum_{\tau=t-T_w/2}^{t+T_w/2}\mathbf{1}[\text{nucleus at }\tau]$$

$\hat R[t] = \frac{R[t] - \mu_R}{\sigma_R}$

• The combined prosody vector per frame is:

$$\mathbf p[t] = [\hat F_0[t],\, \hat E[t],\, \hat R[t]]$$

2. Sparse Sampling, Delta Encoding, and Quantization

Prosody is compressed using sparse keyframes sampled at rate $f_k$ (0.1–1 Hz), typically much lower than the frame extraction rate ($f_p=100$ Hz). Keyframes are spaced by $\Delta t = \lfloor f_p / f_k \rfloor$. Each keyframe encodes the delta from its predecessor:

$\Delta \mathbf p[t] = \mathbf p[t] - \mathbf p[t - \Delta t}$

The first keyframe encodes the absolute value.

Keyframe deltas are quantized using dead-zone uniform quantization. For pitch:

$$\Delta \hat F_0^{(q)}[t] = \begin{cases} 0, & |\Delta \hat F_0[t]| < \tau_F \ \operatorname{sign}(\Delta \hat F_0[t]) \left\lceil |\Delta \hat F_0[t]| / \alpha_F \right\rceil, & \text{otherwise} \end{cases}$$

with threshold $\tau_F=0.05$. Energy and rate follow similar quantization schemes.

Typical bit allocations in balanced mode:

• $b_F=6$ bits for pitch
• $b_E=5$ bits for energy
• $b_R=5$ bits for rate Total: 16 bits/keyframe (plus entropy coding).

The resulting prosody bitrate:

$$b_{\mathrm{prosody}} = f_k \times (\text{bits per keyframe})$$

Modes span minimal (0.1 Hz, $\sim$0.7 bps), balanced (0.5 Hz, $\sim$5.5 bps), and high-quality (1 Hz, $\sim$13.9 bps).

3. Prosody Reconstruction via TTS Interpolation

On decoding, the sparse prosody keyframes are interpolated to reconstruct continuous 100 Hz contours suitable for TTS model prosody conditioning. Cubic-spline interpolation is applied:

$$\tilde{\mathbf p}(t) = \sum_k S_k(t) \mathbf p[t_k]$$

where $S_k(t)$ are spline basis functions and $\{t_k\}$ are keyframe times. The interpolated contours are consumed by the TTS prosody-conditioning layers to synthesize expressive, natural speech.

4. Empirical Quality Characteristics and Bimodal Distribution

Evaluation on LibriSpeech demonstrates a distinct bimodal quality distribution with respect to prosody sampling rate ($f_k$):

• Sparse regime ($\sim$0.05–1 Hz): Yields high perceptual quality (NISQA MOS $\approx$4.30–4.36) at ultra-low total bitrates (132–154 bps including text).
• Dense regime ($>$7 Hz): Maintains high MOS ($\approx$4.32) but at significantly higher bitrates (≈410 bps).
• Intermediate frequencies (1–6 Hz): Create an "uncanny valley," with perceptual discontinuities manifesting as pitch jitter or unnatural transitions. NISQA MOS decreases by $\sim$0.2–0.3 points. This suggests that mid-range update rates induce unnatural artifacts, while sparse and dense extremes avoid these discontinuities.

Speaker similarity is unaffected by sampling rate, confirming that speaker identity ("timbre") is robust across transmission modes, while expressive prosody is sensitive to update rate.

5. Bitrate Efficiency and Comparisons

The STCTS framework delivers a substantial bitrate reduction compared to conventional codecs, summarised as follows:

Method	Total bps	Text bps	Prosody bps
Minimal mode	71.6 ± 8.8	70.9 ± 8.8	0.7 ± 0.1
Balanced mode	76.5 ± 8.8	71.0 ± 8.8	5.5 ± 0.2
High-quality	79.6 ± 8.9	65.8 ± 8.9	13.9 ± 0.2
Opus (6 kbps)	6407.3 ± 217.2	—	—
EnCodec (1 kbps)	999.9 ± 0.1	—	—

Measured against Opus (≈6 kbps), STCTS achieves a 75x reduction in bitrate; versus EnCodec (1 kbps), an 8x reduction. Perceptual quality remains high—NISQA MOS exceeds 4.26 for all tested configurations (Wang et al., 29 Nov 2025).

6. Configuration Guidelines and Perceptual Tradeoffs

Optimal configuration avoids the 1–6 Hz mid-range where perceptual discontinuities are prominent. For maximal compression under extreme constraints, minimal mode at 0.1 Hz is recommended. For best quality-per-bit, balanced mode at 0.5 Hz leverages the sparse regime's "good" peak in the bimodal distribution. High-quality mode at 1 Hz is appropriate when additional prosody bandwidth ($\sim$10–20 bps extra) is available. Word error rate (WER) remains stable across these regimes, while prosody bitrate grows linearly with $f_k$.

7. Visualization and Experimental Metrics

Key results are summarized in representative figures (as described in the source):

• Bitrate vs. NISQA MOS curves reveal peaks at both low and high sampling rates, with a dip in the mid-range ("uncanny valley").
• WER and bitrate components plotted against $f_k$ show prosody bitrate scaling with update rate, while WER is rate-invariant.
• PESQ and STOI mirror the bimodal pattern observed in NISQA.
• Sub-score analysis shows that perceptual discontinuity sharply increases in the mid-range, confirming the nonlinear quality dependence on sampling rate.
• Detailed tables document objective metrics (bps, MOS, PESQ, STOI, WER, speaker similarity), substantiating STCTS's efficacy in ultra-low bitrate scenarios.

Sparse prosody transmission, as established by STCTS, constitutes a practical solution for semantic speech compression with rigorous empirical justification for keyframe selection, interpolation methods, and bitrate allocation. The bimodal quality phenomenon determines the operational regimes most suitable for expressive, high-fidelity, bandwidth-efficient speech synthesis (Wang et al., 29 Nov 2025).