CodecSlime: Dynamic Neural Speech Codec

• CodecSlime is a dynamic frame rate technique for neural speech codecs that adaptively allocates tokens based on the varying information density of natural speech.
• It utilizes the ScheDFR algorithm and a Melt-and-Cool training procedure to optimize segmentation and improve compression without relying on supervised labels.
• Empirical results demonstrate significant reductions in Word Error Rate and enhanced intelligibility compared to fixed frame rate codecs, making it ideal for low-bandwidth applications.

CodecSlime is a plugin-style method for neural speech codec temporal redundancy compression that introduces, for the first time, unsupervised and architecture-agnostic dynamic frame rate (DFR) capabilities to neural codecs. While mainstream codecs employ fixed-frame-rate (FFR) quantization—where each time slice is assigned the same number of tokens—CodecSlime dynamically adjusts token allocation in alignment with the non-uniform information density of natural speech, yielding substantial efficiency gains and improved performance at ultra-low bitrates.

1. Motivation and Principle

Neural speech codecs compress audio by mapping segments to discrete token representations. Traditionally, these codecs use a FFR scheme: every uniform-duration slice (e.g., 20 ms) receives an identical token budget regardless of perceptual or informational redundancy. This uniform structure is mismatched to speech, which often contains long steady-state segments (e.g., vowels, silence) alongside temporally dense transitions (e.g., plosives, consonant clusters) (2506.21074).

CodecSlime addresses this inefficiency by supporting dynamic frame rate quantization, allocating tokens adaptively with respect to the local information density of the audio. This results in fewer tokens on redundant segments and more on information-dense transitions, compressing temporal redundancy in ways unreachable for FFR codecs—all without requiring supervision, explicit segmentation labels, or modifications to the backbone codec architecture.

2. Core Methodologies: ScheDFR and Melt-and-Cool

CodecSlime’s functionality is enabled by two key, architecture-agnostic components:

2.1 ScheDFR (Schedulable Dynamic Frame Rate)

ScheDFR adaptively downsamples the encoder output feature sequence at inference time. Given encoder features $\mathbf{h} \in \mathbb{R}^{T \times d_h}$, ScheDFR finds an “optimal” segmentation $\mathbf{s}^*$, where each segment, potentially spanning several original frames, is merged if its frames are similar (as measured in feature space). The segmentation $\mathbf{s}^*$ is defined as

$$\mathbf{s}^* = \operatorname*{arg\,max}_{\mathbf{s}\in\mathcal{S}}\, \mathcal{J}_h(\mathbf{h}, \mathbf{s})$$

where $\mathcal{S}$ is the set of all valid segmentations, and

$$\mathcal{J}_h(\mathbf{h}, \mathbf{s}) = -\sum_{i=1}^{T'} L(\sigma_i, s_i),$$

with

$$L(j, s) = \frac{1}{s} \sum_{a=j}^{j+s-2} \sum_{b=a+1}^{j+s-1} \lVert h_a - h_b \rVert_2,$$

quantifying intra-segment similarity. The solution is found efficiently using dynamic programming, with constraints on maximum segment length and target downsampling ratio. This permits real-time, utterance-specific token rate adaptation during inference.

2.2 Melt-and-Cool Training Procedure

CodecSlime’s Melt-and-Cool recipe modifies a pretrained FFR backbone to operate stably and optimally under DFR:

• Melt phase: The FFR codec is continued to be trained (“melted”) with randomly sampled downsampling schedules, exposing the model to various merged segment patterns and forcing robustness to a range of variable frame-rate conditions.
• Cool phase: The model is fine-tuned with the “optimal” (ScheDFR/DP-generated) downsampling schedules for each training example. Typically, the encoder is frozen and only the decoder and quantizer are updated for stability and specialization. Ablation and empirical results indicate this two-stage “melt then cool” approach produces better generalization and practical DFR operation than single-stage or static strategies.

Both procedures are unsupervised and independent of explicit phone, word, or event boundaries.

3. Implementation and Codec Integration

CodecSlime is compatible with modern neural codec backbones such as VQ-GAN, using finite scalar quantization (FSQ) for code assignments. The encoder initially generates high-rate features (e.g., 80 Hz), which are adaptively downsampled. For each downsampled segment, compressed codes replace the original token assignment, and the segment duration is encoded (for correct expansion during decoding).

Quantization employs FSQ rather than classical vector quantization (VQ), providing high code utilization and smooth transitions. At decoding, concatenated codes plus associated durations expand merged tokens back to the waveform domain.

This architecture-agnostic approach allows easy retrofit of CodecSlime onto existing neural codec infrastructures with minimal implementation overhead. Downstream latency is determined by the backbone’s hop size and the computational complexity of the DP scheduler, which is negligible (on modern hardware) relative to audio synthesis and can be batched for streaming.

4. Empirical Results and Performance Metrics

Integration of CodecSlime with a standard VQ-GAN codec backbone at 40 Hz frame rate (≈600 bps) demonstrates significant improvements over FFR analogues:

• Word Error Rate (WER): CodecSlime achieves a WER of 3.82% compared to 5.59% for a BigCodec-FSQ FFR model at the same bitrate, corresponding to a 46% reduction.
• STOI (Short-Time Objective Intelligibility): 0.900, matching best FFR baselines.
• PESQ/UTMOS (Perceptual Quality): 1.93 / 3.93 ± 0.029, competitive with state-of-the-art.
• MCD (Mel Cepstral Distortion): 1.83.
• SECS (Speaker Similarity): 0.916.

A single CodecSlime model—trained at a given target frame rate—supports multiple inference frame rates and consistently outperforms retrained FFR models at the same rate, indicating effective generalization.

Codec	Frame Rate (Hz)	Bitrate (kbps)	WER (%)	STOI
BigCodec-FSQ FFR	40	0.57	5.59	—
BigCodec-FSQ FFR	84k codes	0.57	4.12	—
CodecSlime (DFR)	40	0.57	3.82	0.900

5. Trade-Offs, Adaptability, and Interpretation

CodecSlime enables flexible quality/bitrate trade-offs from a single model: by varying the target average frame rate at inference, users can prioritize bit-rate savings or fidelity as needed, without retraining. As the frame rate increases, both WER decreases and STOI increases, yet CodecSlime consistently outperforms FFR models at corresponding rates. The system is efficient for streaming and chunked transmission.

All codec adaptation is unsupervised with no need for linguistic labels, boundaries, or explicit phone alignment. The approach generalizes to multilingual and zero-resource settings due to its reliance on feature cohesion rather than supervised annotation.

A plausible implication is that CodecSlime’s adaptive DFR approach may be beneficial for codecs deployed in bandwidth-constrained environments, or in variable-rate generative speech synthesis pipelines, where maintaining naturalness and intelligibility under dynamic resource constraints is paramount.

6. Applications, Generalization, and Resources

Practical deployment scenarios for CodecSlime include:

• Ultra-low bitrate speech transmission and storage.
• Token-based text-to-speech (TTS), speech-to-speech (S2S), and conversational AI pipelines with discrete code representations.
• Real-time or streaming speech transmission systems, enabling efficient bandwidth usage without significant loss of intelligibility.
• Broad compatibility with neural codec architectures (VQ-GAN and others).
• Robust performance in multilingual and zero-resource settings.

Audio samples demonstrating CodecSlime’s output and token alignment visualizations are available at https://acadarmeria.github.io/codecslime/.

7. Significance and Outlook

CodecSlime establishes the first practical, architecture-agnostic, and unsupervised protocol for dynamic frame rate quantization in neural speech codecs. By leveraging DP-based feature similarity segmentation (ScheDFR) and a specialized two-phase training recipe (Melt-and-Cool), it achieves state-of-the-art WER reduction and quality/bitrate trade-offs. CodecSlime outperforms FFR models at comparable rates and enables a single model to serve a spectrum of bitrate and quality configurations without retraining or supervision (2506.21074).

This suggests a broader move towards temporal adaptivity in neural encoding technologies, opening avenues for further research on frame-rate scheduling, unsupervised temporal analysis, and integration with neural LLMs for generative and compression tasks in spoken language processing.