FlexiCodec: Adaptive Neural Audio Compression

• FlexiCodec is an adaptive neural audio codec design that flexibly varies temporal resolution and latent token types based on audio content complexity.
• It employs entropy-based and semantic-guided frame strategies to dynamically adjust frame rates and balance compression efficiency with performance.
• Implementations trade off bitrate, token sequence length, and quality metrics (e.g., WER, PESQ) to support both autoregressive and generative downstream systems.

FlexiCodec denotes neural audio codec designs whose defining feature is flexibility in compressed representation rather than commitment to a single fixed latent cadence or a single latent type. In 2025, the term was used in at least two direct senses: a variable-frame-rate neural speech codec built around Temporally Flexible Coding (TFC), and a dynamic low-frame-rate dual-stream codec for speech LLMs with inference-time controllable rates between 3 and 12.5 Hz. Related work on CoDiCodec also used “FlexiCodec” more generically for a unified codec that emits both low-rate continuous embeddings and discrete tokens from one trained model. Taken together, these usages suggest that FlexiCodec is best understood as a design principle centered on adaptive temporal or representational allocation in neural audio compression (Zhang et al., 22 May 2025, Li et al., 1 Oct 2025, Pasini et al., 11 Sep 2025).

1. Conceptual scope and terminological usage

A common simplification is to treat FlexiCodec as a single canonical architecture. The literature instead uses the label for distinct but related formulations. One formulation defines FlexiCodec as “Temporally Flexible Neural Speech Coding via Variable Frame Rate,” where flexibility is temporal and implemented by varying frame rate according to temporal entropy. A second formulation defines FlexiCodec as a dynamic, dual-stream neural audio codec for very low frame rates, where flexibility is realized by adaptively merging semantically similar frames and by decoupling semantic and acoustic streams. A third, broader usage appears in CoDiCodec, which “embodies a ‘FlexiCodec’” by flexibly supporting continuous embeddings and discrete tokens from the same encoder-decoder system (Zhang et al., 22 May 2025, Li et al., 1 Oct 2025, Pasini et al., 11 Sep 2025).

Work	Flexibility mechanism	Stated operating regime
TFC-based FlexiCodec	Variable Frame Rate via entropy-based tier allocation	Average frame rate between 18.75 Hz and 75 Hz at 24 kHz
Dynamic low-frame-rate FlexiCodec	Semantic-guided frame merging with dual streams	Inference-time controllable frame rates between 3 and 12.5 Hz
CoDiCodec as a FlexiCodec	Unified continuous and discrete latent modeling	Continuous embeddings at ≈11 Hz and discrete tokens at ≈2.38 kbps

The common denominator is explicit adaptation to content complexity. In the TFC work, low-entropy regions are coded at lower frame rate and high-entropy regions at higher frame rate. In the dynamic low-frame-rate work, semantically redundant adjacent frames are merged, reducing token count while attempting to preserve linguistic content. In CoDiCodec, flexibility refers not to temporal rate selection but to the ability to serve both continuous generative models and autoregressive token models from one trained codec.

2. Variable-frame-rate FlexiCodec through Temporally Flexible Coding

The TFC formulation starts from a criticism of constant-frame-rate neural speech codecs such as SoundStream and EnCodec: bitrate is typically adjusted intra-frame, for example by codebook dropout, while the temporal cadence remains fixed. This is suboptimal when speech information density varies strongly over time. TFC therefore introduces Variable Frame Rate into a neural speech codec “for the first time,” using a plug-and-play module atop a DAC backbone at 24 kHz. The system measures temporal information density by a non-parametric entropy estimator over the waveform, builds a hierarchical encoder with three temporal resolutions—fine $F$, medium $F/2$, and coarse $F/4$—allocates frame-rate tiers by entropy percentiles, quantizes each tier with RVQ, and reconstructs with a conditional hierarchical decoder that composes coarse $\rightarrow$ medium $\rightarrow$ fine while using mask conditioning to avoid switching artifacts (Zhang et al., 22 May 2025).

The entropy estimator is defined segmentwise. For a temporal segment $\mathcal{T}$, with normalized samples $x[t]\in[-1,1]$, the paper defines Gaussian affinities to uniformly spaced bins, normalizes them, and computes Shannon entropy

$$H(\mathcal{T}) = - \sum_{i=0}^{N-1} \bar{p}_{\mathcal{T},i} \log \bar{p}_{\mathcal{T},i}.$$

This entropy is used as a proxy for temporal information density. Frame-rate allocation is then performed by quantile thresholding. Given granularity ratios $r_f,r_m,r_c\ge 0$ with $r_f+r_m+r_c=1$, binary masks are constructed as

$F/2$0

$F/2$1

$F/2$2

$F/2$3

The fused latent is aligned to the finest grid:

$F/2$4

The average frame rate is user-tunable:

$F/2$5

with $F/2$6 Hz in the reported implementation. This yields the stated operating range from 18.75 Hz to 75 Hz. The decoder reconstructs hierarchically,

$F/2$7

before passing $F/2$8 to the waveform decoder. The design objective is not merely lower bitrate; it is shorter token sequences for downstream autoregressive systems.

3. Dynamic low-frame-rate FlexiCodec for speech LLMs

The later FlexiCodec formulation targets a different operating regime: very low frame rates for speech LLMs and LM-based TTS. Its starting point is that conventional codecs operate at $F/2$9 Hz, producing more than 50 tokens per second, whereas text is around 4.5 Hz. The paper identifies a specific failure mode below 12.5 Hz: missing semantic information in low-rate tokens, especially in RVQ-1. FlexiCodec addresses this with a dynamic frame rate and a decoupled semantic/acoustic representation. A frozen ASR encoder, SenseVoice-Small, produces semantic features $F/4$0 at 16.67 Hz and these are downsampled to 12.5 Hz by linear interpolation. In parallel, a DAC-style convolutional waveform encoder downsamples 16 kHz audio to 12.5 Hz acoustic features $F/4$1 through 5 strided CNN blocks with strides $F/4$2. The semantic stream uses FSQ with $F/4$3 scalar dimensions and $F/4$4 levels per dimension, giving $F/4$5 codes, while the acoustic stream quantizes the residual $F/4$6 into $F/4$7 RVQ layers of 4096 entries and 512 dimensions each (Li et al., 1 Oct 2025).

Dynamic frame merging is driven by adjacent semantic similarity. For semantic features $F/4$8, the codec computes

$F/4$9

A threshold $\rightarrow$0 defines maximal segments $\rightarrow$1 satisfying

$\rightarrow$2

Frames in a segment are merged by averaging,

$\rightarrow$3

with segment length $\rightarrow$4 and a maximum merged length $\rightarrow$5, so the duration attribute is storable in 3 bits. Local-attention Transformer bottlenecks are inserted in both merging and unmerging paths to mitigate hard boundary artifacts. The merging Transformers are two modules, one per stream, each with 6 layers, 8 heads, FFN dimension 2048, model dimension approximately 512, and local attention window $\rightarrow$6 tokens. The unmerging Transformer is approximately 100M parameters, with 32 layers, 8 heads, FFN 2048, bidirectional attention, and rotary embeddings.

Inference-time controllability is achieved by tuning $\rightarrow$7. The paper reports that $\rightarrow$8 yields the original 12.5 Hz; $\rightarrow$9 yields approximately 8.3 Hz; $\rightarrow$0 yields approximately 6.25 Hz; and $\rightarrow$1 yields 3–4.5 Hz. For an utterance of duration $\rightarrow$2 seconds and merged sequence length $\rightarrow$3, the realized rate is

$\rightarrow$4

This mechanism is designed to merge tokens only in semantically redundant regions. On TIMIT, the utterance-level phoneme rate correlates with realized $\rightarrow$5 at Pearson $\rightarrow$6, and each merged frame roughly covers two phonemes.

4. Quantization, bitrate, and controllability

The two main FlexiCodec formulations solve the rate-control problem differently. In TFC, average rate is determined by how often the model emits fine, medium, and coarse frames. In the dynamic dual-stream FlexiCodec, average rate is determined by semantic-guided frame merging and by the number of acoustic RVQ layers used at decode time. Both approaches are explicit alternatives to constant-frame-rate tokenization and both treat sequence length as a first-class systems concern (Zhang et al., 22 May 2025, Li et al., 1 Oct 2025).

For TFC, the expected bitrate is

$\rightarrow$7

where $\rightarrow$8 bits for the DAC codebooks with 1024 entries. With $\rightarrow$9 Hz and $\mathcal{T}$0 at inference, the paper gives concrete examples:

• 75 Hz with $\mathcal{T}$1 gives approximately 6 kbps.
• 37.5 Hz with $\mathcal{T}$2 gives approximately 3 kbps.
• 18.75 Hz with $\mathcal{T}$3 gives approximately 1.5 kbps.

For the dynamic low-frame-rate FlexiCodec, bitrate is decomposed into semantic and acoustic streams. The semantic FSQ stream uses $\mathcal{T}$4 and $\mathcal{T}$5, so each token carries $\mathcal{T}$6 bits; storing $\mathcal{T}$7 adds 3 bits, giving

$\mathcal{T}$8

This yields approximately 0.23 kbps at 12.5 Hz, 0.15 kbps at 8.3 Hz, and 0.11 kbps at 6.25 Hz. The acoustic stream contributes

$\mathcal{T}$9

because each selected RVQ layer has 4096 entries, or 12 bits. For RVQ1:8, the paper reports approximately 1.2 kbps at 12.5 Hz, 0.85 kbps at 8.3 Hz, and 0.60 kbps at 6.25 Hz.

This difference in rate accounting is conceptually important. TFC retains a conventional RVQ codec view and varies temporal granularity through tier assignment. The dual-stream FlexiCodec simultaneously varies temporal granularity and separates semantic from acoustic payload, permitting downstream systems to use only semantic tokens autoregressively and treat acoustic detail non-autoregressively.

5. Empirical performance and downstream integration

The TFC-based FlexiCodec was trained on LibriTTS 960 hours, using 1-second clips, batch size 20, and 1,000k iterations. At matched bitrates, DAC+TFC improved or matched DAC while reducing frame rate. At 6 kbps, DAC (75 Hz, $x[t]\in[-1,1]$0) achieved Mel 0.594, STFT 1.456, UTMOS 3.948, STOI 0.967, and WER 4.125, whereas DAC+TFC (75 Hz, $x[t]\in[-1,1]$1) achieved Mel 0.601, STFT 1.451, UTMOS 3.994, STOI 0.970, and WER 2.845. At 3 kbps, DAC (75 Hz, $x[t]\in[-1,1]$2) gave Mel 0.755, STFT 1.617, UTMOS 3.628, STOI 0.938, and WER 2.443, while DAC+TFC (37.5 Hz, $x[t]\in[-1,1]$3) gave Mel 0.715, STFT 1.565, UTMOS 3.895, STOI 0.951, and WER 2.193. At 1.5 kbps, DAC (75 Hz, $x[t]\in[-1,1]$4) gave Mel 0.930, STFT 1.792, UTMOS 2.869, STOI 0.892, and WER 2.110, whereas DAC+TFC (18.75 Hz, $x[t]\in[-1,1]$5) gave Mel 0.864, STFT 1.725, UTMOS 2.995, STOI 0.902, and WER 2.058. The same paper also reports that, at an average 3 kbps / 37.5 Hz target, VFR mixtures outperform pure 37.5 Hz CFR and DAC baselines across Mel, UTMOS, and STOI, with the lowest WER achieved using a small fraction (approximately 6%) of 75 Hz frames (Zhang et al., 22 May 2025).

The dynamic low-frame-rate FlexiCodec was trained on 16 kHz, 54k-hour LibriLight-Large for 800k steps on 8 V100 GPUs, with batch 5×5 s segments per GPU and AdamW at $x[t]\in[-1,1]$6. On LibriSpeech test-clean 4–10 s subset, the paper reports that very low fixed frame rates are especially damaging for RVQ-1 semantics in baseline systems: DualCodec’s RVQ-1 WER rises from 5.93% at 12.5 Hz to 31.5% at 6.25 Hz. FlexiCodec instead reports RVQ-1 WER of 4.15% at 6.25 Hz and 2.98% at 8.3 Hz, with RVQ1:8 WER of 2.53% and 2.28%, respectively; ground-truth WER is 2.1%. Relative to fixed-frame-rate variants of the same model, dynamic merging yields RVQ-1 WER reductions of 19% at 8.3 Hz and 26% at 6.25 Hz, ASR probing reductions of 12% and 21%, and RVQ1:8 WER reductions of 6% and 8%. Acoustic metrics are reported as strong per bitrate, with FlexiCodec at 12.5 Hz slightly outperforming Mimi, XYTokenizer, and DualCodec in PESQ, UTMOS, MCD, and SIM, and remaining superior to similarly low-bitrate baselines at approximately 0.85 kbps and 0.6 kbps (Li et al., 1 Oct 2025).

Downstream integration is central in both formulations. TFC explicitly motivates lower frame rates as a way to reduce sequence length and improve real-time feasibility in autoregressive speech systems. The dynamic low-frame-rate FlexiCodec integrates into AR+NAR TTS by using an AR Transformer LM over semantic RVQ-1 tokens and their duration classes, followed by a conditional flow-matching Transformer for higher-rate acoustics. On the E2TTS test suite, the 6.25 Hz AR setting reports WER 3.2%, SIM-o 0.65, NMOS 3.32, QMOS 3.40, RTF(AR) 0.07, and RTF(AR+NAR) 0.18; the paper interprets this as approximately 7.3× speedup versus CosyVoice AR and approximately 3.4× speedup for the combined AR+NAR pipeline.

6. Relation to CoDiCodec, limitations, and broader significance

CoDiCodec broadens the meaning of FlexiCodec beyond variable frame rate. It is presented as “a true FlexiCodec”: one trained model that produces both compressed continuous embeddings and discrete tokens, globally compressed through summary embeddings and trained end-to-end with a single consistency loss. The system operates in complex STFT space on stereo 44.1 kHz audio, converts audio to complex spectrograms treated as 2-channel real/imag tensors, and applies an amplitude transform

$x[t]\in[-1,1]$7

Its encoder retains only $x[t]\in[-1,1]$8 learnable summary embeddings, each projected to $x[t]\in[-1,1]$9. These support a continuous branch with $$H(\mathcal{T}) = - \sum_{i=0}^{N-1} \bar{p}_{\mathcal{T},i} \log \bar{p}_{\mathcal{T},i}.$$0 and a discrete branch with $$H(\mathcal{T}) = - \sum_{i=0}^{N-1} \bar{p}_{\mathcal{T},i} \log \bar{p}_{\mathcal{T},i}.$$1. The continuous representation can be reshaped into an approximately 11 Hz embedding stream, while the discrete representation uses FSQ with $$H(\mathcal{T}) = - \sum_{i=0}^{N-1} \bar{p}_{\mathcal{T},i} \log \bar{p}_{\mathcal{T},i}.$$2 and $$H(\mathcal{T}) = - \sum_{i=0}^{N-1} \bar{p}_{\mathcal{T},i} \log \bar{p}_{\mathcal{T},i}.$$3, giving $$H(\mathcal{T}) = - \sum_{i=0}^{N-1} \bar{p}_{\mathcal{T},i} \log \bar{p}_{\mathcal{T},i}.$$4 combinations per token and a reported bitrate of approximately 2.38 kbps. The paper further introduces FSQ-dropout with best configuration $$H(\mathcal{T}) = - \sum_{i=0}^{N-1} \bar{p}_{\mathcal{T},i} \log \bar{p}_{\mathcal{T},i}.$$5, trains only with a single consistency loss, and reports that parallel decoding with $$H(\mathcal{T}) = - \sum_{i=0}^{N-1} \bar{p}_{\mathcal{T},i} \log \bar{p}_{\mathcal{T},i}.$$6 achieves superior FAD/FAD_clap and lower time per 60 s audio than autoregressive decoding (Pasini et al., 11 Sep 2025).

This broader usage suggests that FlexiCodec had already become a more general label for codecs that adapt representation to downstream generative needs. In the TFC paper, flexibility lies in content-aware temporal granularity. In the dynamic low-frame-rate paper, it lies in semantic-guided rate control plus semantic/acoustic stream separation. In CoDiCodec, it lies in choosing between continuous and discrete compressed representations from the same model. The unifying idea is not one architectural template, but adaptive allocation of coding capacity.

The limitations reported across the literature are correspondingly different. TFC notes that its entropy estimator is waveform-based and non-parametric and may not perfectly track perceptual salience; exact bitstream packetization and signaling overhead for masks are not detailed; broader architecture ablations and downstream evaluations are deferred. The dynamic low-frame-rate FlexiCodec reports that semantics degrade at extremely low rates such as 3 Hz, that merging can introduce artifacts if ASR features misalign with phonetic boundaries, and that the current system is trained on English speech and operates in full-utterance mode. CoDiCodec notes that parallel decoding memory scales linearly with sequence length, that operation in STFT space may limit direct waveform-domain effects unless paired with high-quality inversion, and that pairwise reconstruction metrics such as SI-SDR and ViSQOL can be lower than in adversarial or reconstruction-trained codecs because the objective is a purely generative consistency loss (Zhang et al., 22 May 2025, Li et al., 1 Oct 2025, Pasini et al., 11 Sep 2025).

Within the 2025 codec landscape, FlexiCodec therefore names a family of adaptive neural coding strategies rather than a single immutable system. Its significance lies in reframing codec design around sequence length, latent semantics, and downstream model compatibility: fewer tokens when content permits, finer resolution when content demands it, and, in the broader CoDiCodec sense, the option to select the latent type most suitable for the generator that follows.