Mimi Codec: Neural Audio Streaming

• Mimi Codec is a neural audio codec that compresses high-fidelity speech into compact discrete latent representations using a learned quantization-based bottleneck.
• It integrates a hybrid CNN-Transformer architecture with multi-stage residual vector quantization to achieve low-latency, high-quality speech synthesis, enhancement, and unified language modeling.
• The design balances trade-offs between latency, storage, and fidelity, delivering significant speed-ups and robust acoustic and semantic tokenization.

The Mimi codec is a streaming neural audio codec employing a learned quantization-based bottleneck to compress high-fidelity speech waveforms into compact, discrete latent representations suitable for low-latency, high-quality speech synthesis, enhancement, and language modeling. Architecturally situated between traditional waveform regression and text-based models, Mimi leverages deep convolutional neural networks (CNNs), Transformers, and multi-stage residual vector quantization (RVQ) to represent audio at rates as low as 12.5 Hz, supporting both semantic and acoustic tokenization. The codec underpins state-of-the-art systems in real-time TTS, speech enhancement, and unified speech language modeling, characterized by strong trade-offs between latency, fidelity, and representational compactness (Wu et al., 27 Jan 2026, Hauret et al., 5 Aug 2025, Sugiura et al., 18 Sep 2025, Su et al., 14 Apr 2026).

1. Core Architecture and Quantization

The Mimi codec employs an encoder–quantizer–decoder pipeline, adapting paradigms from SoundStream/EnCodec while introducing streaming constraints and model variants. Audio at 24 kHz is chunked into non-overlapping 80 ms frames (1,920 samples/frame), which are processed by a six-layer, strided 1D CNN front-end with increasing channel widths. This is followed by four residual ResNet blocks, yielding a downsampled latent tensor $Z_e \in \mathbb{R}^{T \times D}$, with $D$ commonly 448 and $T \approx 30$ for short utterances (Hauret et al., 5 Aug 2025).

Quantization is realized via $M$ residual vector quantizer codebooks (RVQ), each with $K$ entries and dimension $d$, such that $M \cdot d = D$. For each time frame $t$, quantization iteratively encodes the residual:

$$z^{(0)} = Z_e(t) \ \text{for } m = 1, ..., M: \ \quad i_m = \arg\min_j \| z^{(m-1)} - e_{m,j} \|_2 \ \quad q_m = e_{m,i_m} \ \quad z^{(m)} = z^{(m-1)} - q_m$$

The reconstructed embedding is $Z_q(t) = \sum_{m=1}^M q_m$ (Hauret et al., 5 Aug 2025, Sugiura et al., 18 Sep 2025). Decoder modules reflect the encoder, using a symmetric sequence of residual and transposed CNN layers (CNN upsampling) or, in recent variants, Transformer-based upsampling (Wu et al., 27 Jan 2026).

Codebook updates follow the EMA assignment mechanism from SoundStream/EnCodec. The codec loss comprises an $D$0 waveform reconstruction term and, when training from scratch, a codebook commitment loss on quantized residuals (Sugiura et al., 18 Sep 2025).

2. Architectural Extensions and Streaming Considerations

Mimi's hybrid Transformer–CNN decoder supports streaming real-time operation, but standard deconvolution upsamplers present latency challenges on mobile hardware (e.g., poor transposed-convolution kernel support in frameworks like XNNPACK). On-device generation of an 80 ms audio chunk requires 42.1 ms with the original Mimi decoder, resulting in a real-time factor (RTF) $D$1—impeding deployment in latency-critical applications (Wu et al., 27 Jan 2026).

The T-Mimi variant replaces all convolutional upsampling with a stack of four additional streaming Transformer layers (for a total of 12) followed by two linear upsamplers, maintaining parameter count ($D$2M) and hidden dimension ($D$3). This reduces on-device decoding latency to 4.4 ms per 80 ms chunk (RTF ≈ 0.055), yielding a $D$4 speed-up and $D$5 storage reduction with negligible perceptual quality loss (Wu et al., 27 Jan 2026).

Pseudocode for the T-Mimi decoder is as follows: $M$9 Further, quantization-aware training (QAT) with selective precision in late layers preserves audio quality at lower storage budgets; the last two Transformer and both linear layers remain in FP32, earlier layers quantized to 8-bit (Wu et al., 27 Jan 2026).

3. Discrete Tokenization and Unified Speech Representations

The Mimi codec's multi-stage RVQ scheme allows for unified tokenization suitable for both semantic and acoustic modeling. In Llama-Mimi, the first quantizer's codebook is distilled from a pretrained WavLM extractor and encodes predominantly semantic/phonetic content, while subsequent quantizers capture pitch, timbre, and background attributes (Sugiura et al., 18 Sep 2025).

For an audio waveform $D$6, non-overlapping frames yield $D$7 tokens per codebook. The quantizers produce interleaved tokens $D$8. During model training and inference, this interleaving preserves semantic-to-acoustic order without enforced constraints; the model learns RVQ ordering (Sugiura et al., 18 Sep 2025). Encoding/decoding logic is:

Encoding:

$K$0 Decoding:

$K$1 Adjusting the number of quantizers $D$9 mediates an inherent trade-off: higher $T \approx 30$0 increases raw audio fidelity and speaker preservation but degrades linguistic coherence (as measured by LLM-based and sWUGGY/sBLIMP metrics), with Llama-Mimi adopting $T \approx 30$1 for optimal balance (Sugiura et al., 18 Sep 2025).

4. Applications in TTS, Enhancement, and Language Modeling

Mimi serves as a backbone for several oral technologies:

• Real-time TTS: Mimi-based architectures have displaced neural vocoders by directly generating waveforms from compressed discrete latents. Integrating Mimi with a FastSpeech 2 frontend and depth-wise sequential RVQ index prediction yields an end-to-end non-autoregressive TTS pipeline. Each 80 ms frame's RVQ code indices are generated in parallel across time, but sequentially over quantization depth, maintaining low latency and resolving phonetic alignment (Su et al., 14 Apr 2026). The system achieves absolute speed-ups of over $T \approx 30$2 versus cascaded neural-vocoder pipelines and TTFB down to 49 ms.
• Speech Enhancement: Mimi's latent "bottleneck" is leveraged for throat-microphone enhancement. Fine-tuning the encoder on paired throat/air-mic data, with the decoder and quantizer frozen, yields a mapping onto codebooks pretrained on wideband natural speech. This facilitates high spectral restoration and intelligibility with minimal processing latency (15 ms on GPU per 80 ms frame) (Hauret et al., 5 Aug 2025).
• Speech Language Modeling (SLM): Llama-Mimi unifies semantic and acoustic modeling using interleaved RVQ token sequences from Mimi. A single transformer decodes these, with speaker identity and raw audio production quality increasing monotonically with $T \approx 30$3, but content coherence peaking at lower quantizer counts (Sugiura et al., 18 Sep 2025).

5. Quantitative Outcomes and Comparative Evaluations

Empirical benchmarks demonstrate Mimi's competitive performance:

• T-Mimi vs. Mimi: Objective metrics on 100 speech samples—PESQ ($T \approx 30$4 vs $T \approx 30$5), STOI (both $T \approx 30$6), SI-SDR ($T \approx 30$7 dB vs $T \approx 30$8 dB)—show on-par perceptual quality for a $T \approx 30$9 real-time decoding speed increase and $M$0 storage reduction (Wu et al., 27 Jan 2026).
• Speech enhancement: Mimi improves raw throat-mic SI-SDR by $M$1 dB and STOI by $M$2, approaching perceptual quality of much larger flow-matching models but with greatly reduced computation (Hauret et al., 5 Aug 2025).
• SLM tasks: Llama-Mimi 1.3B achieves substantially higher speaker similarity (cosine 0.346 with $M$3 quantizers) and acoustic token consistency compared to SSL-based or CFM models. Trade-offs between acoustic and semantic scores are clearly quantified (e.g., PQ $M$4 and Content LLM-Judge $M$5 for $M$6) (Sugiura et al., 18 Sep 2025).
• TTS streaming: Mimi-based systems run at RTF values as low as $M$7, with spectral fidelity surpassing continuous regression baselines—rich Mel-spectrogram striations and lower voicing error (Su et al., 14 Apr 2026).

6. Limitations, Trade-offs, and Future Directions

Present implementations of Mimi, including T-Mimi, display limitations:

• Quantization-aware training tools (e.g., TorchAO) remain limited for non-Transformer modules (transposed convolution), constraining further storage/latency gains for CNN-based codecs (Wu et al., 27 Jan 2026).
• Increased quantizer counts ($M$8) amplify sequence lengths—potentially challenging for long-form SLM inference—but are necessary for maximal acoustic fidelity (Sugiura et al., 18 Sep 2025).
• Mimi’s discrete bottlenecks enable natural-sounding inference even from bandwidth-reduced encoders (throat-mic), but hallucinated content may diverge from ground-truth, especially with sparse inputs (Hauret et al., 5 Aug 2025).
• Current streaming systems do not jointly optimize the entire codec (encoder, quantizer, decoder, and front-end models), leaving cross-module redundancy untapped.

Future work includes developing joint quantization-aware optimizations, exploring 4-bit QAT for early layers, adopting efficient attention variants (Linformer, layer-skipping), and integrating adversarial or perceptual losses for tighter temporal fidelity (Wu et al., 27 Jan 2026, Hauret et al., 5 Aug 2025). Extending the quantization and tokenization methodology to additional modalities (e.g., bone-conduction microphones) and deeper RVQ hierarchies remains an active research area.

7. Significance and Broader Impact

The Mimi codec establishes a compact and expressive intermediate representation for high-fidelity audio, enabling unification of acoustic, semantic, and language modeling tasks with a single tokenization scheme. Its discrete latent space is foundational in advancing real-time speech synthesis, robust enhancement in adverse conditions, and scalable end-to-end language modeling. Mimi’s design explicitly quantifies and operationalizes trade-offs among latency, storage, acoustic fidelity, and semantic preservation, offering clear benchmarks for codec and spoken LLM development (Wu et al., 27 Jan 2026, Hauret et al., 5 Aug 2025, Sugiura et al., 18 Sep 2025, Su et al., 14 Apr 2026).