Llama-Mimi: Unified Speech & Audio Model

• Llama-Mimi is a speech language model that unifies semantic and acoustic tokens using a single-stage Transformer decoder for integrated language and audio generation.
• It introduces an extended vocabulary and an LLM-as-a-Judge evaluation protocol, providing robust metrics for acoustic consistency and speaker identity preservation.
• The model reveals a trade-off between high-fidelity acoustic detail and semantic coherence, offering actionable insights for future multimodal speech synthesis research.

Llama-Mimi is a speech LLM characterized by its unified sequence modeling of interleaved semantic and acoustic tokens. It is distinguished by its use of a single Transformer decoder with an extended vocabulary to directly predict both language and audio content. This approach delivers state-of-the-art results in acoustic consistency and speaker identity preservation, and introduces a new LLM-based evaluation protocol for assessing spoken content quality. The framework is publicly released for open research and reproducibility.

1. Unified Tokenization and Single-Stage Architecture

Llama-Mimi uses a unified tokenizer based on Mimi, a neural audio codec, to convert raw audio waveforms $x \in \mathbb{R}^T$ into a discrete sequence $h$ of both semantic and acoustic tokens. For an input audio segmented into $T'$ frames and employing $Q$ quantizers, the representation follows:

$$h = (y_1^1, y_1^2, \ldots, y_1^Q, y_2^1, \ldots, y_{T'}^Q)$$

where each frame’s $Q$ tokens are ordered by quantizer. This sequence efficiently interleaves linguistic elements (semantic tokens encoding higher-level intent) and fine-grained acoustic details (acoustic tokens enabling signal-level rendering).

A single Transformer decoder based on the Llama 3 architecture is adapted with an extended vocabulary and special <audio> / </audio> tokens, allowing seamless switching between textual and audio modalities. Unlike multi-stage architectures or dual Transformer systems, Llama-Mimi generates the unified token sequence autoregressively, predicting semantic tokens before conditioning subsequent acoustic tokens frame-wise. This single-stream approach simplifies the model and tightly couples linguistic and prosodic conditioning for speech generation.

2. Training Objectives and Modal Interleaving

The model is trained with a standard autoregressive next-token prediction objective, handling mixed semantic and acoustic tokens without distinction. Conditioning acoustic tokens on previously generated semantic tokens within each frame encourages syntactic and prosodic coherence in the resulting speech.

Special tokens <audio> and </audio> demarcate audio segments and enable mixed text-to-speech generation. The extended vocabulary encodes both linguistic and acoustic elements, supporting multimodal generation from unified input streams.

3. Quantizer Scaling: Fidelity and Semantic Coherence

A central finding of the Llama-Mimi analysis is the impact of quantizer count $Q$ on output fidelity and coherence. Increasing $Q$ yields finer acoustic detail, reflected in higher measured speaker similarity and improved audio quality metrics. For example, cosine similarity (WavLM embedding) for speaker identity rises from 0.346 (Q=4) to 0.474 (Q=8), and audio aesthetics metrics show corresponding improvements.

However, this scaling introduces longer token sequences, creating a tension with the Transformer’s ability to model long-term dependency. As $Q$ increases, semantic fidelity measured by LLM-based quality scores degrades ($3.01$ at Q=4 versus $2.54$ at Q=8), indicating a challenge in balancing acoustic detail against linguistic coherence. This suggests an inherent trade-off between high-resolution acoustic rendering and the retention of long-range semantic structure over extended outputs.

4. Acoustic Consistency and Speaker Identity Preservation

Llama-Mimi achieves state-of-the-art accuracy on acoustic consistency tasks using the SALMon benchmark, which tests the model’s likelihood assignments to natural samples with consistent acoustic attributes (sentiment, speaker, background). Cosine similarity scores for speaker preservation consistently outperform competing models, underscoring its efficacy in maintaining identity through speech generation. The joint modeling approach, with interleaved tokens, is crucial for this property—explicitly conditioning each token on the preceding context, including the semantic frames that carry speaker embedding information.

Semantic benchmarks (sWUGGY, sBLIMP, sTopic-StoryCloze) for grammaticality, continuation, and real-word discrimination reveal that while direct trade-offs exist, Llama-Mimi maintains competitive performance on semantic knowledge even in the presence of higher quantizer configurations.

5. LLM-as-a-Judge Evaluation Protocol

Traditional evaluation metrics, such as likelihood and perplexity, are inadequate for synthesized speech sequences due to instability across varying sequence lengths and sampling artifacts. Llama-Mimi introduces an LLM-as-a-Judge protocol for robust evaluation:

• Generated speech samples are transcribed using an ASR system (Whisper Turbo).
• Prompt and completion transcripts are scored by GPT-4o with fixed temperature settings for deterministic assessment.
• The LLM is instructed to rate completions from 1–10 for relevance, coherence, fluency, and informativeness.

This produces more consistent and discriminative spoken content quality scores compared to perplexity-based methods, enabling reliable comparison across models and quantizer configurations.

6. Public Release and Community Impact

Llama-Mimi’s codebase, trained models, and speech samples are made publicly available, promoting transparency and reproducibility. This facilitates independent verification of findings and accelerates collaboration and downstream research in speech modeling, audio-text integration, and evaluation protocol design.

The unified interleaving of modalities, simplified decoder architecture, and introduction of content quality evaluation establish a well-documented baseline for future work in holistic speech generation and multimodal LLMing.

7. Challenges and Future Directions

The main challenge highlighted is maintaining semantic coherence as acoustic token resolution—and sequence length—increases. Possible future directions (suggested by these findings) include:

• Architectural innovations to enhance long-range dependency modeling in unified transformers.
• Adaptive quantizer selection strategies for balancing fidelity and coherence per utterance.
• Extending the LLM-as-a-Judge paradigm to broader multimodal evaluations in speech-driven applications.

A plausible implication is the need for targeted sequence modeling advances to enable high-resolution speech synthesis without eroding the linguistic content quality, especially at greater quantizer depths and longer output spans.

---

Llama-Mimi advances speech LLMing by jointly modeling semantic and acoustic information with a unified transformer pipeline, presenting rigorous evaluations and trade-off analyses. Its publicly released models and code provide a foundation for further exploration of interleaved sequence modeling, speech quality assessment, and scalable multimodal generation.