MiMo-Audio-7B-Instruct Overview

• The paper introduces a 7B-parameter audio model that is fine-tuned for diverse tasks including ASR, TTS, spoken dialogue, and instruction following.
• It employs a three-component architecture—patch encoder, LLM backbone, and patch decoder—with autoregressive next-token prediction and chain-of-thought reasoning for improved audio understanding.
• Benchmark results demonstrate state-of-the-art performance in audio understanding, spoken dialogue, and TTS, highlighting its potential in unified multimodal acoustic modeling.

Searching arXiv for the specified MiMo-Audio and InstructAudio papers to ground the article and confirm bibliographic details. MiMo-Audio-7B-Instruct is a 7-billion-parameter audio LLM introduced as the instruction-tuned variant of MiMo-Audio-7B-Base and described in “MiMo-Audio: Audio LLMs are Few-Shot Learners” (Team et al., 29 Dec 2025). It preserves the base model’s three-component architecture—patch encoder, LLM backbone, and patch decoder—while fine-tuning all parameters for audio understanding, spoken dialogue, ASR, TTS, and instruction-following generation. In the supplied literature, it is also discussed alongside a separate “MiMo-Audio-7B-Instruct: Design and Performance Report,” which frames a 7B-scale realization of the InstructAudio MM-DiT paradigm for unified speech and music generation via natural-language instruction (Qiang et al., 23 Nov 2025). Taken together, these sources position MiMo-Audio-7B-Instruct at the intersection of autoregressive audio language modeling and instruction-controlled acoustic generation, with emphasis on large-scale pretraining, post-training for instruction following, and broad benchmark coverage.

1. Model identity and architectural organization

MiMo-Audio-7B-Instruct builds directly on MiMo-Audio-7B-Base, preserving a three-component design and 7 billion total parameters; no structural changes are made in the Instruct version, and all parameters are simply fine-tuned (Team et al., 29 Dec 2025). The architecture consists of a patch encoder, an LLM backbone initialized from MiMo-7B-Base, and a patch decoder. The patch encoder has 6 transformer blocks, hidden dimension 1,024, FFN inner dimension 4,096, 64 attention heads, and context length 4 frames (6.25 Hz). The LLM backbone has 36 layers, hidden dimension 4,096, FFN inner dimension 11,008, 32 attention heads, context length 8,192 tokens, and text vocabulary 151,680 tokens. The patch decoder has 16 transformer blocks, hidden dimension 1,024, FFN inner dimension 4,096, 64 attention heads, and audio codebooks $(R' = 8)$ with sizes $[1024,1024,128\times6]$ (Team et al., 29 Dec 2025).

This configuration reflects a discrete-audio-token autoregressive design rather than a diffusion model. In the source material, that design choice is tied to large-scale next-token prediction pretraining over interleaved text and audio sequences. A plausible implication is that MiMo-Audio-7B-Instruct inherits the scaling behavior and transfer properties associated with LLM-style autoregressive pretraining, but in the audio domain.

The accompanying design report introduces a distinct but related framing: MiMo-Audio-7B-Instruct as a 7B-scale extension of InstructAudio’s MM-DiT architecture, obtained by combining a large diffusion backbone with a frozen 7B instruction encoder such as Qwen-7B (Qiang et al., 23 Nov 2025). That report describes joint and single diffusion transformer layers, instruction embeddings $E_{\text{instr}}$, phoneme embeddings $E_{\text{phon}}$, and joint attention over concatenated text-audio conditions. Because the two sources describe different architectural lineages, the name “MiMo-Audio-7B-Instruct” is associated in the supplied data with both an autoregressive audio LLM and a projected 7B-scale InstructAudio realization. This is best understood as a nomenclatural overlap rather than a single unified implementation claim.

2. Pretraining regime and scaling behavior

MiMo-Audio’s pretraining follows a two-stage, next-token-prediction paradigm over more than one hundred million hours of speech plus a large text corpus (Team et al., 29 Dec 2025). The total speech volume is reported as $O(10^8)$ hours from podcasts, audiobooks, news, interviews, conferences, and related sources. Stage 1 uses 2.6T tokens, comprising 1.2T text and 1.4T audio at 6.25 Hz; Stage 2 uses 5T tokens, comprising 2.6T text and 2.4T audio.

The pretraining objective is autoregressive modeling over an interleaved sequence $S$ of text tokens and audio patches $P$:

$p(S)=\prod_{i=1}^{L} p(s_i \mid s_{1:i-1}),$

with the standard cross-entropy loss

$$\mathcal{L}_{\mathrm{pretrain}} = -\sum_{i=1}^{L}\log p(s_i \mid s_{<i}).$$

Stage 1 is described as “speech understanding only.” Its tasks are speech-text interleaved modeling, ASR, audio captioning, and text-only language modeling, with loss applied only on text tokens. The optimization configuration is specified as $$\mathrm{LR}_{\text{patch-encoder}} = 2\times10^{-4}$$ and $[1024,1024,128\times6]$0 under a constant schedule, with batch size 16.8M tokens and context length 8,192 (Team et al., 29 Dec 2025).

Stage 2 adds generation-oriented tasks: speech continuation, interleaved speech/text, ASR, TTS, audio captioning, instruct-TTS, and text-only training. Loss weights over text and 8 RVQ layers are given as $[1024,1024,128\times6]$1. Learning rates are $[1024,1024,128\times6]$2 and $[1024,1024,128\times6]$3 with cosine decay, and the delay pattern for RVQ codebooks is $[1024,1024,128\times6]$4 (Team et al., 29 Dec 2025).

The paper associates this scale of pretraining with the emergence of few-shot learning capabilities across diverse audio tasks (Team et al., 29 Dec 2025). More specifically, zero-/few-shot speech-to-speech tasks in 16-shot contexts—including voice conversion, emotion conversion, rate conversion, denoising, and speech translation—are reported to emerge sharply once pretraining tokens exceed approximately 0.7T, which the source describes as a phase transition or “GPT-3 moment.” This suggests that the model’s post-training performance should be interpreted not only as a consequence of supervised instruction tuning but also as an effect of scale-induced generalization.

3. Instruction tuning and post-training objectives

After pretraining, all 7 billion parameters undergo supervised fine-tuning on approximately 100 billion tokens covering six task types (Team et al., 29 Dec 2025). These are ASR, using AISHELL-1 and LibriSpeech; TTS, using SeedTTS and InstructTTSEval EN/ZH; audio understanding and reasoning, using MMSU, MMAU, MMAR, and MMAU-Pro; spoken dialogue, using Big Bench Audio and MultiChallenge Audio; instruction-following TTS, described as audio caption $[1024,1024,128\times6]$5 speech; and text dialogue, for spoken dialogue synthesis.

The optimization setup for post-training is specified as $[1024,1024,128\times6]$6 and $[1024,1024,128\times6]$7, with cosine decay, 1% warmup, batch size 2.1M tokens, context length 8,192, and the same loss weights as Stage 2 pretraining: $[1024,1024,128\times6]$8 (Team et al., 29 Dec 2025). This continuity between Stage 2 and instruction tuning indicates that post-training does not replace the generative objective but further specializes it through supervised alignment across multiple audio and dialogue tasks.

A distinctive element of the post-training setup is the introduction of a “thinking” mechanism into both audio understanding and generation (Team et al., 29 Dec 2025). The source describes this as chain-of-thought examples added for selected audio QA tasks, such as MMSU, prompting the model to generate intermediate reasoning before final answers. An ablation reported in the paper states that chain-of-thought data improves MMSU perception by +4.85 points, with a small trade-off in reasoning, for +1.18 points overall. This places MiMo-Audio-7B-Instruct within a broader line of work exploring explicit intermediate reasoning in multimodal systems, while also making clear that the intervention is selective rather than globally applied.

The InstructAudio design report presents a different instruction-conditioning mechanism centered on a standardized instruction-phoneme representation (Qiang et al., 23 Nov 2025). In that formulation, free-form natural-language prompts are encoded by a frozen LLM to obtain $[1024,1024,128\times6]$9, text or lyrics are converted to phonemes and encoded as $E_{\text{instr}}$0, and the concatenation is fed into joint diffusion transformer layers. Natural-language instruction descriptions specify desired acoustic attributes, and speaker-tag tokens $E_{\text{instr}}$1 are prepended for dialogue. This suggests an alternative route to instruction following based on explicit instruction-text and phoneme-text fusion rather than autoregressive instruction tuning alone.

4. Benchmark performance

MiMo-Audio-7B-Instruct is reported to achieve open-source SOTA on audio understanding benchmarks, spoken dialogue benchmarks, and instruct-TTS evaluations, approaching or surpassing closed-source models (Team et al., 29 Dec 2025). The reported results span audio understanding and reasoning, spoken dialogue, TTS, and ASR.

Benchmark area	Metric or dataset	MiMo-Audio-7B-Instruct
Audio understanding	MMAU Overall	74.90
Audio reasoning	MMAR Overall	63.60
Audio understanding	MMAU-Pro Overall	53.35
Spoken dialogue	Big Bench Audio / MultiChallenge Audio (S2T / S2S)	72.90 / 60.20
TTS	InstructTTSEval-EN Overall	72.59
ASR	LibriSpeech-test-clean / AISHELL WER	3.50% / 1.65%

For audio understanding and reasoning, the paper reports MMAU scores of 68.47 on Speech, 82.58 on Sound, 73.65 on Music, and 74.90 Overall; MMAR 63.60 Overall; MMAU-Pro 53.35 Overall; and MMSU with 46.86 on Perception, 76.98 on Reasoning, and 61.70 Overall, all marked as best among open-source models where indicated (Team et al., 29 Dec 2025). When “thinking” is enabled on MMSU, the reported scores are 51.71 Perception, 74.79 Reasoning, and 62.88 Overall.

For spoken dialogue, MiMo-Audio-Instruct records 72.90 on S2T and 60.20 on S2S, compared with 70.20 and 67.20 for gpt-4o-audio-preview and 50.90 and 47.50 for Step-Audio2-mini (Team et al., 29 Dec 2025). The asymmetry between S2T and S2S scores is important: MiMo-Audio-7B-Instruct leads on S2T in the reported comparison, whereas gpt-4o-audio-preview exceeds it on S2S.

For TTS and ASR, the model scores 1.96 / 5.37 / 14.14 on Seed-TTS-Eval ZH / EN / ZH-H, 72.59 on InstructTTSEval-EN Overall, 3.50% WER on LibriSpeech-test-clean, and 1.65% WER on AISHELL (Team et al., 29 Dec 2025). The paper further states that MiMo-Audio-7B-Instruct approaches or surpasses closed-source models on these evaluations.

The InstructAudio report supplies a separate benchmark profile for its instruction-controlled speech and music generator (Qiang et al., 23 Nov 2025). On TTS, it reports Word Error Rate on Seed-TTS of 1.52% (EN) and 1.35% (ZH), instruction-control classification accuracy of Gender 100%, Age 86.7%, Emotion 83.3%, Style 86.7%, Accent 100%, and Dialog 90%, as well as Speaker Sim 0.76, Emotion Sim 0.71, LSD 1.88 dB, MCD 5.71 dB, MSEP 437.6, MR 0.33, and MOS values QMOS $E_{\text{instr}}$2 and NMOS $E_{\text{instr}}$3 (Qiang et al., 23 Nov 2025). On text-to-music, it reports classification accuracy for Genre 92.8%, Instrument 83.9%, Singer-Gender 98.9%, Singer-Age 97.2%, Rhythm 94.4%, and Atmosphere 95.0%, SongEval values $E_{\text{instr}}$4, and MOS values QMOS $E_{\text{instr}}$5 and MMOS $E_{\text{instr}}$6. These numbers pertain to the InstructAudio framework rather than the MiMo-Audio autoregressive model, but they are relevant because the design report uses them to motivate a 7B-scale instruction-following extension under the MiMo-Audio-7B-Instruct name.

5. Instruction control, modality coverage, and generation capabilities

Within the supplied sources, MiMo-Audio-7B-Instruct is characterized by broad modality coverage. The autoregressive MiMo-Audio paper emphasizes audio understanding, speech intelligence, spoken dialogue, instruct-TTS, and generalization to tasks absent from the training data, including voice conversion, style transfer, and speech editing (Team et al., 29 Dec 2025). It also states that MiMo-Audio-7B-Base demonstrates powerful speech continuation capabilities, capable of generating highly realistic talk shows, recitations, livestreaming, and debates, and that the Instruct version adds instruction-following and thinking mechanisms at post-training time.

The InstructAudio report provides a more fine-grained taxonomy of controllable attributes (Qiang et al., 23 Nov 2025). The listed controllable dimensions are timbre (gender, age), paralinguistic attributes (emotion, style, accent), musical attributes (genre, instrument, rhythm, atmosphere), and dialogue through turn-taking markers $E_{\text{instr}}$7. Instruction integration is described explicitly in three steps: encoding the free-form natural-language prompt with the frozen LLM to obtain $E_{\text{instr}}$8; encoding the phoneme sequence from text or lyrics to obtain $E_{\text{instr}}$9; and concatenating the two before feeding them into joint diffusion transformer layers.

That report further states that the underlying unified framework supports expressive speech, music, and dialogue generation in English and Chinese, trained on 50,000 hours of speech and 20,000 hours of music data (Qiang et al., 23 Nov 2025). Dialogue data is approximately 0.5% of the corpus and is marked with speaker tokens; clip length ranges from 2 to 20 seconds and is standardized to 44.1 kHz. Mixed-batch sampling of speech and music according to dataset proportions enables a single unified MM-DiT model to learn a shared latent diffusion representation for both modalities. The total loss is described as a sum over speech and music diffusion objectives, with sampling proportions approximately 5:2 in hours balanced per batch.

A central conceptual distinction thus emerges. In MiMo-Audio (Team et al., 29 Dec 2025), instruction following is realized through large-scale autoregressive pretraining plus supervised instruction tuning across many task types. In InstructAudio (Qiang et al., 23 Nov 2025), instruction following is realized through explicit conditioning on natural-language instruction and phoneme representations within a diffusion transformer. The two approaches share an interest in free-form natural-language control over audio generation, but they operationalize that control differently.

6. Limitations, ambiguities, and research significance

The MiMo-Audio paper states several limitations directly (Team et al., 29 Dec 2025). In-context learning remains limited on complex audio generation, with examples including background music and polyphonic sound events. Spoken dialogue can suffer timbre discontinuities, mispronunciations, and style-control instability. The thinking mechanism can introduce hallucinations in non-speech audio tasks. Proposed future work includes reinforcement learning to stabilize dialogue, enhance audio generation, and refine chain-of-thought reasoning.

The InstructAudio report likewise frames its 7B-scale extension as a design path rather than a fully benchmarked release under the same evaluation suite as MiMo-Audio (Qiang et al., 23 Nov 2025). Its conclusion states that the MM-DiT design, instruction-phoneme conditioning format, and joint speech/music diffusion training demonstrate a clear path to a 7B-parameter MiMo-Audio-7B-Instruct framework. It also notes that scaling to MiMo-Audio-7B-Instruct is expected to further improve naturalness and control fidelity, particularly in long-form music and multi-speaker dialogues. Because this language is prospective, it should not be conflated with empirically established results for the autoregressive MiMo-Audio-7B-Instruct system.

A common misconception would be to treat the two supplied sources as describing a single identical model. The evidence provided instead supports a narrower interpretation. “MiMo-Audio-7B-Instruct” in (Team et al., 29 Dec 2025) is a concrete post-trained autoregressive audio LLM with reported benchmark results, architectural dimensions, and training schedules. The same name in the design report linked to (Qiang et al., 23 Nov 2025) denotes a proposed or realizable 7B-scale extension of InstructAudio’s MM-DiT framework. This suggests that the term can refer either to an existing MiMo-Audio instruction-tuned model or to a 7B-scale instruction-controlled speech/music MM-DiT realization, depending on source context.

The broader research significance lies in the convergence of two trends. One is the scaling of autoregressive audio LLMs to very large pretraining corpora, yielding few-shot and instruction-following behavior across diverse audio tasks (Team et al., 29 Dec 2025). The other is the unification of speech and music generation under natural-language instruction through joint multimodal acoustic modeling (Qiang et al., 23 Nov 2025). Together, these sources indicate a field-wide movement toward general-purpose audio models that combine instruction following, broad task transfer, and increasingly unified treatment of speech, music, and dialogue.