Mixtral 8x7B Instruct: Sparse MoE Chat Model

• Mixtral 8x7B Instruct is an instruction-tuned chat assistant built on a sparse Mixture-of-Experts backbone with 47B total and 13B active parameters.
• It uses a top-2 expert routing mechanism from 8 experts per layer to enable efficient inference and supports a 32k-token context length.
• Benchmark and adaptation studies indicate that Mixtral outperforms several leading LLMs on human evaluations and specialized domain tasks.

Searching arXiv for the main Mixtral paper and a few later papers on adaptation and analysis to ground the article in published work. Mixtral 8x7B Instruct is the instruction-following, chat-oriented variant of Mistral’s Mixtral 8x7B family. It uses the same Sparse Mixture of Experts backbone as the base Mixtral 8x7B model, retains a fully dense 32k-token context length, and combines a 47B-parameter sparse capacity with only 13B active parameters during inference for a given token. In the original report, the instruct model is obtained by supervised fine-tuning followed by Direct Preference Optimization, is released under the Apache 2.0 license, and is reported to surpass GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B-chat on human benchmarks (Jiang et al., 2024).

1. Model identity and release context

Mixtral 8x7B Instruct is not a separate architecture from Mixtral 8x7B; it is the same underlying model further adapted to follow user instructions and behave like a conversational assistant. The base model is described as a pretrained Sparse Mixture of Experts LLM, while the instruct version is its fine-tuned companion. A recurrent misconception is to read the name as if it denoted eight independent 7B chat models or a dense $56$B system. The paper instead describes a single decoder-only transformer whose feedforward blocks are replaced by mixture-of-experts layers; the full model has 47B total parameters, but only 13B active parameters are used for a given token during inference (Jiang et al., 2024).

The release context is also notable. Both the base and instruct variants are released under the Apache 2.0 license, which positioned the model as an openly available high-capacity assistant in the early 2024 open-weight ecosystem. The instruct model’s significance therefore derives not only from its chat performance, but from the combination of open weights, sparse activation, and long-context support within the same system (Jiang et al., 2024).

2. Sparse MoE design and routing mechanism

Architecturally, Mixtral is described as “the same architecture as Mistral 7B,” except that standard feedforward networks are replaced by Mixture-of-Experts layers. The reported configuration is a decoder-only transformer with dimension $4096$, $32$ layers, $32$ attention heads, $8$ key-value heads, hidden dimension $14336$, vocabulary size $32000$, context length $32768$, $8$ experts per layer, and top-2 expert routing (Jiang et al., 2024).

Component	Reported value
Layers	32
Model dimension	4096
Attention heads	32
Key-value heads	8
Hidden dimension	14336
Vocabulary size	32000
Context length	32768
Experts per layer	8
Active experts per token	2

The routing mechanism is the central technical feature. For each token and each layer, a router network selects only two of the eight expert FFNs. The general MoE output is written as

$\sum_{i=0}^{n-1} G(x)_i \cdot E_i(x),$

where $4096$0 is the output of expert $4096$1, and $4096$2 is the router weight. The router is implemented as

$4096$3

For Mixtral specifically, $4096$4, so the token-level output becomes

$4096$5

Each expert is a standard SwiGLU feedforward block, and the two selected experts are combined additively (Jiang et al., 2024).

This sparsity explains the distinction between total and active parameters. Each token has access to 47B parameters overall, but only uses 13B active parameters during inference. The paper explicitly connects this to compute efficiency: token-by-token compute is closer to that of a much smaller dense model, whereas memory cost is proportional to the full sparse parameter count, $4096$6B, which it notes is still smaller than Llama 2 70B (Jiang et al., 2024).

3. Instruction tuning, alignment, and conversational behavior

The instruct model is trained in two stages: supervised fine-tuning on an instruction dataset, followed by Direct Preference Optimization on paired feedback data. This post-training sequence is the basis for its assistant-style behavior and differentiates it from the pretrained base model, whose strengths are broader language-modeling and benchmark performance rather than alignment to conversational use (Jiang et al., 2024).

The headline chat result in the original paper is an MT-Bench score of $4096$7. The paper states that this made Mixtral 8x7B Instruct the best open-weights model as of December 2023, and further reports that independent human evaluation from LMSys showed that Mixtral Instruct outperformed GPT-3.5-Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B-chat on human evaluation benchmarks (Jiang et al., 2024).

Subsequent human-centered evaluation refined that picture rather than overturning it. In a between-subjects empathy study over 2,000 emotional dialogue prompts and 1,000 participants, Mixtral-8x7B-Instruct received 1,192 “Good” ratings, versus 986 for human responses, corresponding to a 20.89% increase in “Good” ratings relative to the human baseline; this gain was statistically significant, but Mixtral still ranked below GPT-4 and behind or comparable to LLaMA-2-70B-Chat overall (Welivita et al., 2024). This suggests that the instruction-tuning stack yielded strong perceived empathy under explicit prompting, while not making the model uniformly dominant across all forms of human-facing evaluation.

4. Benchmark profile and long-context behavior

The base Mixtral 8x7B model is reported to match or exceed Llama 2 70B and GPT-3.5 on most evaluated benchmarks, with especially strong results on mathematics, code generation, and multilingual tasks. The paper’s benchmark suite includes MMLU, HellaSwag, WinoGrande, PIQA, ARC-Easy, ARC-Challenge, Natural Questions, TriviaQA, HumanEval, MBPP, MATH, and GSM8K. Reported results include MMLU $4096$8, MBPP $4096$9, MATH $32$0, and GSM8K $32$1, and the paper emphasizes that these are achieved with only 13B active parameters, about 5x fewer active parameters than Llama 2 70B. In a separate direct comparison table, the same paper reports Mixtral at 70.6% on MMLU, 85.8% on ARC Challenge, 60.7% on MBPP, and 58.4% on GSM-8K (Jiang et al., 2024).

Long-context behavior is presented as more than nominal context support. The model was pretrained with a 32k-token context size, achieved 100% retrieval accuracy on the passkey retrieval task regardless of where the key appeared in the sequence and regardless of sequence length, and showed perplexity on a Proof-Pile subset that decreased monotonically as context length increased. The paper interprets this as evidence that the model genuinely uses its long context window effectively (Jiang et al., 2024).

Comparative leaderboard work complicated the model’s standing without negating its strengths. In the SOLAR 10.7B-Instruct comparison on the HuggingFace Open LLM Leaderboard suite, Mixtral 8x7B-Instruct-v0.1 is reported at H6 $32$2, ARC $32$3, HellaSwag $32$4, MMLU $32$5, TruthfulQA $32$6, Winogrande $32$7, and GSM8K $32$8. SOLAR 10.7B-Instruct reports a higher H6 of $32$9, but Mixtral remains higher on MMLU by $32$0 points (Kim et al., 2023). The comparison is best read as evidence that Mixtral’s sparse scaling was highly competitive rather than uniquely unchallenged.

5. Adaptation, localization, and downstream reuse

Mixtral 8x7B Instruct became a frequent backbone for language adaptation. In the Chinese setting, the Aurora project instruction-tuned mixtral-8x7b-instruct-v0.1 with LoRA and 4-bit quantization on 176,678 cleaned Chinese instruction pairs, reporting 5-shot scores of C-Eval $32$1, MMLU $32$2, and CMMLU $32$3 while framing the work as evidence that sparse MoE models can be instruction-tuned in the same general manner as dense models (Wang et al., 2023). A separate Chinese adaptation study started from Mixtral-8x7B-v0.1 rather than the instruct checkpoint, performed further pre-training on 20GB of general Chinese corpus, roughly 7B tokens, and then instruction fine-tuning on 5M Chinese instruction samples. That study reported improved Chinese understanding and generation while retaining English abilities, concluded that the foundation model is the better initialization for cross-lingual adaptation, and found that vocabulary extension reduced token count by 32.6% but did not improve downstream benchmark performance (Cui et al., 2024).

Domain adaptation followed a similar pattern. In a clinical adaptation study, Mixtral was continuously pretrained on a 50B-token clinical corpus mixed with 15B tokens from SlimPajama for a total of 65B tokens over 4 epochs, or 260B processed tokens, and then instruction-tuned on a 500M-token clinical QA dataset. The reported Mixtral 8x7b P+F scores were MedQA $32$4, USMLE $32$5, MMLU $32$6, and MedMCQA $32$7, while MedPrompt-style prompting pushed MedQA accuracy from 52.55% to above 75% (Christophe et al., 2024). In Norwegian adaptation, NorwAI-Mixtral-8x7B-instruct is documented as a 47.00B-parameter MoE model with 32k context length, 4096 hidden size, a 68k tokenizer vocabulary, and instruction tuning after continual pretraining on the 51.15B-token NorLLM_Corpus_V2; in a blind human evaluation on 51 Schibsted news articles, it achieved the highest average score, outperforming both GPT-4 and journalist-written summaries (Gulla et al., 6 Jan 2026).

The model was also reused as a component inside hybrid pipelines rather than as a standalone assistant. In biomedical nested NER, Mixtral 8x7B Instruct served as the prompted entity proposer, queried separately for each BioNNE category with two few-shot examples and semicolon-delimited outputs; the full system reached F1 $32$8 on validation and $32$9 on test, while removing UMLS heuristics reduced validation macro-F1 from $8$0 to $8$1, showing that Mixtral’s candidate generation needed heavy semantic filtering (Zhou, 2024). In synthetic-data construction, Intellecta Cognitiva used Mixtral-8x7B-Instruct-v0.1 to generate the synthetic portion of an 11.53B-token corpus, specifically 8.01B synthetic tokens alongside 3.52B textbook tokens, with prompts for textbook-style explanation and reasoning-oriented “thought” expansion (PS et al., 2024).

6. Limitations, safety, and later interpretation

Off-the-shelf instruction tuning did not make Mixtral 8x7B Instruct uniformly strong across constrained extraction and normalization tasks. In knowledge graph completion for a task-oriented dialogue ontology, Mixtral-8x7B-Instruct-v0.1 reached $8$2 strict accuracy / $8$3 flexible accuracy and $8$4 strict F1 / $8$5 flexible F1 on Templates Easy with hand-written prompts, and $8$6 / $8$7 accuracy with $8$8 / $8$9 F1 on Templates Hard, revealing a large strict-versus-flexible gap and frequent format violations (Iga et al., 2024). In Romanian diacritic restoration, its MTAS was $14336$0, below the echo baseline at $14336$1; the paper attributes this largely to over-generation, stating that Mixtral-8x7B-Instr “adds on average 16.35 diacritics per 10-word sentence” (Nadas et al., 17 Nov 2025). In receipt-item categorisation on AWS Bedrock, it achieved precision $14336$2, recall $14336$3, F1 $14336$4, accuracy $14336$5, balanced accuracy $14336$6, mean latency $14336$7 ms, and an approximate array-length mismatch rate of 1.9%, which placed it below the Claude models in correctness and schema reliability but above Mistral 7B Instruct (Sanchez et al., 2 Apr 2026).

These results support a narrower interpretation of the model’s strengths. Mixtral often performs well as a general-purpose generator, assistant, or recall-oriented proposer, but exact schema adherence, orthographic precision, and domain boundary control frequently require additional prompting, filtering, or post-processing. That pattern is visible across biomedical NER, knowledge graph completion, and receipt classification, where downstream systems either constrain outputs tightly or compensate for them algorithmically (Zhou, 2024, Iga et al., 2024, Sanchez et al., 2 Apr 2026).

Later interpretability work also showed that the sparse router does not yield a simple “harmful experts” story. In a routing analysis of Mixtral 8x7B-Instruct under benign and harmful prompts, activation-based expert usage was broad and long-tailed, while gradient-based importance was concentrated. At the expert-classification level, 216 of 256 layer-expert pairs were labeled shared under activation scores and 245 under gradient scores. Suppressing the top five benign-dominant experts reduced restricted responses from 24 to 14 in activation-based experiments and from 34 to 22 in gradient-based experiments, yet the authors concluded that safety-relevant routing is subtle, depth-dependent, and distributed rather than dominated by a fixed set of experts (Siddiky, 22 May 2026).

Mixtral 8x7B Instruct therefore occupies a specific place in the post-2023 open-weight landscape. It established that a sparse MoE assistant with 32k context, 47B total parameters, and 13B active parameters could compete directly with much larger dense and proprietary systems on many benchmarks and human evaluations, while later work showed that its strengths translated unevenly across languages, domains, and structurally constrained tasks.