---
title: Ministral 3 Series Overview
url: https://www.emergentmind.com/topics/ministral-3-series
type: topic
---

# Ministral 3 Series Overview

The Ministral 3 Series comprises a family of parameter-efficient, dense large language models (LLMs) constructed for compute- and memory-constrained applications. Released in three model sizes—3B, 8B, and 14B parameters—each size is available in three variants: a base pretrained model, an instruction-finetuned checkpoint, and a reasoning model optimized for complex problem-solving. All models are derived via Cascade Distillation from the Mistral Small 3.1 24B teacher and incorporate image understanding capabilities, with weights and configurations distributed under the Apache 2.0 license [2601.08584].

## 1. Architectural Overview and Model Sizes

All models in the Ministral 3 Series adopt a decoder-only Transformer backbone featuring Grouped Query Attention, RoPE positional embeddings, SwiGLU activations, and RMSNorm. They support contexts up to 256K tokens (128K for reasoning variants) and utilize a 131K vocabulary. Embedding strategies vary by size, with tied embeddings in the 3B and untied in larger models.

| Model Size  | Layers | Hidden Dim | Attention Heads | FFN Dim | Embedding Size | Context Length | Embeddings          |
|-------------|--------|------------|-----------------|---------|----------------|---------------|---------------------|
| 14B         | ~40    | 5120       | 32 Q / 8 KV     | 16384   | 5120 (untied)  | 256K          | Untied              |
| 8B          | ~34    | 4096       | 32 Q / 8 KV     | 14336   | 4096 (untied)  | 256K          | Untied              |
| 3B          | ~26    | 3072       | 32 Q / 8 KV     | 9216    | 3072 (tied)    | 256K          | Tied (input-output) |

The 14B variant achieves >40% parameter reduction relative to its 24B teacher while retaining near-teacher performance.

## 2. Model Variants and Training Objectives

For each size, three distinct checkpoints are released:

- **Base (Pretrained):** Trained for next-token prediction, using logits distilled from Mistral Small 3.1 (24B). The training set blends text-only and interleaved text+image streams, totaling 1–3T tokens. The distillation is conducted in two stages (short-context, then long-context).
- **Instruct (Instruction-finetuned):** Extended with supervised fine-tuning (SFT) on high-quality multimodal/text instruction datasets. SFT loss ($\mathcal{L}_{\text{SFT}}$) employs fp8 quantization and logit distillation from a stronger teacher (Mistral Medium 3). Online Direct Preference Optimization (ODPO) further refines the model using a pairwise reward model and a two-sided DPO loss:

  $$
  \mathcal{L}_{\text{ODPO}}(\theta) = -\mathbb{E}_{(x, y^+, y^-)}[w \cdot \log \sigma(\Delta_\theta) + (1 - w) \cdot \log(1 - \sigma(\Delta_\theta))]
  $$

  where $\Delta_\theta = \log \pi_\theta(y^+|x) - \log \pi_\theta(y^-|x)$ and $w = \Pr_{\text{PWRM}}(y^+ \succ y^-)$.

- **Reasoning:** Designed to improve chain-of-thought (CoT) performance through SFT on CoT data, Group Relative Policy Optimization (GRPO), and final ODPO polish. GRPO occurs in two stages: STEM RL (math/code/visual reasoning using an internal LLM judge) and General RL (broader chat/instruction with rubric-based rewards). The 3B model receives additional logit distillation from Magistral Small 1.2 to mitigate verbosity.

## 3. Cascade Distillation: Pruning and Knowledge Transfer

The training of Ministral 3 relies on Cascade Distillation, which iteratively prunes and distills from the 24B teacher through progressively smaller models. The procedure is:

```
teacher ← Mistral_Small_3.1
for size in [14B, 8B, 3B]:
    student₀ ← prune(teacher, target=size)
    student₁ ← distill(student₀, teacher, ctx=16K)
    student_final ← distill(student₁, teacher, ctx=256K)
    yield student_final
    teacher ← student₁  # cascade to next size
```

- **Layer Pruning:** Layers are ranked by the mean ratio of output_norm to input_norm.
- **Hidden-dim Pruning:** Principal Component Analysis (PCA) is applied across layer-norm inputs, followed by a global rotation and projection.
- **FFN Pruning:** Each SwiGLU hidden dimension $i$ is scored as $s_i = \mathbb{E}[|\text{SiLU}(W_1 x)_i \cdot (W_3 x)_i|]$, and only the top-$k$ dimensions are retained.
- **Distillation:** The knowledge distillation loss per token $t$ is the forward KL:

  $$
  \mathcal{L}_{\text{KD}} = \mathrm{KL}(p_T(\cdot\,|\,x_{<t}) \Vert p_S(\cdot\,|\,x_{<t}))
  $$

  The process omits a next-token cross-entropy term (i.e., $\lambda=0$).

This iterative pipeline allows each smaller student to inherit knowledge from the previous model in the sequence, optimizing parameter efficiency at every step.

## 4. Evaluation and Comparative Performance

Comprehensive evaluation highlights smooth scaling behavior and strong performance relative to both the teacher and competing LLMs:

- **Pretrained Base vs. Teacher & Peers (5-shot):**

| Model               | MMLU | MATH (CoT 2-shot) |
|---------------------|------|-------------------|
| Mistral Small 24B   | 81.0 | 55.8              |
| Ministral 3 14B     | 79.4 | 67.6              |
| Ministral 3 8B      | 76.1 | 62.6              |
| Ministral 3 3B      | 70.7 | 60.1              |

- **Comparison to Qwen 3 & Gemma 3 (Base):**
  - Ministral 3 14B outperforms Qwen 3 14B on TriviaQA (74.9 vs 70.3) and MATH (67.6 vs 62.0).
  - Ministral 3 8B surpasses Gemma 3 12B on most tasks, underscoring parameter efficiency.

- **Instruction-finetuned Evaluation (Arena Hard / WildBench / MATHmaj@1 / MM-MTBench):**
  - Ministral 3 14B: 55.1 / 68.5 / 90.4 / 84.9
  - Qwen 3 14B: 42.7 / 65.1 / 87.0 / –
  - Gemma 3 12B: 43.6 / 63.2 / 85.4 / 67.0

- **Reasoning Benchmarks (pass@16, LiveCodeBench@5):**
  - AIME 2024: Ministral 3 14B 89.8, Qwen 3 14B 83.7
  - HMMT 2025: 14B 67.5, Qwen 3 14B 55.8
  - LiveCodeBench: 14B 64.6, Qwen 3 14B 59.3

Performance scales smoothly with parameter count; the 14B variant approaches the performance of significantly larger teacher models.

## 5. Vision Capabilities and Multimodal Evaluation

All Ministral 3 variants possess image understanding abilities via a frozen 410M-parameter Vision Transformer (ViT) encoder derived from Mistral Small 3.1 Pixtral. The original projection layer of Pixtral is discarded; a new lightweight projection layer is trained per student. Training leverages interleaved image+text data in both distillation and SFT.

- **Multimodal Benchmark Results (2-shot):**
  - MMMU: 14B achieves 59.9 (teacher 59.1)
  - MathVista: 14B reaches 43.6 (teacher 51.3, attributed to domain shift)

This setup enables efficient multimodal reasoning in a parameter-constrained regime.

## 6. Licensing, Deployment, and Practical Considerations

The entire Ministral 3 Series (three model sizes × three variants) is offered under the Apache 2.0 license, facilitating open-weight usage, modification, and commercial deployment. All checkpoints and configurations are accessible via [HuggingFace](https://huggingface.co/collections/mistralai/ministral-3), with supplementary technical documentation provided on the [Mistral AI website](https://mistral.ai/news/mistral-3).

The models are optimized for practical deployment on devices with limited computational resources and/or constrained GPUs, with native support for fp16 and int8 quantization. This parameter and FLOP efficiency broadens deployment use cases for sophisticated language and multimodal reasoning tasks within resource-limited environments [2601.08584].

Source: https://www.emergentmind.com/topics/ministral-3-series