---
title: 'MiniMax-01: Hybrid Long-Context Model'
url: https://www.emergentmind.com/topics/minimax-01
type: topic
---

# MiniMax-01: Hybrid Long-Context Model

MiniMax-01 is a foundation-model series comprising MiniMax-Text-01 and MiniMax-VL-01, built around a hybrid long-context architecture that combines lightning attention, periodic softmax attention, and Mixture of Experts (MoE). The series is presented as comparable to top-tier models while offering superior capabilities in processing longer contexts. Its central claim is that linear attention can serve as the default sequence-mixing mechanism at commercial scale, provided that it is integrated with occasional softmax-attention layers, MoE capacity scaling, and a systems stack designed for efficient computation-communication overlap. The reported text context window reaches up to 1 million tokens during training and extrapolates to 4 million tokens during inference, while the vision-language variant extends the same design to multimodal training and evaluation [2501.08313].

## 1. Model family and overall organization

The MiniMax-01 series contains two released model families: MiniMax-Text-01 for text and MiniMax-VL-01 for vision-language tasks. The text model is described as a hybrid long-context architecture using lightning attention, periodic softmax attention, and MoE, while the multimodal model is built by extending the text backbone with a visual encoder and projector. The paper states that experiments on both standard and in-house benchmarks show that these models match the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering a 20–32 times longer context window [2501.08313].

| Model | Core composition | Reported long-context property |
|---|---|---|
| MiniMax-Text-01 | Hybrid attention + MoE | 1M tokens in training, 4M in inference |
| MiniMax-VL-01 | MiniMax-Text-01 + ViT-L/14 + 2-layer MLP projector | Multimodal extension with dynamic-resolution vision input |

The organizing design principle is a deliberate departure from the standard “all softmax-attention Transformer” recipe. Lightning attention is the main attention mechanism for efficiency and long-context scaling; occasional softmax-attention blocks are retained to preserve retrieval and in-context learning quality; MoE is used to scale total parameter count without activating the full model on every token. This suggests that MiniMax-01 should be understood less as a single architectural novelty than as a co-designed model-systems stack in which sequence modeling, capacity scaling, and distributed execution are tightly coupled.

## 2. Hybrid attention architecture

MiniMax-Text-01 uses a Transformer-style stack with attention blocks, RMSNorm and residual connections, and MoE feed-forward blocks. The key architectural pattern is **1 softmax-attention block after every 7 lightning-attention blocks**, yielding a repeating **7× lightning + 1× softmax** structure across **80 layers**. The paper gives the exact text-model specification as **80 layers**, **64 attention heads**, **head dimension = 128**, **hidden size = 6144**, **RoPE on half of the attention head dimension**, **RoPE base frequency = 10,000**, **softmax layers use GQA** with **group size 8**, and **MoE with 32 experts**, **top-2 routing**, with **each expert FFN hidden size = 9216** [2501.08313].

The attention design is motivated by the observation that pure linear attention is efficient but weaker at retrieval and in-context learning. The paper therefore makes linear attention the default and retains a small fraction of softmax layers to recover those behaviors. It further provides a recurrent interpretation of lightning attention. Starting from
\[
\mathbf{O}=\mathrm{Norm}((\mathbf{Q}\mathbf{K}^{\top})\mathbf{V}),
\]
the computation is rewritten as
\[
\mathbf{O}=\mathrm{Norm}(\mathbf{Q}(\mathbf{K}^{\top}\mathbf{V})).
\]
For causal attention, the recurrence is
\[
\mathbf {kv}_0=\mathbf 0, \quad \mathbf {kv}_t=\mathbf {kv}_{t-1} + \mathbf k_t\mathbf v_t^\top, \quad \mathbf o_t^{\top} = \mathbf q_t^{\top} \mathbf {kv}_t.
\]
To make this parallelizable, the sequence is partitioned into blocks of size \(B\), with intra-block attention computed locally, inter-block contributions computed from the running \(\mathbf{KV}\) state, and the cache then updated. The resulting time complexity is reported as
\[
O(nd^2 + nBd),
\]
where \(n\) is sequence length, \(d\) is feature dimension, and \(B\) is block size.

The paper also offers a state-capacity comparison: softmax attention is interpreted as having recurrent state capacity \(O(d)\), whereas lightning attention has capacity \(O(d^2/h)\), where \(h\) is the number of heads. Because \(d > h\), the recurrent state of lightning attention is argued to have larger capacity. This is used to motivate the claim that lightning attention can support long-range retrieval and extrapolation, provided that periodic softmax layers remain in the stack.

## 3. MoE scaling and distributed execution

The final text model has **456B total parameters**, with **45.9B activated parameters per token**, using **32 experts**. MoE is defined for token \( \mathbf{x}_t \) by
\[
\mathbf{h}_t = \sum_{i=1}^E \text{Softmax}_i\left(\text{TopK}(\mathbf{x}_t \cdot \mathbf{W}_g)\right) \cdot \text{FFN}_i(\mathbf{x}_t),
\]
where \(E\) is the number of experts, \(\mathbf{W}_g\) is the router weight, \(\text{TopK}(\cdot)\) keeps only the top \(k\) expert scores, and \(\text{FFN}_i\) is expert \(i\). For MiniMax-01, \(E=32\) and \(k=2\) [2501.08313].

The paper identifies routing collapse and unbalanced expert utilization as major large-scale MoE failure modes. To mitigate them, it introduces a global router inspired by GShard, which synchronizes token counts across expert-parallel groups using an extra all-gather step before dispatch. The auxiliary load-balancing loss is
\[
L_{\text{aux}} = \alpha_{\text{aux}} \cdot \frac{1}{E}\sum_{i=1}^{E} f_i \cdot m_i,
\]
where \(f_i\) is the fraction of tokens assigned to expert \(i\), \(m_i\) is the average routing probability of expert \(i\), and \(\alpha_{\text{aux}}\) is the coefficient.

The systems implementation separates MoE-specific parallelism from the rest of the model. The paper introduces **EP** for expert parallelism, **ETP** for expert tensor parallelism, and **EDP** for expert data parallelism, with
\[
world\_size = size_{PP} \times size_{DP} \times size_{CP} \times size_{TP}
\]
and also
\[
world\_size = size_{PP} \times size_{EDP} \times size_{ETP} \times size_{EP}.
\]
This decoupling allows expert sharding to be tuned independently of the remaining model-parallel axes. The paper further reports **EP-ETP overlap** and token-grouping overlap, stating that these optimizations reduce the **pure communication overhead of the MoE component by 50%**.

For long-context execution, two additional systems components are emphasized. First, **varlen ring attention** supports packed variable-length samples for long-context softmax layers without excessive padding. Second, **LASP+** improves earlier LASP implementations by replacing the original serial dependency with local prefix-sum computation on each CP rank, an **AllGather** synchronization, and then global prefix-sum computation. The reported speed can reach about \(1/N_{pcn}\) of the original LASP algorithm’s serialized cost, where \(N_{pcn}\) is the number of parallel computing nodes. On inference, the paper reports batched kernel fusion, separated prefill and decoding execution, multi-level padding, strided batched matmul support, and WGMMA/TMA-oriented CUDA optimizations, claiming **>75% MFU on H20 end-to-end inference**. In **1,024,000-token** settings, **softmax attention accounts for ~95% of latency**, while lightning attention contributes **<12%**. This suggests that, within the reported serving regime, the retained softmax layers—not the linear-attention majority—dominate long-context latency.

## 4. Scaling laws, data pipeline, and long-context training

The paper states that MiniMax-01 did not select its final scale heuristically; instead, it used scaling-law experiments comparing softmax attention, lightning attention, and hybrid-lightning variants. The approximate parameter and FLOP formulas are given as follows [2501.08313].

| Architecture | Parameters | FLOPs |
|---|---:|---:|
| Softmax attention | \(12ld^2\) | \(72bnld^2\left(1+\frac{n}{6d}+\frac{5}{18d}\right)\) |
| Lightning attention | \(12ld^2 + \frac{2ld^2}{h}\) | \(72bnld^2\left(1+\frac{1}{2h}+\frac{5}{18d}\right)\) |
| Hybrid-lightning | \(12ld^2 + \frac{7ld^2}{4h}\) | \(72bnld^2\left(1+\frac{n}{48d}+\frac{7}{16h}+\frac{5}{18d}\right)\) |

The fitted loss-compute relations are reported as
\[
L(C)=3.7087C^{-0.0798}
\]
for softmax,
\[
L(C)=3.5391C^{-0.0768}
\]
for lightning, and
\[
L(C)=3.4797C^{-0.0763}
\]
for hybrid-lightning. The paper also formulates a loss model conditioned on expert count,
\[
L(P_{\text{act}},T\mid E) = d + aP_{\text{act}}^{\alpha} + bT^{\beta} + c(P_{\text{act}}T)^{\gamma},
\]
where \(P_{\text{act}}\) is activated parameter count, \(T\) is training tokens, and \(E\) is the number of experts. The reported final choice—**456B total parameters** and **45.9B activated parameters**—is tied to a hard constraint that total parameters remain under about **500B** so that inference is feasible on a single-node 8-GPU setup with 8-bit quantization for long contexts.

The data pipeline includes web, books, academic text, and code, with heavy cleaning and deduplication, reward-based document quality scoring, mixture balancing, and byte-level BPE with a **200K vocabulary**. The paper explicitly investigates data quality, formatting, repetition, and mixture composition, and introduces a repetition-aware experimental framework. Initial pretraining uses **sequence length = 8192**, with batch size scaled from **16M → 32M → 64M → 128M tokens**, and AdamW with \(\beta_1=0.9,\ \beta_2=0.95,\ \text{wd}=0.1\), with learning-rate warmup to \(2\times10^{-4}\), followed by a long constant phase and then decay.

Long-context training extends the model to **1,032,192 tokens** through a three-stage curriculum: **128K stage** with **300B tokens** and mixture **30% short / 70% medium / 0% long**; **512K stage** with **32B tokens** and mixture **35% short / 35% medium / 30% long**; and **1M stage** with **26B tokens** and mixture **30% short / 30% medium / 40% long**. The paper states that later post-training keeps **RoPE base frequency = 10M** and uses half-dimensional RoPE on softmax layers to support extrapolation. The stated outcome is stable long-context training up to 1M tokens and extrapolation to **4M-token inference**.

## 5. Text-model capabilities and long-context evaluation

On standard text benchmarks, MiniMax-Text-01 is reported as comparable to top closed-source models on many tasks. The paper highlights **MMLU: 88.5**, **MMLU-Pro: 75.7**, **C-SimpleQA: 67.4**, **IFEval: 89.1**, **Arena-Hard: 89.1**, **GPQA Diamond: 54.4**, **MATH: 77.4**, and **HumanEval: 86.9**. It states that the model especially performs well in Chinese factuality, instruction following, reasoning, and alignment-like user tasks, and that on some specific benchmarks—notably **C-SimpleQA**—it is ahead of GPT-4o and Claude-3.5-Sonnet in the paper’s table [2501.08313].

The principal empirical distinction, however, is long-context behavior. On RULER, the paper reports that MiniMax-Text-01 remains strong from 4k up to 1M context length, with example values of **0.943 at 64k**, **0.947 at 128k**, and **0.910 at 1M**. The paper’s qualitative claim is that degradation is slower at very long context than for GPT-4o, Claude-3.5-Sonnet, and Gemini baselines. Long-context evaluation is divided into retrieval, reasoning, and in-context learning. Retrieval is measured with Needle-In-A-Haystack and the harder MR-NIAH benchmark; understanding and reasoning are measured with RULER and LongBench-v2; and long in-context learning is evaluated with MTOB, framed as translation from a grammar book and examples supplied in context.

For **LongBench-v2**, the reported numbers are **56.5 w/ CoT** and **52.9 w/o CoT**, which the paper states beat GPT-4o and Claude-3.5-Sonnet in the table shown. On MTOB, the model is described as learning from long context and performing competitively on half-book and full-book settings, with especially strong delta gains. The paper also reports successful retrieval in a **4M-token NIAH pressure test**. The architectural interpretation offered is that long-context performance is not attributed to a single trick: it is presented as the consequence of hybrid attention, blockwise lightning attention, MoE capacity scaling, customized parallelism, a curriculum-based long-context training schedule, and RoPE choices intended to support extrapolation. This suggests that MiniMax-Text-01’s long-context profile is inseparable from its training and systems recipe.

## 6. Vision-language extension and broader significance

MiniMax-VL-01 extends the text backbone with a **ViT-L/14 visual encoder** containing **303M parameters** and a **2-layer MLP projector**. The vision pipeline uses dynamic resolution, with images resized on a grid from **336×336 up to 2016×2016**, plus a **336×336 thumbnail**, after which patches are encoded and concatenated [2501.08313].

The training description includes **694M unique image-caption pairs** for vision pretraining, refined captions for **180M images**, **100M description images**, and later alignment data and DPO preferences. The multimodal procedure is divided into four stages: **modality alignment** with **80B tokens**; **vision understanding** with **420B multimodal tokens**, mixed with text post-training data at **20:1**; **user-experience enhancement** with **44.8B multimodal tokens**; and **preference optimization** with DPO on **40K image-text pairs**. The abstract separately states that MiniMax-VL-01 is built through continued training with **512 billion vision-language tokens**.

Reported multimodal results include **MMMU: 68.5**, **MMMU-Pro: 52.7**, **ChartQA: 91.7**, **DocVQA: 96.4**, **OCRBench: 865**, **MMLongBench-Doc: 32.5**, **MEGA-Bench: 47.4**, and **In-house benchmark: 56.6**. The paper highlights strong OCR and document QA, good general VLM performance, weaker performance on very hard math and reasoning tasks than the best specialized systems, and competitive user-experience performance, though not universally best.

The broader significance claimed for MiniMax-01 is that top-tier foundation-model performance need not be tied to pure softmax attention. The paper argues instead that linear attention can be made practical at scale if implemented carefully, while also insisting that softmax attention should not be removed entirely because retrieval matters. In that sense, MiniMax-01 is positioned as a hybridization thesis rather than a pure linear-attention thesis: pure linear models are described as insufficient for LLM quality, but a stack that combines lightning attention, periodic softmax attention, MoE, and custom distributed execution is presented as capable of million-token training contexts and multi-million-token inference while remaining competitive on mainstream text and multimodal benchmarks.

Source: https://www.emergentmind.com/topics/minimax-01