MiniMax-01: Hybrid Long-Context Model

• MiniMax-01 is a foundation model series featuring a hybrid long-context design that combines lightning and periodic softmax attention to enable efficient processing of up to 4M tokens.
• The model leverages Mixture of Experts and customized parallelism (EP, ETP, EDP) to scale parameters and reduce communication overhead while maintaining competitive performance.
• Its vision-language variant integrates a ViT-L/14 encoder with dynamic resolution, delivering robust multimodal capabilities such as enhanced OCR and document QA.

MiniMax-01 is a foundation-model series comprising MiniMax-Text-01 and MiniMax-VL-01, built around a hybrid long-context architecture that combines lightning attention, periodic softmax attention, and Mixture of Experts (MoE). The series is presented as comparable to top-tier models while offering superior capabilities in processing longer contexts. Its central claim is that linear attention can serve as the default sequence-mixing mechanism at commercial scale, provided that it is integrated with occasional softmax-attention layers, MoE capacity scaling, and a systems stack designed for efficient computation-communication overlap. The reported text context window reaches up to 1 million tokens during training and extrapolates to 4 million tokens during inference, while the vision-language variant extends the same design to multimodal training and evaluation (MiniMax et al., 14 Jan 2025).

1. Model family and overall organization

The MiniMax-01 series contains two released model families: MiniMax-Text-01 for text and MiniMax-VL-01 for vision-language tasks. The text model is described as a hybrid long-context architecture using lightning attention, periodic softmax attention, and MoE, while the multimodal model is built by extending the text backbone with a visual encoder and projector. The paper states that experiments on both standard and in-house benchmarks show that these models match the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering a 20–32 times longer context window (MiniMax et al., 14 Jan 2025).

Model	Core composition	Reported long-context property
MiniMax-Text-01	Hybrid attention + MoE	1M tokens in training, 4M in inference
MiniMax-VL-01	MiniMax-Text-01 + ViT-L/14 + 2-layer MLP projector	Multimodal extension with dynamic-resolution vision input

The organizing design principle is a deliberate departure from the standard “all softmax-attention Transformer” recipe. Lightning attention is the main attention mechanism for efficiency and long-context scaling; occasional softmax-attention blocks are retained to preserve retrieval and in-context learning quality; MoE is used to scale total parameter count without activating the full model on every token. This suggests that MiniMax-01 should be understood less as a single architectural novelty than as a co-designed model-systems stack in which sequence modeling, capacity scaling, and distributed execution are tightly coupled.

2. Hybrid attention architecture

MiniMax-Text-01 uses a Transformer-style stack with attention blocks, RMSNorm and residual connections, and MoE feed-forward blocks. The key architectural pattern is 1 softmax-attention block after every 7 lightning-attention blocks, yielding a repeating 7× lightning + 1× softmax structure across 80 layers. The paper gives the exact text-model specification as 80 layers, 64 attention heads, head dimension = 128, hidden size = 6144, RoPE on half of the attention head dimension, RoPE base frequency = 10,000, softmax layers use GQA with group size 8, and MoE with 32 experts, top-2 routing, with each expert FFN hidden size = 9216 (MiniMax et al., 14 Jan 2025).

The attention design is motivated by the observation that pure linear attention is efficient but weaker at retrieval and in-context learning. The paper therefore makes linear attention the default and retains a small fraction of softmax layers to recover those behaviors. It further provides a recurrent interpretation of lightning attention. Starting from

$$\mathbf{O}=\mathrm{Norm}((\mathbf{Q}\mathbf{K}^{\top})\mathbf{V}),$$

the computation is rewritten as

$$\mathbf{O}=\mathrm{Norm}(\mathbf{Q}(\mathbf{K}^{\top}\mathbf{V})).$$

For causal attention, the recurrence is

$$\mathbf {kv}_0=\mathbf 0, \quad \mathbf {kv}_t=\mathbf {kv}_{t-1} + \mathbf k_t\mathbf v_t^\top, \quad \mathbf o_t^{\top} = \mathbf q_t^{\top} \mathbf {kv}_t.$$

To make this parallelizable, the sequence is partitioned into blocks of size $B$, with intra-block attention computed locally, inter-block contributions computed from the running $\mathbf{KV}$ state, and the cache then updated. The resulting time complexity is reported as

$O(nd^2 + nBd),$

where $n$ is sequence length, $d$ is feature dimension, and $B$ is block size.

The paper also offers a state-capacity comparison: softmax attention is interpreted as having recurrent state capacity $O(d)$, whereas lightning attention has capacity $$\mathbf{O}=\mathrm{Norm}(\mathbf{Q}(\mathbf{K}^{\top}\mathbf{V})).$$0, where $$\mathbf{O}=\mathrm{Norm}(\mathbf{Q}(\mathbf{K}^{\top}\mathbf{V})).$$1 is the number of heads. Because $$\mathbf{O}=\mathrm{Norm}(\mathbf{Q}(\mathbf{K}^{\top}\mathbf{V})).$$2, the recurrent state of lightning attention is argued to have larger capacity. This is used to motivate the claim that lightning attention can support long-range retrieval and extrapolation, provided that periodic softmax layers remain in the stack.

3. MoE scaling and distributed execution

The final text model has 456B total parameters, with 45.9B activated parameters per token, using 32 experts. MoE is defined for token $$\mathbf{O}=\mathrm{Norm}(\mathbf{Q}(\mathbf{K}^{\top}\mathbf{V})).$$3 by

$$\mathbf{O}=\mathrm{Norm}(\mathbf{Q}(\mathbf{K}^{\top}\mathbf{V})).$$4

where $$\mathbf{O}=\mathrm{Norm}(\mathbf{Q}(\mathbf{K}^{\top}\mathbf{V})).$$5 is the number of experts, $$\mathbf{O}=\mathrm{Norm}(\mathbf{Q}(\mathbf{K}^{\top}\mathbf{V})).$$6 is the router weight, $$\mathbf{O}=\mathrm{Norm}(\mathbf{Q}(\mathbf{K}^{\top}\mathbf{V})).$$7 keeps only the top $$\mathbf{O}=\mathrm{Norm}(\mathbf{Q}(\mathbf{K}^{\top}\mathbf{V})).$$8 expert scores, and $$\mathbf{O}=\mathrm{Norm}(\mathbf{Q}(\mathbf{K}^{\top}\mathbf{V})).$$9 is expert $$\mathbf {kv}_0=\mathbf 0, \quad \mathbf {kv}_t=\mathbf {kv}_{t-1} + \mathbf k_t\mathbf v_t^\top, \quad \mathbf o_t^{\top} = \mathbf q_t^{\top} \mathbf {kv}_t.$$0. For MiniMax-01, $$\mathbf {kv}_0=\mathbf 0, \quad \mathbf {kv}_t=\mathbf {kv}_{t-1} + \mathbf k_t\mathbf v_t^\top, \quad \mathbf o_t^{\top} = \mathbf q_t^{\top} \mathbf {kv}_t.$$1 and $$\mathbf {kv}_0=\mathbf 0, \quad \mathbf {kv}_t=\mathbf {kv}_{t-1} + \mathbf k_t\mathbf v_t^\top, \quad \mathbf o_t^{\top} = \mathbf q_t^{\top} \mathbf {kv}_t.$$2 (MiniMax et al., 14 Jan 2025).

The paper identifies routing collapse and unbalanced expert utilization as major large-scale MoE failure modes. To mitigate them, it introduces a global router inspired by GShard, which synchronizes token counts across expert-parallel groups using an extra all-gather step before dispatch. The auxiliary load-balancing loss is

$$\mathbf {kv}_0=\mathbf 0, \quad \mathbf {kv}_t=\mathbf {kv}_{t-1} + \mathbf k_t\mathbf v_t^\top, \quad \mathbf o_t^{\top} = \mathbf q_t^{\top} \mathbf {kv}_t.$$3

where $$\mathbf {kv}_0=\mathbf 0, \quad \mathbf {kv}_t=\mathbf {kv}_{t-1} + \mathbf k_t\mathbf v_t^\top, \quad \mathbf o_t^{\top} = \mathbf q_t^{\top} \mathbf {kv}_t.$$4 is the fraction of tokens assigned to expert $$\mathbf {kv}_0=\mathbf 0, \quad \mathbf {kv}_t=\mathbf {kv}_{t-1} + \mathbf k_t\mathbf v_t^\top, \quad \mathbf o_t^{\top} = \mathbf q_t^{\top} \mathbf {kv}_t.$$5, $$\mathbf {kv}_0=\mathbf 0, \quad \mathbf {kv}_t=\mathbf {kv}_{t-1} + \mathbf k_t\mathbf v_t^\top, \quad \mathbf o_t^{\top} = \mathbf q_t^{\top} \mathbf {kv}_t.$$6 is the average routing probability of expert $$\mathbf {kv}_0=\mathbf 0, \quad \mathbf {kv}_t=\mathbf {kv}_{t-1} + \mathbf k_t\mathbf v_t^\top, \quad \mathbf o_t^{\top} = \mathbf q_t^{\top} \mathbf {kv}_t.$$7, and $$\mathbf {kv}_0=\mathbf 0, \quad \mathbf {kv}_t=\mathbf {kv}_{t-1} + \mathbf k_t\mathbf v_t^\top, \quad \mathbf o_t^{\top} = \mathbf q_t^{\top} \mathbf {kv}_t.$$8 is the coefficient.

The systems implementation separates MoE-specific parallelism from the rest of the model. The paper introduces EP for expert parallelism, ETP for expert tensor parallelism, and EDP for expert data parallelism, with

$$\mathbf {kv}_0=\mathbf 0, \quad \mathbf {kv}_t=\mathbf {kv}_{t-1} + \mathbf k_t\mathbf v_t^\top, \quad \mathbf o_t^{\top} = \mathbf q_t^{\top} \mathbf {kv}_t.$$9

and also

$B$0

This decoupling allows expert sharding to be tuned independently of the remaining model-parallel axes. The paper further reports EP-ETP overlap and token-grouping overlap, stating that these optimizations reduce the pure communication overhead of the MoE component by 50%.

For long-context execution, two additional systems components are emphasized. First, varlen ring attention supports packed variable-length samples for long-context softmax layers without excessive padding. Second, LASP+ improves earlier LASP implementations by replacing the original serial dependency with local prefix-sum computation on each CP rank, an AllGather synchronization, and then global prefix-sum computation. The reported speed can reach about $B$1 of the original LASP algorithm’s serialized cost, where $B$2 is the number of parallel computing nodes. On inference, the paper reports batched kernel fusion, separated prefill and decoding execution, multi-level padding, strided batched matmul support, and WGMMA/TMA-oriented CUDA optimizations, claiming >75% MFU on H20 end-to-end inference. In 1,024,000-token settings, softmax attention accounts for ~95% of latency, while lightning attention contributes <12%. This suggests that, within the reported serving regime, the retained softmax layers—not the linear-attention majority—dominate long-context latency.

4. Scaling laws, data pipeline, and long-context training

The paper states that MiniMax-01 did not select its final scale heuristically; instead, it used scaling-law experiments comparing softmax attention, lightning attention, and hybrid-lightning variants. The approximate parameter and FLOP formulas are given as follows (MiniMax et al., 14 Jan 2025).

Architecture	Parameters	FLOPs
Softmax attention	$B$3	$B$4
Lightning attention	$B$5	$B$6
Hybrid-lightning	$B$7	$B$8

The fitted loss-compute relations are reported as

$B$9

for softmax,

$\mathbf{KV}$0

for lightning, and

$\mathbf{KV}$1

for hybrid-lightning. The paper also formulates a loss model conditioned on expert count,

$\mathbf{KV}$2

where $\mathbf{KV}$3 is activated parameter count, $\mathbf{KV}$4 is training tokens, and $\mathbf{KV}$5 is the number of experts. The reported final choice—456B total parameters and 45.9B activated parameters—is tied to a hard constraint that total parameters remain under about 500B so that inference is feasible on a single-node 8-GPU setup with 8-bit quantization for long contexts.

The data pipeline includes web, books, academic text, and code, with heavy cleaning and deduplication, reward-based document quality scoring, mixture balancing, and byte-level BPE with a 200K vocabulary. The paper explicitly investigates data quality, formatting, repetition, and mixture composition, and introduces a repetition-aware experimental framework. Initial pretraining uses sequence length = 8192, with batch size scaled from 16M → 32M → 64M → 128M tokens, and AdamW with $\mathbf{KV}$6, with learning-rate warmup to $\mathbf{KV}$7, followed by a long constant phase and then decay.

Long-context training extends the model to 1,032,192 tokens through a three-stage curriculum: 128K stage with 300B tokens and mixture 30% short / 70% medium / 0% long; 512K stage with 32B tokens and mixture 35% short / 35% medium / 30% long; and 1M stage with 26B tokens and mixture 30% short / 30% medium / 40% long. The paper states that later post-training keeps RoPE base frequency = 10M and uses half-dimensional RoPE on softmax layers to support extrapolation. The stated outcome is stable long-context training up to 1M tokens and extrapolation to 4M-token inference.

5. Text-model capabilities and long-context evaluation

On standard text benchmarks, MiniMax-Text-01 is reported as comparable to top closed-source models on many tasks. The paper highlights MMLU: 88.5, MMLU-Pro: 75.7, C-SimpleQA: 67.4, IFEval: 89.1, Arena-Hard: 89.1, GPQA Diamond: 54.4, MATH: 77.4, and HumanEval: 86.9. It states that the model especially performs well in Chinese factuality, instruction following, reasoning, and alignment-like user tasks, and that on some specific benchmarks—notably C-SimpleQA—it is ahead of GPT-4o and Claude-3.5-Sonnet in the paper’s table (MiniMax et al., 14 Jan 2025).

The principal empirical distinction, however, is long-context behavior. On RULER, the paper reports that MiniMax-Text-01 remains strong from 4k up to 1M context length, with example values of 0.943 at 64k, 0.947 at 128k, and 0.910 at 1M. The paper’s qualitative claim is that degradation is slower at very long context than for GPT-4o, Claude-3.5-Sonnet, and Gemini baselines. Long-context evaluation is divided into retrieval, reasoning, and in-context learning. Retrieval is measured with Needle-In-A-Haystack and the harder MR-NIAH benchmark; understanding and reasoning are measured with RULER and LongBench-v2; and long in-context learning is evaluated with MTOB, framed as translation from a grammar book and examples supplied in context.

For LongBench-v2, the reported numbers are 56.5 w/ CoT and 52.9 w/o CoT, which the paper states beat GPT-4o and Claude-3.5-Sonnet in the table shown. On MTOB, the model is described as learning from long context and performing competitively on half-book and full-book settings, with especially strong delta gains. The paper also reports successful retrieval in a 4M-token NIAH pressure test. The architectural interpretation offered is that long-context performance is not attributed to a single trick: it is presented as the consequence of hybrid attention, blockwise lightning attention, MoE capacity scaling, customized parallelism, a curriculum-based long-context training schedule, and RoPE choices intended to support extrapolation. This suggests that MiniMax-Text-01’s long-context profile is inseparable from its training and systems recipe.

6. Vision-language extension and broader significance

MiniMax-VL-01 extends the text backbone with a ViT-L/14 visual encoder containing 303M parameters and a 2-layer MLP projector. The vision pipeline uses dynamic resolution, with images resized on a grid from 336×336 up to 2016×2016, plus a 336×336 thumbnail, after which patches are encoded and concatenated (MiniMax et al., 14 Jan 2025).

The training description includes 694M unique image-caption pairs for vision pretraining, refined captions for 180M images, 100M description images, and later alignment data and DPO preferences. The multimodal procedure is divided into four stages: modality alignment with 80B tokens; vision understanding with 420B multimodal tokens, mixed with text post-training data at 20:1; user-experience enhancement with 44.8B multimodal tokens; and preference optimization with DPO on 40K image-text pairs. The abstract separately states that MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens.

Reported multimodal results include MMMU: 68.5, MMMU-Pro: 52.7, ChartQA: 91.7, DocVQA: 96.4, OCRBench: 865, MMLongBench-Doc: 32.5, MEGA-Bench: 47.4, and In-house benchmark: 56.6. The paper highlights strong OCR and document QA, good general VLM performance, weaker performance on very hard math and reasoning tasks than the best specialized systems, and competitive user-experience performance, though not universally best.

The broader significance claimed for MiniMax-01 is that top-tier foundation-model performance need not be tied to pure softmax attention. The paper argues instead that linear attention can be made practical at scale if implemented carefully, while also insisting that softmax attention should not be removed entirely because retrieval matters. In that sense, MiniMax-01 is positioned as a hybridization thesis rather than a pure linear-attention thesis: pure linear models are described as insufficient for LLM quality, but a stack that combines lightning attention, periodic softmax attention, MoE, and custom distributed execution is presented as capable of million-token training contexts and multi-million-token inference while remaining competitive on mainstream text and multimodal benchmarks.