MInference 1.0: Efficient Sparse Attention

• MInference 1.0 is a dynamic sparse attention framework that targets the prefill stage in long-context LLMs using structured patterns like A-shape, Vertical-Slash, and Block-Sparse.
• It employs an offline kernel-aware search for pattern assignment and an online index construction to achieve up to 10x speedup while maintaining dense-level accuracy.
• The framework reduces FLOPs and memory overhead, effectively mitigating the quadratic complexity bottleneck in long-context autoregressive Transformer decoders.

Searching arXiv for the specified paper and closely related work to ground the article. MInference 1.0, short for Million-tokens Inference, is a dynamic sparse attention framework for accelerating the pre-filling stage of long-context LLM inference. It is designed for contexts from 128K up to 10M tokens and targets the dominant bottleneck in long-prompt inference: dense self-attention with quadratic complexity in sequence length. The method operates as a training-free, inference-time plugin for existing autoregressive Transformer decoders, assigning each attention head to one of three recurring sparse patterns—A-shape, Vertical-Slash, or Block-Sparse—through an offline kernel-aware search, and then building input-dependent sparse indices during inference when required. In the reported evaluation, it reduces prefill latency by up to 10x on a single NVIDIA A100 while maintaining dense-level accuracy across long-context benchmarks and language-modeling measurements (Jiang et al., 2024).

1. Problem setting and design objective

For long-context LLMs, the dominant inference bottleneck is pre-filling, or prefill: computing attention over the entire prompt before any new token is generated. In a standard Transformer attention layer, per head,

$\mathbf{A} = \text{Softmax}\!\left(\frac{1}{\sqrt{d}\mathbf{Q}\mathbf{K}^\top + \text{mask}\right),$

with complexity $O(T^2 d)$ for sequence length $T$ and head dimension $d$. For sufficiently long contexts, this quadratic term dominates runtime.

The reported measurements quantify the scale of the bottleneck. For LLaMA‑3‑8B on a single NVIDIA A100 (80 GB), pre-filling takes approximately 6 minutes for a 300K-token prompt and approximately 30 minutes for a 1M-token prompt. During prefill, the overhead of self-attention computation exceeds 90% of the total pre-filling latency. By contrast, decoding processes one new token attending to previous tokens with per-token cost $O(T d)$, and prior work has already optimized decoding extensively through KV-cache compression and speculative decoding (Jiang et al., 2024).

MInference explicitly targets prefill rather than decoding. Its stated goal is to reduce FLOPs and latency in prefill by exploiting dynamic sparsity in attention while preserving model quality. The method therefore addresses a gap left by four broad families of prior approaches. Dense kernels such as FlashAttention remain $O(T^2 d)$ despite substantial systems optimization. Static sparse architectures such as sliding-window, dilated, clustered, or linear-attention variants typically require pre-training or retraining and are therefore not drop-in replacements for deployed LLMs. Earlier dynamic sparse attention methods incur substantial online overhead in estimating sparse patterns, which becomes counterproductive for 100K–1M-token regimes. Decoding-oriented KV-compression methods do not address the heavy attention burden during prefill.

A common misconception is that long-context inference is already solved by decoding optimizations alone. The reported measurements indicate otherwise: for million-token prompts, time-to-first-token is dominated by prefill, and dense prefill remains extremely slow even when decoding is optimized.

2. Core mechanism: dynamic sparse attention by head

MInference is a dynamic sparse attention framework targeted exclusively at the prefill stage. Its starting point is an empirical observation: long-context LLM attention is highly sparse and patterned, but that sparsity is also input-dependent. At 128K context, using only the top 4K columns of the attention matrix recovers 96.8% of total attention mass. However, top‑4K columns discovered on one 128K prompt recover only approximately 83.7% of attention on another prompt of the same length. This combination of high sparsity and poor cross-prompt transfer motivates a design that is both structured and dynamic (Jiang et al., 2024).

The framework identifies three recurring two-dimensional attention patterns on a per-head basis.

Pattern	Characterization	Dynamic behavior
A-shape	Global tokens at the beginning plus a local diagonal window	Static structured pattern
Vertical-Slash	Specific columns plus slash-like diagonal structures	Input-dependent indices
Block-Sparse	Scattered but spatially clustered dense blocks	Input-dependent block selection

The sparse attention formulation is written as

$\mathbf{A}(\mathbf{M}) = \text{Softmax}\!\left( \frac{1}{\sqrt{d}\mathbf{Q}\mathbf{K}^\top - c(1-\mathbf{M}) \right),$

where $\mathbf{M}\in\{0,1\}^{T\times T}$ is the sparse mask and $c$ is a large constant such as $10^5$. The optimization objective balances approximation quality against total latency: $O(T^2 d)$0

The A-shape pattern corresponds to a StreamingLLM-style global-plus-local structure. Each query attends to a fixed set of global tokens at the beginning of the sequence and a fixed-size local window of recent tokens. In the reported reference configuration, this uses 1,024 global tokens and 4,096 local-window tokens. Its advantage is low overhead and straightforward mapping to block-sparse FlashAttention-like kernels; its limitation is that it cannot adapt to arbitrary long-range retrievals or highly scattered dependencies.

The Vertical-Slash pattern captures heads for which attention concentrates on a small set of specific columns and a set of slash-like diagonal structures. The positions of these structures vary by prompt. This pattern is intended to preserve retrieval and multi-hop behavior that fixed local-plus-global masks often miss.

The Block-Sparse pattern captures heads whose nonzeros are scattered but spatially clustered. The paper reports that the distance to the top‑10 nearest non-zero neighbor on a 128K prompt is mostly around 5, suggesting strong local clustering in the attention matrix. MInference approximates such heads with a small set of dense blocks, for example 64×64 blocks, which can be handled efficiently by GPU block-sparse kernels.

The method’s central claim is not merely that attention is sparse, but that sparsity must be structured in a kernel-aware way. At equal kernel FLOPs, the A-shape, Vertical-Slash, and Block-Sparse patterns achieve substantially higher attention recall than Top‑K token selection, especially on Block-Sparse heads. This suggests that the design is optimized simultaneously for approximation quality and GPU execution efficiency rather than for sparsity in the abstract.

3. Offline pattern assignment and online index construction

MInference does not learn sparse patterns during pre-training or fine-tuning. Instead, it performs an offline kernel-aware sparse pattern search per head. The inputs are representative $O(T^2 d)$1, $O(T^2 d)$2, and $O(T^2 d)$3 samples for a head, pattern families $O(T^2 d)$4, candidate hyperparameters $O(T^2 d)$5, and a target FLOPs budget expressed in kernel terms. For each candidate configuration, the procedure computes actual kernel FLOPs,

$O(T^2 d)$6

adjusts the configuration with a pattern-specific ChangeSpace operation until $O(T^2 d)$7 is satisfied within tolerance, and forms a kernel-aware search space. It then computes the dense reference output

$O(T^2 d)$8

evaluates each sparse candidate

$O(T^2 d)$9

and chooses the pattern and hyperparameters minimizing deviation from the dense result.

This search is run once per head. The reported search time is approximately 15 minutes on one A100 using a single 30K-token synthetic KV-retrieval example. The resulting head assignments reportedly generalize across lengths and domains, and the same pattern configuration is reused for LLaMA‑3‑8B‑262K and LLaMA‑3‑8B‑1M (Jiang et al., 2024).

The method distinguishes sharply between what is fixed offline and what remains dynamic at inference time. Offline, each head receives a pattern type, pattern hyperparameters, and, for A-shape heads, a static mask. Online, the system builds sparse indices only for the heads whose patterns require prompt-specific adaptation.

For Vertical-Slash heads, the online approximation uses only the last $T$0 queries: $T$1 Vertical indices are selected by column sums,

$T$2

and slash indices by slash-direction sums,

$T$3

These are converted into sparse block and column indices for the full attention computation.

For Block-Sparse heads, the online approximation uses mean pooling with block size 64,

$T$4

then computes block-level attention,

$T$5

and selects the top $T$6 blocks,

$T$7

A-shape heads require no dynamic index construction. This distinction is central to the method’s latency profile: pattern selection is offline, while sparse instantiation is dynamic only where input dependence materially affects attention quality.

4. GPU kernels and systems integration

MInference is implemented as a GPU-efficient sparse attention system built on Triton, PIT, and FlashAttention. The implementation emphasis is not only on reducing arithmetic but also on matching the sparse structure to kernels whose memory access and streaming-softmax behavior remain efficient at long sequence lengths (Jiang et al., 2024).

The Vertical-Slash kernel is a hybrid block-sparse plus PIT FlashAttention kernel. For each block row, it loads a $T$8 chunk of $T$9, maintains FlashAttention-style streaming softmax statistics $d$0 and $d$1, processes slash segments through block-sparse operations, and processes vertical segments by grouping column indices into groups of size $d$2 and loading them into dense blocks via PIT. The final normalization step is

$d$3

The corresponding GPU-side index construction performs point–range merge over slash ranges and vertical points; its complexity per row is $d$4, hence linear in the number of VS lines.

The Block-Sparse kernel is simpler. It is based on Triton FlashAttention with a list of active blocks, and each thread block iterates through those active blocks while applying streaming softmax over them. The paper gives a rough speedup factor versus dense FlashAttention as

$d$5

where $d$6 is the block size and $d$7 is the number of retained blocks.

The A-shape kernel is effectively a StreamingLLM-plus-FlashAttention implementation in Triton with sparse block coverage and no dynamic index-building cost.

The reported kernel microbenchmarks show that at 10K tokens all methods are below 1 ms per attention invocation, and index-building overhead can account for up to approximately 30% of total cost, so net speedup is limited. At 1M tokens, the reported per-attention latency is approximately 164 ms for A-shape, about 13× speedup versus dense FlashAttention for Vertical-Slash heads, and about 30× speedup for Block-Sparse heads. Index-building overhead is approximately 5–15% for Vertical-Slash and approximately 25% for Block-Sparse. Memory overhead for indices is reported as at most 160 MB for LLaMA‑3‑8B at 1M tokens. Effective sparsity exceeds 90% for contexts above 200K and exceeds 95% for contexts above 500K.

The integration path is deliberately conservative. MInference assumes a standard autoregressive Transformer decoder with causal masking, standard Q/K/V projections, and ordinary KV-cache layout. It is compatible with positional encoding schemes including RoPE, NTK-aware interpolation, and LongRoPE. The attention module is modified only in the prefill path: each head dispatches to the A-shape kernel, the Vertical-Slash index-builder plus kernel, or the Block-Sparse index-builder plus kernel, while the decoding path can remain dense or be optimized separately with decoding-time KV compression.

For 1M-token prompts on A100‑80G, the implementation notes three single-GPU measures: splitting attention by head and MLP by sequence to avoid OOM, removing explicit attention masks by implementing causal logic in kernels, and computing the LM head only for the final token in prefill.

5. Empirical evaluation

The evaluation uses a single NVIDIA A100 (80 GB) in bfloat16, with kernels implemented in Triton and a custom PyTorch attention implementation adapted from the HuggingFace LLaMA model. The evaluated models are LLaMA‑3‑8B‑Instruct‑262K, LLaMA‑3‑8B‑Instruct‑1M, Yi‑9B‑200K, GLM‑4‑9B‑1M, Phi‑3‑Mini‑128K, and Qwen2‑7B‑128K. All experiments use greedy decoding for determinism (Jiang et al., 2024).

The benchmark suite spans four long-context settings. InfiniteBench comprises 10 tasks and approximately 3,992 examples with average length approximately 214K tokens, including summarization, QA, code debugging, mathematical min/max search, and retrieval tasks such as PassKey, Number, and KV retrieval. RULER is a synthetic benchmark with 13 tasks across retrieval, multi-hop tracing, aggregation, and QA, evaluated from 4K to 128K context lengths; “effective context” is defined as the maximum length at which performance remains at least 85%. Needle In A Haystack scales retrieval to contexts up to 1M tokens across 750 examples. PG‑19 evaluates perplexity on 1,000 samples of length at least 100K.

For prefill latency on LLaMA‑3‑8B‑Instruct‑1M, the reported speedups are 1.8× at 100K tokens, 4.1× at 300K, 6.8× at 500K, and 10× at 1M. The headline latency reduction is from 30 minutes to 3 minutes for a 1M-token prompt on a single A100. With tensor parallel plus context parallel on 8× A100, the estimate is approximately 40 seconds for 1M-token prefill.

On InfiniteBench, MInference matches or slightly exceeds dense full attention. For LLaMA‑3‑8B‑262K, the dense baseline average is 38.2 and MInference reaches 38.8, compared with 23.8 for StreamingLLM, 24.2 for StreamingLLM with dilated attention, 13.3 for StreamingLLM with strided attention, 34.8 for InfLLM, and 31.9 for “Ours w/ static indices.” On Yi‑9B‑200K, the dense average is 37.5 and MInference is 37.7; on GLM‑4‑9B‑1M, the dense average is 46.7 and MInference is 47.0. Retrieval-heavy tasks are especially informative: MInference retains strong PassKey and Number performance and substantially exceeds static or windowed baselines on KV retrieval.

On RULER, MInference preserves or improves effective context. For LLaMA‑3‑8B‑262K, dense attention has average 84.4 and effective context approximately 16K, whereas MInference scores 97.7 at 4K, 91.2 at 8K, 88.5 at 16K, 85.0 at 32K, 82.3 at 64K, and 77.6 at 128K, for average 87.0 and effective context 32K. For GLM‑4‑9B‑1M, dense attention has average 88.0 and effective context 64K, while MInference scores 94.6, 93.1, 91.0, 89.6, 85.5, and 84.0 across 4K through 128K, giving average 89.6 and effective context 64K.

On Needle In A Haystack, MInference maintains near-perfect retrieval across depths up to 1M tokens for LLaMA‑3‑8B‑1M, and its curves for GLM‑4‑9B‑1M, Yi‑9B‑200K, Phi‑3‑Mini‑128K, and Qwen2‑7B‑128K nearly overlap the dense baseline. On PG‑19, for both LLaMA‑3‑8B‑262K and Yi‑9B‑200K, the perplexity difference from dense attention is within at most 0.2 points at 100K tokens, while StreamingLLM is 0.25–0.75 worse than MInference.

These results support a precise interpretation: MInference is not merely a low-latency approximation, but a sparse prefill mechanism whose approximation error remains small enough that downstream task accuracy and language-modeling quality stay at dense-attention levels in the reported regime.

6. Ablations, comparisons, and limitations

The ablation studies indicate that head-level pattern heterogeneity is essential. Restricting all heads to A-shape reduces the method to StreamingLLM behavior: adequate on local tasks but weak on retrieval beyond the windowed region. Restricting all heads to Block-Sparse drops InfiniteBench average on LLaMA‑3‑8B‑262K from 38.8 to 18.7, with PassKey falling from 100 to 59.5. Restricting all heads to Vertical-Slash gives 37.1, which is close but still below the full mixture, and underperforms on highly dynamic tasks such as KV retrieval. Within Vertical-Slash itself, using only vertical lines yields average 18.6, while using only slash lines yields 35.9; both are worse than the full combination (Jiang et al., 2024).

Dynamic indexing is also shown to be critical. The “Ours w/ static” variant fixes VS and BS indices instead of constructing them per prompt. On InfiniteBench for LLaMA‑3‑8B‑262K, the average falls from 38.8 to 31.9. On KV retrieval, performance nearly collapses: 0.2 instead of 12.8 for LLaMA and 1.0 instead of 27.6 for Yi. This directly refutes the misconception that any structured sparse mask suffices so long as it is chosen well offline. The reported evidence indicates that long-context attention is sparse but not static.

The baseline comparisons further clarify the design space. Dense full attention via FlashAttention provides the quality reference but remains too slow at million-token scale. StreamingLLM and its dilated or strided variants are competitive only where local recency and a small set of global tokens are enough; they fail when relevant information lies outside those regions. InfLLM improves on those baselines in some settings but remains consistently below MInference in both accuracy and effective context. The static-index ablation shows that MInference’s gains arise not simply from pattern choice but from dynamic pattern instantiation.

The stated limitations are equally specific. For short contexts below approximately 10K tokens, attention is already inexpensive, while index-building time remains roughly constant; overall latency approaches dense FlashAttention, and speedup is small. If sparsity is pushed further by reducing the FLOPs target, model performance may decline noticeably. The implementation is optimized primarily for NVIDIA GPUs with Triton, so other accelerators require kernel reimplementation. The method is designed for autoregressive decoders, although the authors note that similar patterns also appear in multimodal LLMs such as LLaVA and InternVL and in encoder-decoder models such as T5.

A plausible implication is that MInference should be understood as a regime-specific systems method rather than a universally superior attention replacement. Its advantages emerge most clearly in the long-context prefill regime—roughly 50K to 1M tokens in the reported practical guidance—where dense quadratic attention is both the dominant cost and structurally sparse enough to admit kernel-efficient approximation.