TyphoonMLA: Hybrid Kernel for LLM Attention

• TyphoonMLA is a hybrid kernel for LLMs that combines the compute-efficient naive kernel with the memory-optimal absorb kernel to optimize multi-head latent attention.
• It leverages shared prefix reuse to achieve up to 3–3.24× throughput speedup on NPUs and GPUs while keeping high-bandwidth memory overhead minimal.
• Its adaptive design automatically falls back to absorb-only computations in low data reuse scenarios, ensuring reliable performance in diverse inference settings.

TyphoonMLA is a hybrid kernel implementation designed to accelerate attention computations in LLMs that utilize Multi-Head Latent Attention (MLA). MLA, adopted in models such as DeepSeek-v3 and Kimi K2, maintains the KV-cache in a low-rank latent representation, which allows for alternative computational strategies in both training and inference. TyphoonMLA achieves high throughput by dynamically mixing the compute-efficient naive formulation with the memory-efficient absorb formulation, directly exploiting shared prefix reuse and minimizing high-bandwidth memory (HBM) traffic. This hybrid approach yields up to 3–3.24× speedup on NPUs and GPUs with a negligible HBM overhead, and is structured for compatibility with contemporary LLM serving frameworks.

1. Background: Multi-Head Latent Attention and Kernel Formulations

MLA is an attention mechanism in which the key and value tensors are stored in compressed (latent) form, enabling significant memory bandwidth reduction at inference. There are two mathematically equivalent but operationally distinct kernel implementations for MLA:

• Naive Kernel: For each token, expands the latent KV-cache into the full attention space before computing the attention output. This strategy is compute-efficient and highly suitable for scenarios where shared data can be reused (such as prefill and when multiple queries share a prefix).
• Absorb Kernel: Keeps KV tensors in their low-rank latent form, “absorbing” the projection matrices into the attention computation. This approach minimizes HBM utilization but is more compute-bound, especially because data reuse across queries is less effective.

Standard LLM training uses the naive kernel. During inference, especially for decoding, absorb-based kernels (such as FlashMLA) are preferred due to their lower memory requirements.

2. TyphoonMLA: Hybrid Scheme and Computational Design

TyphoonMLA introduces a mixed kernel for MLA attention calculation that partitions the KV-cache into shared and non-shared segments. The technical workflow is as follows:

• Shared Prefix (Naive Path):
  • The portion of the KV-cache representing the shared prefix is projected up-front using the naive kernel (e.g., using W₍KVb₎).
  • In batched decoding or when prompts are shared (e.g., long system instructions), this yields substantial computational savings since the same prefix data is reused across multiple queries.
• Non-Shared Suffix (Absorb Path):
  • The segment of the KV-cache unique to each query is kept unexpanded in latent space and processed via the absorb kernel (with weight splits like W₍KVb1₎, W₍KVb2₎), thus minimizing HBM reads.

After independent computation, outputs from both the naive and absorb paths are merged, with correct softmax normalization to produce the final attention result. In the prefill stage, only the naive kernel is used, while in decode, TyphoonMLA adaptively applies naive or absorb computations as dictated by run-time context (batch shapes, prefix length, degree of sharing).

If the batch size is small or the prefix reuse is negligible, TyphoonMLA automatically degrades to the memory-optimal absorb kernel, ensuring no performance loss in such scenarios.

3. Throughput and Resource Efficiency

The principal empirical findings for TyphoonMLA are:

• Speedup: Achieves up to 3× faster throughput on NPUs and 3.24× on GPUs compared to absorb-only baselines (e.g., FlashMLA) under conditions of shared prefix reuse.
• Operational Intensity: In compute-bound regions (with high MAC reuse on prefix), the naive kernel can achieve up to 3.4× fewer MAC operations. In memory-bound regions (suffixes unique to each sequence), absorb achieves ~70× lower HBM reads compared to naive.
• Memory Overhead: TyphoonMLA incurs only ~3% increase in high-bandwidth memory usage relative to absorb-only kernels.
• Dynamic Fallback: For small batch sizes or low data reuse scenarios, performance gracefully falls back to that of absorb-only kernels.

A summary table of kernel characteristics in the context of MLA is as follows:

Kernel	Prefill/Shared Prefix	Decode/Unique Suffix	HBM Utilization
Naive	High efficiency	HBM intensive	High
Absorb	Less efficient	HBM optimal	Low
TyphoonMLA	Hybrid (naive+absorb)	Hybrid (naive+absorb)	~3% overhead

4. Comparisons with Existing Kernels

Traditional FlashAttention (naive) kernels excel in training or prefill but are inefficient for memory-bound inference scenarios due to large HBM reads. Absorb (FlashMLA) kernels minimize these reads by operating entirely in the latent KV space but pay a compute penalty and miss potential data reuse advantages when prefixes are common.

TyphoonMLA outperforms both by:

• Reducing compute operations in the shared prefix regime relative to absorb-only.
• Preserving low memory bandwidth for non-shared sequences, matching absorb.
• Maintaining optimality across varying batch sizes via adaptive kernel selection.

This design allows TyphoonMLA to consistently deliver superior throughput in practical LLM serving environments characterized by prompt sharing, speculative decoding, and branch prediction (such as Tree-of-Thought).

5. Deployment Context and Integration

TyphoonMLA is engineered for integration with major LLM inference and serving frameworks, specifically targeting models like DeepSeek-v3 and Kimi K2, and kernel-integration stacks like vLLM and SGLang. The kernel is compatible with tensor and sequence parallelism, as well as advanced batching strategies (e.g., PagedAttention, RadixAttention).

Applications benefiting from TyphoonMLA include:

• High-throughput LLM inference with prompt sharing (system prompts, conversational context).
• Parallel decoding branches in speculative execution pipelines.
• Large-scale, multi-tenant model serving with guaranteed minimum latency.
• Any scenario where HBM bandwidth is a bottleneck but shared computation on input prefixes can be exploited.

6. Future Research Directions

Potential directions suggested include:

• Development of further adaptive hybrid kernels for emerging attention architectures beyond MLA.
• Dynamic runtime scheduling algorithms that balance kernel selection based on operational telemetry.
• Integration with forthcoming hardware architectures offering even lower HBM latencies or optimized caches for LLM workloads.

Open challenges remain in devising general principles for hybrid kernel orchestration and in expanding the TyphoonMLA paradigm to transformer variants with more intricate attention mechanisms.

7. Summary

TyphoonMLA represents a computationally efficient hybrid kernel for MLA-based LLMs, dynamically combining naive and absorb formulations to optimally leverage compute and memory resources depending on the degree of sequence prefix sharing. Empirical results indicate up to a 3.24× throughput gain with only marginal memory overhead, directly addressing key computational bottlenecks in next-generation LLM inference infrastructure (Yüzügüler et al., 25 Sep 2025).