Full-Pipeline Inference Optimization for MiMo-V2.5 Series: Pushing Hybrid SWA Efficiency to the Limit

Abstract: We present a full-pipeline inference optimization for the MiMo-V2.5 model family, which combines Hybrid Sliding Window Attention (Hybrid SWA), sparse Mixture-of-Experts (MoE), and multimodal encoders. While Hybrid SWA can ideally reduce both attention compute and KVCache storage significantly compared to Full Attention, realizing these gains in production requires substantial engineering effort. We systematically optimize the KVCache system with layerwise prefetch, SWA-aware prefix cache trees, and specialized placement strategies, achieving strict $O(W)$ SWA storage and high cache hit rates. We further build GCache, a high-performance distributed cache infrastructure with RDMA-optimized networking, and develop a KVCache-affinity router to reduce computation while preserving load balancing. We also optimize for multimodal inputs, including GPU image preprocessing, parallel video decoding, and multimodal cache sharing. Together, these optimizations constitute the first large-scale LLM serving system in production that efficiently covers the Hybrid SWA + MoE + multimodal composite architecture.

• The paper introduces a full-pipeline production realization of MiMo-V2.5 combining Hybrid SWA, sparse MoE, and multimodal encoders, achieving nearly 7× KVCache efficiency reduction.
• It leverages dual-pool KVCache management with O(W) storage and layerwise prefetch, maintaining cache hit rates of up to 95% and reducing long-request latency by 30.5%.
• The architecture deploys distributed GCache with RDMA-optimized networking and cache-affinity scheduling, significantly boosting throughput and multimodal processing efficiency.

Full-Pipeline Inference Optimization for MiMo-V2.5 Series: Production Realization of Hybrid SWA + MoE + Multimodal Composite Efficiency

Architectural Design and Theoretical Efficiency Bounds

The MiMo-V2.5 series embodies a composite architecture integrating Hybrid Sliding Window Attention (Hybrid SWA), sparse Mixture-of-Experts (MoE), and multimodal encoders. Hybrid SWA interleaves local SWA with global Full Attention across transformer layers: in MiMo-V2.5-Pro (70 layers), 60 layers are SWA ($W=128$), while only 10 retain Full Attention. This yields near-linear scaling in both attention compute and KVCache storage, reducing theoretical costs by about $7\times$ versus Full Attention architectures. Sparse MoE further dilutes per-token computation while maintaining model expressivity. Multimodal encoders natively support vision, audio, and video, positioning MiMo-V2.5 for long-context, cross-modal inference.

Figure 1: Attention FLOPs and KVCache memory scaling; Hybrid SWA achieves approximately $7\times$ reduction compared to Full Attention.

When benchmarked against other models at sub-500B scale, MiMo-V2.5 and -Pro show the second-lowest KVCache overhead, with only DeepSeek-V4-Flash and -Pro demonstrating lower memory requirements.

Figure 2: Comparative KVCache memory footprints for models under 500B parameters.

These theoretical gains are non-trivial to achieve in production due to KVCache management complexities, distributed scheduling, and multimodal pipeline bottlenecks.

KVCache Engineering: Strict $O(W)$ Storage and Layerwise Prefetch

Hybrid SWA’s efficiency hinges on dual-pool KVCache management. Full Attention layers track complete context ($O(N)$), while SWA layers restrict KVCache to the sliding window ($O(W)$). The system physically segregates these pools, supporting independent eviction and strict enforcement of SWA’s window constraint at the device and host tiers.

Layerwise KVCache prefetch exploits SWA’s window-local access: SWA layers can asynchronously prefetch minimal KVCache while computation proceeds, nearly eliminating cache read stalls and maximizing GPU utilization.

Figure 3: Layerwise KVCache prefetch showing compute overlapping SWA-aware loadback; the GPU is never idle waiting for cache.

The prefix cache tree is upgraded to SWA-aware matching semantics: reuse is permitted only for tail tokens within the window, preventing pseudo-hits and ensuring correctness. Each tree node encodes both Full Attention and SWA segment indices; window-out SWA is evicted independently to maintain capacity efficiency and high hit rates.

Figure 4: SWA-aware prefix cache tree with per-token Full Attention and SWA status; nodes track window-valid slots.

These optimizations enable effective KVCache hit rates of 93–95% in production, measured across high-intensity load profiles.

GCache: Distributed Storage and RDMA-Optimized Networking

GCache is deployed as tiered L3 KVCache in MiMo inference. It supports consistent hashing for metadata decentralization, multi-tier memory-disk co-deployment, shared-memory persistence, and RDMA communication. Benchmarking yields 170GB/s throughput at 280$\mu$s latency for 1MB IO sizes; under GDR, throughput approaches 350GB/s. GCache’s single-replica, fault-tolerant design eliminates multi-replica storage overhead and maintains cost-effective operation.

Figure 5: GCache architecture, illustrating sliced request dispatch, RDMA networking, and object storage backend.

Scheduling and TTFT Optimization

Cache-affinity scheduling leverages Radix prefix trees at the router to prioritize nodes with cached prefixes, maintaining load balancing. This increases L2 cache hit rates and per-node throughput by 25–30%. TTFT is further reduced by reordering queued requests by uncached length, dropping P90 latency for long requests by 30.5% without degrading shorter request TTFT.

Figure 6: TTFT comparison; long-request P90 latency drops 30.5% with cache-prioritized scheduling.

Prefill and MoE Load Balancing

SWA KVCache optimization enables smaller EP configurations, improving throughput by 40%. Length bucketing (0–64K/64K–256K/256K–1M) groups requests by sequence length, mitigating DP-attention synchronization and chunking bottlenecks. MoE pre-training yields expert token balance averaging 0.8495 (mean/max), obviating the need for runtime balancing.

Figure 7: Prefill throughput with fixed chunk size; throughput falls sharply as prefix length increases, validating the bucketing strategy.

Figure 8: Per-layer expert balance (mean/max token count ratio) remains close to 1.0.

NUMA kernel parameter conflicts are resolved to eliminate sporadic compute gaps, further enhancing inference efficiency by 10%.

Decode and MTP Optimization

Multi-turn agentic contexts saturate KVCache memory, throttling decode throughput. Optimizations include SWA-aware decode KVCache, GPU/CPU memory preallocation amortization, and CUDA Graph tuning. MTP is now enabled during prefill, yielding up to $2.3\times$ early decode speedup for 0–128 tokens and $1.5\times$ for 128–256 tokens.

Multimodal Pipeline Optimization

Encoder throughput is doubled via EPD disaggregation, data parallelism, cross-request batching, GPU-side preprocessing, parallel image/video decode, and embedding cache sharing across Encoder GPUs. Consistent hashing increases multimodal cache hit rates by 30%. End-to-end latency improvements are observable across modalities and request types.

Conclusion

The paper delivers the first full-pipeline, production-scale implementation of efficient inference for Hybrid SWA + MoE + multimodal composite architectures, systematically refactoring KVCache, distributed caching, scheduling, prefill, decode, and multimodal pipelines. Empirical results confirm the realization of theoretical $O(W)$ storage scaling, high cache hit rates (up to 95%), and large throughput/latency reductions, especially for long-context agentic and multimodal scenarios. This engineering blueprint has immediate applicability in high-volume LLM deployments, with broader implications for scalable inference, distributed caching, and multimodal integration. Future directions include deeper harness-inference co-design, finer dynamic bucketing, and continued open-source community collaboration.

(2607.13095)