vllm-mlx: Apple Silicon LLM/MLLM Inference

• vllm-mlx is a framework that natively optimizes Large Language Model and Multimodal LLM inference on Apple Silicon by leveraging a unified memory architecture.
• It employs continuous batching and prefix caching, achieving up to 3.7× throughput scaling for text models and 28× latency improvements for vision tasks.
• By integrating MLX-backed Metal kernels and LRU-based caching, the framework supports production-grade, real-time inference for both text and vision-language applications.

vllm-mlx is a framework designed for efficient LLM and Multimodal LLM (MLLM) inference, developed natively for Apple Silicon by building directly on Apple’s MLX library. It exploits the unified memory architecture of M-series chips to deliver high-throughput inference for both text-only and vision-language workloads. The framework implements continuous batching, advanced prefix caching, and content-based multimodal optimizations for production-grade serving on consumer Apple hardware (Barrios, 27 Jan 2026).

1. Motivation and Design Goals

The rapid adoption of Apple Silicon in on-device machine learning has highlighted limitations in extant inference frameworks. PyTorch MPS translates CUDA kernels to Metal but retains a CUDA-style execution model, incurring memory-copy and kernel launch overheads unsuited for unified memory. llama.cpp provides high single-stream text inference via hand-tuned Metal kernels and GGUF quantization but lacks support for concurrent batching and cannot address multimodal (vision–language) requests. vllm-mlx addresses these gaps by focusing on:

• Native exploitation of Apple Silicon's unified memory for LLM and MLLM inference.
• Support for both large text-only and vision–LLMs at production scale.
• Provision of serving features crucial for deployment, including continuous batching and OpenAI-compatible APIs.

2. System Architecture

vllm-mlx integrates three primary subsystems:

1. MLX Backend: Compiles tensor operations natively to Metal kernels, eliminating explicit CPU–GPU data copy overhead due to unified memory.
2. Text Serving Layer: Employs a continuous-batching scheduler, streams output, and leverages text-prefix KV-cache reuse on top of MLX-LM.
3. Multimodal Layer: Incorporates vision encoding, content-based image hashing, and prefix caching (embedding plus attention KV state) with LRU eviction.

The logical inference pipeline connects HTTP/gRPC OpenAI-compatible APIs through the vllm-mlx scheduler to the MLX backend and Metal GPU, then returns generated tokens to the client. For multimodal requests, the pipeline adds image decoding and SHA-256 content hashing with cache lookup, vision encoding if needed, and cache storage.

3. Text Model Optimization: Continuous Batching and Prefix Caching

vllm-mlx achieves significant increases in aggregate throughput primarily through continuous batching and KV-cache reuse:

• Continuous Batching: Requests are admitted to an active batch up to maximum size $M$ at each token-generation step. Admission and completion are interleaved, maintaining high hardware utilization. The aggregate throughput scaling for $N$ concurrent streams follows:

$$T(N) \approx \frac{N \cdot T(1)}{1 + \alpha \cdot (N - 1)}$$

where $\alpha$ quantifies contention, such as memory-bandwidth saturation. On models like Qwen3-0.6B, empirical scaling reaches 3.7× at $N=16$.

• Text Prefix Caching: Used when request prefixes are shared, avoiding recomputation of KV states. For a prompt prefix $P$, the cache $C$ stores $$\text{SHA256}(P) \rightarrow (\text{kv\_state}, |P|)$$. New requests with prefix $P$ can skip $|P|$ forward passes if $N$0 is cached, reducing time-to-first-token (TTFT) by up to 5.8× for 512-token shared prefixes.

4. Multimodal Inference: Content-Based Prefix Caching

Multimodal workloads incur substantial latency from repeated vision encoding. vllm-mlx introduces an image content-based cache, hashing the raw decoded image bytes with SHA-256 to identify identical images regardless of format. On a cache hit, both the image embedding and KV state are loaded immediately, bypassing both the VisionEncoder and the prompt forward pass.

The caching policy uses LRU eviction with a configurable total cache limit (default: 512 MB). Performance for repeated images shows up to 28× speedup in end-to-end latency for vision tasks. For video analysis, similar caching applies per frame, producing up to 24.7× speedups.

5. Performance Benchmarks

Empirical benchmarks on Apple M4 Max (128 GB unified memory) with 4-bit quantized models compare vllm-mlx and llama.cpp:

Model	llama.cpp (tok/s)	vllm-mlx (tok/s)	Speedup
Qwen3-0.6B	281.5	525.5	1.87×
Qwen3-4B	118.2	159.0	1.35×
Qwen3-8B	76.9	93.3	1.21×
Nemotron-30B-A3B	85.1	121.8	1.43×

Concurrency scaling for Qwen3-0.6B shows throughput increases from 441 tok/s (1 stream) to 1642 tok/s (16 streams), a scaling factor of 3.7× before bandwidth saturation becomes significant.

For multimodal models (Qwen3-VL-8B, 1024×1024 images), repeated image queries see latency drop from 21.7 s (cold) to 0.78 s (cached), a 28× improvement. Video analysis with 32 frames achieves 9.4 s (cold) to 0.38 s (cached), corresponding to a 24.7× speedup.

6. Memory Management and Unified-Memory Optimizations

MLX’s unified-memory architecture enables model weights and attention caches to reside in a single physical memory pool, facilitating zero-copy access between CPU and GPU. Lazy evaluation with kernel fusion reduces allocation overhead, which is critical for performance in small-tensor workloads typical of autoregressive generation. vllm-mlx augments the MLX allocator with an LRU-backed cache structure for both vision embeddings and KV states, sizing cache entries according to image resolution and context length.

7. Limitations and Future Extensions

Current limitations are specific to platform (macOS/Apple Silicon), model support (restricted to MLX-compatible architectures), and deployment scale (no cross-device parallelism). Potential future enhancements include speculative decoding, distributed inference on networked Apple Silicon clusters, energy-aware scheduling, prefix caching extensions to other modalities (e.g., speech–LLM), and parallelization strategies (tensor or pipeline parallelism) to scale inference across multi-GPU clusters in high-end Apple hardware (Barrios, 27 Jan 2026).

A plausible implication is that on-device, privacy-preserving, real-time LLM and MLLM inference becomes feasible for both consumer and developer use-cases as a result of these hardware–software co-optimizations.