MLX-Transformers: Apple Silicon Inference

• MLX-Transformers is a library that enables native transformer implementations on Apple Silicon using Apple’s MLX framework and Metal acceleration.
• It benchmarks diverse models like BERT, RoBERTa, and LLMs by seamlessly converting Hugging Face checkpoints into MLX-native formats for on-device inference.
• The system demonstrates practical gains in latency and throughput through GPU acceleration, KV caching, mixed-bit quantization, and streamlined serving patterns.

MLX-Transformers is a transformer-model library built on Apple’s MLX framework for Apple Silicon, introduced as a mechanism for running transformer models sourced from Hugging Face directly on Apple devices and benchmarking their on-device inference characteristics against PyTorch on CPU and CUDA (Ajayi et al., 21 Oct 2025). In the literature, the term has two closely related senses. In the narrow sense, it denotes the specific library evaluated in “Benchmarking On-Device Machine Learning on Apple Silicon with MLX,” which implements model families including BERT, RoBERTa, XLM-RoBERTa, T5, LLaMA, Phi, and others in MLX (Ajayi et al., 21 Oct 2025). In a broader, later usage, it can refer to MLX as a transformer and large-language-model inference stack on Apple Silicon, including MLX-LM-style deployment patterns, KV caching, quantization, and serving (Rajesh et al., 9 Oct 2025). Together these usages situate MLX-Transformers within an Apple-centric ecosystem for local transformer inference, experimentation, and, increasingly, production deployment.

1. Definition, scope, and nomenclature

MLX is described as a machine learning framework developed by Apple’s ML research team specifically for Apple Silicon, with Apple-specific optimization and a Unified Memory Model in which CPU and GPU share a single pool of high-bandwidth memory (Ajayi et al., 21 Oct 2025). MLX-Transformers is the library introduced on top of that framework to provide ready-to-use transformer model implementations in MLX and direct loading of Hugging Face models on Apple Silicon (Ajayi et al., 21 Oct 2025).

The specific library scope stated in the benchmark paper includes model families “BERT, Fuyu, LLaMA, M2M-100, OpenELM, Persimmon, Phi, Phi-3, RoBERTa, T5, XLM-RoBERTa” (Ajayi et al., 21 Oct 2025). The benchmarked subset in that study is narrower, focusing on bert-base-uncased, bert-large-uncased, roberta-base, and xlm-roberta-base, with a single CUDA-only T5 runtime entry also reported (Ajayi et al., 21 Oct 2025). This suggests that the library was positioned simultaneously as an implementation framework and as an empirical vehicle for evaluating MLX as an on-device runtime.

Later papers use the label more expansively. “Production-Grade Local LLM Inference on Apple Silicon” states that it does not introduce a separate package named “MLX-Transformers,” but effectively analyzes “MLX as a transformers/LLM inference stack,” including prefill and decode, KV cache, prompt cache, quantization, streaming, batching, and deployment (Rajesh et al., 9 Oct 2025). In the same ecosystem, mlx_lm is treated as the concrete MLX backend for large local models on Apple Silicon, served through mlx_lm.server and compared directly with GGUF and llama.cpp deployments (Leitch, 20 Apr 2026). The terminological overlap is therefore substantive rather than accidental: the library and the broader runtime stack share the same MLX-native design assumptions.

2. Architectural basis in MLX

The benchmark paper states that MLX exposes a NumPy-style array API, autograd, and backends for mlx-cpu and mlx-gpu, with support for operations such as MatMul, Linear, and Softmax required for modern deep networks (Ajayi et al., 21 Oct 2025). MLX-Transformers implements standard transformer architectures in that environment for encoder-only, encoder-decoder, and decoder-only families (Ajayi et al., 21 Oct 2025).

The architectural description follows the conventional transformer decomposition. For encoder-style models such as BERT, RoBERTa, and XLM-RoBERTa, each layer comprises multi-head self-attention, a feed-forward block, and layer normalization with residual connections (Ajayi et al., 21 Oct 2025). The standard scaled dot-product formulation appears explicitly:

$Q = X W_Q,\quad K = X W_K,\quad V = X W_V$

and

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$$

with feed-forward transformation

$\text{FFN}(x) = W_2 \,\sigma(W_1 x + b_1) + b_2$

(Ajayi et al., 21 Oct 2025).

A more formal algorithmic account is supplied by “Formal Algorithms for Transformers,” which defines token embeddings, positional embeddings, masked and unmasked attention, multi-head composition, and encoder-only, decoder-only, and encoder-decoder transformer variants in mathematically explicit terms (Phuong et al., 2022). That paper’s treatment is not specific to MLX, but it matches the class of models implemented by MLX-Transformers. A plausible implication is that MLX-Transformers should be understood less as a novel architectural family than as an MLX-native realization of established transformer algorithms.

The broader MLX ecosystem reinforces this interpretation. In mlx-snn, all tensor operations are written in mlx.core, modules inherit from mlx.nn.Module, training uses mlx.nn.value_and_grad, and dynamic state is passed explicitly rather than hidden inside mutable modules (Qin, 3 Mar 2026). Although that work targets spiking neural networks rather than transformers, it identifies a recurring MLX-native pattern: explicit state, composable function transforms such as mx.grad and mx.compile, unified memory, and lazy evaluation (Qin, 3 Mar 2026). The paper explicitly frames this as directly transferable to transformer-style code. Similarly, the physics-oriented transformer for 2D heat conduction is implemented with nn.TransformerEncoder, nn.SinusoidalPositionalEncoding, nn.Linear, nn.LayerNorm, and mlx.compile, demonstrating that MLX supports nontrivial encoder-only transformer training outside NLP (Kassinos et al., 2024).

3. Hugging Face integration and implementation workflow

A central design goal of MLX-Transformers is direct interoperability with Hugging Face checkpoints and configurations (Ajayi et al., 21 Oct 2025). The benchmark paper states that the library downloads checkpoints directly from Hugging Face Hub and that a user can point MLX-Transformers at a Hugging Face model identifier and obtain an MLX model without a manual conversion pipeline (Ajayi et al., 21 Oct 2025). The paper also notes an apparent dual description: the abstract says the framework downloads PyTorch checkpoints and converts them to MLX format, but also says that MLX-Transformers enables seamless execution of models directly sourced from Hugging Face, eliminating the need for checkpoint conversion often required when porting between frameworks (Ajayi et al., 21 Oct 2025). The practical interpretation given in the same source is that the mapping from PyTorch-format weights to MLX arrays is handled internally by the library (Ajayi et al., 21 Oct 2025).

The typical workflow described is analogous to Hugging Face Transformers. A practitioner installs MLX and the library, selects a model ID such as bert-base-uncased, roberta-base, or xlm-roberta-base, chooses mlx-gpu or mlx-cpu, tokenizes with Hugging Face tokenizers, and measures wall-clock latency across repeated forward passes (Ajayi et al., 21 Oct 2025). Because MLX relies on unified memory, the paper emphasizes that no manual memory transfer or device-placement code is required in user code (Ajayi et al., 21 Oct 2025).

Later inference-oriented work shows how this workflow generalizes to LLMs. In the Apple-Silicon comparative study, MLX v0.26 is used to run Qwen-2.5-Coder-3B and Qwen-2.5-7B-Instruct on a Mac Studio with an M2 Ultra and 192 GB unified memory, with client-side tokenization and end-to-end measurement including prefill and decode (Rajesh et al., 9 Oct 2025). For serving, the same ecosystem exposes OpenAI-style endpoints through community wrappers such as mlx-openai-server, including /v1/completions, /v1/chat/completions, /v1/embeddings, and basic function-calling (Rajesh et al., 9 Oct 2025). In mlx_lm, the server is invoked as python3.13 -m mlx_lm server, and model-specific chat-template arguments may be required for correct reasoning-mode behavior (Leitch, 20 Apr 2026).

This suggests that the library-level loading path and the runtime-level serving path are converging. The narrower library originally emphasized checkpoint compatibility and model parity with PyTorch (Ajayi et al., 21 Oct 2025); the later stack emphasizes deployment and systems behavior on Apple hardware (Rajesh et al., 9 Oct 2025).

4. Benchmarking methodology and reported performance

The original benchmark compares MLX-Transformers inference latency on an Apple M1 MacBook Pro with 8 GB unified memory, an Apple M2 Max MacBook Pro with 32 GB unified memory, and PyTorch on an NVIDIA A10 PCIe GPU with 24 GB VRAM hosted on AWS EC2 (Ajayi et al., 21 Oct 2025). Inputs are taken from Hugging Face datasets and truncated to 50, 100, 200, and 500 “characters,” with batch sizes of 1, 16, and 32, and latency measured as wall-clock time per forward pass over 10 iterations per configuration (Ajayi et al., 21 Oct 2025).

On the M1, MLX-GPU substantially outperforms MLX-CPU. Overall average latency on mlx-gpu is reported as 179.35 ms for BERT-base, 531.27 ms for BERT-large, 401.27 ms for RoBERTa-base, and 142.42 ms for XLM-RoBERTa-base, versus 557.14 ms, 1558.64 ms, 966.13 ms, and 333.52 ms respectively on mlx-cpu (Ajayi et al., 21 Oct 2025). On the M2 Max, overall mlx-gpu averages are reported as 38.23 ms for BERT-base, 94.06 ms for BERT-large, 74.32 ms for RoBERTa-base, and 27.37 ms for XLM-RoBERTa-base (Ajayi et al., 21 Oct 2025). The corresponding CUDA A10 averages are 23.46 ms, 62.55 ms, 39.08 ms, and 16.09 ms (Ajayi et al., 21 Oct 2025). The paper therefore states that, for these model sizes and settings, M2 Max MLX-GPU latency is within roughly a factor of $1.5$–$2\times$ of the A10 CUDA baseline (Ajayi et al., 21 Oct 2025).

The paper also reports operation-level comparisons that contextualize transformer building blocks. On the M1, average MatMul latency is 26.19 ms on MLX-GPU versus 106.08 ms on MLX-CPU; Linear is 18.88 ms versus 60.87 ms; and Softmax is 27.91 ms versus 50.74 ms (Ajayi et al., 21 Oct 2025). This aligns with the observed model-level speedups and with the claim that GPU acceleration is critical for on-device transformer inference.

Latency scaling in the benchmark is reported as roughly linear with input length and sublinear with batch size on the M2 Max (Ajayi et al., 21 Oct 2025). For example, BERT-base on M2 Max mlx-gpu rises from 16.93 ms at 50 characters to 76.40 ms at 500 characters, while batch-size growth from 1 to 32 increases average latency from 8.02 ms to 70.48 ms rather than by a full factor of 32 (Ajayi et al., 21 Oct 2025). The paper interprets this as evidence that MLX-Transformers exploits GPU parallelism on Apple Silicon.

A different performance regime appears in the later study of local large-language-model runtimes on Apple Silicon. Under that study’s settings, MLX reaches approximately 230 tokens/s steady-state decode throughput on an M2 Ultra, with median inter-token latency of 5–7 ms/token, P99 inter-token latency around 12 ms, GPU utilization above 90%, and CPU usage below 3% (Rajesh et al., 9 Oct 2025). Relative to MLC-LLM, llama.cpp, Ollama, and PyTorch MPS, MLX is reported to have the highest sustained throughput, though not the lowest time-to-first-token for moderate prompt sizes (Rajesh et al., 9 Oct 2025). This does not benchmark the original MLX-Transformers package directly, but it materially extends the evidence base for MLX as a transformer runtime.

5. MLX-Transformers as an Apple-Silicon inference stack

The later systems papers recast MLX-Transformers in operational terms. In the comparative LLM study, MLX is characterized as an Apple-optimized array and inference framework using Metal and Apple-tuned kernels, with support for rotating KV cache, on-disk prompt cache, streaming generation, quantization, and deployment via a thin wrapper (Rajesh et al., 9 Oct 2025). In that context, “MLX-Transformers” denotes not only model definitions but also the end-to-end transformer-serving path on Apple hardware.

Long-context handling is described as one of the principal trade-offs. MLX uses a rotating KV cache with a configurable effective window, defaulting to about 4k tokens, and an optional on-disk prompt cache for shared prefixes (Rajesh et al., 9 Oct 2025). At 100k tokens, the paper reports roughly 40–50 GB RAM usage for overlapping large-context runs, approximately linear prefill growth with context length, and stable inter-token latency of about 11–12 ms once decoding begins (Rajesh et al., 9 Oct 2025). The same study recommends MLX primarily for chat and code workloads in the 4k–32k regime, and recommends MLC-LLM instead for extreme contexts such as 64k–128k because MLX does not yet provide paged attention or KV paging (Rajesh et al., 9 Oct 2025).

Quantization is similarly treated as central. MLX is reported to support mixed-bit quantization in 3/4/6/8-bit formats, with GPTQ support described as “maturing but present,” and an ecosystem of pre-quantized MLX models and conversion recipes targeting Apple hardware (Rajesh et al., 9 Oct 2025). The authors conclude that robust 4-bit quantization yields 2–3× memory reduction and equal or higher throughput for the tested workloads, and recommend MLX as a default for Apple-centric deployments because of its tight integration with Apple’s quantization stack (Rajesh et al., 9 Oct 2025).

Deployment, however, remains deliberately lean. The same paper states that local setup is “trivial,” that pip install mlx-LLM or equivalent MLX wheels suffice, and that prebuilt Metal kernels avoid local compilation (Rajesh et al., 9 Oct 2025). Serving typically requires a thin Python wrapper such as FastAPI or Starlette, or a community solution such as mlx-openai-server; no official Docker or Kubernetes manifests are described (Rajesh et al., 9 Oct 2025). MLX is therefore positioned as a high-efficiency core engine rather than a full serving platform.

This characterization is reinforced by the mlx_lm study on System Dynamics assistants. There, MLX models are served through mlx_lm.server on a Mac Studio M3 Ultra with 512 GB unified memory, including MLX-3bit, MLX-4bit, and MLX-6bit quantizations for Kimi K2.5, DeepSeek V3.2, Qwen 3.5 397B, and GLM-5 (Leitch, 20 Apr 2026). The paper finds that backend choice has larger practical impact than quantization level, and identifies a crucial MLX-specific limitation: mlx_lm ignores response_format for json_schema and json_object, so structured outputs require explicit prompt-level JSON instructions rather than grammar-constrained sampling (Leitch, 20 Apr 2026). It also reports a hard long-context failure mode on Metal: prompts above roughly 60–80k tokens for large models can trigger command-buffer out-of-memory failures despite abundant unified memory (Leitch, 20 Apr 2026). These are systems-level facts rather than model-level facts, but they define the operational boundary of MLX-native transformer deployment.

6. Ecosystem expansion, scientific uses, and limitations

MLX-Transformers sits within a larger MLX-native ecosystem that has expanded beyond conventional NLP inference. The heat-conduction transformer study demonstrates that MLX can train and run encoder-only transformers for operator learning in engineering physics, using separate sinusoidal positional encodings in $x$ and $y$, an explicit diffusivity embedding, and physics-informed loss terms for PDE residuals, boundary conditions, and initial conditions (Kassinos et al., 2024). The authors report block-prediction test loss of $1.13\times10^{-6}$ and autoregressive test loss of $3.2\times10^{-7}$ in the base configuration, and note that mlx.compile yields roughly a 25% training speedup (Kassinos et al., 2024). This extends MLX-Transformers from text modeling to scientific computing.

ChipChat provides a second expansion path: a fully MLX-based on-device voice assistant stack in which transformer components are used for ASR, speaker modeling, a state-action augmented Mixtral-style decoder-only LLM, and streaming TTS, all on a Mac Studio M2 Ultra with 192 GB unified memory (Likhomanenko et al., 26 Aug 2025). The paper states that a naive PyTorch-heavy cascaded implementation had more than 4 s latency, whereas the optimized MLX-based system reaches approximately 920 ms end-to-end response latency, with the LLM remaining the dominant cost at approximately 560 ms (Likhomanenko et al., 26 Aug 2025). The explicit use of MLX-LM rotating KV cache in the LLM component shows how MLX-native transformer inference has become a subsystem in larger real-time applications.

The ecosystem also includes MLX-native tooling that, while not itself a transformer library, is directly relevant to transformer workflows. mlx-vis implements UMAP, t-SNE, PaCMAP, TriMap, DREAMS, CNE, and NNDescent entirely in MLX, with a unified fit_transform interface and GPU-accelerated rendering on Apple Silicon (Xiao, 4 Mar 2026). The paper explicitly frames this as useful for transformer embeddings, attention outputs, and training-dynamics visualization, and reports full-pipeline embedding plus 800-frame video generation on Fashion-MNIST in 3.6–5.2 seconds on an M3 Ultra (Xiao, 4 Mar 2026). This suggests a surrounding tooling layer for analysis and diagnostics that remains in the same MLX/Metal environment.

Several limitations recur across the literature. The original benchmark is restricted to full-precision inference for encoder-style NLP models up to BERT-large, with no quantization, mixed precision, or billion-parameter LLM evaluation (Ajayi et al., 21 Oct 2025). The LLM systems papers identify no built-in continuous batching, no first-party multi-tenant scheduler, and limited concurrency without multiple worker processes and external orchestration (Rajesh et al., 9 Oct 2025). Structured-output reliability in mlx_lm depends on prompt engineering rather than schema enforcement (Leitch, 20 Apr 2026). Extreme long-context processing remains bounded by rotating caches or Metal command-buffer constraints rather than paged attention (Rajesh et al., 9 Oct 2025). Even where performance is strong on Apple hardware, the comparative study states that Apple-side MLX and MLC remain about $5$–$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$$0 slower in throughput than vLLM on an NVIDIA A100 (Rajesh et al., 9 Oct 2025).

Taken together, these sources define MLX-Transformers as both a concrete library and a broader methodological pattern: transformer models implemented natively in MLX, loaded from Hugging Face or related ecosystems, executed on Apple Silicon through unified memory and Metal acceleration, and increasingly embedded in local inference, scientific computing, and application-serving pipelines. The dominant theme across the corpus is not architectural novelty in the transformer itself, but the consolidation of an Apple-Silicon-native transformer stack with distinct strengths in on-device deployment, privacy, and developer ergonomics, and distinct constraints in long-context scaling, batching, and structured serving (Ajayi et al., 21 Oct 2025).