Efficient On-Device Diffusion LLM Inference with Mobile NPU

Abstract: Diffusion LLMs (dLLMs) accelerate generation by denoising multiple tokens in parallel, making them attractive for latency-sensitive mobile inference. However, repeated denoising introduces substantial computation on smartphones. Mobile neural processing units (NPUs) offer high-throughput dense matrix computation, but efficiently exploiting them remains challenging: token commitment shrinks per-block effective workloads, token revision complicates KV cache reuse, and limited NPU-visible address space incurs costly remapping and data transfer overheads. In this paper, we propose llada.cpp, the first NPU-aware inference framework for accelerating dLLMs on smartphones. llada.cpp aligns block-wise dLLM inference with the execution characteristics of mobile NPUs through three techniques. (1) Multi-Block Speculative Decoding fills the shrinking workload in late-stage current-block decoding with speculative future-block tokens. (2) Dual-Path Progressive Revision keeps committed tokens revisable until stable and refreshes unstable tokens through a CPU-side path without stalling dense NPU execution. (3) Swap-Optimized Memory Runtime compacts NPU-visible address layouts and overlaps data staging with NPU computation to reduce remapping and transfer overheads. We implement llada.cpp as an end-to-end framework and evaluate it across diverse hardware platforms and dLLM workloads. llada.cpp reduces LLaDA-8B generation latency by 17x-42x over the CPU baseline with prefix KV cache reuse, while preserving generation quality.

• The paper introduces a framework that leverages multi-block speculative decoding to amortize NPU overhead and reduce latency by 17x–42x compared to CPU baselines.
• It implements dual-path progressive revision by offloading irregular updates to CPU kernels, ensuring maintained quality while skipping redundant projections.
• The framework uses a swap-optimized memory runtime to manage buffers dynamically, enabling efficient token processing on memory-constrained mobile NPUs.

Efficient NPU-Aware Diffusion LLM Inference for Mobile Devices

Introduction

The paper "Efficient On-Device Diffusion LLM Inference with Mobile NPU" (2606.13740) systematically tackles the problem of high-latency text generation using diffusion LLMs (dLLMs) on smartphones. While dLLMs enable parallel generation by denoising multiple tokens simultaneously, their iterative nature incurs substantial computational overhead, posing difficulties for mobile platforms limited by memory, bandwidth, and power. The work identifies a critical mismatch between dLLM inference patterns—characterized by dynamic token masking, revision, and variable computation workloads—and the static, regular execution model of modern mobile neural processing units (NPUs). The authors introduce LLADA.CPP, an inference framework that explicitly aligns dLLM decoding with the parallelism and memory semantics available in mobile NPUs, realizing substantial performance improvements for on-device LLM usage.

Technical Contributions of LLADA.CPP

LLADA.CPP embodies three main algorithm-system co-design innovations:

1. Multi-Block Speculative Decoding: Traditional block-wise dLLM decoding suffers from declining hardware utilization in late-stage denoising as uncertain token positions become scarce, leading to short, sub-optimal NPU workloads. LLADA.CPP introduces a speculative execution window that proactively includes tokens from future output blocks during current NPU forward passes. These speculative tokens are used as draft candidates—they neither influence prefix KV cache updates nor affect final commitment order until their proper turn—thereby preserving correctness while amortizing NPU execution overhead across larger batches. The runtime adaptively expands this window based on device-specific latency profiles and remaining masked token thresholds, maximizing throughput without exceeding hardware or memory constraints.

2. Dual-Path Progressive Revision: dLLM block-wise decoding creates the need for both frequent token revision (to guarantee generation quality) and efficient cache reuse. LLADA.CPP implements a three-state token lifecycle (invisible, visible, stable), leveraging two confidence thresholds to incrementally promote tokens. When prefix KV states become revisable but require irregular, sparse updates, LLADA.CPP offloads these to CPU-side, specialized matrix kernels that directly operate on NPU-tiling layouts. This division exploits the NPU’s strength in dense matrix compute and the CPU’s flexibility for sparse updates, avoiding stalling or contention. Delayed merging at denoising step boundaries prevents excessive synchonization cost. Redundant output projection and sampling logic for already-stable tokens is skipped entirely, further accelerating inference.

3. Swap-Optimized Memory Runtime: Mobile NPUs have restrictive device-visible address space with tight allocation and mapping requirements. LLADA.CPP constructs a graph-guided buffer management and staging system: long-lived tensors (weights, KV cache) are prioritized for residency while short-lived or demoted data are staged via pipelined, double-buffered transfer overlap. Tensors are packed in virtual address space based on operator scheduling and live range analysis, minimizing fragmentation and remapping events. By pipelining data transfer and operator execution, LLADA.CPP hides unavoidable latency, thereby keeping buffer management off the critical path even under memory pressure or long-context outputs.

Empirical Evaluation

Comprehensive experiments were conducted on three generations of Qualcomm Snapdragon smartphones, leveraging LLaDA-8B-Instruct and Dream-7B as primary dLLM workloads, both with low-bit quantization for memory efficiency. Tasks included GSM8K, BoolQ, ARC-C, and HellaSwag, with quality measured in token-level accuracy, while performance was tracked via end-to-end latency, power, and energy consumption.

Latency and Quality

• LLADA.CPP achieves 17x-42x generation latency reduction for LLaDA-8B over an optimized CPU baseline (with prefix KV cache reuse) across diverse datasets and devices, with no material degradation in output quality.
• Speedup over autoregressive Llama-3-8B-Instruct is up to 3.9x on recent NPUs, directly highlighting the advantage of sequence-parallel denoising when mapped properly to NPU workloads.
• Ablation studies isolate the effects of speculative decoding and dual-path revision. Speculative decoding alone may modestly decrease quality due to draft token uncertainty, but the dual-path mechanism recovers quality by maintaining revisable context and selective refresh.
• Token-level skipping for logits projection further brings an average 10%+ latency reduction, especially on longer outputs.

Memory and Energy Efficiency

• Without swap-optimized memory runtime, NPU-based systems fail to run large microbatches (e.g., 256/512 tokens), stressing the necessity of tight buffer management.
• The framework remains operational at the memory boundary by combining graph-guided mapping and pipelined staging, whereas naive approaches fail due to mapping overflows.
• In energy measurements, LLADA.CPP consistently lowers both instantaneous power and total energy-per-request compared to CPU-only or naive NPU offloads—demonstrating that effective NPU utilization is a function of adequate workload parallelism.

Implications and Future Outlook

This work demonstrates that on-device dLLM inference can achieve latency parity or superiority over autoregressive LLMs on modern mobile hardware, provided careful co-design with NPU execution semantics. The findings directly challenge the conventional wisdom that parallel denoising is prohibitively expensive for resource-constrained devices. With robust buffer management and speculative decoding aligned to hardware topologies, mobile LLM applications—including personal assistants, multimodal dialogue, and offline task automation—stand to benefit in interactivity, privacy, and accessibility.

In the medium term, LLADA.CPP’s principal methods (windowed speculative scheduling, decoupled sparse-dense compute, runtime-guided staging) are likely extensible to other high-parallelism networks such as Vision Transformers or multimodal diffusion models designed for local inference. As NPUs scale in capability and memory, the balance between on-device and cloud LLM deployment models may shift further toward ubiquitous, privacy-preserving edge AI.

Conclusion

LLADA.CPP operationalizes the core idea that non-autoregressive diffusion decoding, when meticulously engineered for mobile NPU architectures, offers a competitive and practical approach for on-device LLM inference (2606.13740). Through multi-block speculative decoding, dual-path progressive revision, and swap-optimized memory management, the framework translates sequence-level parallelism into significant latency, memory, and energy advantages without compromising generation quality. This advances the feasibility of interactive, secure, and efficient LLM applications on mobile and embedded devices, and provides a template for further system-algorithm co-design in neural inference at the edge.