llada.cpp: NPU-Accelerated dLLM Inference

• llada.cpp is a specialized framework that harnesses mobile NPUs to accelerate diffusion large language model inference using block-wise denoising and algorithmic innovations.
• It integrates multi-block speculative decoding and dual-path progressive revision to reduce generation latency by 17×–42× over CPU baselines while preserving output quality.
• The system combines a host-side Android runtime with a device-side Hexagon NPU library, optimizing memory mapping and scheduling for efficient Transformer operation execution.

llada.cpp is a specialized NPU-aware inference framework designed to accelerate inference for diffusion LLMs (dLLMs) on smartphones by tightly co-designing algorithmic innovations with mobile neural processing unit (NPU) execution patterns. It is the first implementation to align block-wise dLLM denoising with the vectorized, high-throughput characteristics of mobile NPUs, delivering significant reductions in generation latency (17×–42× over CPU baseline with prefix KV reuse) without compromising model output quality (Wang et al., 11 Jun 2026).

1. High-Level Architecture and Data Flow

llada.cpp comprises two integrated modules: a host-side Android runtime (implemented in C/C++ based on llama.cpp) and a device-side Hexagon NPU operator library (leveraging HMX and HVX engines). The host runtime orchestrates the block-wise diffusion scheduler, manages the three-state token machine, controls dual-path progressive revision, executes swap-optimized memory mapping, and issues FastRPC calls to the NPU operator library. The device-side component implements FP16 matrix multiplications (matmuls), Q4₀ dequantization kernels, FlashAttention, RMSNorm, elementwise operations, shared-memory buffer mapping, and worker pooling.

The data flow is as follows:

1. The CPU marshals a decoding window into NPU-visible buffers via RPCMEM and Linux DMABUF.
2. A FastRPC call from the CPU triggers the NPU graph computation for the current window.
3. The NPU executes fused Transformer layers, returning next-step token logits and intermediate KV states.
4. The CPU evaluates returned logits, updates token states, and commits or marks tokens as visible/stable.
5. Asynchronously, the CPU may refresh sparse KV slices and remap/stage buffers while NPU processing continues.

All control-intensive, irregular tasks (token commits, sparse updates, buffer layout changes) remain on the CPU, while bulk matrix and attention operations execute on the NPU in shape-stable kernels.

2. Multi-Block Speculative Decoding

Multi-Block Speculative Decoding addresses throughput collapse in late-stage block-wise denoising, when the set of uncommitted (“masked”) tokens $|M_t|$ becomes small and cannot fully utilize NPU hardware. The key strategy is to supplement the shrinking current-block workload by speculatively denoising tokens from future blocks within the decoding window $W_t$.

Formal Model:

• At denoising step $t$, let $M_t \subset \{1,\ldots,B\}$ denote masked positions. $|M_t|$ decreases as $t \rightarrow T$.
• Decoding window $W_t = \{c_t,...,c_t+W-1\}$ spans current and possibly future blocks; $c_t$ is the leftmost uncommitted position.
• Each forward pass computes a Transformer over $W_t$, mixing masked and draft tokens.
• After execution, the CPU commits any position $i \in W_t \cap$ current block with $W_t$0.

Throughput Gain Model:

Let $W_t$1 be NPU latency for $W_t$2 tokens (stepwise constant per hardware tile), $W_t$3 is the number of speculative tokens.

• Baseline throughput: $W_t$4
• Speculative throughput: $W_t$5
• Per-step gain:

$W_t$6

By moving $W_t$7 into a higher tile-aligned region, the marginal latency is minimized.

Algorithmic Structure:

The process iteratively selects a window $W_t$8, invokes NPU execution, computes confidences $W_t$9, and commits tokens when above threshold $t$0. When the block completes, KV cache is warm-started for the next block.

3. Dual-Path Progressive Revision

llada.cpp employs a token-state machine with three states per token: IV (invisible), V (visible), and S (stable), with two confidence thresholds $t$1. Transitions are as follows:

1. $t$2
2. $t$3
3. If no token crosses $t$4 at a step, promote highest-$t$5 token to V.

Dual-Path Implementation:

• NPU path: Dense Transformer passes over $t$6, handling both masked and visible positions.
• CPU revision path: For visible tokens in the prefix with changed $t$7, use custom CPU kernels to recompute logits and KV slices. Merges are delayed to end of current denoising step, avoiding per-token synchronization.

Threshold Optimization:

Given error rate $t$8 decreases with $t$9, the thresholds are chosen to optimize progress subject to prefix error constraint,

$M_t \subset \{1,\ldots,B\}$0

where $M_t \subset \{1,\ldots,B\}$1 is expected number of steps to make a token visible, and the regularization penalizes erroneously committed tokens.

4. Swap-Optimized Memory Runtime

Prior to llada.cpp, each layer’s weights, KV cache, and activations were scattered across NPU virtual address (VA) regions, causing translation lookaside buffer (TLB) misses and volatile mapping overhead. Graph-guided buffer compaction assigns long-lived tensors (weights, prefix KV) to fixed, contiguous VA slabs and consolidates short-lived activations and scratch buffers into small pools with lifetime-based reuse, reducing VA mappings by 40–60%.

Producer-Consumer Pipelining:

• Utilizes double-buffer staging. While NPU processes layer $M_t \subset \{1,\ldots,B\}$2 with buffer $M_t \subset \{1,\ldots,B\}$3, the CPU prepares buffer $M_t \subset \{1,\ldots,B\}$4 for layer $M_t \subset \{1,\ldots,B\}$5.
• After layer completion, buffer swap occurs and $M_t \subset \{1,\ldots,B\}$6 is dispatched without waiting.

Performance Modeling:

• Unoptimized: $M_t \subset \{1,\ldots,B\}$7
• Pipelined: $M_t \subset \{1,\ldots,B\}$8 Profiling demonstrates that mapping overhead is reduced by up to 80%, and double-buffering hides approximately 70% of data copy time.

5. Empirical Results and Model Fidelity

Latency and Throughput

For LLaDA-8B on SM8750 (128 tokens), the following timings were observed:

Method	32 tok	64 tok	128 tok
CPU only	860 ms	1695 ms	2996 ms
+ prefix KV cache	190 ms	395 ms	607 ms
+ NPU offload (vanilla dLLM)	110 ms	231 ms	341 ms
+ llada.cpp (all features)	10 ms	12 ms	15.8 ms

The overall speedup is 17×–42× over the CPU+KV cache baseline, and llada.cpp achieves up to 3.9× faster generation than a comparable-size autoregressive Llama-3-8B.

Cross-model and cross-device evaluation (Dream-7B, SM8750, OnePlus 12/15) show scaling with NPU TOPS and relative reductions of similar magnitude.

Output Quality

Accuracy on 200-sample benchmarks:

	GSM8K	BoolQ	ARC-C	HellaSwag
LLaDA CPU	39.0	82.5	84.0	51.0
LLaDA llada.cpp (full)	43.5	80.5	85.0	49.5
Ablation: NPU+MB SpecDec	37.5	79.5	82.0	46.0
+ Dual-Path Prog Revision	43.5	80.5	85.0	49.5

Moving to NPU does not degrade quality; speculative decoding alone introduces a 2–4 point accuracy drop, while dual-path revision recovers over 3 points.

6. Implementation and Integration Details

Key Code Structures

• src/dllm_diffusion.cpp: Manages block loops, window selection, NPU invocation
• src/token_state.h/.cpp: Three-state machine, thresholding, confidence computation
• src/revision_manager.cpp: Sparse KV refresh management, CPU-side recomputation, cache merging
• src/mem_runtime.cpp: Buffer graph, profiling, memory placement, mapping/copy scheduling
• hexagon_dsp/*.C: Operator kernels, weight conversion for NPU execution

Memory Management

RPCMEM allocates DMABUF-tagged buffers, visible to both CPU and NPU. Memory mapping is validated minimally prior to kernel launch. FastRPC manages file descriptor marshalling with no extra userland copies.

Integration with Pre-Existing Runtimes

llada.cpp extends llama.cpp's GGML backend to support dLLM mode, intercepting matmul/attention layers for selective routing to CPU fallback or Hexagon NPU shared objects. It is compatible with standard GGUF-converted model files, but a dedicated layout pass is used to re-tile weights to the required NPU-friendly format (32×32 HMX, 4-bit Q4₀ for HVX).

In summary, llada.cpp achieves substantial on-device dLLM inference acceleration by integrating speculative multi-block decoding, dual-path token revision, and memory runtime optimization, together translating block-wise diffusion model structure into concrete mobile performance gains without compromising fidelity (Wang et al., 11 Jun 2026).