On-Device LLM Inference

Updated 18 August 2025

On-device LLM inference is a paradigm where large models are executed directly on client devices, emphasizing privacy, low latency, and cost-efficiency.
It employs innovations like staged speculative decoding, advanced quantization, and memory-aware offloading to overcome resource constraints.
The approach integrates hybrid device–server architectures, robust security methods, and efficient fine-tuning to optimize throughput and energy use.

On-device LLM inference refers to the execution of LLM forward passes, and—increasingly—fine-tuning or adaptation, directly on client or edge devices such as smartphones, IoT/embedded boards, and laptops, without recourse to server-side computation. This paradigm is driven by privacy, latency, connectivity, and cost considerations, motivating the development of algorithms, quantization strategies, memory management protocols, hardware/software co-design, security protections, and robust benchmarking specifically for resource-constrained environments.

1. Algorithmic Acceleration for Resource Efficiency

A key challenge in on-device LLM inference is to minimize latency and bandwidth consumption under constraints of small batch sizes and limited compute. Several innovations target this problem:

Staged Speculative Decoding: An algorithmic framework in which speculative batches are arranged in a tree structure rather than a linear sequence. Each internal node expands to the top-k most probable tokens, so the number of candidate sequences grows exponentially with depth, but the cost grows only linearly. The oracle model (e.g., a 762M parameter GPT-2-L) verifies only a "frontier" of plausible continuations, amortizing batch verification and increasing expected accepted tokens per batch ( $E[T] = \sum_l p(l)$ , where $p(l)$ is the probability the leaf sequence matches the oracle). A second speculative stage ("speculate the speculator") applies the same logic to the draft model, further reducing decoding time. Experimental results demonstrate up to $3.16\times$ lower single-batch latency with no loss in output quality (Spector et al., 2023).
Token Trees and Model Collaboration: LLMCad organizes candidate tokens into a dynamic tree, verified in parallel by a high-precision LLM, while a compact in-memory LLM generates “easy” tokens. Confidence-based thresholds and a fallback strategy ensure mistakes are caught early without repeated full-model invocation. This approach yields $2.9$– $9.3\times$ speedup on IoT devices and $3.5$– $4.7\times$ on smartphones, even with large-scale LLMs that would not otherwise fit in RAM (Xu et al., 2023).
Hybrid Device–Server Architectures: Methods such as U-shaped or "hat-shaped" inference (e.g., HAT) partition the LLM so shallow input/output submodels run on-device, with the bulk of computation handled by the server. Only hidden states are exchanged, not raw tokens, substantially reducing privacy risk and communication overhead. Speculative decoding and prompt chunking further accelerate throughput, with empirical reductions in time-to-first-token (TTFT) and time-between-tokens (TBT) exceeding $40\%$ in real deployments (Xie et al., 23 Mar 2025).

2. Quantization, Compression, and Memory Management

On-device inference is often memory-bound. State-of-the-art quantization and memory assignment strategies are central:

Quantization: Weight-only post-training quantization (PTQ) is widely deployed to reduce model footprints while minimizing accuracy loss. The process quantizes the weights (e.g., using 4-, 6-, or 8-bit representations), enabling LLMs of multiple billion parameters to fit into 2–3 GB RAM or less (Fassold, 24 Apr 2024, Çöplü et al., 2023). Experiments on edge platforms (Raspberry Pi 4, smartphones) show substantial decreases in latency and energy, with up to $79\%$ reduction in per-token energy usage at negligible accuracy loss up to moderate quantization levels (Husom et al., 4 Apr 2025).
Memory-Aware Offloading: FlexInfer uses asynchronous prefetching to overlap I/O and compute, balanced memory locking across layers, and dynamic tensor preservation (caching critical weights, choosing attention/FFN tensors per available memory) to accommodate large models on devices with less RAM than required for the full weight set. Under tight constraints, FlexInfer achieves $4.2$– $12.5\times$ faster throughput compared to mmap or synchronous approaches (Du et al., 4 Mar 2025).
Chiplet and NPU/Flash Hybridization: Cambricon-LLM leverages a chiplet design with a Neural Processing Unit (NPU) and a dedicated NAND flash chip capable of in-flash GeMV computation and integrated error correction (ECC). Weights are tiled onto the flash per its hardware granularity, and only the KV cache and critical intermediates reside in DRAM. On-die ECC protects outlier weights most critical for inference robustness. The system achieves $3.44$ tokens/s for 70B models—exceeding flash-offloading baselines by $22$– $45\times$ (Yu et al., 24 Sep 2024).
Memory Bandwidth Utilization (MBU): The ELIB benchmarking suite formalizes MBU as: $\text{MBU} = \frac{\text{Achieved Memory Bandwidth}}{\text{Peak Memory Bandwidth}}$ where Achieved Memory Bandwidth is computed as $(\text{Model Param Size} + \text{KV Cache Size}) / \text{TPOT}$ . Maximizing MBU directly correlates with improved throughput and efficiency, guiding quantization and prefetching strategies on various edge hardware (Chen et al., 15 Aug 2025).

3. Hardware Acceleration and Operator Scheduling

CPU, GPU, and NPU Scheduling: Despite common assumptions, CPU-based inference with multi-threaded, graph-level parallel scheduling can outperform GPUs for small GEMMs in certain settings (e.g., iPhone 15 Pro, llama.cpp F16, 2-thread CPU at $17$ tokens/sec vs. Metal GPU at $12.8$ tokens/sec). Bottlenecks arise from GPU kernel launch and memory transfer overhead for small batch sizes, while CPU parallelism can exploit compute graph independence among matrix multiplications (Zhang et al., 9 May 2025).
NPU Offloading: LLM.npu demonstrates multi-level graph optimization: prompt chunking decouples static from dynamic operations for prompt prefill, quantized tensor-level "shadow outlier execution" offloads dense int8 computation to the NPU while processing rare large activations with the CPU/GPU, and out-of-order block scheduling reduces bubble-induced stall. The approach yields up to $22.4\times$ faster prefill and $1000+$ tokens/s for billion-parameter models (Xu et al., 8 Jul 2024).
Browser-Native Inference: WebLLM compiles models via MLC-LLM and Apache TVM for execution in JavaScript/WebGPU/WebAssembly environments, supporting fusion (e.g., kernel fusion, GEMM tiling), and achieves $71$– $80\%$ of native throughput in browser contexts, paving the way for completely local, privacy-preserving web LLM agents (Ruan et al., 20 Dec 2024).

4. Security, Privacy, and Collaborative Scheduling

Privacy Controls: On-device inference provides robust privacy by retaining all user data locally, avoiding network transmission and potential breach or surveillance (Çöplü et al., 2023, Fassold, 24 Apr 2024). However, intermediate values (e.g., KV caches) in GPU memory can leak conversation content.
KV-Shield: Protects against KV leakage by permuting the linear weights in attention layers, ensuring the cache stored on insecure accelerators is randomly permuted. Only the trusted execution environment (TEE) knows the permutation, and applies its inverse before emitting outputs, preserving correctness:

$\text{(1) }\quad \text{Weight}^{p} = \text{Weight} \cdot \text{RPM}; \qquad \text{(2) } \{q, k, v\}^{p} = x \cdot \text{Weight}^{p} = \{q, k, v\} \cdot \text{RPM};$

$\text{(3) } a^{p} = \text{Softmax}(q^{p} K^{p{\top}} / \sqrt{d_k}) V^{p}; \qquad \text{(4) } a = a^{p} \cdot \text{RPM}^{\top}$

Experimental results show significantly lower overhead than fully homomorphic encryption or TEE-only inference (Yang et al., 6 Sep 2024).

Collaborative and Distributed Scheduling: DiSCo dynamically migrates inference between device and server at the token level using a cost-aware scheduler. Migration transfers only token IDs, not caches, minimizing bandwidth and ensuring smooth time-between-token (TBT) even across endpoints. This scheme reduces mean TTFT by up to $78\%$ and serving costs by $84\%$ vs. monolithic deployments (Sun et al., 17 Feb 2025). For large models, tensor parallelism with over-the-air computation or AirComp allows aggregation of intermediate values directly via the wireless channel's superposition property. Mixed-timescale stochastic optimization (semidefinite relaxation + stochastic SCA) minimizes mean-squared transmission error under device-specific power constraints, delivering up to $5\times$ faster generation on multiple devices (Zhang et al., 18 Feb 2025, Zhang et al., 19 Mar 2025).

5. Customization, Personalization, and On-Device Fine-Tuning

Parameter-Efficient Fine-Tuning: On-device, real-time adaptation is enabled with gradient-free zeroth-order methods (e.g., P-RGE) that use randomized perturbations with only forward passes, updated in parallel outer and inner loops. Coupled with LoRA-FA adapters, this approach achieves $4.3\times$ training speedup and up to $9.87\%$ fine-tuning accuracy improvements over full-parameter ZO baselines, with low memory usage (<$2$ GB) and no backprop required (Gao et al., 23 Sep 2024).
Adapter Blending for User Customization: The Crayon approach constructs a pool of LoRA adapters from clustered query embeddings, then instantly blends adapters using cosine similarities from a few user examples. Device–server hybrid inference routes out-of-domain or uncertain queries to a server model only as needed, combining user privacy, low latency, and high customization accuracy (Bang et al., 11 Jun 2024).

6. Benchmarking, Performance Modeling, and Deployment Insights

Comprehensive Benchmarking: ELIB, a modular benchmarking tool, systematically benchmarks quantized LLMs on diverse edge platforms (IoT, mobile, PC) and reports FLOPS, throughput, latency (TTFT, TTLM), MBU, and accuracy (perplexity) (Chen et al., 15 Aug 2025). High MBU/throughput is achieved by matching quantization strategy with optimal acceleration framework (e.g., q4_0 + OpenBLAS/Metal).
Hybrid Analytical Modeling: LIFE (LLM Inference Forecast Engine) abstracts operator-level compute and memory requirements, forecasting TTFT, TPOT, and TPS by combining modeled workload with hardware characteristics (TOPS, bandwidth), operator efficiency, and software-level optimizations (quantization, KV compression, LoRA, operator fusion). LIFE supports performance prediction and optimization across CPUs, NPUs, iGPUs, and GPUs, without hardware or dataset-specific benchmarks (Patwari et al., 29 Jul 2025).
Energy Efficiency: Integrated measurements on hardware like Raspberry Pi with high-resolution tools (e.g., Joulescope) reveal that post-training quantization with PTQ and weight-only low-bit representations can yield up to $79\%$ reduction in energy per token, with careful Pareto analysis identifying quantization levels that optimally balance speed, energy, and accuracy (Husom et al., 4 Apr 2025).

The field of on-device LLM inference is driving algorithmic, hardware, and systems research across speculative decoding, quantization, hybrid architectures, security, adaptation, collaborative scheduling, and comprehensive benchmarking. Key innovations such as tree-structured speculative decoding, adapter blending, asynchronous offloading, and on-die computing enable efficient, scalable, and privacy-preserving LLM deployment on a broad spectrum of edge and client devices.