Mobile LLMs: On-Device AI & Deployment
- Mobile LLMs are AI models specifically engineered for mobile devices, balancing compute constraints and performance through on-device, cloud, or hybrid deployments.
- They employ compact architectures, quantization, and adaptation strategies that yield measurable gains in accuracy and efficiency under latency, battery, and memory limits.
- Deployment frameworks span on-device, cloud, and mobile-edge collaborations, enabling applications such as privacy-sensitive assistance, real-time translation, and smart recommendations.
Mobile LLMs are LLMs designed to run directly on mobile devices or in mobile-edge collaborative settings, bringing capabilities such as conversational AI, summarization, recommendation, and personalized assistants to smartphones, tablets, wearables, and edge-connected mobile systems (Xu et al., 2 Sep 2025). In the recent literature, the topic is defined less by model scale alone than by deployment constraints: latency, memory, battery, privacy, bandwidth, and hardware heterogeneity shape both architecture and systems design. The resulting field spans compact sub-billion models for on-device use, 3B-class interactive assistants on phones, mobile multimodal LLMs, elastic serving systems, mobile benchmarks, and privacy- and edge-oriented deployment frameworks (Liu et al., 2024, Fassold, 2024, Qu et al., 2024).
1. Deployment rationale and system models
The principal motivation for mobile LLMs is to move inference closer to the user. Survey and systems papers consistently describe the advantages of on-device execution as lower latency, better offline robustness, improved privacy and security, reduced dependence on cloud connectivity, and more reliable user experience (Liu et al., 2024). Concrete use cases include privacy-sensitive healthcare and recommendation, smart home control, private voice interactions, and question answering in low-connectivity or monitored environments; one smartphone QA study explicitly highlights journalists who require local answering on the device without cloud transmission (Fassold, 2024).
This motivation does not imply a single deployment architecture. The literature distinguishes three broad modes. On-device inference is preferred for privacy-sensitive tasks, offline use cases, and low-latency interaction. Cloud-based inference remains the baseline for larger models, but is associated with latency, network dependency, and privacy risks from data transmission. Edge/cloud collaborative inference or mobile-edge computing is presented as an intermediate strategy in which mobile devices handle lightweight or privacy-sensitive computation locally while edge servers execute heavier components (Liu et al., 2024). The mobile-edge survey formalizes this broader perspective as MEI4LLM, with four core components: AI-native architecture, parameter-sharing LLM caching and delivery, distributed LLM training or fine-tuning, and distributed LLM inference (Qu et al., 2024).
The deployment rationale is therefore twofold. First, mobile devices are increasingly expected to be active hosts of intelligence rather than mere terminals for cloud intelligence (Liu et al., 2024). Second, purely local execution is constrained by compute scarcity, storage and memory scarcity, and energy limits, which motivates hybridization with edge servers, compression, and architecture/runtime co-design (Qu et al., 2024).
2. Model architectures and adaptation strategies
A central finding of the mobile LLM literature is that compact models for phones are not merely smaller versions of datacenter LLMs. In MobileLLM, the design target is explicitly “fewer than a billion parameters,” and the paper argues that at sub-billion scale architecture matters more than the conventional emphasis on data and parameter count. The proposed recipe combines deep-and-thin transformers, SwiGLU, input-output embedding sharing, grouped-query attention, and immediate block-wise weight sharing. Reported gains are substantial: MobileLLM attains a 2.7%/4.3% accuracy boost over preceding 125M/350M state-of-the-art models, and MobileLLM-LS adds a further 0.7%/0.8% over MobileLLM 125M/350M (Liu et al., 2024). This directly challenges the misconception that scaling parameter count alone determines mobile usefulness.
The same architecture-first logic appears in multimodal mobile deployment. GenieBlue begins from the observation that representative mobile MLLMs such as Qwen2.5-VL-3B, InternVL2.5-4B, and Wings-Qwen1.5 show clear drops on text-only benchmarks relative to their base LLMs, with more than 10% degradation on MATH, AlignBench, and MT-Bench for several models (Lu et al., 8 Mar 2025). The paper attributes part of the difficulty to a hardware mismatch: architectures that preserve language ability often rely on MoE routing, yet the MediaTek and Qualcomm NPUs discussed in the paper do not support MoE deployment. GenieBlue therefore freezes the original LLM parameters during multimodal training, duplicates selected transformer blocks for full fine-tuning, and inserts LoRA modules into the remaining blocks. Under the non-shared-base strategy, the paper reports 100% of the original LLM performance on pure language tasks while retaining 97% of BlueLM-V-3B’s multimodal performance, with deployment on the iQOO 13 with Qualcomm Snapdragon 8 Elite (Gen 4) through the QNN SDK using W8A16 for the ViT and projector, W4A16 for the LLM, and W8A16 for LoRA parameters (Lu et al., 8 Mar 2025).
Specialized mobile applications adopt further adaptation strategies. RecGPT-Mobile uses a lightweight LLM-based intent understanding agent rather than a general chatbot, deploys Qwen3-0.6B-Quant, applies LoRA and quantized LoRA, and triggers inference only when intent drift exceeds a threshold (Zhang et al., 6 May 2026). MAPLE reformulates next-app prediction as a text-to-text problem by converting mobile context into contextual sentences processed by T5 or BART, with installed-app similarity used to address cold start (Khaokaew et al., 2023). Together, these works suggest that mobile LLM architectures are frequently specialized around a narrow use case—assistant behavior, multimodal perception, recommendation, or behavioral prediction—rather than deployed as unrestricted general-purpose models.
3. Runtime systems, quantization, and hardware-aware inference
Mobile LLM deployment depends as much on runtime engineering as on model architecture. A representative end-to-end smartphone deployment is “Porting LLMs to Mobile Devices for Question Answering”, which uses llama.cpp rather than the pipeline
because the authors found the TFLite route cumbersome, brittle, and dependent on legacy TensorFlow 1.x components (Fassold, 2024). Their selected model is Orca-Mini-3B, with 3 billion parameters, quantized to about 5.6 bits per parameter (described as a 6-bit quantized version), requiring approximately 2.2 GB of CPU RAM on-device. The implementation uses Termux, wget, git, cmake, and clang, builds the latest llama.cpp source with CMake and make, runs inference on the CPU only, and exposes both an interactive chat binary and a server application with a REST API similar to the OpenAI API. On a Samsung Galaxy S21, the paper reports interactive speed and qualitative answer quality across politics, geography, history, and other general knowledge questions (Fassold, 2024).
For instruct-style behavior, prompt fidelity matters. The same deployment paper specifies the exact prompt format for Orca-Mini-3B:
1 2 3 |
### System: You are an AI assistant that follows instruction extremely well. Help as much as you can. ### User: What is the smallest state in India ? ### Response: |
This implementation detail is treated as necessary for answer quality because the model must be prompted in the same format used during finetuning (Fassold, 2024).
A broader systems picture emerges from dedicated mobile engines.
| System | Core mechanism | Representative result |
|---|---|---|
| Transformer-Lite | dynamic shapes, operator optimization, M0E4 FP4, KV sub-tensors | 330 token/s prefill and 30 token/s decode for Gemma 2B |
| ELMS | model elasticity + prompt elasticity | less than 1% TTFT switching overhead |
| MNN-LLM | quantization + DRAM-Flash hybrid storage | up to 8.6x speed increase |
Transformer-Lite targets mobile phone GPUs on Qualcomm Adreno and MediaTek/ARM Mali through OpenCL. Its four optimizations are symbolic expression-based dynamic shape inference, operator optimizations and execution priority scheduling, the FP4 quantization method M0E4, and a sub-tensor KV cache strategy. Reported examples include 121 token/s prefill and 14 token/s decoding for ChatGLM2 6B, and 330 token/s prefill and 30 token/s decoding for Gemma 2B, with over 10x speedup for the prefill speed and 2~3x speedup for the decoding speed relative to CPU-based FastLLM and GPU-based MLC-LLM (Li et al., 2024).
ELMS reframes mobile inference as a service-level objective problem. Its key latency relation is
To satisfy per-request latency targets, ELMS combines a one-shot neuron reordering technique that exploits permutation-consistent units with a dual-head Tiny LLM for prompt compression and orchestration. On synthesized end-to-end traces and standalone datasets, the paper reports absolute accuracy improvements of up to 16.83% and 11.04% on average, with less than 1% Time-To-First-Token switching overhead, comparable memory usage, and fewer than 100 offline GPU hours (Yin et al., 2024).
MNN-LLM addresses the memory bottleneck directly through DRAM-Flash hybrid storage, asymmetric quantization, hardware-aware data reordering, multicore load balancing, mixed float precision, geometry-based operator fusion, and LoRA-efficient execution. For Qwen2 7B, the paper notes that Embedding parameters are about 15% of the total, making them attractive for Flash storage, and states that storing Embedding in Flash adds only about 1.4% total inference overhead while reducing DRAM usage by about 15% (Wang et al., 12 Jun 2025). This system-level emphasis reflects a recurring theme: on phones, LLM decode is often dominated by memory movement rather than arithmetic throughput.
4. Interaction paradigms and application domains
The application space of mobile LLMs is broad, but the literature converges on a small set of recurring domains: voice assistants, real-time translation, intelligent/personalized recommendations, augmented reality, smart home, and healthcare (Liu et al., 2024). These domains are united by low-latency interaction, personal context, and privacy-sensitive data.
Several papers propose mobile-native interaction techniques rather than transplanting desktop chat interfaces to phones. “Exploring Mobile Touch Interaction with LLMs” replaces prompt entry with continuous two-finger gestures performed directly on text. The implemented mappings are spread-to-generate and pinch-to-shorten, with the explicit operational mapping
A within-subject study with 14 participants compared NoVis, Lines, and Bubbles feedback. The paper reports overall completion times of 16.38 s, 16.30 s, and 14.41 s, SUS scores of 63.46, 76.96, and 85.54, and NASA-TLX scores of 2.794, 2.154, and 1.976, respectively. In a second comparison, direct gesture interaction outperformed a chatbot-like UI, with 56.35 s versus 134.86 s, SUS 81 versus 52.5, and NASA-TLX 2.06 versus 3.15 (Zindulka et al., 11 Feb 2025). These results indicate that mobile LLM usability depends not only on model quality but also on interaction design and feedback loops that mask generation latency.
Assistant-like applications remain prominent. “Revolutionizing Mobile Interaction: Enabling a 3 Billion Parameter GPT LLM on Mobile” deploys RedPajama-INCITE-Chat-3B-v1 as an Android assistant with text-to-actions. The model is fine-tuned with LoRA, PEFT, and bitsandbytes, converted to GGML, quantized to 4-bit q4_0, and executed through native C++ via the Android NDK. The size reduction is from 5.17 GB in 16-bit to 1.6 GB in 4-bit, and the paper states that the system runs on phones with as little as 4 GB RAM, while 6 GB RAM gives reasonable performance (Carreira et al., 2023). The application can emit structured actions such as <call>John<call>, <search>Highest building in the world<search>, and <calendar\>2023-05-20T09:00:00/Meeting<calendar> (Carreira et al., 2023).
Mobile UI understanding is another major branch. “Enabling Conversational Interaction with Mobile UI using LLMs” converts Android view hierarchies into HTML-like text and uses few-shot prompting with PaLM for four tasks: Screen Question-Generation, Screen Summarization, Screen Question-Answering, and Mapping Instruction to UI Action. The method is competitive without dedicated datasets and training, and the reported results include, for Screen QA, 66.7% exact match and 64.8% micro-F1 in the 2-shot setting, compared with 36.0% exact match and 37.2% micro-F1 for DistilBERT (Wang et al., 2022).
Testing and recommendation are also active application areas. In Android GUI testing, nine LLMs were evaluated for text-input generation across 114 UI pages from 62 open-source Android apps, with page-pass-through rate ranging from 50.58% to 66.67% among stronger models; GPT-4 reached 66.67% (228/342) and GPT-3.5 63.45% (217/342), while invalid-input bug detection rates remained less than 23% (Cui et al., 2024). In production recommendation, RecGPT-Mobile ran for one month on four mobile Taobao feed scenarios with tens of millions of users, using Qwen3-0.6B-Quant, and reported average online improvements of +1.8% CLICK, +2.7% PAY, and +2.5% GMV (Zhang et al., 6 May 2026).
5. Benchmarking and empirical performance
The field’s evaluation problem is explicit: conventional server-oriented benchmarks do not capture mobile information needs or mobile constraints. Mobile-MMLU addresses this gap with 16,186 questions across 80 mobile-related fields, 27 subcategories, and 9 major categories, while Mobile-MMLU-Pro provides a harder subset of 9,497 questions (Bsharat et al., 26 Mar 2025). The benchmark uses multiple-choice, order-invariant questions centered on realistic mobile scenarios such as recipe suggestions, travel planning, and essential daily tasks. Its Mobile Relevance Score is reported as 5.88 for Mobile-MMLU, versus 3.13 for MMLU and 3.00 for MMLU-Pro, and the paper emphasizes mobile-specific dimensions including inference latency, energy consumption, memory usage, response quality, privacy, on-device processing, and personalization (Bsharat et al., 26 Mar 2025). This benchmark institutionalizes the argument that “mobile intelligence” is a distinct evaluation target.
Hardware studies reinforce that benchmark accuracy alone is insufficient. “Understanding LLMs in Your Pockets: Performance Study on COTS Mobile Devices” evaluates local 7B inference across six devices and three SoC vendors using llama.cpp and MLC LLM, with additional NPU analysis on Snapdragon-class devices (Xiao et al., 2024). On CPU, Dimensity 9300 is reported as best overall, achieving >3× prefill speedup and nearly 5× decode speedup relative to Snapdragon 870, while Snapdragon 8 Gen 3 reaches about 80% of Dimensity 9300 throughput (Xiao et al., 2024). Memory footprint is approximately 3.8 GB for llama.cpp across devices and around 4.2 GB to 4.4 GB on GPUs (Xiao et al., 2024).
A notable empirical result is that mobile GPUs do not automatically outperform CPUs. The study finds that MLC LLM on GPUs can perform worse in prefill than llama.cpp on CPUs, and that Adreno 750 is 1.6× faster than Mali-G720 for decoding while Mali shows <3% arithmetic unit utilization during prefill (Xiao et al., 2024). By contrast, the Hexagon NPU on Snapdragon 8 Gen 3 reaches 690 tokens/s prefill, about 50× faster than CPU and GPU solutions, but decode improves only slightly because decode remains memory-bound (Xiao et al., 2024). Sustained performance is also unstable: on Snapdragon 8 Gen 3, throughput falls from 12.7 tok/s to 8.9 tok/s over 20 continuous inference rounds, and latency increases by up to 30% due to DVFS and thermal effects (Xiao et al., 2024).
These studies collectively show that mobile LLM evaluation must be multidimensional. Accuracy, TTFT, TPOT, energy, memory footprint, cold-start latency, runtime portability, DVFS behavior, and accelerator utilization all materially affect usability on phones.
6. Limitations, security, and future directions
The current literature is candid about limitations. Smartphone QA deployments often rely on qualitative rather than benchmark-heavy evaluation and may omit exact latency, tokens-per-second, energy, and benchmark accuracy metrics (Fassold, 2024). Mobile assistants with action tags remain brittle: quantized models can choose the wrong action tag or emit malformed outputs such as <call>John Castro<calendar> (Carreira et al., 2023). In GUI testing, the average PPTR is only 44.87%, and direct invalid-input generation is insufficient for robust bug detection (Cui et al., 2024). These are not marginal caveats; they define the present maturity level of the field.
Privacy and security introduce a second layer of constraints. The dedicated survey on trustworthy mobile LLMs categorizes data anonymization, prompt encryption, differential privacy, federated learning, secure aggregation, noise injection, local processing, and knowledge distillation as major privacy-preserving methods, and identifies adversarial attacks, membership inference, model inversion, and side-channel attacks as key threats (Xu et al., 2 Sep 2025). Because mobile devices process location, contacts, messages, browsing history, health data, financial records, and voice commands, security failures are amplified by the sensitivity of the data (Xu et al., 2 Sep 2025).
Future work is correspondingly system-wide. Specific deployment papers mention phi-2 and GPU acceleration via OpenCL or Vulkan as next steps for smartphone QA (Fassold, 2024). GenieBlue notes that more SoC platforms must be tested beyond its current Snapdragon 8 Elite deployment (Lu et al., 8 Mar 2025). The mobile-edge survey emphasizes green edge LLM, secure edge LLM, and joint optimization across caching, training, and inference, alongside improved support for sparse models, mixed precision, parameter sharing, and distributed execution (Qu et al., 2024).
Taken together, the literature suggests that mobile LLMs are best understood as a co-design problem. Model architecture, quantization, runtime layout, cache management, interaction design, benchmark construction, privacy mechanisms, and edge collaboration are all first-order determinants of performance. The practical significance of the field lies precisely in this convergence: a mobile LLM is not merely a compressed LLM on a smaller computer, but a language system engineered for the latency, memory, energy, privacy, and interface realities of personal devices.
