- The paper demonstrates an end-to-end RAG pipeline running embedding, reranking, and generation on the Snapdragon X Elite NPU, delivering 9.1× faster embedding, 18.1× faster LLM prefilling, and 4.0× lower query energy than the CPU baseline.
- The system reduces indexing energy by 12.3× and query energy by 4.0×, while the integrated GPU performs worse than the CPU for this workload despite active OpenCL offload.
- The paper finds answer quality remains broadly comparable across backends, but static NPU graphs impose smaller chunks, limited context, strict model-loading order, and platform-specific deployment constraints.
Overview and motivation
This paper presents an end-to-end Retrieval-Augmented Generation (RAG) system in which every neural inference stage—embedding generation, cross-encoder reranking, and LLM answer generation—executes on the Qualcomm Hexagon NPU of the Snapdragon X Elite, orchestrated through the Qualcomm AI Runtime (QAIRT/QNN) SDK. The authors argue, plausibly, that no prior work has benchmarked a complete multi-model RAG pipeline on a mobile NPU; the closest predecessor, LLM.npu (Seo et al., 18 Feb 2025), addresses only single-model LLM prefilling. The gap matters for two reasons the paper identifies: a RAG pipeline co-resident models create memory-allocation and scheduling challenges under the NPU's static computation graph model, and energy savings compound across pipeline stages in ways that have not been quantified.
The evaluation platform is a Dell XPS 13 9345 with the Snapdragon X Elite X1E80100 (12 Oryon cores), Adreno X1-85 integrated GPU, Hexagon NPU rated at 45 TOPS INT8, and 64 GB LPDDR5x. Power is measured via HWiNFO64 shared-memory telemetry at 500 ms granularity, with total energy obtained by numerical integration of instantaneous power. The pipeline uses EmbeddingGemma 300M for embeddings, Jina Reranker v2 Base Multilingual as cross-encoder, and Qwen3-4B-Instruct as generator; CPU/GPU baselines run GGUF-quantized equivalents through llama.cpp behind a common backend interface. Notably, the reranker remains on the NPU in all configurations because the OpenCL backend exhibits a batched-scoring defect—this isolates the LLM/embedder as the only varying component but also means the GPU configuration is not fully end-to-end OpenCL.
System design and engineering constraints
The indexing pipeline parses documents into 1,000-character chunks with 500-character overlap (constrained by NPU context limits; a CPU-only deployment would use 2,500/1,250), embeds them into 1,024-dimensional vectors in batches of up to 32, and builds a FAISS IndexFlatL2 index over a SQLite/FTS5 store. The query pipeline performs optional LLM query rewriting, hybrid retrieval fusing FTS5 keyword results with FAISS semantic results via Reciprocal Rank Fusion (k=60), NPU reranking with score-threshold and cliff-cutoff filtering to top-7 chunks, and streaming LLM generation within a fixed 4,096-token context window.
Three engineering findings are documented that will be immediately familiar to anyone deploying on current NPUs:
- Model loading order: models must be loaded largest-first (LLM, then embedding, then reranker); violating this order causes static-graph memory allocation failures, attributed to contiguous memory allocation strategy.
- Static context lengths: ahead-of-time compilation fixes maximum context length, forcing smaller chunks and fewer retrieved passages than CPU/GPU deployments.
- Windows ARM64 ecosystem immaturity: many Python packages lack ARM64 Windows wheels, requiring manual compilation.
These constraints are honest limitations of the current QAIRT/QNN stack rather than fundamental hardware properties, and they directly shape the quality analysis discussed below.
On indexing (10 SEC 10-K filings, 9,324 chunks, ~2.04M tokens), the NPU achieves 3,325 tokens/s embedding throughput versus 367 tokens/s on CPU—a 9.1× speedup—yielding a 7.9× total-pipeline speedup (710.7 s vs. 5,648.2 s). Total system energy drops from 241 kJ to 19.6 kJ, a 12.3× reduction, aided by a super-linear effect: average system power falls from 42.7 W to 27.6 W because CPU clusters sit near idle during NPU execution.
Query results on the wiki_minirag benchmark (120 queries over 3,187 Wikipedia passages) are summarized below:
| Metric |
NPU |
CPU |
GPU/OpenCL |
NPU/CPU |
| LLM TTFT (s) |
1.30 |
24.77 |
42.24 |
19.1× |
| Prefill speed (tok/s) |
786.7 |
43.4 |
25.2 |
18.1× |
| Decode speed (tok/s) |
14.2 |
8.2 |
4.7 |
1.74× |
| Total query time (s) |
9.48 |
37.98 |
63.61 |
4.0× |
| P95 latency (s) |
17.90 |
69.56 |
107.40 |
3.9× |
| Total energy (kJ) |
37.83 |
150.12 |
246.14 |
4.0× |
The prefilling result is the strongest claim in the paper: 18.1× over CPU and 31.2× over GPU, consistent with the NPU's advantage on dense matrix-multiply-dominated compute. Decoding shows only 1.74× improvement, which the authors correctly attribute to the memory-bandwidth-bound, sequential nature of autoregressive decoding—an important nuance, since it means NPU benefits are concentrated in TTFT rather than steady-state token throughput.
The most contrarian finding concerns the integrated GPU. Despite sustaining 71.2% D3D utilization (confirming functional OpenCL offload), the Adreno X1-85 is 1.7× slower than the CPU and consumes 6.5× more energy than the NPU on this workload. The authors characterize this as a hardware ceiling—the small iGPU versus 12 high-performance CPU cores—not a runtime defect. This directly contradicts the conventional x86 intuition that GPU offload is the default acceleration path, and practitioners porting assumptions from discrete-GPU or Apple-silicon settings should take note.
Energy analysis
A key methodological observation is that average system power is essentially identical across backends during query processing (33.24, 32.94, 32.25 W)—the platform's idle baseline dominates the system rail. Consequently, the query-phase energy ratio tracks the wall-clock ratio almost exactly. The super-linear energy effect appears only in the compute-dense indexing workload, where NPU offload reduces average system power by 35%. Per-query energy is 315 J (NPU), 1,251 J (CPU), and 2,051 J (GPU). At 1,000 queries/day, the authors project ~94.9 kWh/year saved per device relative to CPU. This projection assumes sustained query volume at that rate on a single device, which is an assumption about usage patterns rather than a measured quantity.
Answer quality
Because the NPU configuration is deliberately constrained (smaller chunks, top-7 vs. top-10 retrieval, 4,096-token context), quality regression is a legitimate concern. GPT-4.1 LLM-as-judge scoring (temperature 0.0, 1–10 rubric, judge blind to backend) on wiki_minirag yields mean scores of 9.32 (NPU), 8.95 (CPU), and 9.03 (GPU), with 86.7% of queries scored identically across all three backends. Paired exact Wilcoxon signed-rank tests give p=0.039 for NPU−CPU, p=0.121 for NPU−GPU, and p=0.313 for CPU−GPU. The nominally significant NPU−CPU difference favoring the NPU is, per manual spot-checks, dominated by judge sensitivity: nearly identical answers receive scores differing by 6 points, and reference-answer errors in the dataset cause correct direct answers to be scored 1. The paper's conclusion—parity within evaluator noise—is reasonable given this evidence, though it rests on a single judge with no human annotation, a caveat the authors state explicitly.
On FinDER financial QA (heavily abbreviated queries against a tiny 10-document corpus), both backends predominantly refuse rather than fabricate: the NPU returns substantive answers for 9.2% of queries versus 44.2% for CPU, with comparable mean quality among substantive answers (6.45 vs. 7.47). The paper attributes the substantive-rate gap to the CPU's larger context window admitting borderline-relevant passages—not a quality difference—which is consistent but not independently verified.
Limitations
The paper concedes several limitations plainly. Static computation graphs prevent dynamic context expansion, capping chunk size and assembled context. The largest-first loading requirement reflects current NPU memory management. Evaluation is single-device (one Snapdragon X Elite SKU), so specific speedup and energy ratios are platform-dependent even if the qualitative finding generalizes. The quality evaluation uses one LLM judge without human grading, leaving open whether residual score differences reflect judge bias. Additionally, the GPU baseline cannot execute the reranker due to an OpenCL batched-scoring defect, so the GPU comparison is not a fully homogeneous offload; and the FinDER refusal-rate comparison conflates backend with context-window configuration. An open question the paper leaves unanswered is whether dynamic-shape NPU runtimes would eliminate the context-length penalty without sacrificing the measured efficiency gains.
Conclusion
This paper delivers the first documented end-to-end NPU-resident RAG pipeline, with careful controlled baselines and fine-grained power telemetry. Its headline numbers—9.1× embedding throughput, 18.1× prefilling speedup, 12.3× and 4.0× system-energy reductions for indexing and querying respectively—are substantial, and the finding that the integrated GPU is worse than the CPU on this workload is a practically useful corrective. Quality parity holds within evaluator noise despite the NPU's constrained context budget. The main caveats are single-platform evaluation, static-graph context limits, and a single-judge quality assessment; generalization to other NPU architectures (Apple Neural Engine, Intel NPU, MediaTek APU) remains asserted rather than demonstrated.
