Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU

Abstract: Running LLMs in the browser presents a unique opportunity to build efficient, private, and portable AI applications, but requires contending with constrained memory availability and heterogeneous hardware targets. To realize this opportunity, we present Llamas on the Web (LlamaWeb), a WebGPU backend for llama$.$cpp that enables memory-efficient and performance-portable LLM inference across a wide range of model weight formats in the browser. Our design significantly reduces memory overhead through static memory planning and efficient model loading, addresses cross-device variability through a tunable kernel library, and introduces templated GPU kernels that support performant implementations of numerous quantization formats, enabling broad model support and extensibility to new formats. We evaluate LlamaWeb on 16 devices from 8 vendors, collecting data from 10 LLMs and four model weight formats. We compare LlamaWeb against existing browser-based LLM frameworks and find that LlamaWeb requires 29-33% less memory across several combinations of device, browser, and operating system. We also evaluate LlamaWeb's performance against these frameworks and find that it increases decode throughput by 45-69% across four GPUs from separate vendors. In addition, we compare LlamaWeb's performance against other llama$.$cpp backends, where it is competitive with and even beats vendor-specific backend performance on some devices.

• The paper introduces LlamaWeb, achieving memory-efficient LLM inference through static memory planning and zero-copy buffer management.
• It implements performance-portable kernels using empirical device tuning and cached pipeline compilation to optimize matrix operations across diverse hardware.
• The work supports 21 quantization schemes via kernel co-design, enabling accurate, multi-precision inference in browser environments.

LlamaWeb: Portable, Memory-Efficient, and Multi-Precision LLM Inference via WebGPU

Introduction

The proliferation of consumer-grade and edge devices necessitates efficient, private, and locally executable LLMs. Browser-based inference offers an accessible, cross-platform avenue for deploying LLMs without device-specific packaging or user installation friction. However, the constraints of browser execution—including memory limitations, device heterogeneity, and the need for robust quantization support—pose significant systems and performance challenges. “Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU” (2605.20706) addresses these constraints by introducing LlamaWeb, an open-source WebGPU backend for the highly-adopted llama.cpp inference engine. The paper rigorously details technical innovations in memory management, performance-portable kernel design, and quantization-aware GPU computation.

System Design

Static Memory Planning and Model Loading

LlamaWeb statically allocates all required runtime memory at initialization, including tensor data, intermediate buffers for operations such as FlashAttention/FlashDecoding, and a fixed-size parameter buffer “arena” for runtime kernel dispatch variables. This static allocation eliminates the dynamic buffer fragmentation and unpredictable allocation behavior seen in frameworks like WebLLM and Transformers.js, and is critical for conforming to strict per-tab browser memory ceilings, especially on mobile-class systems (e.g., iOS/Safari). The model loading process is optimized to minimize redundant memory materialization by streaming model weights directly from persistent browser storage (OPFS) to WebGPU buffers, leveraging non-blocking asynchronous interfaces to avoid excess allocations in the WASM heap—a crucial consideration given its unreclaimable, grow-only semantics.

Kernel Library and Specialization Framework

Recognizing the extensive hardware and software variability across WebGPU-enabled devices, LlamaWeb factors the kernel library from runtime orchestration. All compute-intensive operations, including matrix multiplication, normalization, activation functions, rotary position embeddings, and FlashAttention variants, are implemented as templated WGSL shaders. The kernel dispatching infrastructure empirically specializes device-tunable parameters (e.g., tiling granularity, workgroup/vector widths, subgroup usage) according to detected hardware features, providing both performance portability and extensibility. Kernel code generation and pipeline compilation are cached using compile-time configuration keys, allowing amortization of specialization overhead after the first forward pass.

Quantization-Aware Computation

Unlike prior browser-based inference solutions that either restrict quantization format diversity or treat kernel-level dequantization simplistically, LlamaWeb generalizes across 21 quantization schemes supported by llama.cpp with a kernel co-design approach. Quantized model weights are treated as flat u32 buffers, with dequantization routines chosen at compile time and fused into the main compute kernels (e.g., for matmul or matrix-vector multiplication). This approach supports both legacy (q4_0/q8_0), blockwise (K-quants), vector-quantized (I-quants), and ultra-low-bitwidth (q1_0) schemes, with manual interventions only required in the addition of new dequantization routines.

Experimental Results

Benchmark Suite and Device Coverage

The LlamaWeb evaluation utilizes 10 LLMs (spanning transformers, SSMs, and 1-bit architectures) and 16 devices across eight vendors (NVIDIA, AMD, Intel, Apple, ARM, Qualcomm, Samsung, Imagination Tech), encompassing both desktop and resource-constrained mobile hardware. Cross-comparisons are conducted against existing browser LLM frameworks (WebLLM, Transformers.js), browser-native backends, and system-native backends (CUDA, Vulkan, Metal, HIP, SYCL).

Memory Consumption

LlamaWeb achieves peak memory usage reductions of 29–33% over WebLLM and Transformers.js (geometric mean across device-browser-OS tuples), attributable to static allocation and zero-copy buffer management. On Safari/iOS (the most constrained class), LlamaWeb enables inference with state-of-the-art models that crash or run unsatisfactorily in baseline engines. Memory leaks were observed in both competing frameworks under certain configurations, but not in LlamaWeb.

Throughput and Performance-Portability

LlamaWeb’s median decode token-per-second (tok/s) throughput exceeds WebLLM by 54% and Transformers.js by 69% across four major GPUs. On certain device/backend combinations—particularly on Intel GPUs with the WebGPU backend—LlamaWeb surpasses even native system libraries. The portable kernel tuning approach enables this performance parity, despite browser-imposed runtime safety checks, with only a moderate (~14–23%) slowdown compared to unchecked native execution.

Perf-portability was quantitatively validated via k-means device clustering; LlamaWeb adapts to high-, mid-, and low-tier GPUs, delivering substantial absolute throughput (e.g., 3k tok/s prefill and 100 tok/s decode for smaller models on RTX 5080; 4–17 tok/s decode on mobile GPUs). Notably, in the prefill phase, LlamaWeb is outperformed by WebLLM and Transformers.js due to the latter’s application of kernel fusion and subgroup-matrix optimizations; LlamaWeb’s portable kernels do not currently exploit subgroup-matrix in browser, but do achieve competitive scores when enabled natively.

Quantization Format Scaling

Templated dequantization kernels enable near-optimal decode throughput for high-utility quantization formats (e.g., q4_k_m, q8_0). For low-bitwidth quantization (q2_k), the performance advantage diminishes and can even reverse due to increased dequantization overhead relative to the saved memory bandwidth; decode throughput increases 20–53% when moving from f16 to q8_0, but regresses by ~17% for q2_k on mid-tier GPUs. Prefill performance is relatively invariant to weight format. LlamaWeb’s quantization-aware design delivers uniform and predictable scaling as KV-cache grows, validating its robustness for long-context inference.

Practical and Theoretical Implications

Deployment and Ecosystem Impact

LlamaWeb extends the GGUF/open-weight LLM ecosystem to the browser in a memory- and performance-efficient manner. This enables accessible, private, and portable deployment of the substantial llama.cpp model zoo—including recent agentic and multilingual architectures—on any device with a modern browser. The browser’s ubiquity removes installation and configuration barriers, democratizing access to LLM technology and expanding privacy-respecting computation beyond server-centric architectures.

Theoretical Directions

The successes and bottlenecks observed suggest multiple avenues for future research:

• Dynamic Kernel Fusion: Integrating kernel fusion techniques, as done in TVM, to minimize kernel launch/dispatch overhead can yield substantial further improvements, especially in the prefill phase.
• Auto-tuning and Meta-learning for Performance Portability: Incorporation of meta-parameter optimization, drawing from auto-tuning frameworks, can refine device-specific performance further.
• Deeper Co-Design of Quantization and Hardware: The limits of portable quantization-aware kernels are set by the dequantization compute/memory tradeoff; deeper integration with subgroup and tensor-core hardware or evolving the quantization formats themselves for hardware-awareness is a critical area for both LLM and systems research.
• WebGPU Specification and Implementation: Addressing runtime safety check overheads, specialization for sub-16b integer types (u16/u8), and improved floating-point semantics will be important for the maturation of browser compute APIs as viable LLM deployment targets.

Conclusion

“Llamas on the Web” establishes LlamaWeb as a reference architecture for browser-based LLM inference, demonstrating that aggressive systems-level design—static memory planning, kernel specialization, and quantization-aware execution—enables significant gains in both efficiency and portability without sacrificing numerical correctness or extensibility. LlamaWeb sets a new operational baseline for quantized LLM deployment within browsers, and provides a flexible platform for future research in cross-device LLM optimization and quantized inference systems (2605.20706).