Llamafile: Lightweight Inference Runtime

• Llamafile is a lightweight, single-file LLM inference runtime that bundles model weights for energy-efficient and portable on-device execution on SBCs.
• It outperforms Ollama on 4 cores by achieving up to 3.8× faster token generation with lower memory usage and reduced power draw.
• Llamafile’s design simplifies deployment on constrained ARM hardware, though it shows limited scalability when additional cores are enabled.

Searching arXiv for the cited paper and related Llamafile context. Llamafile is a lightweight, experimental inference runtime that bundles LLM weights into portable, single-file executables. In the benchmark study "An Evaluation of LLMs Inference on Popular Single-board Computers" (Tung et al., 20 Oct 2025), it is examined as a runtime for on-device inference on single-board computers (SBCs), specifically in comparison with Ollama under constrained ARM hardware. Within that study, Llamafile is characterized by high single-threaded and low-core throughput, better memory efficiency, and lower power draw relative to Ollama, alongside limited scalability as additional threads or heterogeneous cores are enabled. The reported evidence is restricted to the Orange Pi 5 Pro because compatibility constraints prevented direct benchmarking on Raspberry Pi 4 and Raspberry Pi 5 in the authors’ experimental setup (Tung et al., 20 Oct 2025).

1. Runtime model and software characteristics

In the study, Llamafile is defined operationally by its packaging model: LLM weights are bundled into portable, single-file executables. The authors state that this design eliminates external dependencies, reduces startup latency, and lets the operating system directly manage optimization and memory usage during inference (Tung et al., 20 Oct 2025). That description situates Llamafile as a deployment-oriented runtime rather than merely a model-serving wrapper.

All evaluated models in the study were quantized using the q4_k_m scheme to balance memory efficiency and speed. Llamafile supported only a smaller subset of the 25 models compared to Ollama. The paper cites Llamafile [llamafile2024] as accessed May 9, 2025, but the experimental conclusions are not framed as a general evaluation of every supported Llamafile configuration; rather, they concern the subset that could be executed under the authors’ SBC test harness.

The runtime-level profile reported for Llamafile is asymmetric. Its strengths lie in high single-threaded and low-core throughput, lower runtime RAM demand in qualitative terms, and lower power draw relative to Ollama, especially on the Orange Pi 5 Pro. Its limitation is multithread scalability: performance peaks on four big cores and declines when additional cores are enabled. This suggests that, in the tested setting, Llamafile is best understood as favoring low-core, high-efficiency operation rather than maximum many-core scaling (Tung et al., 20 Oct 2025).

2. Experimental context on single-board computers

The benchmark spans three SBCs: Raspberry Pi 4, Raspberry Pi 5, and Orange Pi 5 Pro. Raspberry Pi 4 is described as a quad-core Cortex-A72 (ARMv8, 64-bit) system with 4GB LPDDR4-3200 running Raspberry Pi OS (64-bit Bookworm). Raspberry Pi 5 is a quad-core Cortex-A76 system with 8GB LPDDR4X-4267 running the same OS. Orange Pi 5 Pro is a Rockchip RK3588S octa-core big.LITTLE platform with four high-performance cores and four energy-efficient cores, 6GB LPDDR5, and Ubuntu Server 24.04 (64-bit) (Tung et al., 20 Oct 2025).

Llamafile was not benchmarked on Raspberry Pi 4 or Raspberry Pi 5 because of compatibility limitations in the experimental setup. Although the software discussion notes that Llamafile demonstrated strong performance potential on Pi 5 in principle, no measured Llamafile throughput, memory, or power results are reported for either Raspberry Pi board. Consequently, all direct evidence about Llamafile in the paper comes from the Orange Pi 5 Pro (Tung et al., 20 Oct 2025).

On the Orange Pi 5 Pro, Llamafile was evaluated under 4-, 6-, and 8-core configurations, with the 4-core configuration explicitly targeting only the four high-performance cores. Across all setups, CPU frequency scaling was fixed to performance to avoid dynamic clock adjustments, a USB-C inline power meter was used, and thermal throttling was monitored. No specific cooling hardware is described beyond that monitoring. This methodological choice is significant because the reported throughput and power characteristics depend strongly on thread count and core selection.

The paper’s model pool contains 25 open-source LLMs from eight families, with parameter sizes from 135M to 7B, all quantized with q4_k_m. The models specifically reported under Llamafile on Orange Pi 5 Pro are smollm2:1.7B, llama3.2:3B, tinyllama-1B, and llama3.2:1B. Prompting included three workload types: a paragraph-length prompt, a short factual Q&A prompt, and a short creative prompt. The direct Llamafile-versus-Ollama comparison uses Prompt 1, “Write a 5-paragraph paper on American history,” consistently for fairness (Tung et al., 20 Oct 2025).

3. Measured throughput on Orange Pi 5 Pro

The central throughput result is that Llamafile achieves its best performance at four cores on the Orange Pi 5 Pro and deteriorates substantially at six and eight cores. Figure 1 of the study reports output tokens per second (TPS) for the four Llamafile-tested models under Prompt 1 (Tung et al., 20 Oct 2025).

Model	Llamafile TPS	Ollama TPS
smollm2:1.7B (4 cores)	15.96	5.01
llama3.2:3B (4 cores)	8.88	2.91
tinyllama-1B (4 cores)	27.51	7.21
llama3.2:1B (4 cores)	21.77	6.78

Among the reported cases, the best Llamafile result is 27.51 TPS for TinyLlama-1B at 4 cores, while the worst is 2.34 TPS for LLaMA3.2-3B at 6 cores. The median Llamafile result at 4 cores across the four models is 18.87 TPS, computed in the study as $[(15.96 + 21.77) / 2]$ (Tung et al., 20 Oct 2025).

The degradation beyond four cores is sharp. For smollm2:1.7B, Llamafile falls from 15.96 TPS at 4 cores to 4.40 at 6 cores and 4.57 at 8 cores. For llama3.2:3B, it falls from 8.88 to 2.34 and 2.70. For tinyllama-1B, it falls from 27.51 to 7.44 and 7.06. For llama3.2:1B, it falls from 21.77 to 5.99 and 6.51. The study summarizes this as a roughly 3–4× drop when moving from 4 to 6 cores in representative cases such as TinyLlama-1B and LLaMA3.2-1B (Tung et al., 20 Oct 2025).

By contrast, Ollama scales upward with core count on the same board. TinyLlama-1B under Ollama rises from 7.21 TPS at 4 cores to 8.75 at 6 cores and 14.56 at 8 cores; LLaMA3.2-1B rises from 6.78 to 7.21 and 11.65. This contrast underlies the paper’s distinction between Llamafile’s low-core optimization and Ollama’s higher-core scaling behavior.

At equal 4-core settings, Llamafile outperforms Ollama substantially across all four reported models: 3.82× faster for TinyLlama-1B, 3.21× faster for LLaMA3.2-1B, 3.19× faster for SmolLM2-1.7B, and 3.05× faster for LLaMA3.2-3B. These values instantiate the paper’s “up to ~3.8×” throughput advantage at four cores (Tung et al., 20 Oct 2025).

4. Power consumption, energy efficiency, and metric definitions

The study evaluates throughput, peak RAM, and peak power, with peak power measured using a USB-C inline power meter. For Llamafile on Orange Pi 5 Pro, Table 3 reports peak power of 9 W at 4 cores and 14 W at 8 cores. The authors emphasize that 4-core Llamafile operation can achieve 3–4× faster generation while reducing power by 30–40% compared to higher-core configurations; the reported 9 W versus 14 W corresponds to a 35.7% reduction (Tung et al., 20 Oct 2025).

The metric definitions used in the study are:

$T = \frac{N_{tokens}}{t}$

for throughput in tokens/s,

$$e_{token} = \frac{E}{N_{tokens}}, \quad \text{with} \quad E = \int P(t)\,dt \approx P_{avg} \cdot t$$

for energy per token in joules/token, and

$\eta = \frac{N_{tokens}}{E} = \frac{T}{P_{avg}}$

for efficiency in tokens/J (Tung et al., 20 Oct 2025).

A concrete example reported by the authors uses TinyLlama-1B on Prompt 1. For Llamafile at 4 cores, throughput is 27.51 tokens/s and peak power is approximately 9 W, giving $\eta \approx 27.51 / 9 \approx 3.06$ tokens/J and energy per token of approximately 0.327 J/token. For Ollama at 8 cores on the same model, throughput is 14.56 tokens/s and peak power is approximately 14 W, giving $\eta \approx 1.04$ tokens/J and energy per token of approximately 0.96 J/token. The study summarizes this comparison as approximately 1.9× higher TPS, approximately 36% lower peak power, and approximately 2.9× better tokens/J for Llamafile in that pairing (Tung et al., 20 Oct 2025).

For LLaMA3.2-3B at 4 cores under Llamafile, the study reports 8.88 tokens/s and, using 9 W, estimates $\eta \approx 0.987$ tokens/J and $e_{token} \approx 1.013$ J/token. These examples reinforce the paper’s conclusion that Llamafile is most favorable when evaluated as an energy-efficient, low-core runtime rather than a high-core throughput maximizer.

5. Architectural bottlenecks and runtime trade-offs

The paper identifies an architecture-specific bottleneck for Llamafile on ARM big.LITTLE hardware. On the Orange Pi 5 Pro, performance peaks when only the four high-performance cores are used and declines when the system expands to six or eight cores. The authors describe this as a multithreading inefficiency on heterogeneous ARM cores and suggest that the behavior reflects a bottleneck in parallel token generation or scheduling overhead (Tung et al., 20 Oct 2025).

This interpretation is tightly bounded by the evidence in the paper. The study explicitly does not discuss SIMD/NEON, KV cache management, or file I/O specifics, so no causal claim is made on those dimensions. The contrast with Ollama is therefore framed empirically rather than microarchitecturally: Ollama scales more effectively with core count and trades higher power for higher throughput at eight cores, whereas Llamafile favors low-core, high-efficiency execution (Tung et al., 20 Oct 2025).

The memory result is also asymmetrical. The authors report that Llamafile showed better memory efficiency than Ollama, with lower runtime RAM demand as a qualitative finding, but they do not tabulate detailed numeric RAM figures for Llamafile. Accordingly, any claim beyond the qualitative direction would exceed the study’s evidence. A plausible implication is that Llamafile’s packaging and runtime model may reduce software overheads relevant to constrained devices, but the paper does not quantify that decomposition.

The study also delineates practical model-size limits for SBC inference. It states that, on SBCs, 1–1.5B models are the sweet spot. On Orange Pi 5 Pro, Llamafile ran 1B and 1.7B models comfortably; 3B models ran at lower TPS but remained usable, with 8.88 TPS at 4 cores for llama3.2:3B (Tung et al., 20 Oct 2025).

6. Deployment guidance and scope of applicability

The study’s deployment guidance is specific. For Llamafile, the preferred target SBC is the Orange Pi 5 Pro because it is the only platform in the paper with measured compatibility and direct benchmark data. The recommended configuration is to pin execution to the four high-performance cores, keep the CPU governor fixed to performance, use q4_k_m quantization, and keep prompts moderately sized to avoid KV cache pressure. The authors also recommend adequate cooling for continuous deployment because thermal throttling was monitored and can bias measurements (Tung et al., 20 Oct 2025).

For expected operating ranges on Orange Pi 5 Pro under Prompt 1, the paper reports that 1B–1.7B models deliver approximately 16–28 TPS at 4 cores with approximately 9 W peak power, with TinyLlama-1B at 27.51 TPS and SmolLM2-1.7B at 15.96 TPS. A 3B model operates at approximately 9 TPS at 4 cores and is described as usable for background tasks. Energy efficiency for 1B-class models is reported as approximately 3 tokens/J at 4 cores (Tung et al., 20 Oct 2025).

The paper is careful about scope. Raspberry Pi 5 is described as a promising candidate for 1–1.5B models with Llamafile, but no direct Llamafile results are provided for Raspberry Pi 4 or 5 because of compatibility limitations in the test harness. Any broader claim about Llamafile performance on those boards would therefore be inferential rather than measured. This suggests that the article’s strongest conclusions are about the interaction between Llamafile and the RK3588S-based Orange Pi 5 Pro, not about all ARM SBCs in general.

Within that scope, the study presents a consistent picture: Llamafile excels when priority is given to responsiveness and energy efficiency at low core counts, while Ollama is preferable when scaling to higher core counts is the dominant objective. The resulting trade-off is not merely software preference but a runtime–architecture interaction shaped by heterogeneous ARM cores, quantized model size, and power constraints (Tung et al., 20 Oct 2025).