PICO-TINYML Benchmark: TinyML on Embedded Linux

• PICO-TINYML-BENCHMARK is a modular framework that benchmarks TinyML models on embedded Linux by measuring inference latency, CPU load, memory usage, and prediction confidence.
• It employs a five-module architecture to ensure standard, repeatable evaluations over 100 iterations across models like gesture classification, keyword spotting, and MobileNet V2 on platforms such as Raspberry Pi 4 and BeagleBone AI64.
• The framework provides actionable insights into system-level performance tradeoffs by focusing on CPU-only inference, highlighting differences in resource utilization and real-time behavior between platforms.

PICO‑TINYML‑BENCHMARK is a modular, platform‑agnostic framework for benchmarking TinyML models on embedded systems under realistic, real‑time conditions. “PICO” denotes Performance of Inference, CPU, and Operations, where Performance of Inference covers latency and latency distributions, CPU covers average processor load during inference, and Operations covers memory usage and prediction behavior, specifically confidence and stability. The framework evaluates four core metrics—inference latency, CPU utilization, memory utilization, and prediction confidence or stability—and is presented as a system‑level benchmark for TinyML deployments on embedded Linux platforms rather than as an accuracy‑only benchmark (Dey et al., 5 Sep 2025).

1. Definition and Scope

PICO‑TINYML‑BENCHMARK was introduced to provide standard, repeatable, on‑device benchmarking of TinyML models on embedded Linux systems. Its stated motivation is to capture real‑time behavior beyond accuracy, including latency distributions, CPU load, memory footprint, and prediction stability, and to enable cross‑platform comparison under identical execution conditions. In this framing, the benchmark addresses the gap between model‑centric optimization techniques such as quantization or pruning and the system‑level behavior that emerges during deployment (Dey et al., 5 Sep 2025).

The framework is explicitly platform‑agnostic in the sense that its core logic uses generic Linux and TensorFlow Lite interfaces, avoids vendor‑specific accelerators or SDKs, and restricts the reported experiments to CPU execution only. This restriction is methodologically important: the reported results characterize generic CPU inference rather than accelerator‑assisted inference, even on hardware that includes AI‑specific co‑processors (Dey et al., 5 Sep 2025).

Its benchmark scope is deliberately narrow in one sense and broad in another. It is narrow because it focuses on embedded Linux single‑board systems, three representative TinyML models, and 100 repeated inferences per configuration. It is broad because it unifies latency, CPU, memory, and prediction confidence within one measurement workflow. This suggests that PICO is best understood as a deployment benchmark for runtime behavior, not as a comprehensive benchmark for model quality in the statistical learning sense (Dey et al., 5 Sep 2025).

2. Modular Architecture and Execution Workflow

The framework is organized into five loosely coupled modules. The Dataset Preprocessing Module handles input loading and task‑specific preprocessing, including image resizing and normalization for vision workloads and spectrogram generation for audio keyword spotting. The Model Inference Module uses TensorFlow Lite Runtime, loads quantized .tflite models, and encapsulates interpreter initialization, tensor setup, and output extraction. The Metrics Collection Module uses OS‑level tools, specifically psutil in Python, to record wall‑clock inference latency per iteration, CPU utilization, memory utilization, and predicted labels with confidence scores. The Experiment Orchestration Module iterates over platforms, models, and 100 inference iterations per model/platform pair while enforcing identical configurations and disabling accelerators for fairness. The Visualization & Analysis Module produces latency distributions, CPU and memory trends, and confidence trends and prediction distributions, and computes comparative summary statistics (Dey et al., 5 Sep 2025).

The benchmark workflow begins with preprocessing a raw image or audio sample so that it matches the input specification of the selected model. Gesture inputs are resized and normalized; keyword spotting inputs are converted to spectrograms; MobileNet V2 inputs are resized and normalized to the model’s expected image format. The quantized TensorFlow Lite model is then loaded and the interpreter allocates tensors. The benchmark proceeds through 100 inference iterations, recording timestamps, sampling CPU and memory usage around the inference window, and extracting the predicted label $y_i$ and confidence score $p_i$ for each iteration. All measurements are logged for later aggregation (Dey et al., 5 Sep 2025).

Because the modules are independent, new preprocessing pipelines, models, or datasets can be added without modifying the metric collection core. The paper describes this modularity as a practical mechanism for extensibility rather than as a new algorithmic contribution. In operational terms, the same logic is intended to run unchanged across BeagleBone AI64 and Raspberry Pi 4 because only CPU execution and generic Linux interfaces are used (Dey et al., 5 Sep 2025).

3. Metrics, Instrumentation, and Measurement Model

PICO‑TINYML‑BENCHMARK centers on four metrics. Inference latency is defined as the time in milliseconds required to complete one inference, measured inside the inference loop on every iteration. CPU utilization is the average CPU usage during inference, expressed as a percentage and sampled with psutil or an equivalent tool. Memory utilization is the percentage of system memory used before or after inference, again averaged over iterations. Prediction confidence or stability is the confidence associated with the predicted label, typically taken as the maximum output probability or class score (Dey et al., 5 Sep 2025).

Over $N=100$ iterations, the framework aggregates metrics using averages such as

$\bar{L} = \frac{1}{N}\sum_{i=1}^{N} L_i$

for latency and

$\bar{p} = \frac{1}{N}\sum_{i=1}^{N} p_i$

for confidence. The paper reports averages in its tables, while the plots expose full per‑iteration distributions. It also notes that median, percentiles, and variance could be computed similarly even though the text and tables primarily report averages. A standard deviation for latency,

$$\sigma_L = \sqrt{\frac{1}{N}\sum_{i=1}^{N}(L_i-\bar{L})^2},$$

is described as a derivable measure of jitter, and an analogous variance for confidence is presented as a way to quantify prediction stability, although these derived statistics are not the primary reported outputs (Dey et al., 5 Sep 2025).

Instrumentation is intentionally lightweight. TensorFlow Lite Runtime supplies CPU‑only inference, and psutil provides CPU and memory monitoring on Linux. Measurements are taken inside the inference loop rather than through coarse external profiling. No composite benchmark score is defined. Instead, the framework treats latency, CPU, memory, and confidence as separate observables and interprets results through trade‑off narratives, which distinguishes it from benchmark styles that collapse performance into a single scalar (Dey et al., 5 Sep 2025).

4. Benchmark Workloads, Datasets, and Platforms

The benchmark evaluates three representative TinyML models. Gesture Classification is a visual hand‑gesture recognition task using a quantized TensorFlow Lite model sourced from the TensorFlow Lite gesture classification example. Its dataset consists of 100 RGB images sampled from a public Kaggle hand gesture dataset, and preprocessing uses resizing and normalization to match the model input requirements. Keyword Spotting uses a pretrained, quantized TensorFlow Lite model from Silicon Labs MLTK keyword spotting, with 100 audio clips drawn from the MLTK framework’s test set based on Speech Commands V2; preprocessing generates spectrograms aligned to the model input. MobileNet V2 is a quantized image classification model derived from MobileNet V2 and adapted to TensorFlow Lite; it is evaluated on 100 web‑sourced images resized and normalized to the model specification (Dey et al., 5 Sep 2025).

All three models are quantized. The paper does not detail pruning or other compression methods for these workloads. The datasets are deliberately small because the purpose is repeated inference for system behavior measurement rather than exhaustive statistical estimation of accuracy. This design choice is consistent with the benchmark’s emphasis on runtime distributions and resource usage rather than on dataset‑scale evaluation (Dey et al., 5 Sep 2025).

The two hardware platforms are BeagleBone AI64 and Raspberry Pi 4. BeagleBone AI64 is described as an AI‑oriented embedded platform with an ARM Cortex‑A72 CPU up to 2.0 GHz, 8 GB LPDDR4, Debian‑based Linux, and PowerVR GPU plus AI co‑processors that are present but disabled in the reported experiments. Raspberry Pi 4 is described as a cost‑effective, general‑purpose single‑board computer with a quad‑core Cortex‑A72 at 1.5 GHz, up to 4 GB LPDDR4, and Raspberry Pi OS/Linux. Both systems use the same TensorFlow Lite runtime and the same measurement procedure under CPU‑only execution to preserve fairness (Dey et al., 5 Sep 2025).

5. Experimental Protocol and Reported Results

The experimental protocol runs 100 inference iterations per model per platform. For each iteration, the benchmark records timestamps, executes inference with TensorFlow Lite, immediately samples CPU and memory usage, and captures the output label and confidence score. The same quantized model binaries, runtime environment, and measurement logic are used on both platforms, and BeagleBone accelerators are explicitly disabled so that both boards are compared under generic CPU inference (Dey et al., 5 Sep 2025).

Latency results show lower average inference time on Raspberry Pi 4 for all three workloads. Gesture Classification averages 9.49 ms on BeagleBone AI64 and 1.76 ms on Raspberry Pi 4. Keyword Spotting averages 0.74 ms on BeagleBone AI64 and 0.16 ms on Raspberry Pi 4. MobileNet V2 averages 2125.04 ms on BeagleBone AI64 and 513.60 ms on Raspberry Pi 4. The paper summarizes this as the first key insight: Raspberry Pi 4 achieved lower latency across all models, and the latency distributions are also described as favorable on Raspberry Pi 4 (Dey et al., 5 Sep 2025).

Resource utilization shows the same directional pattern. For Gesture Classification, CPU utilization is 40.38% on BeagleBone AI64 and 8.88% on Raspberry Pi 4, while memory utilization is 15.10% and 11.00%, respectively. For Keyword Spotting, CPU utilization is 18.50% versus 5.00%, and memory utilization is 18.96% versus 11.00%. For MobileNet V2, CPU utilization is 51.80% versus 28.59%, and memory utilization is 19.75% versus 11.81%. The paper identifies this as the second key insight: Raspberry Pi 4 used significantly less CPU and memory, which it describes as making the platform more suitable for battery‑operated or resource‑constrained use cases (Dey et al., 5 Sep 2025).

Prediction confidence remains nearly identical across platforms. Gesture Classification reports an average confidence of 0.98 on both platforms. Keyword Spotting reports 0.99 on both. MobileNet V2 reports 17.16 on both, with the paper noting that this is likely a task‑specific aggregate metric rather than a probability in $[0,1]$. Combined with the confidence and prediction distribution plots, the conclusion is that both platforms exhibit highly stable and identical prediction confidence across iterations. The trade‑off, therefore, is primarily between latency and resource utilization rather than between platform and predictive stability (Dey et al., 5 Sep 2025).

A common misconception would be to read the BeagleBone AI64 hardware profile as implying superior raw performance in these experiments. The paper’s own abstract instead states that BeagleBone AI64 demonstrates consistent inference latency for AI‑specific tasks. In the detailed CPU‑only results, this consistency refers to stability rather than to speed, because Raspberry Pi 4 is numerically faster across all three workloads under the benchmark’s fairness constraints (Dey et al., 5 Sep 2025).

6. Position Within TinyML Benchmarking

PICO‑TINYML‑BENCHMARK is positioned against several existing benchmark traditions. The framework contrasts itself with Edge Impulse, which focuses on building and deploying TinyML models but does not provide detailed on‑device system metrics; with MLPerf Tiny, which standardizes workloads and focuses mainly on accuracy and some latency numbers rather than full CPU, memory, and stability tracking on general embedded platforms; and with narrower individual studies that benchmark a single platform or a smaller metric set (Dey et al., 5 Sep 2025).

This placement aligns PICO with a broader benchmark literature that has emphasized complementary dimensions. The MLPerf Tiny position paper argues for TinyML benchmarks built around representative workloads, latency, energy, and stringent memory budgets on MCU‑class devices (Banbury et al., 2020). MCU‑oriented comparisons such as “TinyML Platforms Benchmarking” concentrate on inference time and memory footprint for frameworks like TensorFlow Lite Micro and STM32Cube.AI, rather than on prediction stability or general embedded Linux system metrics (Osman et al., 2021). “Benchmarking Energy and Latency in TinyML” separates pre‑inference, inference, and post‑inference phases to expose energy–latency trade‑offs at a finer granularity (Bartoli et al., 21 May 2025). “MLonMCU” emphasizes automated end‑to‑end retargeting across frameworks and targets, especially for TFLM and TVM (Kempen et al., 2023). Domain‑specific suites such as BiomedBench define complete end‑to‑end biomedical workloads, including acquisition and idle phases, to expose application‑dependent energy behavior on low‑power boards (Samakovlis et al., 2024).

Against that background, PICO’s distinctive contribution is its explicit combination of latency, CPU utilization, memory utilization, and prediction confidence within a CPU‑only embedded Linux setting. This suggests that PICO occupies a system‑behavior niche between MCU‑centric inference benchmarks and broader end‑to‑end energy benchmarks. It does not attempt to replace workload standards such as MLPerf Tiny; rather, it complements them by emphasizing on‑device runtime observables that are often omitted from model‑centric evaluations (Dey et al., 5 Sep 2025).

7. Limitations and Planned Extensions

The framework has several stated limitations. Hardware coverage is limited to two higher‑end single‑board computers, BeagleBone AI64 and Raspberry Pi 4, with no MCU‑class devices in the reported study. Model diversity is limited to Gesture Classification, Keyword Spotting, and MobileNet V2; object detection, time‑series regression, and multi‑modal models are not yet included. Power consumption, energy per inference, and thermal behavior are not measured. The datasets contain only 100 samples per model, which is suitable for repeated inference benchmarking but not for rigorous accuracy analysis. The framework also assumes relatively stable operating conditions and does not explicitly model environmental variability or concurrent workloads (Dey et al., 5 Sep 2025).

Planned future work follows directly from these limitations. The paper proposes extending support to microcontrollers and ultra‑low‑power processors, integrating power and thermal profiling, and adding metrics such as energy per inference and thermal headroom over time. It also proposes expanding the model set to object detection, time‑series analysis, and multi‑modal models; improving usability through automated deployment pipelines and real‑time dashboards; and evaluating optimization techniques such as dynamic quantization, adaptive resource allocation, and accelerator usage, including GPU or NPU support on BeagleBone AI64 (Dey et al., 5 Sep 2025).

In practical terms, the implementation is described in enough detail to support replication: TensorFlow Lite Runtime and psutil form the software substrate, quantized .tflite models are loaded directly, data preprocessing is task‑specific, and metrics can be stored in CSV or similar formats with configuration captured in JSON or YAML. This suggests that the framework is intended not only as a one‑off experimental study but also as a reusable methodology for system‑level TinyML benchmarking on embedded platforms (Dey et al., 5 Sep 2025).