---
title: MLPerf Tiny Benchmark
url: https://www.emergentmind.com/topics/mlperf-tiny-benchmark
type: topic
---

# MLPerf Tiny Benchmark

MLPerf Tiny Benchmark is an industry-standard suite for measuring the inference performance—accuracy, latency, and energy—of ultra-low-power neural networks on resource-constrained embedded platforms. Developed through collaboration among over fifty organizations from academia and industry, MLPerf Tiny fills a previously unmet need for reproducible, apples-to-apples benchmarks targeting sub-milliWatt microcontroller-class devices and specialized inference accelerators. The suite supports both strictly controlled (closed) and flexible (open) benchmarking divisions and has directly enabled research into model compression, quantization, and efficient on-device execution [2106.07597, 2003.04821].

## 1. Motivations and Design Objectives

MLPerf Tiny addresses the absence of standardized evaluation infrastructure for TinyML, defined as ML inference under 1 mW active power on embedded hardware. Its primary goal is to enable fair comparison across hardware, software, and algorithmic stacks by measuring cost–performance trade-offs with high granularity.

Key requirements covered by the benchmark are:
- Sub-milliWatt operational range (μW–mW), reflecting target application constraints in battery-powered and always-on edge devices.
- Strict RAM (≤512 kB) and flash (≤1 MB) resource thresholds, chosen to be compatible with contemporary MCU-class parts.
- Support for a wide spectrum of hardware (MCUs, DSPs, FPGAs, ASICs, and in-memory compute chips) and software stacks (from hand-optimized C to full compilers and interpreters).
- Explicit handling of measurement challenges, such as end-to-end energy profiling and memory/latency reporting [2106.07597, 2003.04821].

## 2. Benchmark Suite Structure and Workflow

MLPerf Tiny implements a modular benchmarking setup designed for maximal reproducibility and adaptability:
- Each benchmark includes Python training scripts (typically TensorFlow or PyTorch), reference model quantization pipelines (TFLite or ONNX-based), and C/C++ implementations for on-device inference (TFLite Micro or custom backends).
- The reference codebase is divided into five main components: dataset preprocessing, model training, model conversion/quantization, firmware inference code, and the hardware device under test (DUT).
- The measurement harness consists of an energy meter (e.g., Joulescope or LPM01A), host/target communication (serial/GPIO with synchronization), and an automated GUI for control and result logging.

Closed division submissions must use the reference datasets, models, and quality targets with post-training quantization only, ensuring strict comparability. Open division submissions may alter the architecture, quantization, or training but must report performance on the standard test datasets, providing documentation of any changes [2106.07597].

## 3. Benchmark Tasks and Model Reference Set

MLPerf Tiny v0.5/v0.7 comprises four canonical TinyML workloads:

| Task                  | Dataset/Input              | Reference Model                 | Quality Target (Closed)     |
|-----------------------|---------------------------|---------------------------------|-----------------------------|
| Keyword Spotting (KWS)| SpeechCommands v1/v2      | DS-CNN (38.6K params, 49×10 in) | ≥ 90% top-1 accuracy        |
| Visual Wake Words (VWW)| MSCOCO/VWW (96×96 RGB)   | MobileNetV1-0.25× (325K params) | ≥ 80% top-1 accuracy        |
| Image Classification (IC)| CIFAR-10 (32×32×3)     | Mini-ResNet (96K params)        | ≥ 85% top-1 accuracy        |
| Anomaly Detection (AD)| ToyADMOS (5×128)          | FC-autoencoder (270K params)    | ROC-AUC ≥ 0.85              |

- Input preprocessing pipelines are defined, including MFCCs for KWS, single-frame normalization for VWW, image resizing for IC, and spectrogram windowing for AD.
- Baseline models use 8-bit quantization in the closed division. Benchmark-specific architectures include DS-CNN for KWS, MobileNetV1 (depthwise-separable) for VWW, reduced-depth ResNet for IC, and deep FC autoencoders for AD [2106.07597, 2207.06613].
- Open-division examples include binary CNNs and mixed-precision quantization (1–12 bits per layer), as enabled via QAT flows [2206.11791].

## 4. Metrics, Measurement Methodology, and Instrumentation

Performance evaluation centers on:
- **Accuracy:** Top-1 classification or ROC-AUC for AD, computed as 
  $$
  \text{Accuracy} = \frac{\#\{\text{correct predictions}\}}{N}
  $$
- **Latency:** Median time per inference, measured as 
  $$
  L = T_\mathrm{end} - T_\mathrm{start}
  $$
  for $N$ trials (minimum 10), reported in ms.
- **Energy per Inference:** Average energy consumed per sample,
  $$
  E_\text{inf} = \frac{E_\text{total}}{N}
  $$
  with $E_\text{total}$ obtained either by integrating instantaneous power or as $(I_\text{active} - I_\text{idle}) \cdot V_\text{supply} \cdot T_\text{inf}$ [2206.11791].
- **Throughput:** Inferences per second, $\text{IPS} = 1/L$.

Composite metrics such as Energy–Delay Product (EDP: $E_\text{inf} \times L$) are also reported.

The on-device protocol requires strict isolation of power measurement to exclude host/communication overhead, ensures all model weights and activations reside in on-chip memory, and logs full configuration data for reproducibility [2106.07597]. Energy measurement is synchronized via GPIO edge tracing [2206.11791].

## 5. Representative Implementations and Results

The benchmark suite has catalyzed open-source implementations across a variety of platforms:

- **Reference MCUs:** ST NUCLEO-L4R5ZI (Cortex-M4@80 MHz, 1 MB flash, 128 kB RAM) achieves for KWS ≈ 1,000 inf/s and ≈100 µJ/inference; VWW ≈120 inf/s, ≈1.2 mJ/inference; IC ≈100 inf/s, ≈900 µJ; AD ≈25 inf/s, ≈1.5 mJ, all at or above target accuracies [2106.07597].
- **FPGA Open-Division Submissions:** hls4ml and FINN workflows implement, for example, image classification with “tiny-ResNet-8” (58K params, 83.5% accuracy, 8-bit QAT) and binary CNNs (CNV-W1A1, 84.5%), and KWS with MLPs at 3-bit quantization (82.5%) [2206.11791]. Empirical energy per inference for AD/KWS can drop below 30 µJ at ≈20–33 µs latency on Arty A7-100T.
- **Early-Exit Architectures:** T-RecX demonstrates, across ResNet-8/CIFAR-10, DS-CNN/SpeechCommands, and MobileNetV1/VWW, that inserting a single intermediate classifier can reduce average FLOPS by 20%–38% for ≤1% accuracy loss, outperforming BranchyNet and SDN in parameter efficiency and realized FLOPS reductions [2207.06613].

FPGA and NP accelerators demonstrate order-of-magnitude improvements in inference rate (IPS>5K) and energy, at the expense of higher quiescent power and toolchain customization.

## 6. Submission, Scoring, and Reproducibility Protocols

MLPerf Tiny requires all submitters, in both divisions, to:
- Provide full source code, binary builds, and calibration logs.
- Document measurement scripts and dataset/model modifications (open division).
- Report median latency and energy per inference over at least five independent trials.
- Offer a compliance checklist covering model, activation, and stack RAM/flash footprints.

Primary scoring metric is latency (closed division), with energy per inference as tiebreaker, and optional composite EDP ranking. All closed-division entries are validated by independent re-building on pre-specified reference hardware (e.g., STM32H7, Arduino Nano 33 BLE), ensuring exact reproducibility [2106.07597, 2003.04821].

## 7. Research Impact, Challenges, and Future Directions

The MLPerf Tiny benchmark has become a central infrastructure for research on TinyML co-design, model compression, and hardware/software codesign techniques. Its modular, open framework enables:
- Comparative study of quantization granularity effects, e.g., per-layer mixed-precision, binary/ternary networks, quantization-aware training versus post-training quantization.
- Transferability of methods such as layer fusion, spatial streaming, and activation pipelining from FPGAs to MCUs and ASIC accelerators [2206.11791].
- Early-exit and conditional computation empirical study on resource-constrained networks [2207.06613].

Ongoing technical challenges include: accurate ultra-low-power measurement (μW–mW dynamic range), memory and code-size constraints, and fair normalization across diverse hardware and software stacks. Present limitations are the focus on four core tasks, the closed-division restriction to 8-bit quantization, and the lack of multi-core/heterogeneous pipeline or genuinely streaming workloads.

Planned extensions propose: real-time streaming detection (e.g., fall detection), expanded evaluation of binary/mixed-precision models, new task types (object/NLP detection, multi-sensor fusion), refined power domain partitioning, and addition of quality-of-service metrics such as false-alarm rates. All tools, benchmarks, and rules are continuously stewarded via the MLCommons open repository and working group structure [2106.07597, 2003.04821, 2206.11791].

---

**Relevant references:**
- "MLPerf Tiny Benchmark" [2106.07597]
- "Benchmarking TinyML Systems: Challenges and Direction" [2003.04821]
- "Open-source FPGA-ML codesign for the MLPerf Tiny Benchmark" [2206.11791]
- "T-RECX: Tiny-Resource Efficient Convolutional neural networks with early-eXit" [2207.06613]

Source: https://www.emergentmind.com/topics/mlperf-tiny-benchmark