MCUNet: Efficient Deep Learning on MCUs

• MCUNet is a co-design framework that integrates neural architecture search (TinyNAS) with a memory-optimized inference engine (TinyEngine) to enable deep learning on microcontrollers.
• It employs patch-based inference scheduling and hardware-tailored operator inlining to significantly reduce SRAM usage and latency, achieving ImageNet-class performance under strict resource constraints.
• The system demonstrates practical accuracy and efficiency improvements across tasks including image classification, object detection, and wake word recognition on resource-constrained devices.

MCUNet is a system–algorithm co-design framework for deploying deep neural networks on microcontroller units (MCUs), enabling high-accuracy deep learning workloads—including full-scale ImageNet inference—within the severe SRAM (32–512 kB), flash (few MB), and compute constraints of commodity MCUs. The MCUNet series features a joint neural architecture search (TinyNAS) and a memory-optimized inference engine (TinyEngine), systematically pushing the limits of TinyML on IoT hardware (Lin et al., 2024, Lin et al., 2021, Lin et al., 2020).

1. Architectural Overview

MCUNet is structured around two tightly integrated components addressing both model design and deployment on microcontrollers:

• TinyNAS searches for hardware-constrained efficient convolutional neural network (CNN) architectures, focusing on MobileNet-style inverted residual blocks parameterized by expansion ratio ($e\in\{3,4,6\}$), kernel size ($k\in\{3,5,7\}$), per-stage depth ($d\in\{2,3,4\}$), width multiplier ($w$), and input resolution ($r$).
• TinyEngine is a lightweight, code-generation-based inference library that minimizes peak memory via in-place depthwise convolution, patch-based spatial execution, and operator specialization. It compiles only the required kernels into concise binaries, reducing the code size by 4–5× compared to generic interpreters such as TF-Lite Micro or CMSIS-NN.

In MCUNetV2 (Lin et al., 2021), peak activation memory is further reduced by “patch-based inference scheduling,” executing early, memory-heavy layers on spatially disjoint patches, followed by reassembly and standard execution for the remaining layers. This addresses the problem that the first few blocks in CNNs typically dominate the SRAM footprint.

2. System–Algorithm Co-Design and Search Methodology

MCUNet diverges from traditional NAS+kernel optimization by jointly considering both the neural network topology and the deployment schedule as part of a combined search space, directly guided by target MCU constraints.

Two-Stage Search (TinyNAS)

• Stage 1: Search-Space Optimization For a grid of $(w, r)$ tuples (width, resolution), candidate sub-networks are sampled and evaluated for SRAM fit. For each subspace $S$, the CDF of FLOPs among those satisfying the SRAM condition ($M(f_j) \leq M_\text{max}$) is constructed:

$$F_S(F) = (1/ \sum_j 1_j)\sum_{j=1}^m 1_{j} \cdot 1_{FLOPs(f_j) \leq F}$$

The subspace $S^*$ with the rightmost CDF—i.e., highest mean FLOPs under SRAM cap—is selected.

• Stage 2: Resource-Constrained Model Specialization Within $k\in\{3,5,7\}$0, a supernet supporting all block options is trained using weight sharing. An evolutionary search is applied to extract the optimal sub-network $k\in\{3,5,7\}$1 maximizing accuracy

$k\in\{3,5,7\}$2

Joint Architecture and Scheduling (MCUNetV2)

MCUNetV2 augments the search vector to include the number of patch splits ($k\in\{3,5,7\}$3) and how many initial blocks to execute in patch mode ($k\in\{3,5,7\}$4), i.e. $k\in\{3,5,7\}$5 for $k\in\{3,5,7\}$6. Search then solves

$k\in\{3,5,7\}$7

allowing automatic adaptation of both network structure and inference schedule.

3. Memory and Latency Models

MCUNet explicitly couples memory and compute models to hardware effects:

• Latency Modeling:

Empirically, MCU inference latency scales linearly with FLOPs:

$k\in\{3,5,7\}$8

With depthwise-separable model families, FLOPs effectively proxy for latency.

• Memory–Latency Trade-off via Patch Execution:

Patch-based splitting divides high-memory early layers into $k\in\{3,5,7\}$9 spatial tiles, reducing activation memory requirements by $d\in\{2,3,4\}$0. However, patches overlap due to convolutional receptive fields, incurring extra compute:

$d\in\{2,3,4\}$1

where $d\in\{2,3,4\}$2. MCUNetV2 uses receptive-field redistribution—shifting strides/large kernels later—to push down the compute overhead to $d\in\{2,3,4\}$3, and even below $d\in\{2,3,4\}$4 with optimal redistribution (Lin et al., 2021).

4. Deployment on MCUs: Quantization and Runtime Details

MCUNet targets widely deployed STM32 microcontrollers:

Target	SRAM	Flash	Clock	Top-1 Accuracy (ImageNet)	Peak SRAM	Latency
STM32F412 M4	256 kB	1 MB	100 MHz	64.9% (MCUNetV2, patch)	196 kB	463 ms
STM32H743 H7	512 kB	2 MB	480 MHz	71.8% (MCUNetV2, patch)	465 kB	336 ms

All deployed models are post-quantized to INT8 per-tensor (weights and activations). TinyEngine, implemented in C, compiles only used operators, resulting in small binaries (∼450 kB). Optimizations include in-place depthwise convolution (SRAM reduction by 1.6×) and patch scheduling. For wake word tasks, MCUNetV2 demonstrates >90% accuracy with <32 kB SRAM.

5. Experimental Results and Benchmarks

MCUNet establishes new benchmarks for tiny deep learning workloads:

• ImageNet:

MCUNetV2 achieves 71.8% Top-1 on the 512 kB/2 MB STM32H743, exceeding all prior models. MCUNet-M4 fares similarly under 256 kB/1 MB (64.9% Top-1).

• SRAM Reduction:

Patch-based inference reduces peak activation memory by 4–8× in early layers at $d\in\{2,3,4\}$5 FLOPs overhead; with receptive-field redistribution, compute penalty drops to 3–10%.

• Energy and Latency Efficiency:

TinyEngine provides 1.5–22× speedup in latency and 4–8× SRAM decrease compared to TF-Lite Micro, CMSIS-NN, and X-Cube-AI.

• Visual Wake Words (VWW):

MCUNetV2 delivers >90% accuracy under 32 kB SRAM, whereas previous MCUNetV1 required 128 kB SRAM for competitive accuracy.

• Object Detection (Pascal VOC):

MCUNetV2 achieves 68.3% mAP on STM32H743 (SRAM 438 kB), a 16.9% improvement over MCUNetV1, enabled by higher input resolution via patch scheduling (Lin et al., 2021).

6. Technical Innovations and Significance

MCUNet’s advances stem from five principal technical approaches:

1. Automated search-space optimization using FLOPs CDFs under tight SRAM constraints, balancing model capacity and hardware fit.
2. One-shot NAS with weight sharing and evolutionary specialization, enabling fast search under multiple device-specific constraints.
3. Patch-based inference and memory scheduling to eliminate early-block memory bottlenecks, leveraging spatial tiling and receptive-field redistribution.
4. Operator inlining and code generation for a deterministic, concise runtime, avoiding interpreter overhead on resource-limited hardware.
5. Highly memory-compact quantization supporting INT8 (and optionally INT4) weights and activations.

MCUNet’s system–algorithm co-design paradigm demonstrates that ImageNet-class inference, and even object detection, is feasible on $d\in\{2,3,4\}$6 MCUs within sub-megabyte flash and sub-half-megabyte SRAM. Empirical results ensure that these advances are not limited to classification but generalize to low-latency, low-power detection and wake word tasks (Lin et al., 2024, Lin et al., 2021, Lin et al., 2020).

7. Impact and Relevance

MCUNet substantially expands the design space for always-on edge intelligence, bypassing the tradeoffs between network depth/width, input resolution, and hardware compatibility that constrain baseline mobile-class CNNs. By decoupling network and scheduling search, MCUNet overcomes rigid layerwise optimization and makes possible “real” AI tasks on the smallest platforms, marking a significant inflection point in TinyML for IoT and ubiquitous AI (Lin et al., 2024, Lin et al., 2021, Lin et al., 2020).