Split CNN Inference on Networked Microcontrollers

Abstract: Running deep neural networks on microcontroller units (MCUs) is severely constrained by limited memory resources. While TinyML techniques reduce model size and computation, they often fail in practice due to excessive peak Random Access Memory (RAM) usage during inference, dominated by intermediate activations. As a result, many models remain infeasible on standalone MCUs. In this work, we present a fine-grained split inference system for networked MCUs that enables collaborative inference of Convolutional Neural Networks (CNN) models across multiple devices. Our key insight is that breaking the memory bottleneck requires splitting inference at sub-layer granularity rather than at layer boundaries. We reinterpret pre-trained models to enable kernel-wise and neuron-wise partitioning, and distribute both model parameters and intermediate activations across multiple MCUs. A lightweight, resource-aware coordinator orchestrates the inference across MCU devices with heterogeneous resources. We implement the proposed system on a real testbed and evaluate it on up to 8 MCUs using MobileNetV2, a representative CNN model. Our experimental results show that CNN models infeasible on a single MCU can be executed across networked MCUs, reducing the per-MCU peak RAM usage while maintaining the practical end-to-end inference latency. All the source code of this work can be found here: https://github.com/shashsuresh/split-inference-on-MCUs.

• The paper presents a novel fine-grained partitioning method that splits CNN inference at a neuron level across MCUs to drastically lower peak RAM usage.
• It introduces a resource-aware workload allocation using explicit mapping and rating-based strategies to optimize both computation and communication across devices.
• Experimental results demonstrate that this approach enables scalable deployment with up to an 18.9% reduction in end-to-end latency through efficient load balancing.

Fine-Grained Split CNN Inference Across Networked MCUs

Introduction and Motivation

The paper "Split CNN Inference on Networked Microcontrollers" (2605.09357) addresses critical barriers to deploying deep neural networks (DNNs) on microcontroller units (MCUs) imposed by stringent memory and computational constraints. While prior TinyML work enabled model parameter compression via quantization and pruning, real-world deployments remain limited by peak RAM consumption stemming from intermediate activations during inference. This paper posits that sustainable edge AI for MCUs requires collaborative inference, splitting model execution across interconnected devices. The authors’ central proposition is to partition computation at sub-layer granularity—kernel-wise and neuron-wise—allowing both weights and activations to be distributed, surmounting the infeasibility of allocating full layers to individual MCUs.

System Architecture

The proposed system comprises two roles: a coordinator (for orchestration, mapping, and routing) and a set of heterogeneous MCUs acting as workers, each storing fragments of the CNN model. Model splitting is conducted offline, extracting structural metadata (dimensions, kernels, receptive fields) to enable explicit neuron-level dependency mapping. During deployment, each MCU is assigned weight fragments and system parameters reflecting its computational, memory, and communication capabilities. The coordinator uses these metrics to implement a rating-based workload allocation to avoid load imbalance.

Inference is executed in a layer-wise pipeline: the coordinator determines neuron-to-worker assignment, distributes required activations, and aggregates partial outputs for subsequent layers. Notably, this design minimizes per-MCU memory by ensuring no device holds entire layers’ weights or activations.

Fine-Grained Partitioning Mechanisms

Model Reinterpretation

The authors introduce a model reinterpretation pipeline, operational at the neuron level, making receptive fields and dependencies explicit. This reinterpretation is implemented in a lightweight Rust framework, generating portable representations for deployment on constrained hardware.

Convolutional Layer Splitting

For convolutional layers, output feature maps are partitioned at neuron-level granularity, informed by worker capability ratings. The assignment algorithm iterates over output positions and distributes corresponding kernels only as needed, optimizing memory and computational allocation.

Linear Layer Splitting

Linear layers are split column-wise, each column mapped to an output neuron. Workload distribution aligns with MCU capability ratings, maintaining independence across output neuron computations.

Cross-Layer Activation Mapping

The coordinator builds explicit assignment and routing mappings to minimize redundant data transfer and ensure correct activation distribution across successive layers. This mechanism is central to bounding memory usage and communication overhead.

Resource-Aware Workload Allocation

The paper formulates MCU capability ratings based on both computation and communication characteristics, driven by empirical measurements (clock frequency, communication delay, bandwidth). Workload and model fragment allocations are linearly proportional to these ratings, with additional correction for storage constraints. Overflowed allocations are redistributed to underutilized MCUs iteratively, achieving compliance with flash and RAM limits.

System-level optimizations include layer fusion (convolution, BatchNorm, ReLU), computation in-place, and 8-bit quantization to minimize RAM pressure and utilize hardware integer support.

Implementation and Experimental Results

The implementation targets the Teensy 4.1 MCU platform (ARM Cortex-M7, 600 MHz, 1 MB RAM, 8 MB flash, Ethernet). Model fragments are distributed via LittleFS and coordinated with minimal runtime overhead. Communication is orchestrated via TCP using fixed-size packets, with explicit acknowledgments for reliability.

Key numerical findings:

• Using MobileNetV2 (input 112×112×3), single-device inference is infeasible due to several MB peak memory demands. The proposed split inference permits deployment across 3-8 MCUs, maintaining all layers’ RAM usage within budget.
• The rating-based allocation outperforms uniform and frequency-only strategies under heterogeneous compute and communication settings. Incorporation of communication delays results in up to 18.9% reduction in end-to-end latency compared to naive allocations.
• As MCUs increase, computation time decreases monotonically, but communication time becomes the dominant factor (e.g., inference time rises from 42.97 s [3 MCUs] to 56.89 s [8 MCUs], with communication overhead increasing from 27.60 s to 49.82 s, computation declining from 15.37 s to 7.07 s).
• Layer-wise communication and computation time profiles show communication concentration in early layers with large activations, while computation scales effectively with additional MCUs.
• Simulation extending to 120 MCUs demonstrates sharply diminishing per-device peak RAM usage, with saturation of returns beyond 10-20 MCUs—a quantifiable memory-parallelism trade-off.

Implications and Future Directions

The practical implication is the feasibility of deploying previously infeasible CNNs on MCU networks, overcoming RAM bottlenecks without further sacrificing model size or accuracy. This is particularly salient for edge AI scenarios in embedded and IoT systems where scalability, cost, and energy efficiency are critical. The framework's resource-aware coordination and fine-grained splitting set foundations for future model and hardware classes, with potential to support broader DNN architectures and more aggressive quantization/compression strategies.

Theoretically, this work demonstrates the necessity of sub-layer, neuron-wise partitioning for memory-bounded distributed inference, challenging assumptions underlying prior edge-cloud and GPU-centric split inference systems. Future research directions include optimizing inter-device communication, direct MCU-to-MCU forwarding, dynamic load balancing, and extending to models with more heterogeneous layer structures.

Conclusion

This paper establishes a robust methodology for enabling collaborative CNN inference across networked MCUs via fine-grained, resource-aware partitioning. The approach delivers significant reductions in per-device memory usage, maintains practical inference latency, and achieves scalable computation with bounded communication overhead. The system lays groundwork for distributed TinyML, opening new avenues for memory-efficient edge intelligence on ultra-constrained hardware (2605.09357).