CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs (1801.06601v1)

Abstract: Deep Neural Networks are becoming increasingly popular in always-on IoT edge devices performing data analytics right at the source, reducing latency as well as energy consumption for data communication. This paper presents CMSIS-NN, efficient kernels developed to maximize the performance and minimize the memory footprint of neural network (NN) applications on Arm Cortex-M processors targeted for intelligent IoT edge devices. Neural network inference based on CMSIS-NN kernels achieves 4.6X improvement in runtime/throughput and 4.9X improvement in energy efficiency.

• The paper presents CMSIS-NN, a library that optimizes neural network operations on Arm Cortex-M CPUs with up to 4.6X runtime and 4.9X energy efficiency improvements.
• The paper details optimized fixed-point quantization and SIMD-based implementations that reduce memory footprint and computational load on resource-constrained IoT devices.
• The paper demonstrates practical deployment by processing approximately 10.1 images per second on a CIFAR-10 network, underscoring its efficiency for embedded deep learning.

CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs

This paper introduces CMSIS-NN, a collection of optimized neural network kernels specifically designed for Arm Cortex-M processors, which are frequently employed in IoT edge devices. As the proliferation of IoT devices continues to escalate, reaching an expected one trillion by 2035, there is a pressing need for efficient edge computing solutions that reduce latency and energy consumption associated with cloud-dependent data analytics. The authors tackle this challenge by providing highly efficient kernels that not only optimize performance but also significantly decrease memory footprints, catering to the resource-constrained environments typical of IoT edge nodes.

Performance Optimization and Kernels Overview

Deploying neural networks on resource-constrained platforms such as Arm Cortex-M CPUs necessitates significant optimization efforts. The CMSIS-NN library is structured into two main components: NNFunctions and NNSupportFunctions. NNFunctions include implementations of commonly used neural network layers like convolution, depthwise separable convolution, fully-connected layers, and pooling operations. NNSupportFunctions provide utility functions such as data conversion routines and activation function tables, allowing for more complex neural network modules.

Distinct versions of the kernels cater to variable computational requirements and parameters, ensuring adaptability across different layer configurations. A particularly notable aspect is the adoption of fixed-point quantization, specifically supporting 8-bit and 16-bit data types. Fixed-point representation markedly reduces computational complexity and memory requirements compared to traditional 32-bit floating point representations, a crucial consideration on platforms without dedicated floating point units (FPUs).

Implementation Strategies

The optimized kernels leverage the capabilities of Arm Cortex-M processors, such as SIMD instructions, to enhance computational throughput. For instance, the matrix multiplication routines exploit $2 \times 2$ kernels, facilitating efficient data reuse and reducing load instructions. The convolution layers employ a partial im2col approach to balance performance gains with memory footprint constraints, while pooling layers utilize a split x-y pooling strategy to minimize computational overhead.

In terms of activation functions, the paper describes a SWAR (SIMD within a register) approach for the ReLU function that significantly accelerates computation by utilizing hardware-specific operations to eliminate conditional branching. Sigmoid and tanh functions are handled through efficient table-lookups, optimizing these typically expensive computations on fixed-point hardware.

Experimental Validation

The CMSIS-NN kernels are evaluated using a convolutional neural network trained on the CIFAR-10 dataset. The proposed approach achieves substantial improvements over baseline implementations, delivering a 4.6X increase in runtime/throughput and a 4.9X enhancement in energy efficiency. The core application processes approximately 10.1 images per second with high accuracy, demonstrating the feasibility of deploying sophisticated neural networks on low-power, resource-limited platforms.

Implications and Future Prospects

The introduction of CMSIS-NN kernels represents a significant step towards enabling complex neural network applications on IoT edge devices, mitigating the limitations imposed by constrained computational resources. The results suggest further potential for expanding neural network frameworks on microcontroller-class hardware, fostering the development of increasingly intelligent, autonomous edge devices.

This work lays the groundwork for ongoing research in optimizing deep learning inference on embedded systems, challenging researchers to explore novel quantization strategies, kernel optimizations, and cross-layer optimizations to enhance performance further. As edge computing continues to gain traction, CMSIS-NN serves as a robust foundation for advancing deep learning capabilities across diverse IoT applications.