Benchmarking TinyML Systems: Challenges and Direction (2003.04821v4)

Abstract: Recent advancements in ultra-low-power machine learning (TinyML) hardware promises to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted benchmark for these systems. Benchmarking allows us to measure and thereby systematically compare, evaluate, and improve the performance of systems and is therefore fundamental to a field reaching maturity. In this position paper, we present the current landscape of TinyML and discuss the challenges and direction towards developing a fair and useful hardware benchmark for TinyML workloads. Furthermore, we present our four benchmarks and discuss our selection methodology. Our viewpoints reflect the collective thoughts of the TinyMLPerf working group that is comprised of over 30 organizations.

• The paper establishes a standardized benchmark for TinyML systems, addressing challenges in power measurement and memory constraints.
• It details four benchmarks across audio, visual, image classification, and anomaly detection to define clear evaluation protocols.
• The work emphasizes community collaboration to refine benchmarks and drive innovations in low-power AI applications.

Benchmarking TinyML Systems: Challenges and Direction

The paper "Benchmarking TinyML Systems: Challenges and Direction" addresses the emerging field of TinyML, which focuses on enabling machine learning inference on ultra-low-power devices such as microcontrollers. It emphasizes the importance of developing a standardized benchmark to facilitate the evaluation and comparison of TinyML systems. This narrative emerges from the collective insights of the TinyMLPerf working group, consisting of over 30 organizations.

Overview of TinyML

TinyML seeks to perform machine learning inference on devices that operate under strict power constraints, generally below a milliWatt. By integrating inference capabilities directly into these devices, TinyML systems can enhance responsiveness, privacy, and autonomy by minimizing the need for energy-intensive wireless communication. This reduces the dependency on higher-power devices or cloud-based processing, thereby enabling a new class of always-on, battery-powered smart applications.

Challenges in Benchmarking

The paper outlines several unique challenges in developing an effective TinyML benchmark:

1. Power Measurement: Accurately measuring power consumption across varying device types and configurations is complex. The paper notes the significant difference in power budgets between TinyML systems and traditional systems, which complicates consistent measurement.
2. Memory Constraints: TinyML systems have limited memory resources. Unlike systems with gigabyte-scale memory, TinyML devices must manage with kilobytes, significantly affecting model sizes and performance metrics.
3. Hardware and Software Heterogeneity: TinyML hardware ranges from general-purpose microcontrollers to specialized inference engines, complicating standard benchmarking protocols. Software approaches also vary widely, with deployment methods including hand coding, code generation, and ML interpreters.
4. Use Case Diversity: TinyML's application domains, such as audio analysis, image processing, and anomaly detection in industrial settings, require benchmarks that accommodate and accurately represent various computational models and data types.

Proposed Benchmarks

The paper presents four initial benchmarks as a step towards resolving these challenges, each representing a different application domain in TinyML:

• Audio Wake Words: Utilizing the Speech Commands dataset, this benchmark targets keyword spotting tasks typical in smart assistants.
• Visual Wake Words: Based on the Visual Wake Words dataset, this aims to classify images as containing a person or not, highlighting the intersection of computer vision and low-power constraints.
• Image Classification: Using the CIFAR10 dataset, this explores image recognition capabilities within TinyML constraints, employing models like ResNet.
• Anomaly Detection: Focused on audio data, such as the ToyADMOS dataset, to identify abnormal sound patterns, which is crucial for predictive maintenance and fault detection.

Measurement and Metrics

The proposed benchmark suite prioritizes inference latency with optional power consumption metrics. It distinguishes between open and closed divisions to allow flexibility and comparability in results, addressing the heterogeneity challenge. In closed divisions, reference models provide baseline results, while the open division allows for showcasing innovative optimizations.

Future Directions

The authors emphasize the importance of community involvement in refining and expanding the benchmark suite over time. They propose accepting submissions for evaluation as part of a continuous improvement effort.

Implications and Future Developments

The establishment of a robust TinyML benchmark is crucial for accelerating the development and deployment of low-power AI technologies. As TinyML hardware and software evolve, benchmarks will guide performance optimization, resource allocation, and energy efficiency. Looking forward, the integration of more complex tasks and diverse datasets into this framework will enhance the rigorous evaluation of TinyML systems. This field's growth will likely lead to increased computational capabilities at the edge, pushing the frontier of IoT applications and beyond.