Energy-efficiency gap between general-purpose and specialized ML accelerators

Determine how close the energy efficiency of general-purpose accelerators on machine learning workloads can approach that of specialized accelerators, and identify concrete architectural and microarchitectural techniques required to reach this target level of energy efficiency with general-purpose designs.

Background

Specialized machine learning accelerators deliver superior performance and energy efficiency on dense linear algebra workloads but at the cost of flexibility. In contrast, general-purpose accelerators based on programmable processing elements can support a wider range of tasks but typically incur control and memory overheads that reduce energy efficiency.

This paper builds on an optimized RISC-V cluster (Snitch) and introduces two general-purpose enhancements—zero-overhead loop nests and a double-buffering-aware, zero-conflict memory subsystem—to reduce inefficiencies in matrix multiplication. While these results narrow the gap with specialized designs such as OpenGeMM, the broader questions of the fundamental efficiency proximity achievable by general-purpose accelerators and the full set of methods required to reach such proximity remain unresolved.