Co-Design of CNN Accelerators for TinyML using Approximate Matrix Decomposition

Abstract: The paradigm shift towards local and on-device inference under stringent resource constraints is represented by the tiny machine learning (TinyML) domain. The primary goal of TinyML is to integrate intelligence into tiny, low-cost devices under strict resource, energy, and latency constraints. However, the ultra-resource-constrained nature of these devices can lead to increased inference execution time, which can be detrimental in latency critical applications. At the same time, TinyML applications are often associated with sensitive data. As such, latency optimization approaches that rely on training samples are infeasible when such data is unavailable, proprietary, or sensitive, highlighting a pressing need for optimization approaches that do not require access to the training dataset and can be applied directly to pre-trained models. Replacing costly multiplications with more hardware-efficient operations, such as shifts and additions, has been proposed as an effective method for reducing inference latency. However, post-training power-of-two (Po2) approaches are scarce and, in many cases, lead to unacceptable accuracy loss. In this work, we propose a framework that applies approximate matrix decomposition to a given CNN in order to optimize hardware implementations subject to strict constraints and without any need of re-training or fine-tuning steps. The genetic algorithm-driven framework explores different matrix decompositions and resulting multiplier-less CNN accelerator designs for FPGA targets. A comprehensive evaluation of different TinyML benchmarks demonstrates our framework's efficacy in generating latency-optimized implementations that satisfy strict accuracy and resource constraints, achieving an average 33% latency improvement with an average accuracy loss of 1.3% compared to typical systolic array-based FPGA accelerators.

• The paper introduces a novel co-design framework that leverages approximate matrix decomposition to replace multipliers with shift-and-add operations in CNNs.
• It employs NSGA-II optimization to balance accuracy, latency, and energy efficiency, achieving up to 34% latency reduction with minimal accuracy drop.
• The methodology enables post-training conversion of CNN weights into power-of-two values without retraining, facilitating efficient FPGA-based accelerator designs.

Co-Design of CNN Accelerators for TinyML using Approximate Matrix Decomposition

Introduction

The proliferation of TinyML, which deploys machine learning models on ultra-resource-constrained devices, necessitates the development of highly efficient hardware acceleration solutions for deep neural networks. Convolutional Neural Networks (CNNs) are heavily utilized in such domains, but their deployment is challenged by limitations in computational resources, on-device energy, and latency—factors exacerbated by reliance on expensive multiplier operations. The work "Co-Design of CNN Accelerators for TinyML using Approximate Matrix Decomposition" (2604.16113) addresses these constraints through a post-training, hardware-aware co-design framework. The methodology comprises a novel application of approximate matrix decomposition that transforms CNN weights into power-of-two (Po2) values, thereby enabling multiplier-less (shift-and-add based) inference without requiring retraining or fine-tuning, all while systematically co-optimizing accelerator configurations for FPGAs.

Approximate Matrix Decomposition for Multiplier-less CNNs

CNN inference in resource-constrained TinyML scenarios benefits significantly from approaches that replace multiplication with shift-and-add operations, leveraging the hardware efficiency of FPGAs. Prior work in Po2 quantization generally requires retraining to preserve accuracy, an infeasible condition when training data is unavailable or sensitive. This paper leverages approximate Weight Matrix Decomposition (WMD), which acts directly on pre-trained CNN weight matrices by decomposing them into a set of sparse matrices with Po2-constrained nonzero values.

The decomposition process is parameterized by the number of factors ($P$), allowed exponent ($Z$), sparsity level ($E$), matrix width ($M$), and slice width ($S_W$), enabling fine-grained control over the trade-off between representational accuracy and hardware efficiency. Input feature slices are multiplied by the decomposed weight matrices, and partial summations are accumulated to generate output activations.

Figure 1: (a) CNN weight matrix transformation to $N$ and $M$ dimensions; (b) WMD decomposition into Po2-weighted factors; (c) Shift-add multiplication with input and accumulation of partial sums.

Hardware Accelerator Architecture

A programmable CNN accelerator is designed, centered around a 2D systolic array (SA) of lightweight processing elements (PEs) that exclusively perform shift-and-add computations. Each PE is microarchitected with pipelined blocks dedicated to different stages of the matrix decomposition (F-blocks), using only barrel shifters and adders to effect Po2 multiplication.

Key optimizations include:

• Time-multiplexed F-blocks for deeper decompositions.
• Position-encoding savings, via diagonal optimizations in F-matrices.
• Exclusive use of right-shift operations (Po2 exponents negative), minimizing hardware resource requirements.

Input data buffering and result reduction across the 2D array are achieved through on-chip memory, with careful layering of PE grid dimensions to optimize latency/resource trade-offs under fixed FPGA LUT budgets.

Figure 2: (a) Schematic of a WMD-based PE microarchitecture; (b) Systolic array composed of PEs, each supporting pipelined shift-and-add operations within the matrix decomposition framework.

Hardware-Aware Co-Design Framework

The framework performs hardware-software co-design by jointly optimizing the parameters of weight decomposition and corresponding accelerator configurations. A high-level methodology is employed:

1. Inputs: Pre-trained CNN, user-defined accuracy/latency constraints, full decomposition parameter space.
2. Design Space Partition: "Hard" accelerator parameters are fixed network-wide, while "soft" parameters are layer-specific.
3. Surrogate Models: Layerwise analytical models estimate hardware resource utilization and inference latency for candidate configurations.
4. Genetic Algorithm-based Exploration: NSGA-II is used to traverse the constrained search space, maintaining Pareto fronts of optimal accuracy-latency-resource trade-offs.
5. Output: Hardware-compliant, decomposed CNN-accelerator pairs, with all multiplier operations replaced by shift-and-add logic and no accuracy-impacting retraining.

   Figure 3: Framework methodology: accepting a pre-trained CNN and constraints, the NSGA-II driven exploration produces a Pareto front of latency-accuracy optimal, decomposed network-accelerator solutions.

Experimental Evaluation

Benchmark and Setup

The methodology is validated using the MLPerf Tiny benchmark suite, including ResNet, MobileNetV1, and DS-CNN on datasets such as CIFAR-10, VWW, and Speech Commands, respectively. Target hardware is the Artix-7 XC7A100TCSG324-1 FPGA, synthesized via Vivado 2023.1. Custom parameterizable SAs are instantiated both for the WMD-based PEs and baseline MAC-based designs across 4–8 bit precision configurations.

Pareto Frontier and Performance Summary

NSGA-II optimization demonstrates that the majority of evolved Pareto-optimal solutions deliver accuracy drops within 2% of float32 baselines, coupled with significant normalized latency reductions (up to 34%). In some cases (e.g., DS-CNN), decomposed Po2 networks surpass original accuracy at iso-latency, underscoring the expressive efficacy of the decomposition despite the absence of retraining.

Figure 4: NSGA-II discovered Pareto front for various models, showing accuracy loss versus normalized speedup relative to an 8-bit MAC-baseline SA.

Comparison with Post-Training Quantization (PTQ) and Po2 Approaches

The proposed approach consistently outperforms PTQ using MAC-based SAs at comparable accuracy drops, especially in throughput and energy efficiency. Even when compared to ShiftCNN, a state-of-the-art post-training Po2-based FPGA design, the proposed solution attains higher throughput at lower or equivalent accuracy drops with similar resource usage.

Figure 5: Comparison of SA (MAC) and the framework's accelerators, illustrating the superior positioning of co-designed solutions in latency, accuracy, and energy-efficiency space.

Figure 6: Comparison between the framework and Shift-CNN: achieved GOPS and resulting accuracy drop for high-performance points on the Pareto front.

Critical Numerical Results and Contrasts

• Latency: Average 33% reduction in inference latency (as high as 1.82x speedup for DS-CNN), with simultaneous hardware resource compliance.
• Accuracy drop: Mean loss 1.3% relative to floating-point, remaining within the TinyML-acceptable 2% window.
• Throughput: 41% higher throughput on average versus MAC-based SAs, and up to 2.4x the throughput of comparable Po2-based state-of-the-art FPGA implementations under matched resource budgets.
• Energy efficiency: WMD-PE architectures exhibit substantial improvements in GOPS/W, especially prominent on DS-CNN.
• Constraints: The genetic framework maintains strict resource and accuracy constraints throughout, and never relies on retraining or access to training data.

Implications and Future Perspectives

This work demonstrates a robust path to efficient TinyML deployment of CNNs under real-world hardware constraints, eliminating the practical burden of retraining or access to data. The co-optimization method shown here provides both a methodology and architectural principles for the development of next-generation highly customizable neural accelerators. Extensions to this paradigm could be explored for transformer architectures or more heavily pruned/sparse models, where the dimensionality and granularity of decomposition parameters might yield even more pronounced efficiency gains. The inherent generality of the genetic search approach further suggests readiness for rapid adaptation to emerging neural operation primitives and process nodes.

Conclusion

The co-design framework for post-training, hardware-aware Po2 CNN acceleration offers a rigorously validated solution for TinyML deployment on FPGAs, efficiently mapping pre-trained models to multiplier-less architectures. By leveraging approximate matrix decomposition and multi-objective genetic optimization, significant reductions in latency and resource consumption are achieved at negligible accuracy loss, without retraining. The framework consistently assures Pareto-optimal trade-offs in practical deployment settings, advancing the field of energy-aware neural inference in constrained environments.