- The paper demonstrates an end-to-end pipeline that converts trained DNNs into PoT-quantized models for efficient shift-based inference on custom hardware.
- It integrates multiple quantization schemes (QKeras, MSQ, APoT) and supports seamless conversion from PyTorch/TensorFlow to TFLite with negligible accuracy loss.
- The pipeline achieves up to 3.6x speedup and 78% energy reduction by mapping multiplications to LUT-based shifts, validating its hardware-software co-design approach on edge platforms.
PoTAcc: End-to-End Acceleration Pipeline for Power-of-Two Quantized DNNs
Introduction and Motivation
Power-of-two (PoT) quantization provides a compelling methodology for deploying DNNs on resource-constrained edge platforms by converting computationally intensive multiplications into bit-shift and addition operations. This reduction in arithmetic complexity yields substantial improvements in both area and power efficiency, especially when instantiated on custom hardware. Despite established accuracy for PoT-quantized DNNs in tasks like image classification, the principal bottleneck has been the lack of an integrated workflow—from model training, through quantized model conversion, to bit-shift-based hardware execution—suitable for real-world edge devices.
General-purpose CPUs/GPUs lack native shift-kernel support and cannot exploit the intrinsic efficiency benefits of PoT quantization. The absence of inference frameworks supporting PoT-quantized models on custom accelerators, and the design divergence among PoT quantization methods, further fragments this landscape. There is also a dearth of open-source, extensible hardware and toolflow solutions that support diverse PoT quantization strategies, end-to-end.
Technical Contributions
The PoTAcc pipeline addresses these systemic limitations by providing:
- Generalized Shift-Based Hardware: Efficient shift-based processing element (PE) designs for three predominant PoT quantization schemes—QKeras (single-term), MSQ (double-term/mixed), and APoT (multi-term additive)—supporting 4-bit weight quantization, with extensibility to lower precision settings.
- Custom Accelerator Platforms: Hardware implementations targeting CPU+FPGA SoC platforms (PYNQ-Z2 and Xilinx Kria), with designs scalable over varying GEMM unit configurations. The accelerators support integration with TensorFlow Lite (TFLite) via delegate, enabling actual deployment scenarios.
- Modular Model and Inference Framework Integration: Seamless conversion of PoT-quantized models trained in PyTorch or TensorFlow into TFLite format and subsequent preparation for shift-based inference. Model weight preprocessing ensures precision-aligned data representations for hardware execution.
- Quantitative Evaluation: Comprehensive analysis across multiple DNN architectures (CNNs, vision transformers) and datasets (CIFAR-10, ImageNet). The pipeline delivers empirical accuracy, speed, energy, and resource utilization results for the chosen hardware and quantization schemes.
- Open-Source Availability: All code, hardware, and conversion flows are open, facilitating reproducibility and extensibility.
Methodological Details
Quantization Schemes
PoTAcc supports multiple non-uniform quantization methods:
- QKeras: Single PoT term (no zero as a valid level), requiring only a single shift operation for multiplications.
- APoT (Additive PoT) and MSQ: Two PoT terms per level (with zero as a possible term), supported by encoding schemes allowing variable bit allocation to each term. The inclusion of zero-value levels increases PE complexity.
Hardware Accelerator Architecture
Based on the SECDA and SECDA-TFLite methodologies, PoTAcc provides a modular accelerator built around vector MAC (VMAC) units. Three novel shift-PE configurations are mapped onto these units, instantiated on FPGA platforms with scalable GEMM unit counts and buffer sizes. The architecture supports both convolutional and fully-connected layers, partitioning weight and activation data appropriately for efficient utilization. The design eliminates unnecessary multipliers in favor of LUT-mapped shifters, reducing DSP usage.
Optimizations such as selective weight-copy to local buffers, DMA weight preload, and support for buffer scaling are implemented to maximize throughput and minimize energy and memory overhead.
Model Conversion and Preprocessing
The pipeline retains quantization semantics from training to hardware, converting trained models to TFLite via Nobuco (PyTorch-to-TensorFlow) and the native TFLite converter. Nontrivial quantized weights are binary-packed, and bias scales are recomputed post-quantization to ensure inference fidelity. The workflow ensures no significant quantization drift through the conversion and preparation stages.
Experimental Results
Resource Utilization and Hardware Overheads
Compared to the multiplier-based baseline (VMAC), shift-based VSAC designs achieve a mean 20% LUT reduction, with up to 53.8% reduction in PEs on the largest accelerator instantiations (Kria with 8 GEMM units). DSP usage is minimized since PoTAcc maps shifts to LUTs, and precision reduction (from int8 to 4-bit) enables larger effective buffer sizes and higher PE utilization.
Accuracy
End-to-end pipeline accuracy demonstrates negligible degradation: 0.4% on CIFAR-10 (MobileNetV2, QKeras) and 1.9% on ImageNet (MobileNetV2, APoT) after model conversion and four-bit weight preprocessing. Precision discrepancies largely stem from TFLite's post-training quantization of activations to int8, not from hardware or quantization methodology.
Latency and Energy
The VSAC accelerators, when paired with the CPU:
- Achieve up to 3.6x inference speedup and 78.0% energy reduction versus CPU-only execution on the PYNQ-Z2 board.
- Yield a mean 1.0x speedup and 29.4% energy reduction on Kria.
- Deliver a further average 4.9–24.5% energy savings over optimized baseline VMAC accelerators due to the inherent efficiency of shifts over multiplications.
Performance effects are particularly pronounced for DNNs with large and deep layers, which maximize buffer utilization via increased shift-PE density. Reduction in precision directly translates to improved PE efficiency and resource allocation, validating the hardware-software co-design premise.
Implications and Future Directions
This work establishes a new reproducible baseline for efficient, hardware-aware PoT quantized inference on realistic edge device platforms. The PoTAcc pipeline enables systematic and fair comparison of different PoT quantization strategies in practical deployment scenarios, extending beyond CNNs to transformers and other topologies.
From a theoretical standpoint, PoTAcc quantitatively demonstrates that the principal tradeoff for PoT quantization—non-uniform quantization error versus hardware cost—can be tuned with minimal impact to accuracy, while enabling significant deployment scalability. Practically, the open, modular pipeline can accelerate research and industry adoption by demystifying hardware-software integration barriers.
Potential future improvements include extending shift-based acceleration to additional layer types (e.g., depthwise convolution), further optimizing activation/buffer dataflows to increase accelerator offload ratio, and exploiting temporal locality to reduce off-chip memory accesses. Integration with emerging open-source hardware synthesis tools and microarchitectural exploration for even lower-precision PoT quantization stand as promising directions.
Conclusion
PoTAcc delivers a comprehensive, open-source pipeline for end-to-end acceleration of PoT-quantized DNNs, with demonstrated efficacy in terms of accuracy retention, hardware resource improvement, and latency/energy reduction. This work provides both the methodology and the evidence base to facilitate practical deployment and future exploration of PoT quantization as a mainstream approach for efficient deep learning on edge devices (2605.06082).