System Integration of Xilinx DPU and HDMI for Real-Time inference in PYNQ Environment with Image Enhancement (2312.09506v2)
Abstract: Use of edge computing in application of Computer Vision (CV) is an active field of research. Today, most CV applications make use of Convolutional Neural Networks (CNNs) to inference on and interpret video data. These edge devices are responsible for several CV related tasks, such as gathering, processing and enhancing, inferencing on, and displaying video data. Due to ease of reconfiguration, computation on FPGA fabric is used to achieve such complex computation tasks. Xilinx provides the PYNQ environment as a user-friendly interface that facilitates in Hardware/Software system integration. However, to the best of authors' knowledge there is no end-to-end framework available for the PYNQ environment that allows Hardware/Software system integration and deployment of CNNs for real-time input feed from High Definition Multimedia Interface (HDMI) input to HDMI output, along with insertion of customized hardware IPs. In this work we propose an integration of rea\textbf{L}-time image \textbf{E}nancement IP with \textbf{A}I inferencing engine in the \textbf{P}ynq environment (\textbf{LEAP}), that integrates HDMI, AI acceleration, image enhancement in the PYNQ environment for Xilinx's Microprocessor on Chip (MPSoC) platform. We evaluate our methodology with two well know CNN models, Resnet50 and YOLOv3. To validate our proposed methodology, LEAP, a simple image enhancement algorithm, histogram equalization, is designed and integrated in the FPGA fabric along with Xilinx's Deep Processing Unit (DPU). Our results show successful implementation of end-to-end integration using completely open source information.
- X. Feng, Y. Jiang, X. Yang, M. Du, and X. Li, “Computer vision algorithms and hardware implementations: A survey,” Integration, vol. 69, pp. 309–320, 2019.
- X. Wang, Y. Han, V. C. M. Leung, D. Niyato, X. Yan, and X. Chen, “Convergence of edge computing and deep learning: A comprehensive survey,” IEEE Communications Surveys & Tutorials, vol. 22, no. 2, pp. 869–904, 2020.
- A. A. Adeyemo, J. J. Sanderson, T. A. Odetola, F. Khalid, and S. R. Hasan, “Stain: Stealthy avenues of attacks on horizontally collaborated convolutional neural network inference and their mitigation,” IEEE Access, vol. 11, pp. 10520–10534, 2023.
- S. Mittal, S. Gupta, and S. Dasgupta, “Fpga: An efficient and promising platform for real-time image processing applications,” in National Conference On Research and Development In Hardware Systems (CSI-RDHS), 2008.
- T. Sandefur and S. R. Hasan, “Framework to benchmark cnns (fabcnn) for processing real-time hd video streams on fpgas,” in 2022 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1778–1782, IEEE, 2022.
- R. Huang and J. Wang, “A design of object detection system in fog,” in 2022 IEEE 6th Information Technology and Mechatronics Engineering Conference (ITOEC), vol. 6, pp. 604–608, 2022.
- J.-M. Morel, A.-B. Petro, and C. Sbert, “Fast implementation of color constancy algorithms,” Proceedings of SPIE - The International Society for Optical Engineering, vol. 7241, 01 2009.
- J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263–7271, 2017.
- J. Sanderson and S. R. Hasan, “Integrating gstreamer with xilinx’s zcu104 edge platform for real-time intelligent image enhancement,” in 2023 MidWest Symposium on Circuits And Systems (MWSCAS), IEEE. (Accepted, Publishing in progress).
- “DPUCZDX8G for Zynq UltraScale+ MPSoCs Product Guide (PG338).” https://docs.xilinx.com/r/en-US/pg338-dpu?tocId=3xsG16y_QFTWvAJKHbisEw2.
- “HDMI 1.4/2.0 Transmitter Subsystem Product Guide (PG235).” https://docs.xilinx.com/r/en-US/pg235-v-hdmi-tx-ss?tocId=XLSPfpAut4Tw1N_71V9XKQ.
- “ZCU104 VCU HDMI ROI TRD.” https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1268973650/Zynq+UltraScale+MPSoC+ZCU104+VCU+HDMI+ROI+2020.2.
- “PYNQ: Python productivity for Adaptive Computing platforms.” https://pynq.readthedocs.io/en/latest/.
- “PYNQ HDMI Examples.” https://github.com/Xilinx/PYNQ/tree/master/boards/Pynq-Z1/base/notebooks/video.
- “DPU-PYNQ Examples.” https://github.com/Xilinx/DPU-PYNQ/tree/master/pynq_dpu/notebooks.
- “Zynq UltraScale+ MPSoC ZCU104 Evaluation Kit.” https://www.xilinx.com/products/boards-and-kits/zcu104.html.
- Pearson Education Ltd., third ed., 2008.
- “DPU Resnet50.xmodel.” https://github.com/Xilinx/DPU-PYNQ/blob/master/pynq_dpu/tests/models/dpu_resnet50.xmodel.link.
- J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” arXiv preprint arXiv:1804.02767, 2018.
- “Vitis-AI Model Zoo.” https://github.com/Xilinx/Vitis-AI/tree/v2.5/model_zoo.
- “AMBA Specifications.” https://www.arm.com/architecture/system-architectures/amba/amba-specifications.
- “AMBA AXI and ACE Protocol Specification.” https://developer.arm.com/documentation/ihi0022/e.
- “AMBA 4 AXI4-Stream Protocol Specification.” https://developer.arm.com/documentation/ihi0051/a.
- “DPU-PYNQ.” https://github.com/Xilinx/DPU-PYNQ.
- “Vitis 2022.1.” https://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/vivado-design-tools/2022-1.html.
- “PetaLinux Building and System Customization.” https://xilinx.github.io/Vitis-Tutorials/2022-1/build/html/docs/Vitis_Platform_Creation/Feature_Tutorials/02_petalinux_customization/README.html.
- “PYNQ SD Card image.” https://pynq.readthedocs.io/en/v3.0.0/pynq_sd_card.html.
- “DPU YOLOv3.xmodel.” https://github.com/Xilinx/Vitis-AI/tree/v2.5/model_zoo/model-list/tf_yolov4_coco_416_416_60.3G_2.5.
- T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Doll’a r, and C. L. Zitnick, “Microsoft COCO: common objects in context,” CoRR, vol. abs/1405.0312, 2014.
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee, 2009.
- Jonathan Sanderson (2 papers)
- Syed Rafay Hasan (17 papers)