Papers
Topics
Authors
Recent
Search
2000 character limit reached

Mondrian: On-Device High-Performance Video Analytics with Compressive Packed Inference

Published 12 Mar 2024 in cs.CV | (2403.07598v1)

Abstract: In this paper, we present Mondrian, an edge system that enables high-performance object detection on high-resolution video streams. Many lightweight models and system optimization techniques have been proposed for resource-constrained devices, but they do not fully utilize the potential of the accelerators over dynamic, high-resolution videos. To enable such capability, we devise a novel Compressive Packed Inference to minimize per-pixel processing costs by selectively determining the necessary pixels to process and combining them to maximize processing parallelism. In particular, our system quickly extracts ROIs and dynamically shrinks them, reflecting the effect of the fast-changing characteristics of objects and scenes. It then intelligently combines such scaled ROIs into large canvases to maximize the utilization of inference accelerators such as GPU. Evaluation across various datasets, models, and devices shows Mondrian outperforms state-of-the-art baselines (e.g., input rescaling, ROI extractions, ROI extractions+batching) by 15.0-19.7% higher accuracy, leading to $\times$6.65 higher throughput than frame-wise inference for processing various 1080p video streams. We will release the code after the paper review.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. J. Yi, S. Choi, and Y. Lee, “Eagleeye: Wearable camera-based person identification in crowded urban spaces,” in Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, 2020, pp. 1–14.
  2. K. Yang, J. Yi, K. Lee, and Y. Lee, “Flexpatch: Fast and accurate object detection for on-device high-resolution live video analytics,” in IEEE INFOCOM 2022-IEEE Conference on Computer Communications.   IEEE, 2022, pp. 1898–1907.
  3. C. L. School, “Expectation of privacy,” https://www.law.cornell.edu/wex/expectation_of_privacy, accessed: 2022-06-17.
  4. A. W. Services, “Price samples of cloud gpu,” https://aws.amazon.com/ec2/instance-types/g4/, accessed: 2022-06-18.
  5. “Ultralytics YOLO-v5,” https://github.com/ultralytics/yolov5. Accessed: 18 Mar. 2023.
  6. P. Natarajan, P. K. Atrey, and M. Kankanhalli, “Multi-camera coordination and control in surveillance systems: A survey,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 11, no. 4, pp. 1–30, 2015.
  7. K. Apicharttrisorn, X. Ran, J. Chen, S. V. Krishnamurthy, and A. K. Roy-Chowdhury, “Frugal following: Power thrifty object detection and tracking for mobile augmented reality,” in Proceedings of the 17th conference on embedded networked sensor systems, 2019, pp. 96–109.
  8. X. Zeng, B. Fang, H. Shen, and M. Zhang, “Distream: scaling live video analytics with workload-adaptive distributed edge intelligence,” in Proceedings of the 18th Conference on Embedded Networked Sensor Systems, 2020, pp. 409–421.
  9. J. He, G. Baig, and L. Qiu, “Real-time deep video analytics on mobile devices,” in Proceedings of the Twenty-second International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, 2021, pp. 81–90.
  10. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in European conference on computer vision.   Springer, 2016, pp. 21–37.
  11. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861, 2017.
  12. T.-W. Chin, R. Ding, and D. Marculescu, “Adascale: Towards real-time video object detection using adaptive scaling,” Proceedings of Machine Learning and Systems, vol. 1, pp. 431–441, 2019.
  13. L. Liu, H. Li, and M. Gruteser, “Edge assisted real-time object detection for mobile augmented reality,” in The 25th Annual International Conference on Mobile Computing and Networking, 2019, pp. 1–16.
  14. S. Jiang, Z. Lin, Y. Li, Y. Shu, and Y. Liu, “Flexible high-resolution object detection on edge devices with tunable latency,” in Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, 2021, pp. 559–572.
  15. L. N. Huynh, Y. Lee, and R. K. Balan, “Deepmon: Mobile gpu-based deep learning framework for continuous vision applications,” in Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services, 2017, pp. 82–95.
  16. M. Xu, M. Zhu, Y. Liu, F. X. Lin, and X. Liu, “Deepcache: Principled cache for mobile deep vision,” in Proceedings of the 24th Annual International Conference on Mobile Computing and Networking, 2018, pp. 129–144.
  17. J. S. Jeong, J. Lee, D. Kim, C. Jeon, C. Jeong, Y. Lee, and B.-G. Chun, “Band: coordinated multi-dnn inference on heterogeneous mobile processors,” in Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, 2022, pp. 235–247.
  18. W. Seo, S. Cha, Y. Kim, J. Huh, and J. Park, “Slo-aware inference scheduler for heterogeneous processors in edge platforms,” ACM Transactions on Architecture and Code Optimization (TACO), vol. 18, no. 4, pp. 1–26, 2021.
  19. P. Kohl, A. Specker, A. Schumann, and J. Beyerer, “The mta dataset for multi-target multi-camera pedestrian tracking by weighted distance aggregation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 1042–1043.
  20. J. Hartmanis, “Computers and intractability: a guide to the theory of np-completeness (michael r. garey and david s. johnson),” Siam Review, vol. 24, no. 1, p. 90, 1982.
  21. I. Gokarn, H. Sabbella, Y. Hu, T. Abdelzaher, and A. Misra, “Mosaic: Spatially-multiplexed edge ai optimization over multiple concurrent video sensing streams,” in Proceedings of the 14th Conference on ACM Multimedia Systems, 2023, pp. 278–288.
  22. R. Xu, C.-l. Zhang, P. Wang, J. Lee, S. Mitra, S. Chaterji, Y. Li, and S. Bagchi, “Approxdet: content and contention-aware approximate object detection for mobiles,” in Proceedings of the 18th Conference on Embedded Networked Sensor Systems, 2020, pp. 449–462.
  23. R. Xu, J. Lee, P. Wang, S. Bagchi, Y. Li, and S. Chaterji, “Litereconfig: cost and content aware reconfiguration of video object detection systems for mobile gpus,” in Proceedings of the Seventeenth European Conference on Computer Systems, 2022, pp. 334–351.
  24. J. Jiang, G. Ananthanarayanan, P. Bodik, S. Sen, and I. Stoica, “Chameleon: scalable adaptation of video analytics,” in Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, 2018, pp. 253–266.
  25. B. Zhang, X. Jin, S. Ratnasamy, J. Wawrzynek, and E. A. Lee, “Awstream: Adaptive wide-area streaming analytics,” in Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, 2018, pp. 236–252.
  26. P. J. Kolesar, “A branch and bound algorithm for the knapsack problem,” Management science, vol. 13, no. 9, pp. 723–735, 1967.
  27. J. Jylänki, “A thousand ways to pack the bin-a practical approach to two-dimensional rectangle bin packing,” retrived from http://clb. demon. fi/files/RectangleBinPack. pdf, 2010.
  28. A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” arXiv preprint arXiv:2004.10934, 2020.
  29. W. Zhang, Z. He, L. Liu, Z. Jia, Y. Liu, M. Gruteser, D. Raychaudhuri, and Y. Zhang, “Elf: accelerate high-resolution mobile deep vision with content-aware parallel offloading,” in Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, 2021, pp. 201–214.
  30. K. Du, A. Pervaiz, X. Yuan, A. Chowdhery, Q. Zhang, H. Hoffmann, and J. Jiang, “Server-driven video streaming for deep learning inference,” in Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, 2020, pp. 557–570.
  31. S. Lee and S. Nirjon, “Fast and scalable in-memory deep multitask learning via neural weight virtualization,” in Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services, 2020, pp. 175–190.
  32. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al., “{{\{{TensorFlow}}\}}: a system for {{\{{Large-Scale}}\}} machine learning,” in 12th USENIX symposium on operating systems design and implementation (OSDI 16), 2016, pp. 265–283.
  33. Alibaba, “Alibaba mnn,” https://github.com/alibaba/MNN, accessed: 2022-06-18.
  34. Y. Li, A. Padmanabhan, P. Zhao, Y. Wang, G. H. Xu, and R. Netravali, “Reducto: On-camera filtering for resource-efficient real-time video analytics,” in Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, 2020, pp. 359–376.
  35. J. Shi et al., “Good features to track,” in 1994 Proceedings of IEEE conference on computer vision and pattern recognition.   IEEE, 1994, pp. 593–600.
  36. J. Canny, “A computational approach to edge detection,” IEEE Transactions on pattern analysis and machine intelligence, no. 6, pp. 679–698, 1986.
  37. S. Suzuki et al., “Topological structural analysis of digitized binary images by border following,” Computer vision, graphics, and image processing, vol. 30, no. 1, pp. 32–46, 1985.
  38. A. R. Kumar, B. Ravindran, and A. Raghunathan, “Pack and detect: Fast object detection in videos using region-of-interest packing,” in Proceedings of the ACM India Joint International Conference on Data Science and Management of Data, 2019, pp. 150–156.
  39. Google, “Tensorflow lite,” https://www.tensorflow.org/lite, accessed: 2022-06-18.
  40. Q. Zhang, H. Sun, X. Wu, and H. Zhong, “Edge video analytics for public safety: A review,” Proceedings of the IEEE, vol. 107, no. 8, pp. 1675–1696, 2019.
  41. M. Eldib, F. Deboeverie, W. Philips, and H. Aghajan, “Behavior analysis for elderly care using a network of low-resolution visual sensors,” Journal of Electronic Imaging, vol. 25, no. 4, p. 041003, 2016.
  42. A. Prati, C. Shan, and K. I.-K. Wang, “Sensors, vision and networks: From video surveillance to activity recognition and health monitoring,” Journal of Ambient Intelligence and Smart Environments, vol. 11, no. 1, pp. 5–22, 2019.
  43. A. Balasundaram and C. Chellappan, “An intelligent video analytics model for abnormal event detection in online surveillance video,” Journal of Real-Time Image Processing, vol. 17, no. 4, pp. 915–930, 2020.
  44. K. Gautam and S. K. Thangavel, “Video analytics-based intelligent surveillance system for smart buildings,” Soft Computing, vol. 23, no. 8, pp. 2813–2837, 2019.
  45. G. Ananthanarayanan, P. Bahl, P. Bodík, K. Chintalapudi, M. Philipose, L. Ravindranath, and S. Sinha, “Real-time video analytics: The killer app for edge computing,” computer, vol. 50, no. 10, pp. 58–67, 2017.
  46. J. Barthélemy, N. Verstaevel, H. Forehead, and P. Perez, “Edge-computing video analytics for real-time traffic monitoring in a smart city,” Sensors, vol. 19, no. 9, p. 2048, 2019.
  47. M. Naphade, S. Wang, D. C. Anastasiu, Z. Tang, M.-C. Chang, X. Yang, Y. Yao, L. Zheng, P. Chakraborty, C. E. Lopez et al., “The 5th ai city challenge,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 4263–4273.
  48. M.-C. Chang, C.-K. Chiang, C.-M. Tsai, Y.-K. Chang, H.-L. Chiang, Y.-A. Wang, S.-Y. Chang, Y.-L. Li, M.-S. Tsai, and H.-Y. Tseng, “Ai city challenge 2020-computer vision for smart transportation applications,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 2020, pp. 620–621.
  49. S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” Advances in neural information processing systems, vol. 28, 2015.
  50. M. Tan, R. Pang, and Q. V. Le, “Efficientdet: Scalable and efficient object detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 10 781–10 790.
  51. M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in International conference on machine learning.   PMLR, 2019, pp. 6105–6114.
  52. B. Uzkent and S. Ermon, “Learning when and where to zoom with deep reinforcement learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 12 345–12 354.
  53. H. Wang, A. Kembhavi, A. Farhadi, A. L. Yuille, and M. Rastegari, “Elastic: Improving cnns with dynamic scaling policies,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2258–2267.
  54. M. Zhu, K. Han, E. Wu, Q. Zhang, Y. Nie, Z. Lan, and Y. Wang, “Dynamic resolution network,” Advances in Neural Information Processing Systems, vol. 34, 2021.
  55. M. Liu, X. Ding, and W. Du, “Continuous, real-time object detection on mobile devices without offloading,” in 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS).   IEEE, 2020, pp. 976–986.
  56. Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang, “Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,” ACM SIGARCH Computer Architecture News, vol. 45, no. 1, pp. 615–629, 2017.
  57. S. Laskaridis, S. I. Venieris, M. Almeida, I. Leontiadis, and N. D. Lane, “Spinn: synergistic progressive inference of neural networks over device and cloud,” in Proceedings of the 26th annual international conference on mobile computing and networking, 2020, pp. 1–15.
  58. M. Almeida, S. Laskaridis, S. I. Venieris, I. Leontiadis, and N. D. Lane, “Dyno: Dynamic onloading of deep neural networks from cloud to device,” ACM Transactions on Embedded Computing Systems (TECS), 2021.
  59. Z. Wang, L. Zheng, Y. Liu, Y. Li, and S. Wang, “Towards real-time multi-object tracking,” in European Conference on Computer Vision.   Springer, 2020, pp. 107–122.
  60. P. Dendorfer, A. Osep, A. Milan, K. Schindler, D. Cremers, I. Reid, S. Roth, and L. Leal-Taixé, “Motchallenge: A benchmark for single-camera multiple target tracking,” International Journal of Computer Vision, vol. 129, no. 4, pp. 845–881, 2021.
  61. X. Dong, Z. Ouyang, Z. Guo, and J. Niu, “Polarmask-tracker: Lightweight multi-object tracking and segmentation model for edge device,” in 2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom).   IEEE, 2021, pp. 689–696.
  62. B. Shuai, A. Berneshawi, X. Li, D. Modolo, and J. Tighe, “Siammot: Siamese multi-object tracking,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 12 372–12 382.
  63. C.-Y. Tsai and Y.-K. Su, “Mobilenet-jde: a lightweight multi-object tracking model for embedded systems,” Multimedia Tools and Applications, vol. 81, no. 7, pp. 9915–9937, 2022.
  64. N. Corporation, “NVIDIA hyper-q,” http://developer.download.nvidia.com/compute/DevZone/C/html_x64/6_Advanced/simpleHyperQ/doc/HyperQ.pdf, accessed: 2022-06-22.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up