Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DeePref: Deep Reinforcement Learning For Video Prefetching In Content Delivery Networks (2310.07881v1)

Published 11 Oct 2023 in cs.NI, cs.AI, and cs.LG

Abstract: Content Delivery Networks carry the majority of Internet traffic, and the increasing demand for video content as a major IP traffic across the Internet highlights the importance of caching and prefetching optimization algorithms. Prefetching aims to make data available in the cache before the requester places its request to reduce access time and improve the Quality of Experience on the user side. Prefetching is well investigated in operating systems, compiler instructions, in-memory cache, local storage systems, high-speed networks, and cloud systems. Traditional prefetching techniques are well adapted to a particular access pattern, but fail to adapt to sudden variations or randomization in workloads. This paper explores the use of reinforcement learning to tackle the changes in user access patterns and automatically adapt over time. To this end, we propose, DeePref, a Deep Reinforcement Learning agent for online video content prefetching in Content Delivery Networks. DeePref is a prefetcher implemented on edge networks and is agnostic to hardware design, operating systems, and applications. Our results show that DeePref DRQN, using a real-world dataset, achieves a 17% increase in prefetching accuracy and a 28% increase in prefetching coverage on average compared to baseline approaches that use video content popularity as a building block to statically or dynamically make prefetching decisions. We also study the possibility of transfer learning of statistical models from one edge network into another, where unseen user requests from unknown distribution are observed. In terms of transfer learning, the increase in prefetching accuracy and prefetching coverage are [$30%$, $10%$], respectively. Our source code will be available on Github.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (85)
  1. Measurement study of netflix, hulu, and a tale of three cdns. IEEE/ACM Transactions on Networking, 23(6):1984–1997, 2014.
  2. Space: Segment prefetching and caching at the edge for adaptive video streaming. IEEE Access, 11:21783–21798, 2023.
  3. Effectively prefetching remote memory with leap. In 2020 USENIX Annual Technical Conference (USENIX ATC 20), pages 843–857, 2020.
  4. Abs: A low-cost adaptive controller for prefetching in a banked shared last-level cache. ACM Transactions on Architecture and Code Optimization (TACO), 8(4):1–20, 2012.
  5. Benefits and schemes of prefetching from cloud to fog networks. In 2017 IEEE 6th International Conference on Cloud Networking (CloudNet), pages 1–5. IEEE, 2017.
  6. Potentials and limitations of fault-based markov prefetching for virtual memory pages. ACM SIGMETRICS Performance Evaluation Review, 27(1):206–207, 1999.
  7. Machine learning at the mobile edge: The case of dynamic adaptive streaming over http (dash). IEEE Transactions on Network and Service Management, 19(4):4779–4793, 2022.
  8. Laszlo A. Belady. A study of replacement algorithms for a virtual-storage computer. IBM Systems journal, 5(2):78–101, 1966.
  9. Pythia: A customizable hardware prefetching framework using online reinforcement learning. In MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, pages 1121–1137, 2021.
  10. Web caching and zipf-like distributions: Evidence and implications. In IEEE INFOCOM’99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No. 99CH36320), volume 1, pages 126–134. IEEE, 1999.
  11. A study of integrated prefetching and caching strategies. ACM SIGMETRICS Performance Evaluation Review, 23(1):188–197, 1995.
  12. Automatic i/o hint generation through speculative execution. USENIX, 1999.
  13. Effective hardware-based data prefetching for high-performance processors. IEEE transactions on computers, 44(5):609–623, 1995.
  14. An adaptive io prefetching approach for virtualized data centers. IEEE Transactions on Services Computing, 10(3):328–340, 2015.
  15. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
  16. VNI Cisco. Cisco visual networking index: Forecast and trends, 2017–2022. White Paper, 2018.
  17. Framework for tcp throughput testing. Technical report, 2011.
  18. User perception of adapting video quality. International Journal of Human-Computer Studies, 64(8):637–647, 2006.
  19. Proactive instruction fetch. In Proceedings of the 44th Annual IEEE/ACM International Symposium on Microarchitecture, pages 152–162, 2011.
  20. Toward qoe-assured 4k video-on-demand delivery through mobile edge virtualization with adaptive prefetching. IEEE Transactions on Multimedia, 19(10):2222–2237, 2017.
  21. Binny S Gill and Luis Angel D Bathen. Amp: Adaptive multi-stream prefetching in a shared cache. In FAST, volume 7, pages 185–198, 2007.
  22. Sarc: Sequential prefetching in adaptive replacement cache. In USENIX Annual Technical Conference, General Track, pages 293–308, 2005.
  23. Youtube traffic characterization: a view from the edge. In Proceedings of the 7th ACM SIGCOMM conference on Internet measurement, pages 15–28, 2007.
  24. Yao Guo. Compiler-assisted hardware-based data prefetching for next generation processors. University of Massachusetts Amherst, 2007.
  25. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4):19, 2016.
  26. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  27. Musa: Wi-fi ap-assisted video prefetching via tensor learning. In 2017 IEEE/ACM 25th International Symposium on Quality of Service (IWQoS), pages 1–6. IEEE, 2017.
  28. Improving fairness, efficiency, and stability in http-based adaptive video streaming with festive. In Proceedings of the 8th international conference on Emerging networking experiments and technologies, pages 97–108, 2012.
  29. Dulo: an effective buffer cache management scheme to exploit both temporal and spatial locality. In Proceedings of the 4th conference on USENIX Conference on File and Storage Technologies, volume 4, pages 8–8, 2005.
  30. A prefetching scheme exploiting both data layout and access history on disk. ACM Transactions on Storage (TOS), 9(3):1–23, 2013.
  31. {{\{{FAST}}\}}: Quick application launch on {{\{{Solid-State}}\}} drives. In 9th USENIX Conference on File and Storage Technologies (FAST 11), 2011.
  32. Users of the world, unite! the challenges and opportunities of social media. Business horizons, 53(1):59–68, 2010.
  33. Near-optimal parallel prefetching and caching. SIAM Journal on computing, 29(4):1051–1082, 2000.
  34. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238–1274, 2013.
  35. Rdip: Return-address-stack directed instruction prefetching. In 2013 46th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 260–271. IEEE, 2013.
  36. Adam Langley et al. Transport layer security (tls) snap start. Working Draft, 2010.
  37. Transport layer security (tls) false start. Technical report, 2016.
  38. Automatic pool allocation: improving performance by controlling data structure layout in the heap. ACM Sigplan Notices, 40(6):129–142, 2005.
  39. Deep learning. nature, 521(7553):436–444, 2015.
  40. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334–1373, 2016.
  41. Integrated prefetching and caching for adaptive video streaming over http: an online approach. In Proceedings of the 6th ACM Multimedia Systems Conference, pages 142–152. ACM, 2015.
  42. Big data: the management revolution. Harvard business review, 90(10):60–68, 2012.
  43. The nist definition of cloud computing. 2011.
  44. Sparsh Mittal. A survey of recent prefetching techniques for processor caches. ACM Computing Surveys (CSUR), 49(2):1–35, 2016.
  45. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
  46. Insight and perspectives for content delivery networks. Communications of the ACM, 49(1):101–106, 2006.
  47. Biswabandan Panda. Spac: A synergistic prefetcher aggressiveness controller for multi-core systems. IEEE Transactions on Computers, 65(12):3740–3753, 2016.
  48. Caffeine: A utility-driven prefetcher aggressiveness engine for multicores. ACM Transactions on Architecture and Code Optimization (TACO), 12(3):1–25, 2015.
  49. A taxonomy and survey of content delivery networks. Grid Computing and Distributed Systems Laboratory, University of Melbourne, Technical Report, 4:70, 2007.
  50. Informed prefetching and caching. In Proceedings of the fifteenth ACM symposium on Operating systems principles, pages 79–95, 1995.
  51. Semantic locality and context-based prefetching using reinforcement learning. In 2015 ACM/IEEE 42nd Annual International Symposium on Computer Architecture (ISCA), pages 285–297. IEEE, 2015.
  52. Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
  53. Compiler orchestrated prefetching via speculation and predication. ACM SIGARCH Computer Architecture News, 32(5):189–198, 2004.
  54. A taxonomy and survey of cloud computing systems. In 2009 Fifth International Joint Conference on INC, IMS and IDC, pages 44–51. Ieee, 2009.
  55. Artificial intelligence: a modern approach. Pearson Education Limited, 2013.
  56. Fast-track session establishment for tls. In NDSS. Citeseer, 2002.
  57. Revisiting caching in content delivery networks. In The 2014 ACM international conference on Measurement and modeling of computer systems, pages 567–568, 2014.
  58. Intelligent admission control manager for fog-integrated cloud: A hybrid machine learning approach. Concurrency and Computation: Practice and Experience, page e6687, 2021.
  59. Edge computing: Vision and challenges. IEEE Internet of Things Journal, 3(5):637–646, 2016.
  60. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016.
  61. {{\{{Context-Aware}}\}} prefetching at the storage server. In 2008 USENIX Annual Technical Conference (USENIX ATC 08), 2008.
  62. The case for prefetching and prevalidating tls server certificates. In NDSS, volume 12, 2012.
  63. Characterizing the workload of a netflix streaming video server. In 2016 IEEE International Symposium on Workload Characterization (IISWC), pages 1–12. IEEE, 2016.
  64. Automated control of aggressive prefetching for http streaming video servers. In Proceedings of International Conference on Systems and Storage, pages 1–11, 2014.
  65. Methodologies for generating http streaming video workloads to evaluate web server performance. In Proceedings of the 5th Annual International Systems and Storage Conference, pages 1–12, 2012.
  66. Combining prefetch control and cache partitioning to improve multicore performance. In 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 953–962. IEEE, 2019.
  67. Reinforcement learning: An introduction. MIT press, 2018.
  68. Introduction to reinforcement learning, volume 2. MIT press Cambridge, 1998.
  69. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12:1057–1063, 1999.
  70. Gerald Tesauro. Temporal difference learning and td-gammon. Communications of the ACM, 38(3):58–68, 1995.
  71. Analysis of temporal-diffference learning with function approximation. In Advances in neural information processing systems, pages 1075–1081, 1997.
  72. Reducing seek overhead with application-directed prefetching. In USENIX Annual Technical Conference, 2009.
  73. When caches aren’t enough: Data prefetching techniques. Computer, 30(7):23–30, 1997.
  74. A compiler-assisted data prefetch controller. In Proceedings 1999 IEEE International Conference on Computer Design: VLSI in Computers and Processors (Cat. No. 99CB37040), pages 372–377. IEEE, 1999.
  75. The potential costs and benefits of long-term prefetching for content distribution. Computer Communications, 25(4):367–375, 2002.
  76. Implementing cooperative prefetching and caching in a globally-managed memory system. In Proceedings of the 1998 ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems, pages 33–43, 1998.
  77. Efficient {{\{{MRC}}\}} construction with {{\{{SHARDS}}\}}. In 13th USENIX Conference on File and Storage Technologies (FAST 15), pages 95–110, 2015.
  78. Q-learning. Machine learning, 8(3-4):279–292, 1992.
  79. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992.
  80. Optimization of heat-based cache replacement in edge computing system. The Journal of Supercomputing, 77:2268–2301, 2021.
  81. Mithril: mining sporadic associations for cache prefetching. In Proceedings of the 2017 Symposium on Cloud Computing, pages 66–79, 2017.
  82. Tombolo: Performance enhancements for cloud storage gateways. In 2016 32nd Symposium on Mass Storage Systems and Technologies (MSST), pages 1–14. IEEE, 2016.
  83. Ctdgm: A data grouping model based on cache transaction for unstructured data storage systems. arXiv preprint arXiv:2009.14414, 2020.
  84. Characteristics of youtube network traffic at a campus network–measurements, models, and implications. Computer networks, 53(4):501–514, 2009.
  85. Hardware and software cache prefetching techniques for mpeg benchmarks. IEEE Transactions on Circuits and Systems for Video Technology, 10(5):782–796, 2000.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now