Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Temporal-Aware Deep Reinforcement Learning for Energy Storage Bidding in Energy and Contingency Reserve Markets (2402.19110v1)

Published 29 Feb 2024 in eess.SY, cs.LG, cs.SY, and math.OC

Abstract: The battery energy storage system (BESS) has immense potential for enhancing grid reliability and security through its participation in the electricity market. BESS often seeks various revenue streams by taking part in multiple markets to unlock its full potential, but effective algorithms for joint-market participation under price uncertainties are insufficiently explored in the existing research. To bridge this gap, we develop a novel BESS joint bidding strategy that utilizes deep reinforcement learning (DRL) to bid in the spot and contingency frequency control ancillary services (FCAS) markets. Our approach leverages a transformer-based temporal feature extractor to effectively respond to price fluctuations in seven markets simultaneously and helps DRL learn the best BESS bidding strategy in joint-market participation. Additionally, unlike conventional "black-box" DRL model, our approach is more interpretable and provides valuable insights into the temporal bidding behavior of BESS in the dynamic electricity market. We validate our method using realistic market prices from the Australian National Electricity Market. The results show that our strategy outperforms benchmarks, including both optimization-based and other DRL-based strategies, by substantial margins. Our findings further suggest that effective temporal-aware bidding can significantly increase profits in the spot and contingency FCAS markets compared to individual market participation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. S. R. Sinsel, R. L. Riemke, and V. H. Hoffmann, “Challenges and solution technologies for the integration of variable renewable energy sources—a review,” Renewable Energy, vol. 145, pp. 2271–2285, 2020.
  2. K. Abdulla, J. de Hoog, V. Muenzel, F. Suits, K. Steer, A. Wirth, and S. Halgamuge, “Optimal operation of energy storage systems considering forecasts and battery degradation,” IEEE Transactions on Smart Grid, vol. 9, no. 3, pp. 2086–2096, 2018.
  3. D. Krishnamurthy, C. Uckun, Z. Zhou, P. R. Thimmapuram, and A. Botterud, “Energy storage arbitrage under day-ahead and real-time price uncertainty,” IEEE Transactions on Power Systems, vol. 33, no. 1, pp. 84–93, 2018.
  4. R. Weron, “Electricity price forecasting: A review of the state-of-the-art with a look into the future,” International Journal of Forecasting, vol. 30, no. 4, pp. 1030–1081, 2014.
  5. H. Wang and B. Zhang, “Energy storage arbitrage in real-time markets via reinforcement learning,” IEEE Power & Energy Society General Meeting (PESGM), 2018.
  6. V.-H. Bui, A. Hussain, and H.-M. Kim, “Double deep q𝑞qitalic_q -learning-based distributed operation of battery energy storage system considering uncertainties,” IEEE Transactions on Smart Grid, vol. 11, no. 1, pp. 457–469, 2020.
  7. J. Cao, D. Harrold, Z. Fan, T. Morstyn, D. Healey, and K. Li, “Deep reinforcement learning-based energy storage arbitrage with accurate lithium-ion battery degradation model,” IEEE Transactions on Smart Grid, vol. 11, no. 5, pp. 4513–4521, 2020.
  8. H. Xu, X. Li, X. Zhang, and J. Zhang, “Arbitrage of energy storage in electricity markets with deep reinforcement learning,” CoRR, vol. abs/1904.12232, 2019.
  9. X. Wei, Y. Xiang, J. Li, and X. Zhang, “Self-dispatch of wind-storage integrated system: A deep reinforcement learning approach,” IEEE Transactions on Sustainable Energy, vol. 13, no. 3, pp. 1861–1864, 2022.
  10. B. Huang and J. Wang, “Deep-reinforcement-learning-based capacity scheduling for pv-battery storage system,” IEEE Transactions on Smart Grid, vol. 12, no. 3, pp. 2272–2283, 2021.
  11. M. Anwar, C. Wang, F. de Nijs, and H. Wang, “Proximal policy optimization based reinforcement learning for joint bidding in energy and frequency regulation markets,” IEEE Power & Energy Society General Meeting (PESGM), 2022.
  12. K. Bradbury, L. Pratson, and D. Patiño-Echeverri, “Economic viability of energy storage systems based on price arbitrage potential in real-time u.s. electricity markets,” Applied Energy, vol. 114, pp. 512–519, 2014.
  13. D. McConnell, T. Forcey, and M. Sandiford, “Estimating the value of electricity storage in an energy-only wholesale market,” Applied Energy, vol. 159, pp. 422–432, 2015.
  14. D. Zafirakis, K. J. Chalvatzis, G. Baiocchi, and G. Daskalakis, “The value of arbitrage for energy storage: Evidence from european electricity markets,” Applied Energy, vol. 184, pp. 971–986, 2016.
  15. L. He and J. Zhang, “Energy trading in local electricity markets with behind-the-meter solar and energy storage,” IEEE Transactions on Energy Markets, Policy and Regulation, vol. 1, no. 2, pp. 107–117, 2023.
  16. W. Chen, J. Qiu, J. Zhao, Q. Chai, and Z. Y. Dong, “Bargaining game-based profit allocation of virtual power plant in frequency regulation market considering battery cycle life,” IEEE Transactions on Smart Grid, vol. 12, no. 4, pp. 2913–2928, 2021.
  17. J. Wang, D. K. Mishra, L. Li, and J. Zhang, “Demand side management and peer-to-peer energy trading for industrial users using two-level multi-agent reinforcement learning,” IEEE Transactions on Energy Markets, Policy and Regulation, vol. 1, no. 1, pp. 23–36, 2023.
  18. J. Jeong, S. W. Kim, and H. Kim, “Deep reinforcement learning based real-time renewable energy bidding with battery control,” IEEE Transactions on Energy Markets, Policy and Regulation, vol. 1, no. 2, pp. 85–96, 2023.
  19. P. Böhm, P. Pounds, and A. C. Chapman, “Feature extraction for effective and efficient deep reinforcement learning on real robotic platforms,” in 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 7126–7132.
  20. J. Liu, B.-Y. Tsai, and D.-S. Chen, “Deep reinforcement learning based controller with dynamic feature extraction for an industrial claus process,” Journal of the Taiwan Institute of Chemical Engineers, vol. 146, p. 104779, 2023.
  21. J. Li, C. Wang, Y. Zhang, and H. Wang, “Learn to bid: Deep reinforcement learning with transformer for energy storage bidding in energy and contingency reserve markets,” in NeurIPS 2022 Workshop on Tackling Climate Change with Machine Learning, 2022.
  22. T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in Proceedings of the 35th International Conference on Machine Learning, vol. 80, 10–15 Jul 2018, pp. 1861–1870.
  23. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30, 2017.
  24. M. Lin, Q. Chen, and S. Yan, “Network in network,” 2013. [Online]. Available: http://arxiv.org/abs/1312.4400
  25. O. Lockwood and M. Si, “A review of uncertainty for deep reinforcement learning,” Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 18, no. 1, pp. 155–162, Oct. 2022.
  26. M. D. Badoual and S. J. Moura, “A learning-based optimal market bidding strategy for price-maker energy storage,” in 2021 American Control Conference (ACC), 2021, pp. 526–532.
  27. S. Mitchell, M. OSullivan, and I. Dunning, “PuLP: a linear programming toolkit for python,” The University of Auckland, Auckland, New Zealand, vol. 65, 2011.
  28. P. Donti, B. Amos, and J. Z. Kolter, “Task-based end-to-end model learning in stochastic optimization,” in Advances in Neural Information Processing Systems, vol. 30, 2017.
  29. X. Chen, Y. Liu, and L. Wu, “Towards improving operation economics: A bilevel mip-based closed-loop predict-and-optimize framework for prescribing unit commitment,” ArXiv, vol. abs/2208.13065, 2022.
  30. A. Shrikumar, P. Greenside, and A. Kundaje, “Learning important features through propagating activation differences,” in Proceedings of the 34th International Conference on Machine Learning, vol. 70, 06–11 Aug 2017, pp. 3145–3153.
  31. M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” in Proceedings of the 34th International Conference on Machine Learning - Volume 70, ser. ICML’17, 2017, p. 3319–3328.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now