Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning an Inventory Control Policy with General Inventory Arrival Dynamics (2310.17168v2)

Published 26 Oct 2023 in cs.LG and stat.ML

Abstract: In this paper we address the problem of learning and backtesting inventory control policies in the presence of general arrival dynamics -- which we term as a quantity-over-time arrivals model (QOT). We also allow for order quantities to be modified as a post-processing step to meet vendor constraints such as order minimum and batch size constraints -- a common practice in real supply chains. To the best of our knowledge this is the first work to handle either arbitrary arrival dynamics or an arbitrary downstream post-processing of order quantities. Building upon recent work (Madeka et al., 2022) we similarly formulate the periodic review inventory control problem as an exogenous decision process, where most of the state is outside the control of the agent. Madeka et al., 2022 show how to construct a simulator that replays historic data to solve this class of problem. In our case, we incorporate a deep generative model for the arrivals process as part of the history replay. By formulating the problem as an exogenous decision process, we can apply results from Madeka et al., 2022 to obtain a reduction to supervised learning. Via simulation studies we show that this approach yields statistically significant improvements in profitability over production baselines. Using data from a real-world A/B test, we show that Gen-QOT generalizes well to off-policy data and that the resulting buying policy outperforms traditional inventory management systems in real world settings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (78)
  1. Loss of plasticity in continual deep reinforcement learning. arXiv:2303.07507.
  2. Neural inventory control in networks via hindsight differentiable policy optimization. arXiv:2306.11246.
  3. Studies in the mathematical theory of inventory and production. Stanford University Press.
  4. Belief movement, uncertainty reduction, and rational updating. Tech. rep., Haas School of Business, University of California, Berkeley.
  5. Orl: Reinforcement learning benchmarks for online stochastic optimization problems. arXiv:1911.10641.
  6. Bao, Y. (2006). Supply chain competition. Tech. rep., UNSW Sydney. [PDF]
  7. Myopic heuristics for the random yield problem. Operations Research 47 713–722.
  8. Language models are few-shot learners. In Advances in Neural Information Processing Systems (H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan and H. Lin, eds.), vol. 33. Curran Associates, Inc.
  9. The effects of load smoothing on inventory levels in a capacitated production and inventory system. Tech. rep., Cornell University Operations Research and Industrial Engineering. [PDF]
  10. Model-augmented actor-critic: Backpropagating through paths. In ICLR.
  11. A newsvendor’s procurement problem when suppliers are unreliable. Manufacturing & Service Operations Management 9 9–32.
  12. Solving semi-markov decision problems using average reward reinforcement learning. Management Science 45 560–574.
  13. Dawid, A. (1982). The well calibrated bayesian. Journal of the American Statistical Association 77 605–613.
  14. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT.
  15. Sample-efficient reinforcement learning in the presence of exogenous information. arXiv:2206.04282.
  16. Sparsity in partially controllable linear systems. In International Conference on Machine Learning. PMLR.
  17. MQTransformer: Multi-Horizon Forecasts with Context Dependent and Feedback-Aware Attention. arXiv:2009.14799.
  18. An inventory model with limited production capacity and uncertain demands i. the average-cost criterion. Mathematics of Operations Research 11 193–207.
  19. Reducing the cost of demand uncertainty through accurate response to early sales. Operations research 44 87–99.
  20. Threshold Martingales and the Evolution of Forecasts. arXiv:2105.06834.
  21. Calibrated learning and correlated equilibrium. Games and Economic Behavior 21 40–55.
  22. Deep Learning for Time Series Forecasting: The Electric Load Case. arXiv:1907.09207.
  23. Periodic review production models with variable yield and uncertain demand. Iie Transactions 20 144–150.
  24. Inventory management in supply chains: a reinforcement learning approach. International Journal of Production Economics 78 153–161.
  25. Can deep reinforcement learning improve inventory management? performance on lost sales, dual-sourcing, and multi-echelon problems. Manufacturing & Service Operations Management 24 1349–1368.
  26. Generative adversarial nets. In Advances in Neural Information Processing Systems (Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence and K. Weinberger, eds.), vol. 27. Curran Associates, Inc.
  27. Graves, A. (2012). Sequence transduction with recurrent neural networks. arXiv:1211.3711.
  28. Graves, A. (2013). Generating sequences with recurrent neural networks. arXiv:1308.0850.
  29. The structure of periodic review policies in the presence of random yield. Operations Research 38 634–643.
  30. Chainqueen: A real-time differentiable physical simulator for soft robotics. In 2019 International conference on robotics and automation (ICRA). IEEE.
  31. Learning protein structure with a differentiable simulator. In International Conference on Learning Representations.
  32. Forecasting with trees. International Journal of Forecasting 38 1473–1481. Special Issue: M5 competition.
  33. Kaplan, R. S. (1970). A dynamic inventory model with stochastic lead times. Management Science 16 491–507.
  34. Single item inventory control under periodic review and a minimum order quantity. International Journal of Production Economics 133 280–285.
  35. Auto-encoding variational bayes. arXiv:1312.6114.
  36. An introduction to variational autoencoders. arXiv:1906.02691.
  37. A periodic-review inventory system with supply interruptions. Probability in the Engineering and Informational Sciences 18 33–53.
  38. Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting. arXiv:1912.09363.
  39. Economic order quantity for items with imperfect quality: Revisited. International Journal of Production Economics 112 808–815. Special Section on RFID: Technology, Applications, and Impact on Business Operations.
  40. Sample path generation for probabilistic demand forecasting. In KDD 2018 Workshop on Mining and Learning from Time Series.
  41. Deep inventory management. arXiv:2210.03137.
  42. Multi-echelon inventory management for a non-stationary capacitated distribution network. Tech. rep., SSRN. [PDF]
  43. Asynchronous methods for deep reinforcement learning. arXiv:1602.01783.
  44. Playing atari with deep reinforcement learning. arXiv:1312.5602.
  45. An analysis of multi-agent reinforcement learning for decentralized inventory control systems. arXiv:2307.11432.
  46. Sequence to sequence deep learning models for solar irradiation forecasting. In IEEE Milan PowerTech.
  47. Nahmias, S. (1979). Simple approximations for a variety of dynamic leadtime lost-sales inventory models. Operations Research 27 904–924.
  48. STConvS2S: Spatiotemporal Convolutional Sequence to Sequence Network for weather forecasting. arXiv:1912.00134.
  49. Model-based reinforcement learning with scalable composite policy gradient estimators. In ICML.
  50. Porteus, E. L. (2002). Foundations of stochastic inventory theory. Stanford University Press.
  51. A practical end-to-end inventory management model with deep learning. Management Science 69 759–773.
  52. Inventory management with periodic ordering and minimum order quantities. Journal of the Operational Research Society 49 1085–1094.
  53. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting 36 1181–1191.
  54. Proximal policy optimization algorithms. arXiv:1707.06347.
  55. A two-echelon inventory system with a minimum order quantity requirement. Sustainability 11.
  56. Mastering the game of go with deep neural networks and tree search. Nature 529 484–489.
  57. Hindsight learning for MDPs with exogenous inputs. In Proceedings of the 40th International Conference on Machine Learning, vol. 202 of Proceedings of Machine Learning Research. PMLR.
  58. Inventory control with information about supply conditions. Management Science 42 1409–1419.
  59. Do differentiable simulators give better policy gradients? In International Conference on Machine Learning. PMLR.
  60. LSTM Neural Networks for Language Modeling. In INTERSPEECH.
  61. Reinforcement Learning: An iIntroduction. MIT press.
  62. Szepesvári, C. (2010). Algorithms for Reinforcement Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, Morgan & Claypool Publishers.
  63. Taleb, N. N. (2018). Election predictions as martingales: an arbitrage approach. Quantitative Finance 18 1–5.
  64. All roads lead to quantitative finance. Quantitative Finance 19 1775–1776.
  65. THOMAS, J. D. (2023). Towards cooperative marl in industrial domains .
  66. Wavenet: A generative model for raw audio. arXiv:1609.03499.
  67. Pixel recurrent neural networks. In International conference on machine learning. PMLR.
  68. Veinott, A. F. (1965). The optimal inventory policy for batch ordering. Operations Research 13 424–432.
  69. Deep Generative Quantile-Copula Models for Probabilistic Forecasting. In ICML Time Series Workshop.
  70. A multi-horizon quantile recurrent forecaster. In NIPS Time Series Workshop.
  71. A learning algorithm for continually running fully recurrent neural networks. Neural Computation 1 270–280.
  72. Long-term Forecasting using Higher Order Tensor RNNs. arXiv:1711.00073.
  73. Policy optimization for continuous reinforcement learning. arXiv:2305.18901.
  74. On the structure of optimal ordering policies for stochastic inventory systems with minimum order quantity. Probability in the Engineering and Informational Sciences 20 257–270.
  75. Effective control policies for stochastic inventory systems with a minimum order quantity and linear costs. International Journal of Production Economics 106 523–531.
  76. Zhu, H. (2022). A simple heuristic policy for stochastic inventory systems with both minimum and maximum order quantity requirements. Annals of Operations Research 309 347–363.
  77. Effective inventory control policies with a minimum order quantity and batch ordering. International Journal of Production Economics 168 21–30.
  78. Zipkin, P. (2008). Old and new methods for lost-sales inventory systems. Operations research 56 1256–1263.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now