Data-driven inventory management for new products: An adjusted Dyna-$Q$ approach with transfer learning (2501.08109v4)

Abstract: In this paper, we propose a novel reinforcement learning algorithm for inventory management of newly launched products with no historical demand information. The algorithm follows the classic Dyna-$Q$ structure, balancing the model-free and model-based approaches, while accelerating the training process of Dyna-$Q$ and mitigating the model discrepancy generated by the model-based feedback. Based on the idea of transfer learning, warm-start information from the demand data of existing similar products can be incorporated into the algorithm to further stabilize the early-stage training and reduce the variance of the estimated optimal policy. Our approach is validated through a case study of bakery inventory management with real data. The adjusted Dyna-$Q$ shows up to a 23.7\% reduction in average daily cost compared with $Q$-learning, and up to a 77.5\% reduction in training time within the same horizon compared with classic Dyna-$Q$. By using transfer learning, it can be found that the adjusted Dyna-$Q$ has the lowest total cost, lowest variance in total cost, and relatively low shortage percentages among all the benchmarking algorithms under a 30-day testing.

• The paper introduces an adjusted Dyna-Q algorithm integrating model-based and model-free learning to optimize inventory for new products.
• The algorithm employs warm-start data and a search-then-convergence strategy to effectively tackle cold-start challenges while reducing training time.
• Empirical results demonstrate up to 23.7% cost reduction and 77.5% faster convergence compared to traditional Q-learning methods.

Data-Driven Inventory Management for New Products: A Reinforcement Learning Approach

The paper by Qu et al. presents a reinforcement learning (RL) algorithm tailored to the inventory management of new products, which traditionally lack historical demand data. Herein, the authors propose an enhanced version of the Dyna-$Q$ algorithm that integrates both model-based and model-free RL approaches to generate optimal inventory policies efficiently. The algorithm leverages warm-start information derived from similar existing products, addressing cold-start challenges in inventory management scenarios.

Algorithmic Innovations

The core contribution of the paper is the adjusted Dyna-$Q$ algorithm. This variant diverges from traditional RL methods by adopting a search-then-convergence (STC) strategy, allowing the algorithm to dynamically adjust the balance between exploration and exploitation as training progresses. This approach aims to mitigate model discrepancies that are more pronounced in early training stages and effectively accelerates the convergence when historical product data is sparse or unavailable.

1. Warm-start Information: The algorithm incorporates prior demand estimation from products with similar characteristics. A Bayesian Neural Network (BNN) predicts initial demand distributions, providing a foundation for simulating early-stage inventory policies.
2. Search-Then-Convergence Process: The exploration probability and planning steps dynamically decline through training, facilitating efficient learning. At early stages, the RL agent encourages broader exploration, tapering to focused exploitation as confidence in the model's estimates increases.
3. State Transition Modeling: A detailed strategy is devised for modeling the transition of perishable goods, respecting the constraints of given shelf lives, hence adapting to realistic inventory management scenarios.

Empirical Evaluation

Empirical validation is carried out using a case paper based on a bakery's inventory data. The new product, Boule 400g, lacked historical demand data, warranting a specialized approach to inventory management. The paper demonstrates the algorithm's capability to lower average daily costs by up to 23.7% compared to homogeneous $Q$-learning implementations. Moreover, it achieved up to a 77.5% reduction in training time over classic Dyna-$Q$, underscoring its computational efficiency.

In testing scenarios over a 30-day window, the algorithm exhibited improved demand matching, resulting in lower average costs and a reduction in the incidence of stock shortages. The inclusion of warm-start information led to a notable stabilization of policy outcomes, reflected in reduced variance in total costs.

Discussion of Implications and Future Directions

The results suggest significant benefits from integrating model-based predictions with real-world adjustments informed by model-free learning. This approach not only refines the resulting policy quality but also optimizes the computational overhead typically associated with large-scale inventory problems. The ability of the adjusted Dyna-$Q$ algorithm to adapt dynamically holds promise for various industries facing cold-start issues with newly launched products.

Prospective research could extend this framework to explore different market dynamics, such as varying demand elasticities or supplier lead times. Furthermore, there is potential for extending this approach within multi-echelon inventory systems, or integrating with supply chain management systems incorporating stochastic elements like supplier reliability or logistic constraints.

Overall, this paper delivers a substantive methodology for real-time inventory control, embedding reinforcement learning at its core, thereby representing a scalable and flexible solution for contemporary inventory management challenges.