Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning (1805.10000v1)

Abstract: Applying reinforcement learning in physical-world tasks is extremely challenging. It is commonly infeasible to sample a large number of trials, as required by current reinforcement learning methods, in a physical environment. This paper reports our project on using reinforcement learning for better commodity search in Taobao, one of the largest online retail platforms and meanwhile a physical environment with a high sampling cost. Instead of training reinforcement learning in Taobao directly, we present our approach: first we build Virtual Taobao, a simulator learned from historical customer behavior data through the proposed GAN-SD (GAN for Simulating Distributions) and MAIL (multi-agent adversarial imitation learning), and then we train policies in Virtual Taobao with no physical costs in which ANC (Action Norm Constraint) strategy is proposed to reduce over-fitting. In experiments, Virtual Taobao is trained from hundreds of millions of customers' records, and its properties are compared with the real environment. The results disclose that Virtual Taobao faithfully recovers important properties of the real environment. We also show that the policies trained in Virtual Taobao can have significantly superior online performance to the traditional supervised approaches. We hope our work could shed some light on reinforcement learning applications in complex physical environments.

• The paper introduces Virtual-Taobao, a simulator using historical data, GAN-SD, and MAIL to virtualize a large-scale e-commerce environment for reinforcement learning without high real-world sampling costs.
• Experiments show Virtual-Taobao accurately simulates real-world customer behavior and interaction dynamics, enabling RL policies that achieve over 2% revenue improvement.
• This virtualization framework enables practical reinforcement learning deployment in high-sampling-cost environments and has potential applications beyond online retail, such as finance or recommendations.

Virtualizing E-Commerce: Reinforcement Learning in Online Retail

The paper, "Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning," explores the implementation of reinforcement learning (RL) within the context of a large-scale, real-world online retail platform, specifically Taobao. It addresses the challenges associated with physically implementing RL in such environments, predominantly due to the high costs associated with vast sampling needs inherent to RL methods.

The authors propose a novel simulator, Virtual Taobao, to virtualize the e-commerce environment based on historical customer behavior data. This approach circumvents the physical constraints typically encountered in RL deployment in live environments. The Virtual Taobao simulator is built using two main innovations: GAN-SD (Generative Adversarial Networks for Simulating Distributions) and MAIL (Multi-agent Adversarial Imitation Learning).

Key Methodologies

1. GAN-SD for Customer Simulation:
   • GAN-SD helps generate representative customer distributions by introducing entropy and KL divergence constraints. This allows the model to simulate a broad spectrum of customer profiles observed on Taobao accurately.
   • This technique effectively models customer features and requests, maintaining fidelity to real-world data distributions.
2. MAIL for Interaction Generation:
   • MAIL extends the Generative Adversarial Imitation Learning (GAIL) framework to a multi-agent setting, simulating both customer and engine policies simultaneously.
   • This allows for realistic simulation of customer interactions with the platform, accounting for dynamic strategy changes and customer responses.
3. Action Norm Constraint (ANC) Strategy:
   • ANC addresses the problem of overfitting policies to the virtual environment by implementing action norm restrictions, which help in maintaining policy generalizability to the real-world environment.

Experimental Evaluation

The experimental results indicate that the Virtual Taobao environment can simulate customer distributions and interaction dynamics closely resembling those observed in the real Taobao platform. The comparison of customer- and time-based behavior metrics, such as Rate of Purchase Page (R2P), between the virtual and real environments demonstrates high fidelity.

Furthermore, reinforcement learning strategies developed using Virtual Taobao have shown substantial improvements over traditional supervised learning methods in enhancing platform revenue and performance, achieving over 2% revenue improvement.

Implications and Future Directions

The successful virtualization of the Taobao environment through Virtual Taobao offers significant implications for RL applications in complex, dynamic systems beyond online retail, such as automated financial trading or dynamic content recommendation systems. The proposed framework highlights potential pathways for deploying RL strategies in high-sampling-cost environments by using historical data to create faithful simulators.

Looking forward, the extension of such virtual environments could explore adaptive strategies that progressively align more closely with evolving real-world dynamics, further enhancing the transferability of RL policies. Additionally, integrating robust mechanisms to address evolving customer behavior and platform interactions could yield even more predictive virtual environments.

Overall, this research presents a significant contribution to the use of reinforcement learning in practical, large-scale systems, providing a framework for reducing deployment costs and enhancing efficacy.