Reinforcement Learning from Statistical Feedback: the Journey from AB Testing to ANT Testing
Abstract: Reinforcement Learning from Human Feedback (RLHF) has played a crucial role in the success of large models such as ChatGPT. RLHF is a reinforcement learning framework which combines human feedback to improve learning effectiveness and performance. However, obtaining preferences feedback manually is quite expensive in commercial applications. Some statistical commercial indicators are usually more valuable and always ignored in RLHF. There exists a gap between commercial target and model training. In our research, we will attempt to fill this gap with statistical business feedback instead of human feedback, using AB testing which is a well-established statistical method. Reinforcement Learning from Statistical Feedback (RLSF) based on AB testing is proposed. Statistical inference methods are used to obtain preferences for training the reward network, which fine-tunes the pre-trained model in reinforcement learning framework, achieving greater business value. Furthermore, we extend AB testing with double selections at a single time-point to ANT testing with multiple selections at different feedback time points. Moreover, we design numerical experiences to validate the effectiveness of our algorithm framework.
- Deep learning: a statistical viewpoint. Acta numerica, 30: 87–201.
- Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1): 45–67.
- Better rewards yield better summaries: Learning to summarise without references. arXiv preprint arXiv:1909.01214.
- A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt. arXiv preprint arXiv:2303.04226.
- Customer lifetime value in video games using deep learning and parametric models. In 2018 IEEE international conference on big data (big data), 2134–2140. IEEE.
- Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30.
- A hitchhiker’s guide to statistical comparisons of reinforcement learning algorithms. arXiv preprint arXiv:1904.06979.
- Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729.
- APRIL: Interactively learning to summarise by combining active preference learning and reinforcement learning. arXiv preprint arXiv:1808.09658.
- Offline a/b testing for recommender systems. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, 198–206.
- DeepFM: a factorization-machine based neural network for CTR prediction. arXiv preprint arXiv:1703.04247.
- Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31.
- A survey of community detection approaches: From statistical modeling to deep learning. IEEE Transactions on Knowledge and Data Engineering, 35(2): 1149–1170.
- Designing with data: Improving the user experience with A/B testing. ” O’Reilly Media, Inc.”.
- Reward uncertainty for exploration in preference-based reinforcement learning. arXiv preprint arXiv:2205.12401.
- Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint arXiv:2201.06009.
- Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744.
- Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
- Feature engineering in big data analytics for IoT-enabled smart manufacturing–comparison between deep learning and statistical learning. Computers & Chemical Engineering, 141: 106970.
- Literature survey of statistical, deep and reinforcement learning in natural language processing. In 2017 International conference on computing, communication and automation (ICCCA), 350–354. IEEE.
- A/B testing: The most powerful way to turn clicks into customers. John Wiley & Sons.
- Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 3008–3021.
- Learning rewards from linguistic feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 6002–6010.
- Reinforcement learning for statistical process control in manufacturing. Measurement, 182: 109616.
- DeepReI: Deep learning-based gas chromatographic retention index predictor. Analytica Chimica Acta, 1147: 64–71.
- CL4CTR: A Contrastive Learning Framework for CTR Prediction. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, 805–813.
- Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862.
- A comprehensive analysis of the Elo rating algorithm: Stochastic model, convergence characteristics, design guidelines, and experimental results. arXiv preprint arXiv:2212.12015.
- Deep learning for click-through rate estimation. arXiv preprint arXiv:2104.10584.
- Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, 1059–1068.
- Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
