Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
Gemini 2.5 Pro
GPT-5
GPT-4o
DeepSeek R1 via Azure
2000 character limit reached

Conditions on Preference Relations that Guarantee the Existence of Optimal Policies (2311.01990v2)

Published 3 Nov 2023 in cs.LG

Abstract: Learning from Preferential Feedback (LfPF) plays an essential role in training LLMs, as well as certain types of interactive learning agents. However, a substantial gap exists between the theory and application of LfPF algorithms. Current results guaranteeing the existence of optimal policies in LfPF problems assume that both the preferences and transition dynamics are determined by a Markov Decision Process. We introduce the Direct Preference Process, a new framework for analyzing LfPF problems in partially-observable, non-Markovian environments. Within this framework, we establish conditions that guarantee the existence of optimal policies by considering the ordinal structure of the preferences. We show that a decision-making problem can have optimal policies -- that are characterized by recursive optimality equations -- even when no reward function can express the learning goal. These findings underline the need to explore preference-based learning strategies which do not assume that preferences are generated by reward.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. Advances in preference-based reinforcement learning: A review. In 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 2527–2532.
  2. A definition of continual reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems.
  3. On the expressivity of markov reward. Advances in Neural Information Processing Systems, 34:7799–7812.
  4. Direct preference-based policy optimization without reward modeling. In Thirty-seventh Conference on Neural Information Processing Systems.
  5. Constitutional ai: Harmlessness from ai feedback.
  6. Bellman, R. (1957). Dynamic Programming. Princeton University Press, Princeton, NJ, USA, 1 edition.
  7. Quantifying hypothesis space misspecification in learning from human–robot demonstrations and physical corrections. IEEE Transactions on Robotics, 36(3):835–854.
  8. Settling the reward hypothesis. In International Conference on Machine Learning, pages 3003–3020. PMLR.
  9. Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research. Survey Certification.
  10. On the theory of reinforcement learning with once-per-episode feedback. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 3401–3412. Curran Associates, Inc.
  11. Deep reinforcement learning from human preferences. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  12. Simple agent, complex environment: Efficient reinforcement learning with agent states. J. Mach. Learn. Res., 23:255:1–255:54.
  13. Hutter, M. (2016). Extreme state aggregation beyond markov decision processes. Theoretical Computer Science, 650:73–91. Algorithmic Learning Theory.
  14. Jensen, K. K. (2012). Unacceptable risks and the continuity axiom. Economics & Philosophy, 28(1):31–42.
  15. Beyond reward: Offline preference-guided policy optimization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 15753–15768. PMLR.
  16. Preference transformer: Modeling human preferences using transformers for rl. In The Eleventh International Conference on Learning Representations.
  17. Provably feedback-efficient reinforcement learning via active reward learning. Advances in Neural Information Processing Systems, 35:11063–11078.
  18. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning.
  19. Reinforcement learning, bit by bit. Foundations and Trends® in Machine Learning, 16(6):733–865.
  20. Mitten, L. G. (1974). Preference order dynamic programming. Management Science, 21(1):43–46.
  21. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533.
  22. OpenAI (2023). Gpt-4 technical report.
  23. Deep exploration via bootstrapped dqn. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
  24. Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learning. International Journal of Human-Computer Studies, 160:102772.
  25. Rational multi-objective agents must admit non-markov reward representations. In NeurIPS ML Safety Workshop.
  26. Puterman, M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition.
  27. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems.
  28. Dueling rl: Reinforcement learning with trajectory preferences. In International Conference on Artificial Intelligence and Statistics, pages 6263–6289. PMLR.
  29. Utility theory for sequential decision making. In International Conference on Machine Learning, pages 19616–19625. PMLR.
  30. On the limitations of Markovian rewards to express multi-objective, risk-sensitive, and modal tasks. In Evans, R. J. and Shpitser, I., editors, Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pages 1974–1984. PMLR.
  31. Sobel, M. J. (1975). Ordinal dynamic programming. Management Science, 21(9):967–975.
  32. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021.
  33. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA.
  34. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124–1131.
  35. Theory of games and economic behavior. Princeton University Press.
  36. Weng, P. (2011). Markov decision processes with ordinal rewards: Reference point-based preferences. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 21, pages 282–289.
  37. A survey of preference-based reinforcement learning methods. J. Mach. Learn. Res., 18(1):4945–4990.
  38. Preference-based reinforcement learning with finite-time guarantees. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA. Curran Associates Inc.
  39. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 43037–43067. PMLR.
  40. Fine-tuning language models from human preferences.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now