Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Incentivized Learning in Principal-Agent Bandit Games (2403.03811v1)

Published 6 Mar 2024 in stat.ML, cs.GT, and cs.LG

Abstract: This work considers a repeated principal-agent bandit game, where the principal can only interact with her environment through the agent. The principal and the agent have misaligned objectives and the choice of action is only left to the agent. However, the principal can influence the agent's decisions by offering incentives which add up to his rewards. The principal aims to iteratively learn an incentive policy to maximize her own total utility. This framework extends usual bandit problems and is motivated by several practical applications, such as healthcare or ecological taxation, where traditionally used mechanism design theories often overlook the learning aspect of the problem. We present nearly optimal (with respect to a horizon $T$) learning algorithms for the principal's regret in both multi-armed and linear contextual settings. Finally, we support our theoretical guarantees through numerical experiments.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Associative reinforcement learning using linear probabilistic concepts. In ICML, pp.  3–11, 1999.
  2. Auer, P. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397–422, 2002.
  3. Bandit social learning: Exploration under myopic behavior. arXiv preprint arXiv:2302.07425, 2023.
  4. Principal-agent reward shaping in mdps. arXiv preprint arXiv:2401.00298, 2023.
  5. Solving convex programs by random walks. Journal of the ACM (JACM), 51(4):540–556, 2004.
  6. A survey on multi-player bandits. arXiv preprint arXiv:2211.16275, 2022.
  7. The best of both worlds: Stochastic and adversarial bandits. In Conference on Learning Theory, pp.  42–1. JMLR Workshop and Conference Proceedings, 2012.
  8. Optimal adaptive policies for sequential allocation problems. Advances in Applied Mathematics, 17(2):122–142, 1996.
  9. Doubly high-dimensional contextual bandits: An interpretable model for joint assortment-pricing. arXiv preprint arXiv:2309.08634, 2023.
  10. Online bayesian persuasion. Advances in Neural Information Processing Systems, 33:16188–16198, 2020.
  11. Learning approximately optimal contracts. In International Symposium on Algorithmic Game Theory, pp.  331–346. Springer, 2022.
  12. Feature-based dynamic pricing. Management Science, 66(11):4921–4943, 2020.
  13. Learning algorithms for online principal-agent problems (and selling goods online). In Proceedings of the 23rd international conference on Machine learning, pp.  209–216, 2006.
  14. Stochastic linear optimization under bandit feedback. 2008.
  15. Den Boer, A. V. Dynamic pricing and learning: historical origins, current research, and new directions. Surveys in operations research and management science, 20(1):1–18, 2015.
  16. Estimating and incentivizing imperfect-knowledge agents with hidden rewards. arXiv preprint arXiv:2308.06717, 2023a.
  17. Repeated principal-agent games with unobserved agent rewards and perfect-knowledge agents. arXiv preprint arXiv:2304.07407, 2023b.
  18. From data to decisions: helping crop producers build their actionable knowledge. Critical reviews in plant sciences, 36(2):71–88, 2017.
  19. Gittins, J. C. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society Series B: Statistical Methodology, 41(2):148–164, 1979.
  20. Incentive-aware contextual pricing with non-parametric market noise. In International Conference on Artificial Intelligence and Statistics, pp.  9331–9361. PMLR, 2023.
  21. Geometric algorithms and combinatorial optimization, volume 2. Springer Science & Business Media, 2012.
  22. Nearly optimal algorithms for linear contextual bandits with adversarial corruptions. Advances in Neural Information Processing Systems, 35:34614–34625, 2022.
  23. Incentivizing combinatorial bandit exploration. Advances in Neural Information Processing Systems, 35:37173–37183, 2022.
  24. Dynamic pricing in high-dimensions. The Journal of Machine Learning Research, 20(1):315–363, 2019.
  25. Bayesian persuasion. American Economic Review, 101(6):2590–2615, 2011.
  26. Corruption-tolerant bandit learning. Machine Learning, 108(4):687–715, 2019.
  27. The theory of incentives: the principal-agent model. In The Theory of Incentives. Princeton University Press, 2009.
  28. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6(1):4–22, 1985.
  29. Bandit Algorithms. Cambridge University Press, 2020.
  30. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on the World Wide Web, pp.  661–670, 2010.
  31. Multidimensional binary search for contextual decision-making. Operations Research, 66(5):1346–1361, 2018.
  32. Stochastic bandits robust to adversarial corruptions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pp.  114–122, 2018.
  33. Bayesian incentive-compatible bandit exploration. Operations Research, 68(4):1132–1161, 2020.
  34. Contextual pricing for lipschitz buyers. Advances in Neural Information Processing Systems, 31, 2018.
  35. Myerson, R. B. Mechanism design. Springer, 1989.
  36. Rademacher, L. A. Approximating the centroid is hard. In Proceedings of the Twenty-Third Annual Symposium on Computational Geometry, pp.  302–305, 2007.
  37. Linearly parameterized bandits. Mathematics of Operations Research, 35(2):395–411, 2010.
  38. The price of incentivizing exploration: A characterization via thompson sampling and sample complexity. In Proceedings of the 22nd ACM Conference on Economics and Computation, pp.  795–796, 2021.
  39. Exploration and incentives in reinforcement learning. Operations Research, 2023.
  40. Slivkins, A. et al. Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning, 12(1-2):1–286, 2019.
  41. How small is a unit ball? Mathematics Magazine, 62(2):101–107, 1989.
  42. Thompson, W. R. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285–294, 1933.
  43. Woodroofe, M. A one-armed bandit problem with a concomitant variable. Journal of the American Statistical Association, 74(368):799–806, 1979.
  44. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1–36, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Antoine Scheid (5 papers)
  2. Daniil Tiapkin (24 papers)
  3. Etienne Boursier (27 papers)
  4. Aymeric Capitaine (5 papers)
  5. El Mahdi El Mhamdi (12 papers)
  6. Eric Moulines (151 papers)
  7. Michael I. Jordan (438 papers)
  8. Alain Durmus (98 papers)
Citations (5)
View on Semantic Scholar