Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Adversarial Contextual Bandits Go Kernelized (2310.01609v1)

Published 2 Oct 2023 in stat.ML and cs.LG

Abstract: We study a generalization of the problem of online learning in adversarial linear contextual bandits by incorporating loss functions that belong to a reproducing kernel Hilbert space, which allows for a more flexible modeling of complex decision-making scenarios. We propose a computationally efficient algorithm that makes use of a new optimistically biased estimator for the loss functions and achieves near-optimal regret guarantees under a variety of eigenvalue decay assumptions made on the underlying kernel. Specifically, under the assumption of polynomial eigendecay with exponent $c>1$, the regret is $\widetilde{O}(KT{\frac{1}{2}(1+\frac{1}{c})})$, where $T$ denotes the number of rounds and $K$ the number of actions. Furthermore, when the eigendecay follows an exponential pattern, we achieve an even tighter regret bound of $\widetilde{O}(\sqrt{T})$. These rates match the lower bounds in all special cases where lower bounds are known at all, and match the best known upper bounds available for the more well-studied stochastic counterpart of our problem.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011.
  2. Corralling a band of bandit algorithms. In Conference on Learning Theory, pages 12–38, 2017.
  3. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1):48–77, 2002.
  4. Label optimal regret bounds for online local learning. In Proceedings of The 28th Conference on Learning Theory (COLT), pages 150–166, 2015.
  5. High-probability regret bounds for bandit online linear optimization. In Proceedings of the 21st Annual Conference on Learning Theory-COLT 2008, pages 335–342, 2008.
  6. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 19–26. JMLR Workshop and Conference Proceedings, 2011.
  7. Sparsity, variance and curvature in multi-armed bandits. In Algorithmic Learning Theory, pages 111–127, 2018.
  8. Gaussian process optimization with adaptive sketching: Scalable and no regret. In Conference on Learning Theory, pages 533–557, 2019.
  9. Near-linear time gaussian process optimization with adaptive batching and resparsification. In International Conference on Machine Learning, pages 1295–1305, 2020.
  10. Online learning with kernel losses. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 971–980. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/chatterji19a.html.
  11. On kernelized multi-armed bandits. In International Conference on Machine Learning, pages 844–853. PMLR, 2017.
  12. Paul Christiano. Provably manipulation-resistant reputation systems. In Proceedings of the 29th Annual Conference on Learning Theory (COLT), pages 670–697, 2016.
  13. Support Vector Machines. Springer New York, NY, 2008.
  14. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 208–214. JMLR Workshop and Conference Proceedings, 2011.
  15. Refined regret for adversarial mdps with linear function approximation. arXiv preprint arXiv:2301.12942, 2023.
  16. Information-theoretic metric learning. In Proceedings of the 24th international conference on Machine learning, pages 209–216. ACM, 2007.
  17. Learning in games: Robustness of fast convergence. Advances in Neural Information Processing Systems, 29, 2016.
  18. Adapting to misspecification in contextual bandits. Advances in Neural Information Processing Systems, 33:11478–11489, 2020.
  19. Online metric learning and fast similarity search. In Advances in neural information processing systems, pages 761–768, 2009.
  20. Implicit online learning. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 575–582, 2010.
  21. Nearly minimax-optimal regret for linearly parameterized bandits. In Conference on Learning Theory, pages 2173–2174. PMLR, 2019.
  22. Bypassing the simulator: Near-optimal adversarial linear contextual bandits. arXiv preprint arXiv:2309.00814, 2023.
  23. Efficient online portfolio with logarithmic regret. Advances in neural information processing systems, 31, 2018.
  24. J. Mercer. Functions of positive and negative type, and their connection with the theory of integral equations. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 209:415–446, 1909. ISSN 02643952. URL http://www.jstor.org/stable/91043.
  25. Efficient and robust algorithms for adversarial linear contextual bandits. In Conference on Learning Theory, pages 3049–3068. PMLR, 2020.
  26. Early stopping and non-parametric regression: an optimal data-dependent stopping rule. The Journal of Machine Learning Research, 15(1):335–366, 2014.
  27. Gaussian processes for machine learning. MIT press Cambridge, MA, 2006.
  28. Lower bounds on regret for noisy gaussian process bandit optimization. In Conference on Learning Theory, pages 1723–1742, 2017.
  29. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2002.
  30. Information consistency of nonparametric gaussian process methods. IEEE Transactions on Information Theory, 54(5):2376–2382, 2008. 10.1109/TIT.2007.915707.
  31. Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995, 2009.
  32. From ads to interventions: Contextual bandits in mobile health. In Mobile Health - Sensors, Analytic Methods, and Applications, 2017. URL https://api.semanticscholar.org/CorpusID:18778220.
  33. Finite-time analysis of kernelised contextual bandits. In Uncertainty in Artificial Intelligence, 2013.
  34. More adaptive algorithms for adversarial bandits. In Conference On Learning Theory, pages 1263–1291. PMLR, 2018.
  35. On function approximation in reinforcement learning: Optimism in the face of large state spaces. Advances in Neural Information Processing Systems, 2020, 2020.
  36. On early stopping in gradient descent learning. Constructive Approximation, 26:289–315, 2007.
  37. Efficient kernelized ucb for contextual bandits. In International Conference on Artificial Intelligence and Statistics, pages 5689–5720, 2022.
  38. Return of the bias: Almost minimax optimal high probability bounds for adversarial linear bandits. In Proceedings of Thirty Fifth Conference on Learning Theory, pages 3285–3312, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Gergely Neu (52 papers)
  2. Julia Olkhovskaya (11 papers)
  3. Sattar Vakili (37 papers)