Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Batched Stochastic Bandit for Nondegenerate Functions (2405.05733v2)

Published 9 May 2024 in stat.ML and cs.LG

Abstract: This paper studies batched bandit learning problems for nondegenerate functions. We introduce an algorithm that solves the batched bandit problem for nondegenerate functions near-optimally. More specifically, we introduce an algorithm, called Geometric Narrowing (GN), whose regret bound is of order $\widetilde{{\mathcal{O}}} ( A_{+}d \sqrt{T} )$. In addition, GN only needs $\mathcal{O} (\log \log T)$ batches to achieve this regret. We also provide lower bound analysis for this problem. More specifically, we prove that over some (compact) doubling metric space of doubling dimension $d$: 1. For any policy $\pi$, there exists a problem instance on which $\pi$ admits a regret of order ${\Omega} ( A_-d \sqrt{T})$; 2. No policy can achieve a regret of order $ A_-d \sqrt{T} $ over all problem instances, using less than $ \Omega ( \log \log T ) $ rounds of communications. Our lower bound analysis shows that the GN algorithm achieves near optimal regret with minimal number of batches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24.
  2. Optimization algorithms on matrix manifolds. Princeton University Press.
  3. Learning with limited rounds of adaptivity: coin tossing, multi-armed bandits, and ranking from pairwise comparisons. In Conference on Learning Theory, pages 39–75. PMLR.
  4. Batched dueling bandits. arXiv preprint arXiv:2202.10660.
  5. Agrawal, R. (1995). The continuum-armed bandit problem. SIAM Journal on Control and Optimization, 33(6):1926–1951.
  6. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory, pages 39–1. JMLR Workshop and Conference Proceedings.
  7. The multiplicative weights update method: a meta-algorithm and applications. Theory of computing, 8(1):121–164.
  8. Auer, P. (2002). Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397–422.
  9. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235–256.
  10. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1):48–77.
  11. Improved rates for the stochastic continuum-armed bandit problem. In Conference on Computational Learning Theory, pages 454–468. Springer.
  12. Bandit problems: sequential allocation of experiments (monographs on statistics and applied probability). London: Chapman and Hall, 5(71-87):7–7.
  13. Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press.
  14. Boumal, N. (2023). An introduction to optimization on smooth manifolds. Cambridge University Press.
  15. Estimation des densités : Risque minimax. In Dellacherie, C., Meyer, P. A., and Weil, M., editors, Séminaire de Probabilités XII, pages 342–363, Berlin, Heidelberg. Springer Berlin Heidelberg.
  16. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning, 5(1):1–122.
  17. Online optimization in 𝒳𝒳\mathcal{X}caligraphic_X-armed bandits. Advances in Neural Information Processing Systems, 22:201–208.
  18. 𝒳𝒳\mathcal{X}caligraphic_X-armed bandits. Journal of Machine Learning Research, 12(5):1655–1695.
  19. Lipschitz bandits without the Lipschitz constant. In International Conference on Algorithmic Learning Theory, pages 144–158. Springer.
  20. Online learning with switching costs and other adaptive adversaries. Advances in Neural Information Processing Systems, 26:1160–1168.
  21. Proximal gradient method for nonsmooth optimization over the stiefel manifold. SIAM Journal on Optimization, 30(1):210–239.
  22. Robust dynamic pricing with demand learning in the presence of outlier customers. Operations Research, 71(4):1362–1386.
  23. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 208–214. JMLR Workshop and Conference Proceedings.
  24. Gaussian process optimization with mutual information. In International Conference on Machine Learning, pages 253–261. PMLR.
  25. Cope, E. W. (2009). Regret and convergence bounds for a class of continuum-armed bandit problems. IEEE Transactions on Automatic Control, 54(6):1243–1253.
  26. The price of bandit information for online optimization. Advances in Neural Information Processing Systems, 20.
  27. Regret bounds for batched bandits. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7340–7348.
  28. Lipschitz bandits with batched feedback. IEEE Transactions on Information Theory.
  29. A new first-order algorithmic framework for optimization problems with orthogonality constraints. SIAM Journal on Optimization, 28(1):302–332.
  30. Riemannian optimization on the symplectic stiefel manifold. SIAM Journal on Optimization, 31(2):1546–1575.
  31. Batched multi-armed bandits problem. Advances in Neural Information Processing Systems, 32:503–513.
  32. Approximating nash equilibria in normal-form games via stochastic optimization. In The Twelfth International Conference on Learning Representations.
  33. Gittins, J. C. (1979). Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society Series B: Statistical Methodology, 41(2):148–164.
  34. Sequential batch learning in finite-action linear contextual bandits. arXiv preprint arXiv:2004.06321.
  35. A broyden class of quasi-newton methods for riemannian optimization. SIAM Journal on Optimization, 25(3):1660–1685.
  36. Top arm identification in multi-armed bandits with batch arm pulls. In Artificial Intelligence and Statistics, pages 139–148. PMLR.
  37. Collaborative top distribution identifications with limited interaction. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS), pages 160–171. IEEE.
  38. Kleinberg, R. (2005). Nearly tight bounds for the continuum-armed bandit problem. Advances in Neural Information Processing Systems, 18:697–704.
  39. Multi-armed bandits in metric spaces. In Proceedings of the fortieth annual ACM symposium on Theory of computing, pages 681–690.
  40. Contextual bandits with continuous actions: Smoothing, zooming, and adapting. The Journal of Machine Learning Research, 21(1):5402–5446.
  41. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6(1):4–22.
  42. Bandit algorithms. Cambridge University Press.
  43. Stochastic zeroth-order riemannian derivative estimation and optimization. Mathematics of Operations Research, 48(2):1183–1211.
  44. Zeroth-order riemannian averaging stochastic approximation algorithms. arXiv preprint arXiv:2309.14506.
  45. Gaussian process bandit optimization with few batches. In International Conference on Artificial Intelligence and Statistics, pages 92–107. PMLR.
  46. Optimal algorithms for Lipschitz bandits with heavy-tailed rewards. In International Conference on Machine Learning, pages 4154–4163.
  47. Lipschitz bandits: Regret lower bound and optimal algorithms. In Conference on Learning Theory, pages 975–999. PMLR.
  48. Efficient contextual bandits with continuous actions. Advances in Neural Information Processing Systems, 33:349–360.
  49. Dynamic pricing with unknown nonparametric demand and limited price changes. Operations Research.
  50. Batched bandit problems. The Annals of Statistics, 44(2):660–681.
  51. Petersen, P. (2006). Riemannian geometry, volume 171. Springer.
  52. Adaptive discretization for adversarial lipschitz bandits. In Belkin, M. and Kpotufe, S., editors, Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 of Proceedings of Machine Learning Research, pages 3788–3805. PMLR.
  53. Robbins, H. (1952). Some aspects of the sequential design of experiments. Bull. Amer. Math. Soc., 58:527–535.
  54. Linear bandits with limited adaptivity and learning distributional optimal design. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 74–87.
  55. Ruszczyński, A. (2021). A stochastic subgradient method for nonsmooth nonconvex multilevel composition optimization. SIAM Journal on Control and Optimization, 59(3):2301–2320.
  56. Riemannian stochastic variance reduced gradient algorithm with retraction and vector transport. SIAM Journal on Optimization, 29(2):1444–1472.
  57. Shamir, O. (2013). On the complexity of bandit and derivative-free stochastic convex optimization. In Conference on Learning Theory, pages 3–24. PMLR.
  58. Slivkins, A. (2014). Contextual bandits with similarity information. Journal of Machine Learning Research, 15(1):2533–2568.
  59. Slivkins, A. (2019). Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning, 12(1-2):1–286.
  60. Information-theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE transactions on information theory, 58(5):3250–3265.
  61. Collaborative learning with limited interaction: tight bounds for distributed exploration in multi-armed bandits. In 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS), pages 126–146. IEEE.
  62. Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285–294.
  63. Stochastic simultaneous optimistic optimization. In International Conference on Machine Learning, pages 19–27. PMLR.
  64. Yau, S.-T. (1974). Non-existence of continuous convex functions on certain riemannian manifolds. Mathematische Annalen, 207(4):269–270.
  65. Improved dynamic regret for non-degenerate functions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 732–741, Red Hook, NY, USA. Curran Associates Inc.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now