Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Functional Graphical Models: Structure Enables Offline Data-Driven Optimization (2401.05442v3)

Published 8 Jan 2024 in cs.LG and cs.AI

Abstract: While machine learning models are typically trained to solve prediction problems, we might often want to use them for optimization problems. For example, given a dataset of proteins and their corresponding fluorescence levels, we might want to optimize for a new protein with the highest possible fluorescence. This kind of data-driven optimization (DDO) presents a range of challenges beyond those in standard prediction problems, since we need models that successfully predict the performance of new designs that are better than the best designs seen in the training set. It is not clear theoretically when existing approaches can even perform better than the naive approach that simply selects the best design in the dataset. In this paper, we study how structure can enable sample-efficient data-driven optimization. To formalize the notion of structure, we introduce functional graphical models (FGMs) and show theoretically how they can provide for principled data-driven optimization by decomposing the original high-dimensional optimization problem into smaller sub-problems. This allows us to derive much more practical regret bounds for DDO, and the result implies that DDO with FGMs can achieve nearly optimal designs in situations where naive approaches fail due to insufficient coverage of the offline data. We further present a data-driven optimization algorithm that inferes the FGM structure itself, either over the original input variables or a latent variable representation of the inputs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32, 2019.
  2. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599, 2010.
  3. Conditioning by adaptive sampling for robust design. In International conference on machine learning, pp. 773–782. PMLR, 2019.
  4. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning, pp. 1042–1051. PMLR, 2019.
  5. Markov fields on finite graphs and lattices. 1971.
  6. Diagnosing and enhancing vae models. arXiv preprint arXiv:1903.05789, 2019.
  7. Erdogdu, M. A. Newton-stein method: A second order method for glms via stein’s lemma. Advances in Neural Information Processing Systems, 28, 2015.
  8. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132–20145, 2021.
  9. Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 4(2):268–276, 2018.
  10. Generalized additive models: some applications. Journal of the American Statistical Association, 82(398):371–386, 1987.
  11. Decentralized high-dimensional bayesian optimization with factor graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  12. Geometric programming for aircraft design optimization. AIAA Journal, 52(11):2414–2426, 2014.
  13. Jordan, M. I. Learning in graphical models. 1999.
  14. High dimensional bayesian optimisation and bandits via additive models. In International conference on machine learning, pp. 295–304. PMLR, 2015.
  15. Confuciux: Autonomous hardware resource assignment for dnn accelerators using reinforcement learning. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp.  622–636. IEEE, 2020.
  16. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  17. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  18. Probabilistic graphical models: principles and techniques. MIT press, 2009.
  19. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021.
  20. Model inversion networks for model-based optimization. Advances in Neural Information Processing Systems, 33:5126–5137, 2020.
  21. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191, 2020.
  22. Data-driven offline optimization for architecting hardware accelerators. arXiv preprint arXiv:2110.11346, 2021.
  23. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
  24. Pearl, J. Causality. Cambridge university press, 2009.
  25. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
  26. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp.  745–750, 2007.
  27. Data-driven offline decision-making via invariant representation learning. arXiv preprint arXiv:2211.11349, 2022.
  28. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34:11702–11716, 2021.
  29. Variational inference with normalizing flows. In International conference on machine learning, pp. 1530–1538. PMLR, 2015.
  30. High-dimensional bayesian optimization via additive models with overlapping groups. In International conference on artificial intelligence and statistics, pp.  298–307. PMLR, 2018.
  31. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):148–175, 2015.
  32. Use of exchangeable pairs in the analysis of simulations. Lecture Notes-Monograph Series, pp.  1–26, 2004.
  33. Studeny, M. Probabilistic conditional independence structures. Springer Science & Business Media, 2006.
  34. Conservative objective models for effective offline model-based optimization. In International Conference on Machine Learning, pp. 10358–10368. PMLR, 2021.
  35. Design-bench: Benchmarks for data-driven offline model-based optimization. In International Conference on Machine Learning, pp. 21658–21676. PMLR, 2022.
  36. Wainright, M. High dimensional statistics. a non-asymptotic point of view, 2019.
  37. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.
  38. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683–6694, 2021.
  39. Are random decompositions all we need in high dimensional bayesian optimisation? arXiv preprint arXiv:2301.12844, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Jakub Grudzien Kuba (12 papers)
  2. Masatoshi Uehara (49 papers)
  3. Pieter Abbeel (372 papers)
  4. Sergey Levine (531 papers)
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets