Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
157 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Exploring validation metrics for offline model-based optimisation with diffusion models (2211.10747v3)

Published 19 Nov 2022 in stat.ML and cs.LG

Abstract: In model-based optimisation (MBO) we are interested in using machine learning to design candidates that maximise some measure of reward with respect to a black box function called the (ground truth) oracle, which is expensive to compute since it involves executing a real world process. In offline MBO we wish to do so without assuming access to such an oracle during training or validation, with makes evaluation non-straightforward. While an approximation to the ground oracle can be trained and used in place of it during model validation to measure the mean reward over generated candidates, the evaluation is approximate and vulnerable to adversarial examples. Measuring the mean reward of generated candidates over this approximation is one such `validation metric', whereas we are interested in a more fundamental question which is finding which validation metrics correlate the most with the ground truth. This involves proposing validation metrics and quantifying them over many datasets for which the ground truth is known, for instance simulated environments. This is encapsulated under our proposed evaluation framework which is also designed to measure extrapolation, which is the ultimate goal behind leveraging generative models for MBO. While our evaluation framework is model agnostic we specifically evaluate denoising diffusion models due to their state-of-the-art performance, as well as derive interesting insights such as ranking the most effective validation metrics as well as discussing important hyperparameters.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. Stochastic simulation: algorithms and analysis, volume 57. Springer, 2007.
  2. Borji, Ali. Pros and cons of GAN evaluation measures: New developments. Computer Vision and Image Understanding, 215:103329, 2022.
  3. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018.
  4. Conditioning by adaptive sampling for robust design. In International conference on machine learning, pp. 773–782. PMLR, 2019.
  5. Understanding disentangling in beta-vae. arXiv preprint arXiv:1804.03599, 2018.
  6. Bidirectional learning for offline infinite-width model-based optimization. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.  29454–29467. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/bd391cf5bdc4b63674d6da3edc1bde0d-Paper-Conference.pdf.
  7. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  8. Generating images with perceptual similarity metrics based on deep networks. Advances in neural information processing systems, 29, 2016.
  9. The Fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 12(3):450–455, 1982.
  10. Autofocused oracles for model-based design. Advances in Neural Information Processing Systems, 33:12945–12956, 2020.
  11. Offline model-based optimization via normalized maximum likelihood estimation. arXiv preprint arXiv:2102.07970, 2021.
  12. Jumbo: Scalable multi-task bayesian optimization using offline data. arXiv preprint arXiv:2106.00942, 2021.
  13. Hamidieh, Kam. A data-driven statistical model for predicting the critical temperature of a superconductor. Computational Materials Science, 154:346–354, 2018.
  14. GANs trained by a two time-scale update rule converge to a nash equilibrium. CoRR, abs/1706.08500, 2017. URL http://arxiv.org/abs/1706.08500.
  15. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
  16. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  17. Huszár, Ferenc. How (not) to train your generative model: Scheduled sampling, likelihood, adversary? arXiv preprint arXiv:1511.05101, 2015.
  18. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  19. Model inversion networks for model-based optimization. Advances in Neural Information Processing Systems, 33:5126–5137, 2020.
  20. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
  21. Local latent space bayesian optimization over structured inputs. Advances in Neural Information Processing Systems, 35:34505–34518, 2022.
  22. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pp. 7176–7185. PMLR, 2020.
  23. Making sense of reinforcement learning and probabilistic inference. arXiv preprint arXiv:2001.00805, 2020.
  24. Implicit offline reinforcement learning via supervised learning. arXiv preprint arXiv:2210.12272, 2022.
  25. Data-driven offline decision-making via invariant representation learning. arXiv preprint arXiv:2211.11349, 2022.
  26. Assessing generative models via precision and recall. Advances in neural information processing systems, 31, 2018.
  27. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256–2265. PMLR, 2015.
  28. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
  29. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438–12448, 2020.
  30. A note on the evaluation of generative models. arXiv preprint arXiv:1511.01844, 2015.
  31. Conservative objective models for effective offline model-based optimization. In International Conference on Machine Learning, pp. 10358–10368. PMLR, 2021.
  32. Design-bench: Benchmarks for data-driven offline model-based optimization. In International Conference on Machine Learning, pp. 21658–21676. PMLR, 2022.
  33. Vapnik, Vladimir. Principles of risk minimization for learning theory. Advances in neural information processing systems, 4, 1991.
  34. Pre-trained gaussian processes for bayesian optimization. arXiv preprint arXiv:2109.08215, 2021.
  35. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp.  681–688, 2011.
  36. Weng, Lilian. What are diffusion models? lilianweng.github.io, Jul 2021. URL https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.
  37. Few-shot bayesian optimization with deep kernel surrogates. In International Conference on Learning Representations, 2021.
  38. Roma: Robust model adaptation for offline model-based optimization. Advances in Neural Information Processing Systems, 34:4619–4631, 2021.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now