Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators (2405.17708v2)

Published 27 May 2024 in cs.LG, cs.AI, and stat.ML

Abstract: Offline policy evaluation (OPE) allows us to evaluate and estimate a new sequential decision-making policy's performance by leveraging historical interaction data collected from other policies. Evaluating a new policy online without a confident estimate of its performance can lead to costly, unsafe, or hazardous outcomes, especially in education and healthcare. Several OPE estimators have been proposed in the last decade, many of which have hyperparameters and require training. Unfortunately, choosing the best OPE algorithm for each task and domain is still unclear. In this paper, we propose a new algorithm that adaptively blends a set of OPE estimators given a dataset without relying on an explicit selection using a statistical procedure. We prove that our estimator is consistent and satisfies several desirable properties for policy evaluation. Additionally, we demonstrate that when compared to alternative approaches, our estimator can be used to select higher-performing policies in healthcare and robotics. Our work contributes to improving ease of use for a general-purpose, estimator-agnostic, off-policy evaluation framework for offline RL.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Cao, R. (1993). Bootstrapping the mean integrated squared error. Journal of Multivariate Analysis, 45(1):137–160.
  2. Learning bellman complete representations for offline policy evaluation. In International Conference on Machine Learning, pages 2938–2971. PMLR.
  3. Chen, Y.-C. (2017a). Introduction to resampling methods. lecture 5: Bootstrap. https://faculty.washington.edu/yenchic/17Sp_403/Lec5-bootstrap.pdf.
  4. Chen, Y.-C. (2017b). Introduction to resampling methods. lecture 9: Introduction to the bootstrap theory. https://faculty.washington.edu/yenchic/17Sp_403/Lec9_theory.pdf.
  5. Double/debiased machine learning for treatment and causal parameters. arXiv preprint arXiv:1608.00060.
  6. Using a bootstrap method to choose the sample fraction in tail index estimation. Journal of Multivariate analysis, 76(2):226–248.
  7. Bootstrap bandwidth selection in kernel density estimation from a contaminated sample. Annals of the Institute of Statistical Mathematics, 56(1):19–47.
  8. Cvxpy: A python-embedded modeling language for convex optimization. The Journal of Machine Learning Research, 17(1):2909–2913.
  9. Model-based offline reinforcement learning with local misspecification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7423–7431.
  10. Bootstrap for correcting the mean square error of prediction and smoothed estimates in structural models.
  11. Efron, B. (1990). More efficient bootstrap computations. Journal of the American Statistical Association, 85(409):79–89.
  12. Efron, B. (1992). Bootstrap methods: another look at the jackknife. Springer.
  13. An introduction to the bootstrap. CRC press.
  14. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning, pages 1447–1456. PMLR.
  15. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219.
  16. Benchmarks for deep off-policy evaluation. arXiv preprint arXiv:2103.16596.
  17. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR.
  18. Bootstrapping statistical functionals. Statistics & probability letters, 39(3):229–236.
  19. Variational latent branching model for off-policy evaluation. arXiv preprint arXiv:2301.12056.
  20. A note on bootstrapping the sample median. The Annals of Statistics, 12(3):1130–1135.
  21. Combining parametric and nonparametric models for off-policy evaluation. In International Conference on Machine Learning, pages 2366–2375. PMLR.
  22. Hall, P. (1990). Using the bootstrap to estimate mean squared error and select smoothing parameter in nonparametric problems. Journal of multivariate analysis, 32(2):177–203.
  23. Hong, H. (1999). Lecture 11: Bootstrap.
  24. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pages 652–661. PMLR.
  25. Confounding-robust policy evaluation in infinite-horizon reinforcement learning. Advances in neural information processing systems, 33:22293–22304.
  26. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine, 24(11):1716–1720.
  27. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169.
  28. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191.
  29. Batch policy learning under constraints. In International Conference on Machine Learning, pages 3703–3712. PMLR.
  30. Oracle inequalities for model selection in offline reinforcement learning. arXiv preprint arXiv:2211.02016.
  31. Bootstrap variance estimation of nonlinear functions of parameters: an application to long-run elasticities of energy demand. Review of Economics and Statistics, 81(4):728–733.
  32. Breaking the curse of horizon: Infinite-horizon off-policy estimation. Advances in Neural Information Processing Systems, 31.
  33. Representation balancing mdps for off-policy policy evaluation. Advances in neural information processing systems, 31.
  34. Offline policy evaluation across representations with applications to educational games. In AAMAS, volume 1077.
  35. Non-parametric bootstrap mean squared error estimation for m-quantile estimators of small area averages, quantiles and poverty indicators. Computational Statistics & Data Analysis, 56(10):2889–2902.
  36. Mikusheva, A. (2013). Time series analysis. lecture 9: Bootstrap. https://ocw.mit.edu/courses/14-384-time-series-analysis-fall-2013/resources/mit14_384f13_lec9/.
  37. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32.
  38. A spectral approach to off-policy evaluation for pomdps. arXiv preprint arXiv:2109.10502.
  39. Off-policy policy evaluation for sequential decisions under unobserved confounding. Advances in Neural Information Processing Systems, 33:18819–18831.
  40. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299–319.
  41. Counterfactual off-policy evaluation with gumbel-max structural causal models. In International Conference on Machine Learning, pages 4881–4890. PMLR.
  42. Păduraru, C. (2007). Planning with approximate and learned models of markov decision processes.
  43. Paduraru, C. (2013). Off-policy evaluation in markov decision processes.
  44. Precup, D. (2000). Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80.
  45. Reinforcement learning tutor better supported lower performers in a math task. Machine Learning, pages 1–26.
  46. Shao, J. (1990). Bootstrap estimation of the asymptotic variances of statistical functionals. Annals of the Institute of Statistical Mathematics, 42:737–752.
  47. A minimax learning approach to off-policy evaluation in confounded partially observable markov decision processes. In International Conference on Machine Learning, pages 20057–20094. PMLR.
  48. Shi, X. (2012). Econ 715. lecture 10: Bootstrap. https://www.ssc.wisc.edu/~xshi/econ715/Lecture_10_bootstrap.pdf.
  49. Adaptive estimator selection for off-policy evaluation. In International Conference on Machine Learning, pages 9196–9205. PMLR.
  50. Off-policy evaluation in partially observable environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10276–10283.
  51. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning, pages 2139–2148. PMLR.
  52. High confidence policy improvement. In International Conference on Machine Learning, pages 2380–2388. PMLR.
  53. Improved estimator selection for off-policy evaluation. Workshop on Reinforcement Learning Theory at the 38th International Conference on Machine Learning.
  54. Empirical study of off-policy policy evaluation for reinforcement learning. arXiv preprint arXiv:1911.06854.
  55. Empirical study of off-policy policy evaluation for reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
  56. Optimal and adaptive off-policy evaluation in contextual bandits. In International Conference on Machine Learning, pages 3589–3597. PMLR.
  57. Williams, C. J. (2010). The bootstrap method for estimating mse. https://www.webpages.uidaho.edu/~chrisw/stat514/bootstrap1.pdf.
  58. Wolpert, D. H. (1992). Stacked generalization. Neural networks, 5(2):241–259.
  59. Batch value-function approximation with only realizability. In International Conference on Machine Learning, pages 11404–11413. PMLR.
  60. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems, 33:6551–6561.
  61. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129–14142.
  62. Sope: Spectrum of off-policy estimators. Advances in Neural Information Processing Systems, 34:18958–18969.
  63. Non-parametric methods for partial identification of causal effects. Columbia CausalAI Laboratory Technical Report.
  64. Towards hyperparameter-free policy selection for offline reinforcement learning. Advances in Neural Information Processing Systems, 34.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets