Multiple-policy Evaluation via Density Estimation (2404.00195v2)

Abstract: We study the multiple-policy evaluation problem where we are given a set of $K$ policies and the goal is to evaluate their performance (expected total reward over a fixed horizon) to an accuracy $\epsilon$ with probability at least $1-\delta$. We propose an algorithm named $\mathrm{CAESAR}$ for this problem. Our approach is based on computing an approximate optimal offline sampling distribution and using the data sampled from it to perform the simultaneous estimation of the policy values. $\mathrm{CAESAR}$ has two phases. In the first we produce coarse estimates of the visitation distributions of the target policies at a low order sample complexity rate that scales with $\tilde{O}(\frac{1}{\epsilon})$. In the second phase, we approximate the optimal offline sampling distribution and compute the importance weighting ratios for all target policies by minimizing a step-wise quadratic loss function inspired by the DualDICE \cite{nachum2019dualdice} objective. Up to low order and logarithmic terms $\mathrm{CAESAR}$ achieves a sample complexity $\tilde{O}\left(\frac{H^4}{\epsilon^2}\sum_{h=1}^H\max_{k\in[K]}\sum_{s,a}\frac{(d_h^{\pi^k}(s,a))^2}{\mu^*_h(s,a)}\right)$, where $d^{\pi}$ is the visitation distribution of policy $\pi$, $\mu^*$ is the optimal sampling distribution, and $H$ is the horizon.

• The paper introduces CAESAR, a novel algorithm that evaluates multiple RL policies simultaneously using optimal sampling and density estimation techniques.
• It efficiently computes an optimal offline sampling distribution by leveraging coarse visitation estimates to reduce sample complexity compared to naive methods.
• Theoretical results guarantee non-asymptotic sample efficiency, paving the way for faster development and refinement of RL policies in practical applications.

Multiple-policy Evaluation via Density Estimation

Introduction to Multiple-policy Evaluation

Policy evaluation, a central problem within Reinforcement Learning (RL), seeks to estimate the expected total rewards from following a given policy. This is essential both for assessing the performance of existing policies and for guiding the development of new ones. The multiple-policy evaluation scenario, where the objective is to evaluate the performance of not just one, but a set of $K$ target policies, presents an interesting challenge. The naive approach of applying single-policy evaluation methods $K$ times does not leverage the potential overlap in the policies' behavior, leading to inefficiencies. This paper introduces a novel algorithm, CAESAR, which targets this gap by proposing an efficient means of evaluating multiple policies simultaneously.

Key Innovations of CAESAR

Efficient Computation of Optimal Sampling Distribution

CAESAR operates in two main phases, the first of which involves generating coarse estimates of the visitation distributions of each target policy using a sample complexity that scales with $\tilde{O}(\frac{1}{\epsilon})$. These estimations then inform the computation of an optimal offline sampling distribution. Notably, this distribution is approximated to ensure that it lies within the convex hull of the target policies' visitation distributions, facilitating feasible sample generation for the estimation process. Through this, CAESAR leverages similarities among target policies to ensure sample efficiency.

Importance Weighting for Policy Evaluation

Building on the established sampling distribution, CAESAR employs a novel application of importance weighting for multi-policy evaluation. Inspired by DualDICE, the algorithm minimizes a step-wise quadratic loss function to estimate importance weighting ratios accurately. Essentially, it tailors the density estimation technique for the finite-horizon, tabular MDP settings, enabling the accurate estimation of policy values with non-asymptotic sample complexity guarantees.

Theoretical Contributions

The paper establishes a finite sample complexity result for the problem of multi-policy evaluation, showcasing how CAESAR significantly outperforms naive uniform sampling over target policies. Specifically, under certain conditions, it achieves a sample complexity of $$\tilde{O}\left(\frac{H^4}{\epsilon^2}\sum_{h=1}^H\max_{k\in[K]}\sum_{s,a}\frac{(d_h^{\pi^k}(s,a))^2}{\mu^*_h(s,a)}\right)$$. Importantly, it also demonstrates that the estimated sampling distribution, derived from coarse estimations of the visitation distributions, approaches the efficiency of an optimal distribution. This establishes a theoretical foundation for sample-efficient multi-policy evaluation without requiring additional complexity introduced by an extensive search across all deterministic policies.

Practical Implications and Future Directions

CAESAR paves the way for more effective and efficient multi-policy evaluations, crucial for scenarios where multiple policies (e.g., resulting from different configurations or hyperparameters) must be assessed concurrently. This can significantly speed up the iterative process of RL algorithm development and policy refinement, especially in domains where data collection is expensive or time-consuming.

However, the developed methodology introduces new questions and potential research directions. For instance, the approach's dependence on the horizon $H^4$ suggests that further refinements could yield more scalable solutions, especially for problems with long horizons. Additionally, exploring how reward-dependent sample complexities can further optimize the evaluation process remains an open area, particularly in sparse reward environments where focusing on significant state-action pairs could lead to efficiency gains.

Conclusion

The CAESAR algorithm presents a significant step forward in the domain of multiple-policy evaluation, providing a methodologically sound and theoretically backed approach to efficiently estimate the performance of multiple policies. Its development not only addresses an existing gap in the literature but also opens avenues for further research into more efficient and effective policy evaluation methods within RL.