Log-Sum-Exponential Estimator for Off-Policy Evaluation and Learning (2506.06873v1)

Abstract: Off-policy learning and evaluation leverage logged bandit feedback datasets, which contain context, action, propensity score, and feedback for each data point. These scenarios face significant challenges due to high variance and poor performance with low-quality propensity scores and heavy-tailed reward distributions. We address these issues by introducing a novel estimator based on the log-sum-exponential (LSE) operator, which outperforms traditional inverse propensity score estimators. Our LSE estimator demonstrates variance reduction and robustness under heavy-tailed conditions. For off-policy evaluation, we derive upper bounds on the estimator's bias and variance. In the off-policy learning scenario, we establish bounds on the regret -- the performance gap between our LSE estimator and the optimal policy -- assuming bounded $(1+\epsilon)$-th moment of weighted reward. Notably, we achieve a convergence rate of $O(n^{-\epsilon/(1+ \epsilon)})$ for the regret bounds, where $\epsilon \in [0,1]$ and $n$ is the size of logged bandit feedback dataset. Theoretical analysis is complemented by comprehensive empirical evaluations in both off-policy learning and evaluation scenarios, confirming the practical advantages of our approach. The code for our estimator is available at the following link: https://github.com/armin-behnamnia/lse-offpolicy-learning.

• The paper introduces a novel LSE estimator that reduces variance in off-policy evaluation and outperforms traditional IPS methods.
• Theoretical analysis shows the estimator is asymptotically unbiased with optimal regret bounds under both bounded and heavy-tailed reward scenarios.
• Empirical results demonstrate enhanced stability and predictive accuracy, making the estimator robust for reinforcement learning applications.

Log-Sum-Exponential Estimator for Off-Policy Evaluation and Learning

The paper introduces a novel estimator inspired by the log-sum-exponential (LSE) operator to address high variance and robustness challenges in off-policy evaluation (OPE) and off-policy learning (OPL) scenarios. These scenarios often suffer due to low-quality propensity scores and heavy-tailed reward distributions, which traditional inverse propensity score (IPS) estimators struggle with. The LSE estimator leverages the robustness properties of the log-sum-exponential function, demonstrating reduced variance and improved performance under heavy-tailed conditions, thereby effectively outperforming existing estimators.

Theoretical Contributions

1. Bias and Variance Analysis:
   • The paper explores the LSE estimator's bias and variance, providing both bounds and asymptotic properties. Results show that the LSE becomes asymptotically unbiased when $\lambda$ is selected as a function of the sample size $n$, with a convergence rate of $O(n^{-\epsilon/(1+\epsilon)})$ provided $\epsilon \in [0,1]$.
   • Variance comparison highlights LSE's reduced variance compared to IPS, particularly under no robust constraints of heavy-tailed reward distributions.
2. Regret Bound:
   • An upper bound on regret is derived for learning policies using the LSE estimator. Notably, the convergence rate of the regret bounds is shown to be optimal, especially under bounded second moments, achieving $O(n^{-1/2})$. In essence, LSE is theoretically equipped to handle both bounded and unbounded weighted reward scenarios efficiently.
3. Robustness:
   • Under scenarios of noisy rewards and noisy (estimated) propensity scores, the LSE estimator showcases robustness. The upper bound on regret incorporates a cost associated with noise, demonstrating that the estimator effectively mitigates this with appropriate tuning of parameter $\lambda$.

Practical Implications

The empirical evaluations confirm that the LSE estimator consistently outperforms traditional methods across different datasets, including synthetic setups and real-world scenarios such as recommendation systems. Particularly, in environments with heavy-tailed reward distributions and unreliable propensity scores, LSE demonstrates significant advantages in stability and predictive accuracy. This positions the estimator as a formidable tool in reinforcement learning applications where robustness and variance control are critical.

Future Directions

The integration of the LSE estimator with model-based approaches, such as doubly robust methods, opens avenues for further enhancements in both propositions of theory and empirical results. Expansion into reinforcement learning settings lacking i.i.d. assumptions promises valuable insights into dependencies and correlations within data structures.

In conclusion, the LSE estimator offers a compelling alternative to extant methods within off-policy evaluation and learning, particularly under challenging scenarios involving heavy-tailed and noisy reward distributions. The paper’s comprehensive theoretical analysis, augmented by empirical validation, provides a robust foundation for future exploration in AI and reinforcement learning domains.