Analyze iterate-dependent sampling policies

Extend the convergence and finite-time analysis of the residual-preconditioned advantage-based value-learning algorithm from a fixed behavior policy to iterate-dependent sampling policies such as epsilon-greedy exploration.

Background

The paper’s theoretical analysis assumes that samples are generated by a fixed behavior policy with full support and an ergodic induced state chain. Practical reinforcement-learning procedures commonly use policies that change with the iterates, including epsilon-greedy exploration. Establishing guarantees under such iterate-dependent sampling is explicitly identified as an unresolved direction.