Data-Driven Non-Parametric Model Learning and Adaptive Control of MDPs with Borel spaces: Identifiability and Near Optimal Design

Abstract: We consider an average cost stochastic control problem with standard Borel spaces and an unknown transition kernel. We do not assume a parametric structure on the unknown kernel. We present topologies on kernels which lead to their identifiability and ensures near optimality through robustness to identifiability errors. Following this, we present two data-driven identifiability results; the first one being Bayesian (with finite time convergence guarantees) and the second one empirical (with asymptotic convergence guarantees). The identifiability results are, then, used to design near-optimal adaptive control policies which alternate between periods of exploration, where the agent attempts to learn the true kernel, and periods of exploitation where the knowledge accrued throughout the exploration phase is used to minimize costs. We will establish that such policies are near optimal. Moreover, we will show that near optimality can also be achieved through policies that simultaneously explore and exploit the environment. Thus, our primary contributions are to present very general conditions ensuring identifiability, as well as adaptive learning and control which leads to optimality. Key tools facilitating our analysis are a general measurability theorem, robustness to model learning, and continuity of expected average cost in stationary policies under Young topology.