A complexity analysis of Policy Iteration through combinatorial matrices arising from Unique Sink Orientations

Abstract: Unique Sink Orientations (USOs) are an appealing abstraction of several major optimization problems of applied mathematics such as for instance Linear Programming (LP), Markov Decision Processes (MDPs) or 2-player Turn Based Stochastic Games (2TBSGs). A polynomial time algorithm to find the sink of a USO would translate into a strongly polynomial time algorithm to solve the aforementioned problems---a major quest for all three cases. In addition, we may translate MDPs and 2TBSGs into the problem of finding the sink of an acyclic USO of a cube, which can be done using the well-known Policy Iteration algorithm (PI). The study of its complexity is the object of this work. Despite its exponential worst case complexity, the principle of PI is a powerful source of inspiration for other methods. As our first contribution, we disprove Hansen and Zwick's conjecture claiming that the number of steps of PI should follow the Fibonacci sequence in the worst case. Our analysis relies on a new combinatorial formulation of the problem---the so-called Order-Regularity formulation (OR). Then, for our second contribution, we (exponentially) improve the $\Omega(1.4142^n)$ lower bound on the number of steps of PI from Schurr and Szab\'o in the case of the OR formulation and obtain an $\Omega(1.4269^n)$ bound.