Last-Iterate Convergence: Zero-Sum Games and Constrained Min-Max Optimization (1807.04252v4)

Abstract: Motivated by applications in Game Theory, Optimization, and Generative Adversarial Networks, recent work of Daskalakis et al \cite{DISZ17} and follow-up work of Liang and Stokes \cite{LiangS18} have established that a variant of the widely used Gradient Descent/Ascent procedure, called "Optimistic Gradient Descent/Ascent (OGDA)", exhibits last-iterate convergence to saddle points in {\em unconstrained} convex-concave min-max optimization problems. We show that the same holds true in the more general problem of {\em constrained} min-max optimization under a variant of the no-regret Multiplicative-Weights-Update method called "Optimistic Multiplicative-Weights Update (OMWU)". This answers an open question of Syrgkanis et al \cite{SALS15}. The proof of our result requires fundamentally different techniques from those that exist in no-regret learning literature and the aforementioned papers. We show that OMWU monotonically improves the Kullback-Leibler divergence of the current iterate to the (appropriately normalized) min-max solution until it enters a neighborhood of the solution. Inside that neighborhood we show that OMWU is locally (asymptotically) stable converging to the exact solution. We believe that our techniques will be useful in the analysis of the last iterate of other learning algorithms.

Last-Iterate Convergence: Zero-Sum Games and Constrained Min-Max Optimization

The paper, "Last-Iterate Convergence: Zero-Sum Games and Constrained Min-Max Optimization," authored by Constantinos Daskalakis and Ioannis Panageas, rigorously investigates the last-iterate convergence of Optimistic Multiplicative-Weights Update (OMWU) in the context of constrained convex-concave min-max problems, specifically focusing on applications within game theory and optimization domains, including Generative Adversarial Networks (GANs).

Core Contributions

The primary contribution of this work is the extension of the last-iterate convergence result for unconstrained systems shown in earlier works to constrained systems using OMWU. The authors establish that OMWU exhibits last-iterate convergence to the saddle points for constrained min-max problems by introducing techniques fundamentally differing from the standard no-regret learning literature. Specifically, they demonstrate that OMWU consistently improves the Kullback-Leibler divergence relative to the optimal solution until convergence.

Numerics and Theory

The paper includes robust numerical results indicating the scale of convergence rate versus dimensions in zero-sum games, solidifying practical implications. A deterministic decrease of the Kullback-Leibler divergence is analytically shown, implying steady progression towards optimality. The proposed OMWU mechanism integrates aspects of no-regret learning with differential dynamics, ensuring that, under a particular choice of learning rate, the algorithm will converge, even in constrained scenarios.

Implications

This work bears significant implications for the theoretical understanding of learning dynamics in strategic interactions. By proving convergence under constraints, the authors offer a promising approach for real-world scenarios where such constraints are ubiquitous, such as economic markets or AI training frameworks like GANs. The establishment of last-iterate convergence rather than average convergence enhances the applicability of these methods, guaranteeing stability and predictability in dynamic settings.

Future Directions

Unanswered questions generated by this paper include exploring whether OMWU can lead to polynomial-time convergence rates and extending these results beyond strictly bilinear structures. Furthermore, optimizing the choice of step size for various instances could yield improvements in efficiency and independently serve as an interesting direction for further theoretical investigation.

The techniques presented for analyzing convergence can potentially be applied to other learning algorithms beyond OMWU, given the novel approach to asserting stability and convergence without reliance on traditional methods focusing solely on averaged iterates, thus broadening the horizon for research in stability of complex systems.