Do Not Discretize, Optimize: Almost Greedy Fictitious Play

Abstract: Our work revolves around Fictitious Play, one of the first iterative methods that is known to converge to a Nash equilibrium in zero-sum games. In recent years, there has been a revived interest, due to applications in various machine learning problems, which has motivated a line of work on its convergence properties and on proposing new variants of the initial algorithm. Our paper is along this direction and introduces one new variant, which we refer to as Almost Greedy Fictitious Play. The proposed algorithm greedily attempts to find the optimal stepsize at each iteration but its search space is constrained and includes almost all the line between the cumulative mixed strategy and the current best response. Our main result is that the method achieves an instance dependent convergence rate of $\mathcal{O}(1/T)$ with respect to the duality gap. This matches the rate of Continuous Fictitious Play, and offers an alternative to discretization. We complement our theoretical findings with experiments that demonstrate the effectiveness of the method.

• The paper introduces AGFP, an adaptive stepsize method that almost minimizes the duality gap in zero-sum games, achieving an O(1/T) convergence rate.
• It reformulates classic Fictitious Play by integrating a constrained line search over [δ, 1] to avoid stagnation, with both theoretical guarantees and empirical validation.
• Empirical results show significant improvements over standard FP, including up to 50x speedup in large-scale Gaussian games, demonstrating AGFP’s practical impact.

Almost Greedy Fictitious Play: Instance-Optimal Step Selection for Zero-Sum Games

Introduction

This work introduces Almost Greedy Fictitious Play (AGFP), a variant of Fictitious Play (FP) designed for efficient convergence to Nash equilibrium in zero-sum games without recourse to discretization artifacts. Building on the classic FP framework—where agents update their mixed strategies by best responding to opponent history and then averaging—AGFP employs an adaptive stepsize at each iteration, greedily chosen to (almost) minimize the duality gap along the line segment to the current best responses. The key innovation is constraining the stepsize search to $[\delta,1]$, with $\delta>0$ serving as a regularization parameter to circumvent stalling at non-progressing points due to numerical or tie-breaking issues.

Algorithmic Framework

The core insight is that standard Fictitious Play (with step $1/(t+1)$) can be viewed as Frank-Wolfe with a canonical schedule. In contrast, the Greedy Fictitious Play variant replaces this schedule with a line search determining the step along the direction from the current iterate towards the best response in the strategy space, minimizing the duality gap. However, directly minimizing over $[0,1]$ is problematic due to potential stagnation when the minimizer is at zero. AGFP avoids this by enforcing $\eta_t \geq \delta$.

Figure 1: Greedy Fictitious Play progression in the Rock-Paper-Scissors game.

Figure 2: Greedy stepsize $\eta_t$ selected in the initial iterations.

Formally, denoting the current average strategies by $(x_{t-1}, y_{t-1})$ and the best responses by $(e_{i_t}, e_{j_t})$, AGFP computes

$$\eta_t = \operatorname*{arg\,min}_{\eta \in [\delta,1]} \psi\big((1-\eta)z_{t-1} + \eta(e_{i_t}, e_{j_t})\big),$$

where $\psi$ is the duality gap metric. The updated profile is set as the aforementioned convex combination.

Theoretical Properties

The primary theoretical result establishes that AGFP attains an $\delta>0$0 rate with respect to the duality gap, where the hidden constant is game-dependent and reflects a novel "condition-like number" $\delta>0$1. This rate matches that of continuous-time Fictitious Play without the discretization degradation inherent in the classic FP. The analysis hinges on the conditional decrease in the duality gap, the identification of tie-breaking and stability regimes for best responses, and structuring the sequence of iterates to avoid pathological stagnation.

Importantly, the results hold for last-iterate convergence, not merely on averaged sequences, and the adaptive stepsize is shown to be optimal up to game constants. The duality gap decrease is often geometric along segments where best responses remain stable and is maintained by $\delta>0$2-truncated steps elsewhere.

Figure 3: AGFP performance on Rock-Paper-Scissors with $\delta>0$3; duality gap approaches plateau determined by $\delta>0$4.

Figure 4: Duality gap per iteration, illustrating $\delta>0$5 behavior across iterations in random Gaussian games.

Empirical Evaluation

AGFP was evaluated on canonical and high-dimensional random zero-sum games. The algorithm consistently demonstrated superior convergence over classic FP:

• Dimensionality Scaling: Empirical rates suggest $\delta>0$6 scaling in $\delta>0$7 games, matching the best known rates for continuous FP and outperforming FP's $\delta>0$8.
• Wall-clock Speed: Despite a more complex per-iteration computation due to line search, AGFP outstrips FP in time-to-accuracy for moderate accuracy requirements. For example, in $\delta>0$9 Gaussian games, AGFP was up to 50x faster at reaching $1/(t+1)$0.

  Figure 5: AGFP achieves $1/(t+1)$1 scaling in a $1/(t+1)$2 random Gaussian zero-sum game.

  

  Figure 6: Wall-clock comparison: AGFP vs. classical FP for reaching the same duality gap accuracy.

In Rock-Paper-Scissors and its generalizations, the optimal stepsize schedule closely follows $1/(t+1)$3 when resolving in the correct tie-breaking regime, and $1/(t+1)$4-induced plateaus emerge as predicted by theory.

Implications and Future Directions

AGFP eliminates the reliance on coarse discretization, providing an instance-optimal, last-iterate convergent method for zero-sum games. This property is relevant for large-scale, distributed, or online equilibrium computation. The $1/(t+1)$5 constant, akin to a local condition number, invites further investigation, especially with respect to smoothed analysis and connections to recent advances in the convergence theory of gradient-based equilibrium algorithms (2606.12149).

Figure 7: Stepsize trajectory in higher-dimensional RPS; emergence of line segments with stepsizes distinct from $1/(t+1)$6.

The methodology opens several avenues:

1. Extension to Potential Games: AGFP presents a promising template for efficient equilibrium finding where jumps to pure Nash equilibria can be exploited.
2. Smoothed Complexity: Bounding or characterizing $1/(t+1)$7 in generic settings could bridge gaps in longstanding conjectures concerning Fictitious Play, especially in random matrices.
3. Non Zero-Sum and General-sum Extensions: Adapting the AGFP framework and duality gap minimization strategies to broader classes of games.
4. Tie-Breaking and Memory-Augmented Dynamics: Unexplained empirical phenomena in high-dimensional RPS suggest further study on history-sensitive or stochastic tie-breaking rules.

Conclusion

AGFP leverages adaptive, instance-optimal step selection to achieve theoretically backed and empirically validated convergence in zero-sum games. Its regularization mechanism ensures robust progress, and its performance suggests AGFP as a compelling tool for equilibrium computation in high-dimensional, practical settings. Future research should focus on the theoretical understanding of $1/(t+1)$8, extensions beyond zero-sum settings, and integration with modern policy optimization and adversarial learning frameworks.