When is the average number of saddle points typical?

Abstract: A common measure of a function's complexity is the count of its stationary points. For complicated functions, this count grows exponentially with the volume and dimension of their domain. In practice, the count is averaged over a class of functions (the annealed average), but the large numbers involved can produce averages biased by extremely rare samples. Typical counts are reliably found by taking the average of the logarithm (the quenched average), which is more difficult and not often done in practice. When most stationary points are uncorrelated with each other, quenched and anneals averages are equal. Equilibrium heuristics can guarantee when most of the lowest minima will be uncorrelated. We show that these equilibrium heuristics cannot be used to draw conclusions about other minima and saddles by producing examples among Gaussian-correlated functions on the hypersphere where the count of certain saddles and minima has different quenched and annealed averages, despite being guaranteed `safe' in the equilibrium setting. We determine conditions for the emergence of nontrivial correlations between saddles, and discuss the implications for the geometry of those functions and what out-of-equilibrium settings might be affected.