Universality of first-order methods on random and deterministic matrices

Abstract: General first-order methods (GFOM) are a flexible class of iterative algorithms which update a state vector by matrix-vector multiplications and entrywise nonlinearities. A long line of work has sought to understand the large-n dynamics of GFOM, mostly focusing on "very random" input matrices and the approximate message passing (AMP) special case of GFOM whose state is asymptotically Gaussian. Yet, it has long remained unknown how to construct iterative algorithms that retain this Gaussianity for more structured inputs, or why existing AMP algorithms can be as effective for some deterministic matrices as they are for random matrices. We analyze diagrammatic expansions of GFOM via the limiting traffic distribution of the input matrix, the collection of all limiting values of permutation-invariant polynomials in the matrix entries, to obtain the following results: 1. We calculate the traffic distribution for the first non-trivial deterministic matrices, including (minor variants of) the Walsh-Hadamard and discrete sine and cosine transform matrices. This determines the limiting dynamics of GFOM on these inputs, resolving parts of longstanding conjectures of Marinari, Parisi, and Ritort (1994). 2. We design a new AMP iteration which unifies several previous AMP variants and generalizes to new input types, whose limiting dynamics are Gaussian conditional on some latent random variables. The asymptotic dynamics hold for a large and natural class of traffic distributions (encompassing both random and deterministic input matrices) and the algorithm's analysis gives a simple combinatorial interpretation of the Onsager correction, answering questions posed recently by Wang, Zhong, and Fan (2022).

• The paper introduces a combinatorial framework that unifies the behavior of GFOM across random and deterministic matrices.
• It rigorously proves that after puncturing, structured matrices (e.g., Hadamard) exhibit Gaussian AMP dynamics akin to random orthogonal models.
• The analysis leverages diagrammatic expansions and cactus properties to explain the Onsager corrections and state evolution in iterative algorithms.

Universality of First-Order Methods on Random and Deterministic Matrices

Introduction and Motivation

The paper "Universality of first-order methods on random and deterministic matrices" (2604.11729) addresses a core theoretical gap in understanding the behavior of General First-Order Methods (GFOM)—that is, iterative algorithms built on alternating matrix-vector multiplications and entrywise nonlinearities—in high-dimensional settings where input matrices may be either random or highly structured and deterministic.

While Approximate Message Passing (AMP), a principal subclass of GFOM, is well-characterized for random i.i.d. matrices (notably, its iterates are asymptotically Gaussian and described by state evolution), key questions have persisted regarding the dynamics of AMP and general GFOM when the input $\bA$ is deterministic or lacks full randomness. These questions become critical for applications in statistical inference, optimization over pseudorandom structures, and compressed sensing, where fast, structured transforms (e.g., Hadamard or Fourier-type) are often employed.

Traffic Distributions and Diagrammatic Expansions

The paper advances a unified combinatorial diagrammatic framework to analyze GFOM iterates, drawing from the concept of the traffic distribution—the collection of limiting values of all permutation-invariant polynomials in the matrix entries. A principal technical achievement is the reduction of the study of GFOM to questions about these limits, indexed by finite (multi-)graphs (called diagrams) and their associated graph polynomials.

The connection to strong/weak cactus properties and factorization over cactus graphs becomes the central universality principle: If two matrix ensembles (random or deterministic) share the same traffic (or diagonal) distribution and satisfy a suitable cactus property, then for any fixed iteration depth $t$, all pGFOM observables converge to the same limiting behavior. Concretely, the analysis demonstrates that for a large natural class of input matrices—encompassing Wigner, Haar-invariant, and punctured structured orthogonal matrices—the nontrivial limits are fully determined by their behavior on cactus diagrams (Figure 1).

Figure 1: A cactus graph in $\mathcal{C}$; intuitively, a “tree of cycles.”

Universality for Delocalized Deterministic Matrices

A critical result is the rigorous extension of universality from random to deterministic matrices under delocalization conditions. The authors prove that punctured (i.e., centered) versions of highly delocalized orthogonal matrices—including the Walsh–Hadamard, discrete sine, and cosine transform matrices—exhibit the same traffic distribution as an associated random orthogonal model (ROM), upon removing the effect of the all-ones direction. This resolves longstanding conjectures in the AMP/statistical mechanics literature about the pseudorandomness of fast transform ensembles and identifies the necessity of puncturing to avoid divergence due to alignment with the all-ones vector. After puncturing, the AMP (and thus GFOM) dynamics, including state evolution, are identical to those for the ROM [(2604.11729), Theorem 1].

Combinatorial State Evolution and Onsager Corrections

The heart of the technical contribution is a diagrammatic expansion for vector-valued iterations: Each GFOM iterate decomposes as a linear combination over rooted, treelike diagrams. The asymptotic empirical law of iterates is governed by a process indexed by these diagrams, where only treelike terms—those built from trees with hanging cactuses—are non-vanishing in the limit. This leads to an explicit, general Gaussian description, conditional on latent "cactus variables" derived from the diagonal distribution.

Figure 2: A treelike diagram in $\mathcal{T}_1$, showing the rooted tree with hanging cactuses.

A new treelike AMP iteration is then constructed, generalizing all prior AMP and OAMP variants: The Onsager correction is determined explicitly via combinatorics on diagrams, yielding a state evolution that holds for any input ensemble where the (strong) cactus property and diagonal distribution exist, even when the matrix is deterministic or block-structured. The Onsager terms, which are crucial for ensuring the iterates retain conditional Gaussianity, are explained combinatorially as the analytic counterparts of eliminating intersecting walks in the diagram algebra.

Explicit Results and Examples

• Walsh–Hadamard and fast transform matrices: After centering (puncturing), their traffic and diagonal distributions coincide with ROM, and thus standard AMP state evolution holds. This directly explains empirical observations of AMP divergence with unpunctured structured matrices and validates the folklore pseudo-randomness hypothesis.
• Orthogonally invariant random matrices: The cactus property is shown to be equivalent to invariance under the action of $O(n)$, and the entire structure of state evolution (including covariance recursion for AMP iterates) can be computed via combinatorial formulas on diagrams.
• Block-structured models: The explicit limiting process is a mixture (indexed by block membership) of Gaussian processes whose covariance recursively involves the free cumulants of individual blocks. The work thus unifies AMP on standard and block Wigner, block GOE, as well as more general ensembles.

Figure 3: $\Tr(\bA^4)^2$ represented as two squares in the combinatorial notation (left) versus the 't Hooft double line notation (right), illustrating the duality between Feynman diagram expansions and the traffic framework.

Figure 4: The twelve Feynman diagrams associated to $\langle \Tr(\bA^4) \rangle$; gluing diagrams demonstrates the combinatorics behind cactus dominance and the emergence of “taco” and degenerate polyhedra contributions.

Implications and Future Directions

The paper provides an explicit and general universality theorem for AMP and GFOM: the asymptotic dynamics and achievable estimation/optimization limits are fully characterized, for both random and highly-structured deterministic (delocalized) matrices, by their (diagonal) traffic distribution. This result closes critical theoretical gaps regarding the optimality and applicability of AMP-type algorithms beyond i.i.d. ensembles and provides a combinatorial foundation for designing optimal iterative algorithms tailored to arbitrary matrix structures admissible within the cactus universality class.

The findings have direct implications for algorithms in high-dimensional inference, optimization under structural constraints, and the judicious selection of fast transforms in signal processing and learning. The combinatorial method further offers a blueprint for analyzing iterates of broader classes of permutation- or block-invariant algorithms and sets a theoretical pathway for extending universality principles to even more structured and dependent ensembles.

Conclusion

This work rigorously bridges the gap between random matrix models and structured deterministic transforms in the analysis of first-order iterative algorithms, providing a complete combinatorial theory for their universality. The explicit diagrammatic expansion, the role of cactus-type structures, and the derivation of detailed Onsager corrections yield a sharp and unifying perspective on the Gaussian dynamics of GFOM and AMP—not only explaining but also predicting their behavior across both random and deterministic matrix environments. The combinatorial and algebraic framework presented here is poised to serve as a foundation for future research in both theoretical and applied large-scale optimization, inference, and learning settings.