Per-Shot Evaluation of QAOA on Max-Cut: A Black-Box Implementation Comparison with Goemans-Williamson

Abstract: The Quantum Approximate Optimization Algorithm (QAOA) has emerged as a promising approach for addressing combinatorial optimization problems on near-term quantum hardware. In this work, we conduct an empirical evaluation of QAOA on the Max-Cut problem, using the Goemans-Williamson (GW) algorithm as a classical baseline for comparison. Unlike many prior studies, our methodology treats QAOA implementations as black-box optimizers, relying solely on default parameter settings without manual fine-tuning. We evaluate specific off-the-shelf QAOA implementations under default settings, not the algorithmic potential of QAOA with optimized parameters. This reflects a more realistic use case for end users who may lack the resources or expertise for instance-specific optimization. To facilitate fair and informative evaluation, we construct benchmark instances using well-known graph generation models that emulate practical graph structures, avoiding synthetic constructions tailored to either quantum or classical algorithms. A central component of our analysis is a per-shot statistical framework, which tracks the quality of QAOA outputs as a function of the number of circuit executions. This enables probabilistic comparisons with the GW algorithm by examining when and how frequently QAOA surpasses classical performance baselines such as the GW expectation and lower bound. Our results provide insight into the practical applicability of QAOA for Max-Cut and highlight its current limitations, offering a framework that can guide the assessment and development of future QAOA implementations.

• The paper implements a per-shot statistical framework to evaluate QAOA on Max-Cut, revealing that under default settings, fewer than 10% of QAOA runs meet the Goemans-Williamson expectation value.
• The benchmarking methodology employs a noiseless statevector simulation, ensuring the results are indicative of algorithmic, implementation-based performance, not hardware effects
• The paper finds that GW algorithm requires on average only three random hyperplane samplings to exceed its own expectation, starkly contrast to QAOA

Overview

This paper presents an empirical benchmarking study of the Quantum Approximate Optimization Algorithm (QAOA) applied to Max-Cut, evaluated against the Goemans–Williamson (GW) semidefinite-programming-based approximation algorithm (2604.08367). The central methodological contribution is a per-shot statistical framework: rather than reporting best-achieved or tuned performance, the authors treat QAOA implementations as black-box optimizers run under default parameter settings, and track solution quality as a function of circuit executions. This design choice deliberately measures what an end user without parameter-tuning expertise would experience, and it yields a stark conclusion: under default settings and realistic shot budgets, fewer than 10% of QAOA runs reach even the GW expectation value, whereas GW itself requires on average fewer than three random hyperplane samples to exceed its own expectation.

Motivation and positioning

QAOA's performance is known to be highly sensitive to the variational parameters $(\boldsymbol{\gamma}, \boldsymbol{\beta})$, and obtaining good parameters typically requires instance-specific classical optimization whose cost is often excluded from reported comparisons. The authors argue that this exclusion is misleading: if extensive parameter sweeps are required to beat a classical solver, that overhead must either be included in runtime accounting or the claimed advantage is not available to typical users. Their benchmark therefore fixes all parameters at implementation defaults — initial angles, circuit depth $p$, optimizer settings — for both quantum and classical solvers, and runs on noiseless statevector simulation to isolate algorithmic-implementation behavior from hardware effects.

The study also positions itself against prior empirical work with mixed outcomes: some studies report competitive or superior QAOA performance [e.g., (Boulebnane et al., 2021, Montanez-Barrera et al., 2024)], while others show local classical heuristics outperforming low-depth QAOA (Hsieh et al., 2022). The present work contributes a benchmarking protocol rather than a claim about QAOA's theoretical potential; the authors are explicit that their negative results concern specific implementations, not the paradigm.

Benchmarking methodology

Three components define the framework.

Black-box evaluation semantics. The paper distinguishes a shot (a single evaluation producing one candidate bitstring) from a run (a complete execution comprising many shots). For QAOA, shots are repeated measurements of a fixed circuit; for GW, each shot is an independent random hyperplane rounding of the SDP solution. The per-shot statistic tracked is the best-so-far cut value $C_{\max}^{(s)} = \max_{r \le s} C^{(r)}$, aggregated across $R$ independent runs via empirical percentiles $P_{100-\epsilon}(s)$. This best-so-far tracking matches how QAOA is operationally used — the classical post-processing retains the highest-quality sample — so it constitutes an optimistic but faithful reading of shot-based performance.

Instance generation with hardness guards. Because standard Gset instances are too large for tractable QAOA simulation and contain mixed-sign weights, the authors generate random graphs with $|V| = 29$ from three networkx models (Connected Watts–Strogatz, Barabási–Albert, Erdős–Rényi), subject to three constraints: (i) the GW expectation satisfies $\mathbb{E}[\alpha_{\mathrm{GW}}] \le 0.97$, ensuring a non-trivial gap to optimality at this small scale; (ii) the 99.9th percentile of the cut-value distribution lies below the GW lower bound, ruling out random sampling as a competitive baseline; and (iii) between 29 and $2^7$ distinct cuts exceed the GW expectation, guaranteeing enough "good" solutions for meaningful frequency statistics. Exact optima are computed by brute-force enumeration, giving reliable normalization.

Statistical comparison targets. QAOA trajectories are compared against two GW reference points: the GW lower bound ($\alpha \approx 0.878$ guarantee) and the instance-specific GW expectation $\mathbb{E}[C_{\mathrm{GW}}]$ computed analytically from the SDP solution. Confidence intervals on percentiles are obtained via non-parametric bootstrapping.

Experimental results

The experimental configuration uses $p$0 runs per instance, a shot budget $p$1, and a GW sample budget of $p$2. Three findings emerge consistently across all instances:

1. Rapid passage of the lower bound. Most runs exceed the GW lower bound within the first 2,000 shots (two Erdős–Rényi instances require roughly 6,000). Default-parameter QAOA thus quickly escapes trivially bad cuts.
2. Stagnation below the GW expectation. Even after the full budget, fewer than 1% of runs exceed the GW expectation on every instance except one Barabási–Albert graph reaching 1.2%; in the worst case the fraction never exceeds 0.2%. No instance shows convergence toward the GW expectation as shot count increases, and for no instance does $p$3 reach it; on three instances even $p$4 remains below.
3. Near-optimal cuts are essentially absent. Across 7,000 total runs over the whole dataset, exactly one run produced a cut exceeding 99% of the optimum.

The GW side of the comparison is equally quantified: on average at most 3 hyperplane samples suffice to beat the GW expectation, at most 15 suffice to find an optimal cut, and only two instances fail to hit $p$5 within $p$6 samples while still achieving $p$7. The resulting asymmetry — GW surpasses its own expectation in about three samplings while QAOA never reliably reaches it within ~23,000 shots — is the paper's strongest quantitative claim.

Limitations and open questions

The authors are explicit about scope. First, results apply to a single publicly available QAOA implementation under default settings; they do not bound what optimized parameters, adaptive depth selection, or problem-aware ansätze could achieve, and the paper concedes that bridging the observed gap requires improved parameter-setting strategies and shot-allocation methods. Second, instances are limited to $p$8 with positive weights, chosen partly because small graphs force high GW performance (hence the 0.97 expectation cap); whether the qualitative stagnation persists at larger sizes or other graph families is not established here. Third, simulations are noiseless, so hardware decoherence effects — which would only worsen QAOA's position — are outside the measured scope. Fourth, the classical overhead of QAOA's own parameter optimization loop is deliberately excluded, meaning the comparison addresses fixed-circuit sampling behavior rather than full hybrid-loop cost. An open question the framework poses but does not answer is whether any off-the-shelf QAOA implementation can pass the same per-shot percentile tests without manual tuning.

Conclusion

The paper delivers a reproducible, statistically grounded benchmarking protocol for shot-based solvers on combinatorial optimization, together with a guarded instance-generation scheme suitable for small-scale studies. Applied to Max-Cut at $p$9, it shows that a representative black-box QAOA implementation, run as end users would run it, is decisively outperformed by GW under matched evaluation semantics. The result is best read as a measurement of the current implementation-to-user gap rather than a verdict on QAOA's asymptotic potential, and the released benchmark suite provides a concrete instrument for testing whether future implementations close it.