Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards Robust Benchmarking of Quantum Optimization Algorithms (2405.07624v1)

Published 13 May 2024 in quant-ph and cs.ET

Abstract: Benchmarking the performance of quantum optimization algorithms is crucial for identifying utility for industry-relevant use cases. Benchmarking processes vary between optimization applications and depend on user-specified goals. The heuristic nature of quantum algorithms poses challenges, especially when comparing to classical counterparts. A key problem in existing benchmarking frameworks is the lack of equal effort in optimizing for the best quantum and, respectively, classical approaches. This paper presents a comprehensive set of guidelines comprising universal steps towards fair benchmarks. We discuss (1) application-specific algorithm choice, ensuring every solver is provided with the most fitting mathematical formulation of a problem; (2) the selection of benchmark data, including hard instances and real-world samples; (3) the choice of a suitable holistic figure of merit, like time-to-solution or solution quality within time constraints; and (4) equitable hyperparameter training to eliminate bias towards a particular method. The proposed guidelines are tested across three benchmarking scenarios, utilizing the Max-Cut (MC) and Travelling Salesperson Problem (TSP). The benchmarks employ classical mathematical algorithms, such as Branch-and-Cut (BNC) solvers, classical heuristics, Quantum Annealing (QA), and the Quantum Approximate Optimization Algorithm (QAOA).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (75)
  1. M. Kuramata, R. Katsuki et al., “Solving large break minimization problems in a mirrored double round-robin tournament using quantum annealing,” PLOS ONE, vol. 17, no. 4, p. e0266846, Apr. 2022, publisher: Public Library of Science.
  2. J. R. Finžgar, P. Ross et al., “QUARK: A Framework for Quantum Computing Application Benchmarking,” in 2022 IEEE International Conference on Quantum Computing and Engineering (QCE), Sep. 2022, pp. 226–237, doi: 10.1109/QCE53715.2022.00042.
  3. K. Blekos, D. Brand et al., “A review on Quantum Approximate Optimization Algorithm and its variants,” Physics Reports, vol. 1068, pp. 1–66, Jun. 2024, doi: 10.1016/j.physrep.2024.03.002.
  4. T. Lubinski, C. Coffrin et al., “Optimization Applications as Quantum Performance Benchmarks,” Feb. 2024, doi: 10.48550/arXiv.2302.02278.
  5. B. Tasseff, T. Albash et al., “On the Emerging Potential of Quantum Annealing Hardware for Combinatorial Optimization,” Oct. 2022, doi: 10.48550/arXiv.2210.04291.
  6. J. King, S. Yarkoni et al., “Benchmarking a quantum annealing processor with the time-to-target metric,” Aug. 2015, doi: 10.48550/arXiv.1508.05087.
  7. E. Pelofske, A. Bärtschi et al., “Quantum Annealing vs. QAOA: 127 Qubit Higher-Order Ising Problems on NISQ Computers,” in High Performance Computing, A. Bhatele, J. Hammond et al., Eds.   Cham: Springer Nature Switzerland, 2023, pp. 240–258, doi: 10.1007/978-3-031-32041-5_13.
  8. R. Au-Yeung, N. Chancellor et al., “NP-hard but no longer hard to solve? Using quantum computing to tackle optimization problems,” Frontiers in Quantum Science and Technology, vol. 2, p. 1128576, Feb. 2023, doi: 10.3389/frqst.2023.1128576.
  9. G. E. Crooks, “Performance of the Quantum Approximate Optimization Algorithm on the Maximum Cut Problem,” Nov. 2018, doi: 10.48550/arXiv.1811.08419.
  10. S. Zielinski, J. Nüßlein et al., “Influence of Different 3SAT-to-QUBO Transformations on the Solution Quality of Quantum Annealing: A Benchmark Study,” in Proceedings of the Companion Conference on Genetic and Evolutionary Computation.   Lisbon Portugal: ACM, Jul. 2023, pp. 2263–2271, doi: 10.1145/3583133.3596330.
  11. E. Taveres-Cachat and F. Goia, “Exploring the impact of problem formulation in numerical optimization: A case study of the design of PV integrated shading systems,” Building and Environment, vol. 188, p. 107422, Jan. 2021, doi: 10.1016/j.buildenv.2020.107422.
  12. O. Şeker, N. Tanoumand et al., “Digital Annealer for quadratic unconstrained binary optimization: A comparative performance analysis,” Applied Soft Computing, vol. 127, no. C, Sep. 2022, doi: 10.1016/j.asoc.2022.109367.
  13. M. Padberg and G. Rinaldi, “A Branch-and-Cut Algorithm for the Resolution of Large-Scale Symmetric Traveling Salesman Problems,” SIAM Review, vol. 33, no. 1, pp. 60–100, Mar. 1991, doi: 10.1137/1033004.
  14. A. Abbas, A. Ambainis et al., “Quantum Optimization: Potential, Challenges, and the Path Forward,” Dec. 2023, doi: 10.48550/arXiv.2312.02279.
  15. J. Preskill, “Quantum Computing in the NISQ era and beyond,” Quantum, vol. 2, p. 79, Aug. 2018, publisher: Verein zur Förderung des Open Access Publizierens in den Quantenwissenschaften. doi: 10.22331/q-2018-08-06-79.
  16. H. Oshiyama and M. Ohzeki, “Benchmark of quantum-inspired heuristic solvers for quadratic unconstrained binary optimization,” Scientific Reports, vol. 12, no. 1, p. 2146, Feb. 2022, number: 1 Publisher: Nature Publishing Group. doi: 10.1038/s41598-022-06070-5.
  17. T. Albash and D. A. Lidar, “Adiabatic Quantum Computing,” Reviews of Modern Physics, vol. 90, no. 1, p. 015002, Jan. 2018, doi: 10.1103/RevModPhys.90.015002.
  18. D. Wolpert and W. Macready, “No free lunch theorems for optimization,” IEEE Transactions on Evolutionary Computation, vol. 1, no. 1, pp. 67–82, Apr. 1997, doi: 10.1109/4235.585893.
  19. F. Glover, “Future paths for integer programming and links to artificial intelligence,” Computers & Operations Research, vol. 13, no. 5, pp. 533–549, Jan. 1986, doi: 10.1016/0305-0548(86)90048-1.
  20. F. Glover, “Tabu Search—Part I,” ORSA Journal on Computing, vol. 1, no. 3, pp. 190–206, Aug. 1989, publisher: ORSA. doi: 10.1287/ijoc.1.3.190.
  21. F. Glover, “Tabu Search—Part II,” ORSA Journal on Computing, vol. 2, no. 1, pp. 4–32, Feb. 1990, publisher: ORSA. doi: 10.1287/ijoc.2.1.4.
  22. G. Palubeckis, “Multistart Tabu Search Strategies for the Unconstrained Binary Quadratic Optimization Problem,” Annals of Operations Research, vol. 131, no. 1-4, pp. 259–282, Oct. 2004, doi: 10.1023/B:ANOR.0000039522.58036.68.
  23. “dwave-samplers,” Oct. 2023, https://github.com/dwavesystems/dwave-samplers.
  24. S. Kirkpatrick, C. D. Gelatt et al., “Optimization by Simulated Annealing,” Science, vol. 220, no. 4598, pp. 671–680, May 1983, publisher: American Association for the Advancement of Science. doi: 10.1126/science.220.4598.671.
  25. Y. Nourani and B. Andresen, “A comparison of simulated annealing cooling strategies,” Journal of Physics A: Mathematical and General, vol. 31, no. 41, pp. 8373–8385, Oct. 1998, doi: 10.1088/0305-4470/31/41/011.
  26. IBM Corp., “IBM ILOG CPLEX 22.1.0 User Manual,” 2022, https://www.ibm.com/docs/en/icos/22.1.1.
  27. Gurobi Optimization, LLC, “Gurobi Optimizer Reference Manual,” 2023, https://www.gurobi.com.
  28. G. Dantzig, R. Fulkerson et al., “Solution of a Large-Scale Traveling-Salesman Problem,” Journal of the Operations Research Society of America, vol. 2, no. 4, pp. 393–410, 1954, publisher: INFORMS.
  29. G. Gamrath, “Enhanced predictions and structure exploitation in branch-and-bound,” Ph.D. dissertation, Technische Universität Berlin, 2020, doi: 10.14279/DEPOSITONCE-10710.
  30. G. B. Dantzig, “Maximization of a linear function of variables subject to linear inequalities,” Activity analysis of production and allocation, vol. 13, pp. 339–347, 1951.
  31. I. I. Dikin, “Iterative solution of problems of linear and quadratic programming,” Sov. Math., Dokl., vol. 8, pp. 674–675, 1967.
  32. A. Makhorin, “GLPK (GNU Linear Programming Kit),” 2012, https://www.gnu.org/software/glpk/.
  33. T. Achterberg, “SCIP: solving constraint integer programs,” Mathematical Programming Computation, vol. 1, no. 1, pp. 1–41, Jul. 2009, doi: 10.1007/s12532-008-0001-1.
  34. M. Berkelaar, “lp_solve,” 2021, https://lpsolve.sourceforge.net/.
  35. R. Lougee-Heimer, “The Common Optimization INterface for Operations Research: Promoting open-source software in the operations research community,” IBM Journal of Research and Development, vol. 47, no. 1, pp. 57–66, Jan. 2003, doi: 10.1147/rd.471.0057.
  36. H. Mittelmann, “Decison Tree for Optimization Software,” https://plato.asu.edu/bench.html.
  37. M. Sun, T. Li et al., “MindOpt Adapter for CPLEX Benchmarking Performance Analysis,” Jan. 2024, doi: 10.48550/arXiv.2312.13527.
  38. T. Kadowaki and H. Nishimori, “Quantum annealing in the transverse Ising model,” Physical Review E, vol. 58, no. 5, pp. 5355–5363, Nov. 1998, doi: 10.1103/PhysRevE.58.5355.
  39. M. Born and V. Fock, “Beweis des Adiabatensatzes,” Zeitschrift für Phys., vol. 51, no. 3, pp. 165–180, 1928, doi: 10.1007/BF01343193.
  40. E. Farhi, J. Goldstone et al., “A Quantum Approximate Optimization Algorithm,” Nov. 2014, arXiv:1411.4028 [quant-ph]. doi: 10.48550/arXiv.1411.4028.
  41. S. H. Sack and M. Serbyn, “Quantum annealing initialization of the quantum approximate optimization algorithm,” Quantum, vol. 5, p. 491, Jul. 2021, doi: 10.22331/q-2021-07-01-491.
  42. K. Mitarai, M. Negoro et al., “Quantum circuit learning,” Physical Review A, vol. 98, no. 3, p. 032309, Sep. 2018, doi: 10.1103/PhysRevA.98.032309.
  43. S. Hadfield, Z. Wang et al., “From the Quantum Approximate Optimization Algorithm to a Quantum Alternating Operator Ansatz,” Algorithms, vol. 12, no. 2, p. 34, Feb. 2019, number: 2 Publisher: Multidisciplinary Digital Publishing Institute. doi: 10.3390/a12020034.
  44. D. Cruz, R. Fournier et al., “Efficient quantum algorithms for G⁢H⁢Z𝐺𝐻𝑍GHZitalic_G italic_H italic_Z and W𝑊Witalic_W states, and implementation on the IBM quantum computer,” Advanced Quantum Technologies, vol. 2, no. 5-6, p. 1900015, Jun. 2019, doi: 10.1002/qute.201900015.
  45. Z. Wang, N. C. Rubin et al., “X⁢Y𝑋𝑌XYitalic_X italic_Y-mixers: analytical and numerical results for QAOA,” Physical Review A, vol. 101, no. 1, p. 012320, Jan. 2020, doi: 10.1103/PhysRevA.101.012320.
  46. J. Stein, F. Chamanian et al., “Evidence that PUBO outperforms QUBO when solving continuous optimization problems with the QAOA,” in Proceedings of the Companion Conference on Genetic and Evolutionary Computation, ser. GECCO ’23 Companion.   New York, NY, USA: Association for Computing Machinery, Jul. 2023, pp. 2254–2262, doi: 10.1145/3583133.3596358.
  47. A. Glos, A. Krawiec et al., “Space-efficient binary optimization for variational computing,” Sep. 2020, doi: 10.48550/arXiv.2009.07309.
  48. A. Bärtschi and S. Eidenbenz, “Grover Mixers for QAOA: Shifting Complexity from Mixer Design to State Preparation,” in 2020 IEEE International Conference on Quantum Computing and Engineering (QCE), Oct. 2020, pp. 72–82, doi: 10.1109/QCE49297.2020.00020.
  49. L. K. Grover, “A fast quantum mechanical algorithm for database search,” in Proceedings of the twenty-eighth annual ACM symposium on Theory of Computing, ser. STOC ’96.   New York, NY, USA: Association for Computing Machinery, Jul. 1996, pp. 212–219, doi: 10.1145/237814.237866.
  50. F. Glover, G. Kochenberger et al., “Quantum bridge analytics I: a tutorial on formulating and using QUBO models,” Annals of Operations Research, vol. 314, no. 1, pp. 141–183, Jul. 2022, doi: 10.1007/s10479-022-04634-2.
  51. M. A. F. Romero, E. A. R. García et al., “A heuristic algorithm based on tabu search for the solution of flexible job shop scheduling problems with lot streaming,” in Proceedings of the Genetic and Evolutionary Computation Conference, ser. GECCO ’18.   New York, NY, USA: Association for Computing Machinery, Jul. 2018, pp. 285–292, doi: 10.1145/3205455.3205534.
  52. V. F. Yu, H. Susanto et al., “A Simulated Annealing Algorithm for the Vehicle Routing Problem With Parcel Lockers,” IEEE Access, vol. 10, pp. 20 764–20 782, 2022, conference Name: IEEE Access. doi: 10.1109/ACCESS.2022.3152062.
  53. B. Poggel, N. Quetschlich et al., “Recommending Solution Paths for Solving Optimization Problems with Quantum Computing,” in 2023 IEEE International Conference on Quantum Software (QSW), Jul. 2023, pp. 60–67, doi: 10.1109/QSW59989.2023.00017.
  54. T. Bartz-Beielstein, C. Doerr et al., “Benchmarking in Optimization: Best Practice and Open Issues,” Dec. 2020, doi: 10.48550/arXiv.2007.03488.
  55. I. Dunning, S. Gupta et al., “What Works Best When? A Systematic Evaluation of Heuristics for Max-Cut and QUBO,” INFORMS Journal on Computing, vol. 30, no. 3, pp. 608–624, Aug. 2018, doi: 10.1287/ijoc.2017.0798.
  56. M. Kowalsky, T. Albash et al., “3-regular three-XORSAT planted solutions benchmark of classical and quantum heuristic optimizers,” Quantum Science and Technology, vol. 7, no. 2, p. 025008, Apr. 2022, doi: 10.1088/2058-9565/ac4d1b.
  57. L. Zhou, S.-T. Wang et al., “Quantum Approximate Optimization Algorithm: Performance, Mechanism, and Implementation on Near-Term Devices,” Physical Review X, vol. 10, no. 2, p. 021067, Jun. 2020, doi: 10.1103/PhysRevX.10.021067.
  58. S. Sharma and V. Kumar, “A Comprehensive Review on Multi-objective Optimization Techniques: Past, Present and Future,” Archives of Computational Methods in Engineering, vol. 29, no. 7, pp. 5605–5633, Nov. 2022, doi: 10.1007/s11831-022-09778-9.
  59. P. M. Lerman, “Fitting Segmented Regression Models by Grid Search,” Applied Statistics, vol. 29, no. 1, p. 77, 1980, doi: 10.2307/2346413.
  60. J. Bergstra, B. Komer et al., “Hyperopt: a Python library for model selection and hyperparameter optimization,” Computational Science & Discovery, vol. 8, no. 1, p. 014008, Jul. 2015, publisher: IOP Publishing. doi: 10.1088/1749-4699/8/1/014008.
  61. C. Erden, “Genetic algorithm-based hyperparameter optimization of deep learning models for PM2.5 time-series prediction,” International Journal of Environmental Science and Technology, vol. 20, no. 3, pp. 2959–2982, Mar. 2023, doi: 10.1007/s13762-023-04763-6.
  62. T. Akiba, S. Sano et al., “Optuna: A Next-generation Hyperparameter Optimization Framework,” Jul. 2019, doi: 10.48550/arXiv.1907.10902.
  63. J. A. Montanez-Barrera, D. Willsch et al., “Transfer learning of optimal QAOA parameters in combinatorial optimization,” Feb. 2024, arXiv:2402.05549 [quant-ph]. doi: 10.48550/arXiv.2402.05549.
  64. C. McGeoch and P. Farré, “The Advantage System: Performance Update,” 2020, doi: https://www.dwavesys.com/media/kjtlcemb/14-1054a-a_advantage_system_performance_update.pdf.
  65. F. Barahona and A. R. Mahjoub, “On the cut polytope,” Mathematical Programming: Series A and B, vol. 36, no. 2, pp. 157–173, Jun. 1986.
  66. M. X. Goemans and D. P. Williamson, “Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming,” Journal of the ACM, vol. 42, no. 6, pp. 1115–1145, Nov. 1995, doi: 10.1145/227683.227684.
  67. S. Diamond and S. Boyd, “CVXPY: A Python-embedded modeling language for convex optimization,” Journal of Machine Learning Research, vol. 17, no. 83, pp. 1–5, 2016.
  68. P. Erdős and A. Rényi, “On random graphs. I.” Publicationes Mathematicae Debrecen, vol. 6, no. 3-4, pp. 290–297, Jul. 1959, doi: 10.5486/PMD.1959.6.3-4.12.
  69. J. Nocedal, “Updating quasi-Newton matrices with limited storage,” Mathematics of Computation, vol. 35, no. 151, pp. 773–782, 1980, doi: 10.1090/S0025-5718-1980-0572855-7.
  70. R. Herrman, J. Ostrowski et al., “Lower bounds on circuit depth of the quantum approximate optimization algorithm,” Quantum Information Processing, vol. 20, no. 2, p. 59, Feb. 2021, doi: 10.1007/s11128-021-03001-7.
  71. A. C. Atkinson, “A segmented algorithm for simulated annealing,” Statistics and Computing, vol. 2, no. 4, pp. 221–230, Dec. 1992, doi: 10.1007/BF01889682.
  72. D. Willsch, M. Willsch et al., “Benchmarking Advantage and D-Wave 2000Q quantum annealers with exact cover problems,” Quantum Information Processing, vol. 21, no. 4, p. 141, Apr. 2022, doi: 10.1007/s11128-022-03476-y.
  73. A. Lucas, “Ising formulations of many NP problems,” Frontiers in Physics, vol. 2, 2014, doi: 10.3389/fphy.2014.00005.
  74. A. Montanez-Barrera, D. Willsch et al., “Unbalanced penalization: A new approach to encode inequality constraints of combinatorial problems for quantum optimization algorithms,” Jun. 2023, doi: 10.48550/arXiv.2211.13914.
  75. H. Schawe and A. K. Hartmann, “Phase Transitions of Traveling Salesperson Problems solved with Linear Programming and Cutting Planes,” EPL (Europhysics Letters), vol. 113, no. 3, p. 30004, Feb. 2016, arXiv:1512.08554 [cond-mat]. doi: 10.1209/0295-5075/113/30004.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now