Frictionless Hamiltonian Descent and Coordinate Hamiltonian Descent for Strongly Convex Quadratic Problems and Beyond (2402.13988v3)

Abstract: We propose an optimization algorithm called Frictionless Hamiltonian Descent, which is a direct counterpart of classical Hamiltonian Monte Carlo in sampling. We find that Frictionless Hamiltonian Descent for solving strongly convex quadratic problems exhibits a novel update scheme that involves matrix-power-vector products. We also propose Frictionless Coordinate Hamiltonian Descent and its parallelizable variant, which turns out to encapsulate the classical Gauss-Seidel method, Successive Over-relaxation, Jacobi method, and more, for solving a linear system of equations. The result not only offers a new perspective on these existing algorithms but also leads to a broader class of update schemes that guarantee the convergence. Finally, we also highlight the potential of Frictionless Hamiltonian Descent beyond quadratics by studying solving certain non-convex functions, where Frictionless Hamiltonian Descent can find a global optimal point.