On the Relation Between the Randomized Extended Kaczmarz Algorithm and Coordinate Descent

Abstract: In this note we compare the randomized extended Kaczmarz (EK) algorithm and randomized coordinate descent (CD) for solving the full-rank overdetermined linear least-squares problem and prove that CD needs less operations for satisfying the same residual-related termination criteria. For the general least-squares problems, we show that running first CD to compute the residual and then standard Kaczmarz on the resulting consistent system is more efficient than EK.