Iterative Hessian sketch: Fast and accurate solution approximation for constrained least-squares (1411.0347v1)

Abstract: We study randomized sketching methods for approximately solving least-squares problem with a general convex constraint. The quality of a least-squares approximation can be assessed in different ways: either in terms of the value of the quadratic objective function (cost approximation), or in terms of some distance measure between the approximate minimizer and the true minimizer (solution approximation). Focusing on the latter criterion, our first main result provides a general lower bound on any randomized method that sketches both the data matrix and vector in a least-squares problem; as a surprising consequence, the most widely used least-squares sketch is sub-optimal for solution approximation. We then present a new method known as the iterative Hessian sketch, and show that it can be used to obtain approximations to the original least-squares problem using a projection dimension proportional to the statistical complexity of the least-squares minimizer, and a logarithmic number of iterations. We illustrate our general theory with simulations for both unconstrained and constrained versions of least-squares, including $\ell_1$-regularization and nuclear norm constraints. We also numerically demonstrate the practicality of our approach in a real face expression classification experiment.

• The paper demonstrates that classical randomized sketching is suboptimal for solution approximation due to the need for sketch sizes proportional to the sample size.
• It introduces the iterative Hessian sketch (IHS) method that refines approximations iteratively to achieve ε-accurate solutions in log(1/ε) steps.
• The method is validated both theoretically and numerically, showing geometric convergence and practical scalability for high-dimensional, constrained least-squares problems.

Iterative Hessian Sketch for Fast and Accurate Solution Approximation in Constrained Least-Squares Problems

The paper addresses the limitations of classical randomized sketching methods when applied to constrained least-squares problems. It introduces the iterative Hessian sketch (IHS) as a method to achieve fast and accurate solution approximations, focusing on minimizing the distance between the approximate and true minimizers rather than merely approximating the objective function cost.

Main Contributions

1. Sub-optimality of Classical Sketching: The paper provides a lower bound for any method sketching the data matrix and vector in a least-squares problem. It highlights that the most widely used classical least-squares sketch, which involves both the data matrix and vector, is sub-optimal for solution approximation. The results reveal that a sketch size proportional to the sample size $n$ is necessary, which negates the computational benefits when $n \gg d$.
2. Iterative Hessian Sketch (IHS): The primary contribution is the introduction of the IHS method, which iteratively refines an approximation to the least-squares solution. This method involves computing multiple sketches and refining the solution iteratively, exploiting log $(1/\varepsilon)$ steps to achieve $\varepsilon$-accurate solutions. It addresses the shortcoming of the classical approach by only sketching the data matrix and iteratively reducing the solution error.
3. Numerical and Theoretical Validation: The IHS is supported by theoretical guarantees that constrain error bounds in terms of the statistical complexity of the problem. Numerical experiments, including unconstrained and constrained scenarios with $\ell_1$-regularization and nuclear norm constraints, empirically demonstrate the efficiency and practicality of the IHS compared to classical methods.

Strong Results and Claims

• The IHS algorithm achieves geometric convergence in the solution approximation error, contingent on the projection dimension and number of iterations. It efficiently matches the optimal solution error rates attained by full least-squares solutions, with computational requirements that scale with problem complexity rather than the ambient dimension.
• Theoretical analysis shows that for a range of least-squares problems, including high-dimensional scenarios like sparse and low-rank estimations, the sketch size and iterations provided by IHS lead to a significant decrease in computational burden while maintaining solution accuracy.

Implications and Speculation on Future Development

The findings challenge conventional strategies used in constrained least-squares approximation, highlighting the gap between computational savings and solution accuracy in high-dimensional settings. This discrepancy prompted the development of the IHS approach, which not only offers theoretical guarantees for solution proximity but also suggests a broader application scope in statistical learning and signal processing.

Future developments in AI and machine learning can incorporate such iterative techniques to handle growing datasets within manageable computational bounds without compromising inferential precision. The targeted reduction of solution error through adaptive sketching methods could reshape applications in regression analysis, especially in fields requiring real-time data processing and decision-making.

The paper's methodological advancements underscore the potential for iterative algorithms in refining solution-based optimization, setting the stage for their integration into mainstream numerical computing frameworks. Research could further explore the versatility of the IHS approach by extending its application to non-convex settings and more complex machine learning models. Such extensions would expand its utility beyond least-squares problems, potentially addressing a wide array of optimization challenges in AI.