Geometric Approach and Closed Exact Formulae for the Lasso

Abstract: We provide a geometric approach to the lasso. We study the tangency of the level sets of the least square objective function with the polyhedral boundary sets $B(t)$ of the parameters in $\mathbb R^p$ with the $\ell_1$ norm equal to $t$. Here $t$ decreases from the value $\hat t$, which corresponds to the actual, nonconstrained minimizer of the least square objective function, denoted by $\hat\beta$. We derive closed exact formulae for the solution of the lasso under the full rank assumption. Our method does not assume iterative numerical procedures and it is, thus, computationally more efficient than the existing algorithms for solving the lasso. We also establish several important general properties of the solutions of the lasso. We prove that each lasso solution form a simple polygonal chain in $\mathbb{R}^p$ with $\hat\beta$ and the origin as the endpoints. There are no two segments of the polygonal chain that are parallel. We prove that such a polygonal chain can intersect interiors of more than one orthant in $\mathbb{R}^p$, but it cannot intersect interiors of more than $p$ orthants, which is, in general, the best possible estimate for non-normalized data. We prove that if a polygonal chain passes from the interior of one to the interior of another orthant, then it never again returns to the interior of the former. The intersection of a chain and the interior of an orthant coincides with a segment minus its end points, which belongs to a ray having $\hat\beta$ as its initial point. We illustrate the results using real data examples as well as especially crafted examples with hypothetical data. Already in $p=2$ case we show a striking difference in the maximal number of quadrants a polygonal chain of a lasso solution can intersect in the case of normalized data, which is $1$ vs. nonnormalized data, which is $2$.