Subspace Embedding and Linear Regression with Orlicz Norm (1806.06430v1)

Abstract: We consider a generalization of the classic linear regression problem to the case when the loss is an Orlicz norm. An Orlicz norm is parameterized by a non-negative convex function $G:\mathbb{R}+\rightarrow\mathbb{R}+$ with $G(0)=0$: the Orlicz norm of a vector $x\in\mathbb{R}^n$ is defined as $ |x|G=\inf\left{\alpha>0\large\mid\sum{i=1}^n G(|x_i|/\alpha)\leq 1\right}. $ We consider the cases where the function $G(\cdot)$ grows subquadratically. Our main result is based on a new oblivious embedding which embeds the column space of a given matrix $A\in\mathbb{R}^{n\times d}$ with Orlicz norm into a lower dimensional space with $\ell_2$ norm. Specifically, we show how to efficiently find an embedding matrix $S\in\mathbb{R}^{m\times n},m<n$ such that $\forall x\in\mathbb{R}^{d},\Omega(1/(d\log n)) \cdot |Ax|G\leq |SAx|_2\leq O(d^2\log n) \cdot |Ax|_G.$ By applying this subspace embedding technique, we show an approximation algorithm for the regression problem $\min{x\in\mathbb{R}^d} |Ax-b|_G$, up to a $O(d\log^2 n)$ factor. As a further application of our techniques, we show how to also use them to improve on the algorithm for the $\ell_p$ low rank matrix approximation problem for $1\leq p<2$.