Performance of empirical risk minimization in linear aggregation

Abstract: We study conditions under which, given a dictionary $F={f_1,\ldots ,f_M}$ and an i.i.d. sample $(X_i,Y_i){i=1}^N$, the empirical minimizer in $\operatorname {span}(F)$ relative to the squared loss, satisfies that with high probability [R\bigl(\tilde{f}^{\mathrm{ERM}}\bigr)\leq\inf{f\in\operatorname {span}(F)}R(f)+r_N(M),] where $R(\cdot)$ is the squared risk and $r_N(M)$ is of the order of $M/N$. Among other results, we prove that a uniform small-ball estimate for functions in $\operatorname {span}(F)$ is enough to achieve that goal when the noise is independent of the design.