Importance sampling for coordinate blocks

Extend randomized block coordinate descent generalization analysis from uniform sampling of coordinate blocks to importance sampling schemes, in order to determine whether nonuniform block-selection probabilities yield improved theoretical guarantees.

Background

The paper analyzes randomized block coordinate descent (RBCD) when each coordinate block is selected uniformly at random. Its high-probability optimization and generalization guarantees rely on this sampling scheme and on identities relating uniformly sampled partial gradients to the full gradient.

The conclusion explicitly identifies extension to importance sampling as a direction for further investigation. Such sampling may exploit differences among block-wise Lipschitz constants or gradient magnitudes and could potentially improve the dependence of the guarantees on the number and geometry of coordinate blocks.