Adaptive parameter selection for the Optimal Tensor Method

Develop parameter selection and adaptation strategies for the Optimal Tensor Method (the optimal acceleration scheme of Kovalev et al., 2022; Carmon et al., 2022, implemented here as Algorithm “Optimal”) that improve its empirical efficiency by reducing the number of inner iterations and enhancing overall progress when theoretical parameter settings perform poorly in practice.

Background

In the paper’s experimental comparison of high-order acceleration schemes, the Optimal Tensor Method (referred to as Optimal Acceleration) exhibited the weakest practical performance relative to NATA, Near-Optimal Tensor Acceleration, and other methods.

The authors attribute this underperformance primarily to the internal parameters of the Optimal Tensor Method, which were set to their theoretical values in the implementation. This choice led to many inner-loop iterations with limited global progress, suggesting that practical performance is sensitive to parameter tuning and may benefit from adaptive strategies.

Consequently, the authors explicitly identify improving these internal parameters as an open question, motivating research into principled, adaptive parameter selection that preserves theoretical guarantees while providing strong empirical performance.