Automatic tuning of AFOR hyperparameters

Develop an automatic procedure for adjusting AFOR’s three additional hyperparameters—the lower bound eta_2^{min}, direction weight w, and warm-up length T_w—in order to reduce manual tuning.

Background

AFOR introduces three hyperparameters beyond Adam: the lower bound on the adaptive second-moment decay coefficient, the direction weight used by the Residual-Alignment Signal Scorer, and the warm-up length used by the Adaptive Forgetting Controller. In the experiments, these values are fixed using a sensitivity study and are not automatically adapted.

The conclusion explicitly states that automatically adjusting these parameters remains open. An automatic tuning mechanism could improve the optimizer’s portability across EEG datasets, model backbones, and other nonstationary optimization settings while reducing reliance on manual configuration.

References

While AFOR addresses the fixed second-moment memory limitation, two aspects remain open for future investigation. First, the guarantee in Section~VI is conditional on the gradient-drift requirement and rests on worst-case constants, so it is qualitative rather than a quantitative certificate for the reported runs. Second, the three additional hyperparameters are fixed to defaults validated by our sensitivity study, and adjusting them automatically would reduce manual tuning.

— Adaptive Forgetting for Nonstationary Optimization: Towards Robust EEG Decoding  (2609.24233 - Zhu et al., 21 Sep 2026) in Section VI, CONCLUSION

Also, BLEND’s performance depends on hyperparameters (e.g., $\gamma$ and $\lambda$) that govern the personalization–generalization trade-off, and their adaptive selection in new environments remains an open problem.

— BLEND: Balancing Personalization vs. Generalization in Federated Vision-Language Models  (2610.12907 - Abdisarabshali et al., 9 Oct 2026) in Section Limitations