Papers
Topics
Authors
Recent
Search
2000 character limit reached

Optimization with Access to Auxiliary Information

Published 1 Jun 2022 in cs.LG and math.OC | (2206.00395v4)

Abstract: We investigate the fundamental optimization question of minimizing a target function ff, whose gradients are expensive to compute or have limited availability, given access to some auxiliary side function hh whose gradients are cheap or more available. This formulation captures many settings of practical relevance, such as i) re-using batches in SGD, ii) transfer learning, iii) federated learning, iv) training with compressed models/dropout, Et cetera. We propose two generic new algorithms that apply in all these settings; we also prove that we can benefit from this framework under the Hessian similarity assumption between the target and side information. A benefit is obtained when this similarity measure is small; we also show a potential benefit from stochasticity when the auxiliary noise is correlated with that of the target function.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. On the convergence of sgd with biased gradients. arXiv:2008.00051 [cs.LG], 2020.
  2. Ac/dc: Alternating compressed/decompressed training of deep neural networks. arXiv:2106.12379 [cs.LG], 2019.
  3. Masked training of neural networks with partial gradients. arXiv:2106.08895 [cs.LG], 2021.
  4. Auxiliary learning by implicit differentiation. ICLR 2021. URL https://arxiv.org/pdf/2007.02693.pdf.
  5. Practical coreset constructions for machine learning. arXiv:1703.06476 [stat.ML]https://arxiv.org/abs/1703.06476, 2017.
  6. Auxiliary task reweighting for minimum-data learning. 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada. URL https://arxiv.org/pdf/2010.08244.pdf.
  7. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
  8. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27, 2011. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.
  9. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3):542–542, 2009.
  10. Linear speedup in personalized collaborative learning. arXiv:2111.05968 [cs.LG], 2021.
  11. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597–1607. PMLR, 2020.
  12. Momentum-based variance reduction in non-convex sgd. arXiv:1905.10018 [cs.LG], 2019.
  13. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.
  14. Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv:1905.11881 [math.OC]. URL https://arxiv.org/abs/1905.11881.
  15. Accelerating stochastic gradient descent using predictive variance reduction. NeurIPS, 2013.
  16. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv:2008.03606 [cs.LG], 2020a.
  17. SCAFFOLD: Stochastic controlled averaging for on-device federated learning. In 37th International Conference on Machine Learning (ICML), 2020b.
  18. Adaptive balancing of gradient and update computation times using global geometry and approximate subproblems. In International Conference on Artificial Intelligence and Statistics, pp.  1204–1213. PMLR, 2018.
  19. J. Kiefer and J. Wolfowitz. Stochastic estimation of the maximum of a regression function. Ann. Math. Statist. Volume 23, Number 3, 462-466, 1952.
  20. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  21. Federated optimization : Distributed machine learning for on-device intelligence. arxiv.org/abs/1610.02527, 2016.
  22. MNIST handwritten digit database. 2010. URL http://yann.lecun.com/exdb/mnist/.
  23. Deep learning. nature, 521(7553):436–444, 2015.
  24. Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML, 2013.
  25. Non-convex finite-sum optimization via scsg methods. arXiv:1706.09156 [math.OC]. URL https://arxiv.org/abs/1706.09156.
  26. Communication-efficient learning of deep networks from decentralized data. In Proceedings of AISTATS, pp. 1273–1282, 2017a.
  27. Communication-efficient learning of deep networks from decentralized data. In Proceedings of AISTATS, pp.  1273–1282, 2017b.
  28. Agnostic federated learning. arXiv preprint arXiv:1902.00146, 2019.
  29. Cubic regularization of newton method and its global performance. https://link.springer.com/content/pdf/10.1007/s10107-006-0706-8.pdf, 2006.
  30. Sarah: A novel method for machine learning problems using stochastic recursive gradient. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp.  2613–2621. JMLR. org, 2017.
  31. Stochastic variance reduction for nonconvex optimization. arXiv:1603.06160 [math.OC]https://arxiv.org/abs/1603.06160, 2016.
  32. A stochastic approximation method the annals of mathematical statistics. Vol. 22, No. 3. pp. 400-407, 1951a.
  33. A stochastic approximation method. The annals of mathematical statistics, pp.  400–407, 1951b.
  34. Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
  35. Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61:85–117, 2015.
  36. Descending through a crowded valley–benchmarking deep learning optimizers. arXiv preprint arXiv:2007.01547, 2020.
  37. Communication efficient distributed optimization using an approximate newton-type method. arXiv:1312.7853 [cs.LG], 2013.
  38. meprop: Sparsified back propagation for accelerated deep learning with reduced overfitting. In International Conference on Machine Learning, pp. 3299–3308. PMLR, 2017.
  39. Adaptive auxiliary task weighting for reinforcement learning. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. URL https://openreview.net/pdf?id=rkxQFESx8S.
  40. How transferable are features in deep neural networks? arXiv preprint arXiv:1411.1792, 2014.
  41. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  1803–1811, 2019.
  42. Xiaojin Jerry Zhu. Semi-supervised learning literature survey. Technical report, University of Wisconsin-Madison Department of Computer Sciences, 2005.
  43. A comprehensive survey on transfer learning. arXiv:1703.06476 [stat.ML]https://arxiv.org/abs/1911.02685, 2020.
Citations (10)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now