Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 37 tok/s
Gemini 2.5 Pro 41 tok/s Pro
GPT-5 Medium 10 tok/s Pro
GPT-5 High 15 tok/s Pro
GPT-4o 84 tok/s Pro
Kimi K2 198 tok/s Pro
GPT OSS 120B 448 tok/s Pro
Claude Sonnet 4 31 tok/s Pro
2000 character limit reached

Online Multi-Task Learning with Recursive Least Squares and Recursive Kernel Methods (2308.01938v2)

Published 3 Aug 2023 in stat.ML and cs.LG

Abstract: This paper introduces two novel approaches for Online Multi-Task Learning (MTL) Regression Problems. We employ a high performance graph-based MTL formulation and develop two alternative recursive versions based on the Weighted Recursive Least Squares (WRLS) and the Online Sparse Least Squares Support Vector Regression (OSLSSVR) strategies. Adopting task-stacking transformations, we demonstrate the existence of a single matrix incorporating the relationship of multiple tasks and providing structural information to be embodied by the MT-WRLS method in its initialization procedure and by the MT-OSLSSVR in its multi-task kernel function. Contrasting the existing literature, which is mostly based on Online Gradient Descent (OGD) or cubic inexact approaches, we achieve exact and approximate recursions with quadratic per-instance cost on the dimension of the input space (MT-WRLS) or on the size of the dictionary of instances (MT-OSLSSVR). We compare our online MTL methods to other contenders in a real-world wind speed forecasting case study, evidencing the significant gain in performance of both proposed approaches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (26)
  1. Y. Zhang and Q. Yang, “A survey on multi-task learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 12, pp. 5586–5609, 2022.
  2. S. H. G. Oliveira, A. R. Gonçalves, and F. J. Von Zuben, “Asymmetric multi-task learning with local transference,” ACM Trans. Knowl. Discov. Data, vol. 16, no. 5, apr 2022. [Online]. Available: https://doi.org/10.1145/3514252
  3. A. Argyriou, T. Evgeniou, and M. Pontil, “Multi-task feature learning,” in Advances in Neural Information Processing Systems, B. Schölkopf, J. Platt, and T. Hoffman, Eds., vol. 19.   MIT Press, 2006. [Online]. Available: https://proceedings.neurips.cc/paper/2006/file/0afa92fc0f8a9cf051bf2961b06ac56b-Paper.pdf
  4. J. Chen, L. Tang, J. Liu, and J. Ye, “A convex formulation for learning shared structures from multiple tasks,” in Proceedings of the 26th Annual International Conference on Machine Learning, ser. ICML ’09.   New York, NY, USA: Association for Computing Machinery, 2009, p. 137–144. [Online]. Available: https://doi.org/10.1145/1553374.1553392
  5. J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang, “Learning under concept drift: A review,” IEEE Transactions on Knowledge and Data Engineering, vol. 31, no. 12, pp. 2346–2363, 2019.
  6. G. R. Lencione and F. J. Von Zuben, “Wind speed forecasting via multi-task learning,” in 2021 International Joint Conference on Neural Networks (IJCNN), 2021, pp. 1–8.
  7. G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, “Extreme learning machine: a new learning scheme of feedforward neural networks,” in 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541), vol. 2, 2004, pp. 985–990 vol.2.
  8. S. Ji and J. Ye, “An accelerated gradient method for trace norm minimization,” in Proceedings of the 26th Annual International Conference on Machine Learning, ser. ICML ’09.   New York, NY, USA: Association for Computing Machinery, 2009, p. 457–464. [Online]. Available: https://doi.org/10.1145/1553374.1553434
  9. L. Vandenberghe and S. Boyd, “Semidefinite programming,” SIAM Review, vol. 38, no. 1, pp. 49–95, 1996. [Online]. Available: https://doi.org/10.1137/1038003
  10. A. O. C. Ayres and F. J. Von Zuben, “The extreme value evolving predictor,” IEEE Transactions on Fuzzy Systems, pp. 1–1, 2020.
  11. A. Gonçalves, F. Von Zuben, and A. Banerjee, “Multi-task sparse structure learning with gaussian copula models,” Journal of Machine Learning Research, vol. 17, pp. 1–30, 04 2016.
  12. S. C. Hoi, D. Sahoo, J. Lu, and P. Zhao, “Online learning: A comprehensive survey,” Neurocomputing, vol. 459, pp. 249–289, 2021.
  13. E. Hazan, “Introduction to online convex optimization,” CoRR, vol. abs/1909.05207, 2019. [Online]. Available: http://arxiv.org/abs/1909.05207
  14. O. Dekel, P. M. Long, and Y. Singer, “Online multitask learning,” in Learning Theory, G. Lugosi and H. U. Simon, Eds.   Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, pp. 453–467.
  15. G. Cavallanti, N. Cesa-Bianchi, and C. Gentile, “Linear algorithms for online multitask classification,” Journal of Machine Learning Research, vol. 11, no. 97, pp. 2901–2934, 2010. [Online]. Available: http://jmlr.org/papers/v11/cavallanti10a.html
  16. X. Cao and K. J. R. Liu, “Decentralized sparse multitask rls over networks,” IEEE Transactions on Signal Processing, vol. 65, no. 23, pp. 6217–6232, 2017.
  17. S. P. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn., vol. 3, no. 1, pp. 1–122, 2011. [Online]. Available: https://doi.org/10.1561/2200000016
  18. J. Xu, P.-N. Tan, J. Zhou, and L. Luo, “Online multi-task learning framework for ensemble forecasting,” IEEE Transactions on Knowledge and Data Engineering, vol. 29, no. 6, pp. 1268–1280, 2017.
  19. D. Jin, J. Chen, C. Richard, and J. Chen, “Online proximal learning over jointly sparse multitask networks with ℓ∞,1subscriptℓ1\ell_{\infty,1}roman_ℓ start_POSTSUBSCRIPT ∞ , 1 end_POSTSUBSCRIPT regularization,” IEEE Transactions on Signal Processing, vol. 68, pp. 6319–6335, 2020.
  20. R. Nassif, S. Vlaski, C. Richard, J. Chen, and A. H. Sayed, “Multitask learning over graphs: An approach for distributed, streaming machine learning,” IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 14–25, 2020.
  21. J. D. A. Santos and G. A. Barreto, “A regularized estimation framework for online sparse lssvr models,” Neurocomputing, vol. 238, pp. 114–125, 2017. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231217301169
  22. Y. Engel, S. Mannor, and R. Meir, “The kernel recursive least-squares algorithm,” IEEE Transactions on Signal Processing, vol. 52, no. 8, pp. 2275–2285, 2004.
  23. G. R. Lencione and F. J. V. Zuben, “Reformulation of a regularized estimation framework for online sparse lssvr models,” 2023. [Online]. Available: https://doi.org/10.2139/ssrn.4474505
  24. G. R. Lencione, A. O. C. Ayres, and F. J. Von Zuben, “Online convex optimization of a multi-task fuzzy rule-based evolving system,” in 2022 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), 2022, pp. 1–8.
  25. C. Draxl, A. Clifton, B.-M. Hodge, and J. McCaa, “The wind integration national dataset (wind) toolkit,” Applied Energy, vol. 151, pp. 355–366, 2015. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0306261915004237
  26. D. Pereira, A. Afonso, and F. Medeiros, “Overview of friedman’s test and post-hoc analysis,” Aug 2015. [Online]. Available: http://hdl.handle.net/10174/16113

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
Lightbulb On Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets