Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
173 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Matrix Completion via Nonsmooth Regularization of Fully Connected Neural Networks (2403.10232v1)

Published 15 Mar 2024 in cs.IT, cs.LG, and math.IT

Abstract: Conventional matrix completion methods approximate the missing values by assuming the matrix to be low-rank, which leads to a linear approximation of missing values. It has been shown that enhanced performance could be attained by using nonlinear estimators such as deep neural networks. Deep fully connected neural networks (FCNNs), one of the most suitable architectures for matrix completion, suffer from over-fitting due to their high capacity, which leads to low generalizability. In this paper, we control over-fitting by regularizing the FCNN model in terms of the $\ell_{1}$ norm of intermediate representations and nuclear norm of weight matrices. As such, the resulting regularized objective function becomes nonsmooth and nonconvex, i.e., existing gradient-based methods cannot be applied to our model. We propose a variant of the proximal gradient method and investigate its convergence to a critical point. In the initial epochs of FCNN training, the regularization terms are ignored, and through epochs, the effect of that increases. The gradual addition of nonsmooth regularization terms is the main reason for the better performance of the deep neural network with nonsmooth regularization terms (DNN-NSR) algorithm. Our simulations indicate the superiority of the proposed algorithm in comparison with existing linear and nonlinear algorithms.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. E. J. Candès and B. Recht, “Exact matrix completion via convex optimization,” Foundations of Computational mathematics, vol. 9, no. 6, pp. 717–772, 2009.
  2. E. J. Candès and T. Tao, “The power of convex relaxation: Near-optimal matrix completion,” IEEE Transactions on Information Theory, vol. 56, no. 5, pp. 2053–2080, 2010.
  3. G. Liu and P. Li, “Low-rank matrix completion in the presence of high coherence,” IEEE Transactions on Signal Processing, vol. 64, no. 21, pp. 5623–5633, 2016.
  4. X. Li, H. Zhang, and R. Zhang, “Matrix completion via non-convex relaxation and adaptive correlation learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  5. J. Fan and T. W. Chow, “Non-linear matrix completion,” Pattern Recognition, vol. 77, pp. 378–394, 2018.
  6. J. Fan and T. Chow, “Deep learning based matrix completion,” Neurocomputing, vol. 266, pp. 540–549, 2017.
  7. C. Guillemot and O. Le Meur, “Image inpainting: Overview and recent advances,” IEEE signal processing magazine, vol. 31, no. 1, pp. 127–144, 2013.
  8. M. Le Pendu, X. Jiang, and C. Guillemot, “Light field inpainting propagation via low rank matrix completion,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 1981–1993, 2018.
  9. F. Xiao, W. Liu, Z. Li, L. Chen, and R. Wang, “Noise-tolerant wireless sensor networks localization via multinorms regularized matrix completion,” IEEE Transactions on Vehicular Technology, vol. 67, no. 3, pp. 2409–2419, 2017.
  10. T. L. Nguyen and Y. Shin, “Matrix completion optimization for localization in wireless sensor networks for intelligent iot,” Sensors, vol. 16, no. 5, p. 722, 2016.
  11. H. Steck, “Training and testing of recommender systems on data missing not at random,” in Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, 2010, pp. 713–722.
  12. Y. Koren, “Factorization meets the neighborhood: a multifaceted collaborative filtering model,” in Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, 2008, pp. 426–434.
  13. B. Recht, M. Fazel, and P. A. Parrilo, “Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization,” SIAM review, vol. 52, no. 3, pp. 471–501, 2010.
  14. J.-F. Cai, E. J. Candès, and Z. Shen, “A singular value thresholding algorithm for matrix completion,” SIAM Journal on optimization, vol. 20, no. 4, pp. 1956–1982, 2010.
  15. Z. Lin, M. Chen, and Y. Ma, “The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices,” arXiv preprint arXiv:1009.5055, 2010.
  16. Y. Hu, D. Zhang, J. Ye, X. Li, and X. He, “Fast and accurate matrix completion via truncated nuclear norm regularization,” IEEE transactions on pattern analysis and machine intelligence, vol. 35, no. 9, pp. 2117–2130, 2012.
  17. Q. Liu, Z. Lai, Z. Zhou, F. Kuang, and Z. Jin, “A truncated nuclear norm regularization method based on weighted residual error for matrix completion,” IEEE Transactions on Image Processing, vol. 25, no. 1, pp. 316–330, 2015.
  18. F. Nie, H. Huang, and C. Ding, “Low-rank matrix recovery via efficient schatten p-norm minimization,” in Twenty-sixth AAAI conference on artificial intelligence, 2012.
  19. K.-C. Toh and S. Yun, “An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems,” Pacific Journal of optimization, vol. 6, no. 615-640, p. 15, 2010.
  20. K. Mohan and M. Fazel, “Iterative reweighted algorithms for matrix rank minimization,” The Journal of Machine Learning Research, vol. 13, no. 1, pp. 3441–3473, 2012.
  21. N. Srebro, J. Rennie, and T. Jaakkola, “Maximum-margin matrix factorization,” Advances in neural information processing systems, vol. 17, 2004.
  22. J. D. Rennie and N. Srebro, “Fast maximum margin matrix factorization for collaborative prediction,” in Proceedings of the 22nd international conference on Machine learning, 2005, pp. 713–719.
  23. Z. Wen, W. Yin, and Y. Zhang, “Solving a low-rank factorization model for matrix completion by a non-linear successive over-relaxation algorithm,” Mathematical Programming Computation, vol. 4, no. 4, pp. 333–361, 2012.
  24. E. J. Candès, X. Li, Y. Ma, and J. Wright, “Robust principal component analysis?” Journal of the ACM (JACM), vol. 58, no. 3, pp. 1–37, 2011.
  25. X. Hu, Y. Han, and Z. Geng, “A novel matrix completion model based on the multi-layer perceptron integrating kernel regularization,” IEEE Access, vol. 9, pp. 67 042–67 050, 2021.
  26. T. Zhou, H. Shan, A. Banerjee, and G. Sapiro, “Kernelized probabilistic matrix factorization: Exploiting graphs and side information,” in Proceedings of the 2012 SIAM international Conference on Data mining.   SIAM, 2012, pp. 403–414.
  27. R. Salakhutdinov, A. Mnih, and G. Hinton, “Restricted boltzmann machines for collaborative filtering,” in Proceedings of the 24th international conference on Machine learning, 2007, pp. 791–798.
  28. S. Sedhain, A. K. Menon, S. Sanner, and L. Xie, “Autorec: Autoencoders meet collaborative filtering,” in Proceedings of the 24th international conference on World Wide Web, 2015, pp. 111–112.
  29. C.-Y. Wu, A. Ahmed, A. Beutel, A. J. Smola, and H. Jing, “Recurrent recommender networks,” in Proceedings of the tenth ACM international conference on web search and data mining, 2017, pp. 495–503.
  30. D. M. Nguyen, E. Tsiligianni, R. Calderbank, and N. Deligiannis, “Regularizing autoencoder-based matrix completion models via manifold learning,” in 2018 26th European Signal Processing Conference (EUSIPCO).   IEEE, 2018, pp. 1880–1884.
  31. X. P. Li, M. Wang, and H. C. So, “An interpretable bi-branch neural network for matrix completion,” Signal Processing, p. 108640, 2022.
  32. S. Mehrdad and M. H. Kahaei, “Deep learning approach for matrix completion using manifold learning,” Signal Processing, vol. 188, p. 108231, 2021.
  33. S. Amini and S. Ghaemmaghami, “A new framework to train autoencoders through non-smooth regularization,” IEEE Transactions on Signal Processing, vol. 67, no. 7, pp. 1860–1874, 2019.
  34. S. Amini, M. Soltanian, M. Sadeghi, and S. Ghaemmaghami, “Non-smooth regularization: Improvement to learning framework through extrapolation,” IEEE Transactions on Signal Processing, vol. 70, pp. 1213–1223, 2022.
  35. Y. Xu and W. Yin, “A globally convergent algorithm for nonconvex optimization based on block coordinate update,” Journal of Scientific Computing, vol. 72, no. 2, pp. 700–734, 2017.
  36. N. Parikh, S. Boyd et al., “Proximal algorithms,” Foundations and trends® in Optimization, vol. 1, no. 3, pp. 127–239, 2014.
  37. S. Amini and S. Ghaernmaghami, “Sparse autoencoders using non-smooth regularization,” in 2018 26th European Signal Processing Conference (EUSIPCO).   IEEE, 2018, pp. 2000–2004.
  38. H. Attouch, J. Bolte, and B. F. Svaiter, “Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized gauss–seidel methods,” Mathematical Programming, vol. 137, no. 1-2, pp. 91–129, 2013.
  39. W. H. Young, “On classes of summable functions and their fourier series,” Proceedings of the Royal Society of London. Series A, Containing Papers of a Mathematical and Physical Character, vol. 87, no. 594, pp. 225–229, 1912.
  40. H. Attouch, J. Bolte, P. Redont, and A. Soubeyran, “Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the kurdyka-łojasiewicz inequality,” Mathematics of operations research, vol. 35, no. 2, pp. 438–457, 2010.
  41. J. Ba and R. Caruana, “Do deep nets really need to be deep?” Advances in neural information processing systems, vol. 27, 2014.
  42. A. Golubeva, B. Neyshabur, and G. Gur-Ari, “Are wider nets better given the same number of parameters?” arXiv preprint arXiv:2010.14495, 2020.
  43. H. Wang, S. Agarwal, and D. Papailiopoulos, “Pufferfish: communication-efficient models at no extra cost,” Proceedings of Machine Learning and Systems, vol. 3, pp. 365–386, 2021.
  44. Y. E. Nesterov, “A method of solving a convex programming problem with convergence rate o(1/k2)1superscript𝑘2(1/k^{2})( 1 / italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ),” in Doklady Akademii Nauk, vol. 269, no. 3.   Russian Academy of Sciences, 1983, pp. 543–547.
  45. M. C. Mukkamala, F. Westerkamp, E. Laude, D. Cremers, and P. Ochs, “Bregman proximal framework for deep linear neural networks,” arXiv preprint arXiv:1910.03638, 2019.
  46. H. H. Bauschke, J. Bolte, and M. Teboulle, “A descent lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications,” Mathematics of Operations Research, vol. 42, no. 2, pp. 330–348, 2016.
  47. H. Lu, R. M. Freund, and Y. Nesterov, “Relatively smooth convex optimization by first-order methods, and applications,” SIAM Journal on Optimization, vol. 28, no. 1, pp. 333–354, 2018.
  48. M. Ahookhosh, L. T. K. Hien, N. Gillis, and P. Patrinos, “Multi-block Bregman proximal alternating linearized minimization and its application to orthogonal nonnegative matrix factorization,” Computational Optimization and Applications, vol. 79, no. 3, pp. 681–715, 2021.
  49. M. Ahookhosh, L. T. K. Hien, N. Gillis, and P. Patrinos, “A block inertial Bregman proximal algorithm for nonsmooth nonconvex problems with application to symmetric nonnegative matrix tri-factorization,” Journal of Optimization Theory and Applications, vol. 190, no. 1, pp. 234–258, 2021.
  50. L. T. Khanh Hien, D. N. Phan, N. Gillis, M. Ahookhosh, and P. Patrinos, “Block Bregman majorization minimization with extrapolation,” SIAM Journal on Mathematics of Data Science, vol. 4, no. 1, pp. 1–25, 2022.
  51. J. Fan, M. Zhao, and T. W. Chow, “Matrix completion via sparse factorization solved by accelerated proximal alternating linearized minimization,” IEEE Transactions on Big Data, vol. 6, no. 1, pp. 119–130, 2018.
  52. H. Ye, H. Li, F. Cao, and L. Zhang, “A hybrid truncated norm regularization method for matrix completion,” IEEE Transactions on Image Processing, vol. 28, no. 10, pp. 5171–5186, 2019.
  53. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
  54. J. Chong, “Math 742: Geometric analysis.”

Summary

We haven't generated a summary for this paper yet.

X Twitter Logo Streamline Icon: https://streamlinehq.com