Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Fast Updating Truncated SVD for Representation Learning with Sparse Matrices (2401.09703v1)

Published 18 Jan 2024 in math.NA and cs.NA

Abstract: Updating a truncated Singular Value Decomposition (SVD) is crucial in representation learning, especially when dealing with large-scale data matrices that continuously evolve in practical scenarios. Aligning SVD-based models with fast-paced updates becomes increasingly important. Existing methods for updating truncated SVDs employ Rayleigh-Ritz projection procedures, where projection matrices are augmented based on original singular vectors. However, these methods suffer from inefficiency due to the densification of the update matrix and the application of the projection to all singular vectors. To address these limitations, we introduce a novel method for dynamically approximating the truncated SVD of a sparse and temporally evolving matrix. Our approach leverages sparsity in the orthogonalization process of augmented matrices and utilizes an extended decomposition to independently store projections in the column space of singular vectors. Numerical experiments demonstrate a remarkable efficiency improvement of an order of magnitude compared to previous methods. Remarkably, this improvement is achieved while maintaining a comparable precision to existing approaches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (28)
  1. Implicit svd for graph representation learning. Advances in Neural Information Processing Systems, 34:8419–8431, 2021.
  2. Matthew Brand. Fast low-rank modifications of the thin singular value decomposition. Linear algebra and its applications, 415(1):20–30, 2006.
  3. Lightgcl: Simple yet effective graph contrastive learning for recommendation. In The Eleventh International Conference on Learning Representations, 2022.
  4. Performance of recommender algorithms on top-n recommendation tasks. In Proceedings of the fourth ACM conference on Recommender systems, pp.  39–46, 2010.
  5. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391–407, 1990.
  6. Accelerating dynamic network embedding with billions of parameter updates to milliseconds. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp.  414–425, 2023.
  7. Time-sensitive recommendation from recurrent user activities. Advances in neural information processing systems, 28, 2015.
  8. Calculating the singular values and pseudo-inverse of a matrix. Journal of the Society for Industrial and Applied Mathematics, Series B: Numerical Analysis, 2(2):205–224, 1965.
  9. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217–288, 2011.
  10. The movielens datasets: History and context. ACM Transactions on Interactive Intelligent Systems (TiiS), 5(4):19:1–19:19, 2015. doi: 10.1145/2827872.
  11. Projection techniques to update the truncated svd of evolving matrices with applications. In International Conference on Machine Learning, pp.  5236–5246. PMLR, 2021.
  12. Matrix factorization techniques for recommender systems. Computer, 42(8):30–37, 2009.
  13. Community structure in large networks: Natural cluster sizes and the absence of large well-defined clusters. Internet Mathematics, 6(1):29–123, 2009.
  14. Neural word embedding as implicit matrix factorization. Advances in neural information processing systems, 27, 2014.
  15. Randomized block krylov methods for stronger and faster approximate singular value decomposition. Advances in neural information processing systems, 28, 2015.
  16. Eigenrec: generalizing puresvd for effective and efficient top-n recommendations. Knowledge and Information Systems, 58:59–81, 2019.
  17. Compressive spectral embedding: sidestepping the svd. Advances in neural information processing systems, 28, 2015.
  18. Trust management for the semantic web. In ISWC, 2003.
  19. Incremental singular value decomposition algorithms for highly scalable recommender systems. In Fifth international conference on computer and information science, volume 1, pp.  27–8. Citeseer, 2002.
  20. Relational learning via latent social dimensions. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp.  817–826, 2009.
  21. Eigenfaces for recognition. Journal of cognitive neuroscience, 3(1):71–86, 1991.
  22. Sampling and multilevel coarsening algorithms for fast matrix approximations. Numerical Linear Algebra with Applications, 26(3):e2234, 2019.
  23. Low rank approximation using error correcting coding matrices. In International Conference on Machine Learning, pp.  702–710. PMLR, 2015.
  24. Fast updating algorithms for latent semantic indexing. SIAM Journal on Matrix Analysis and Applications, 35(3):1105–1131, 2014.
  25. Randomized algorithms to update partial singular value decomposition on a hybrid cpu/gpu cluster. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp.  1–12, 2015.
  26. On updating problems in latent semantic indexing. SIAM Journal on Scientific Computing, 21(2):782–791, 1999.
  27. Timers: Error-bounded svd restart on dynamic networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  28. High-order proximity preserved embedding for dynamic networks. IEEE Transactions on Knowledge and Data Engineering, 30(11):2134–2144, 2018.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now