Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Distributed Matrix Multiplication with a Smaller Recovery Threshold through Modulo-based Approaches (2309.10249v1)

Published 19 Sep 2023 in cs.IT and math.IT

Abstract: This paper considers the problem of calculating the matrix multiplication of two massive matrices $\mathbf{A}$ and $\mathbf{B}$ distributedly. We provide a modulo technique that can be applied to coded distributed matrix multiplication problems to reduce the recovery threshold. This technique exploits the special structure of interpolation points and can be applied to many existing coded matrix designs. Recently studied discrete Fourier transform based code achieves a smaller recovery threshold than the optimal MatDot code with the expense that it cannot resist stragglers. We also propose a distributed matrix multiplication scheme based on the idea of locally repairable code to reduce the recovery threshold of MatDot code and provide resilience to stragglers. We also apply our constructions to a type of matrix computing problems, where generalized linear models act as a special case.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (30)
  1. A. Buluc and J. R. Gilbert, “Challenges and advances in parallel sparse matrix-matrix multiplication,” in Proc. 2008 37th Int. Conf. Parallel Process.   IEEE, 2008, pp. 503–510.
  2. J. Dean and L. A. Barroso, “The tail at scale,” Commun. ACM, vol. 56, no. 2, pp. 74–80, 2013.
  3. S. Li, S. Avestimehr, et al., “Coded computing: Mitigating fundamental bottlenecks in large-scale distributed computing and machine learning,” Foundations and Trends® in Communications and Information Theory, vol. 17, no. 1, pp. 1–148, 2020.
  4. K. Lee, M. Lam, R. Pedarsani, D. Papailiopoulos, and K. Ramchandran, “Speeding up distributed machine learning using codes,” IEEE Trans. Inf. Theory, vol. 64, no. 3, pp. 1514–1529, 2017.
  5. Q. Yu, M. Maddah-Ali, and S. Avestimehr, “Polynomial codes: an optimal design for high-dimensional coded matrix multiplication,” in Adv. Neural Inf. Process. Syst., vol. 30, 2017.
  6. S. Dutta, M. Fahim, F. Haddadpour, H. Jeong, V. Cadambe, and P. Grover, “On the optimal recovery threshold of coded matrix multiplication,” IEEE Trans. Inf. Theory, vol. 66, no. 1, pp. 278–301, 2019.
  7. Q. Yu, M. A. Maddah-Ali, and S. A. Avestimehr, “Straggler mitigation in distributed matrix multiplication: Fundamental limits and optimal coding,” IEEE Trans. Inf. Theory, vol. 66, no. 3, pp. 1920–1933, 2020.
  8. S. Dutta, Z. Bai, H. Jeong, T. M. Low, and P. Grover, “A unified coded deep neural network training strategy based on generalized polydot codes,” in Proc. 2018 IEEE Int. Symp. Inf. Theory (ISIT).   IEEE, 2018, pp. 1585–1589.
  9. W.-T. Chang and R. Tandon, “On the capacity of secure distributed matrix multiplication,” in 2018 IEEE Global Communications Conference (GLOBECOM).   IEEE, 2018, pp. 1–6.
  10. R. G. L. D’Oliveira, S. E. Rouayheb, and D. Karpuk, “Gasp codes for secure distributed matrix multiplication,” IEEE Trans. Inf. Theory, vol. 66, no. 7, pp. 4038–4050, 2020.
  11. N. Mital, C. Ling, and D. Gündüz, “Secure distributed matrix computation with discrete fourier transform,” IEEE Trans. Inf. Theory, vol. 68, no. 7, pp. 4666–4680, 2022.
  12. J. Zhu, Q. Yan, and X. Tang, “Improved constructions for secure multi-party batch matrix multiplication,” IEEE Trans. Commun., vol. 69, no. 11, pp. 7673–7690, 2021.
  13. H. A. Nodehi, S. R. H. Najarkolaei, and M. A. Maddah-Ali, “Entangled polynomial coding in limited-sharing multi-party computation,” in Proc. 2018 IEEE Inf. Theory Workshop (ITW).   IEEE, 2018, pp. 1–5.
  14. Q. Yu and A. S. Avestimehr, “Entangled polynomial codes for secure, private, and batch distributed matrix multiplication: Breaking the ”cubic” barrier,” in Proc. 2020 IEEE Int. Symp. Inf. Theory (ISIT).   IEEE, 2020, pp. 245–250.
  15. M. Aliasgari, O. Simeone, and J. Kliewer, “Private and secure distributed matrix multiplication with flexible communication load,” IEEE Trans. Inf. Forensics Security, vol. 15, pp. 2722–2734, 2020.
  16. J. Li and C. Hollanti, “Private and secure distributed matrix multiplication schemes for replicated or mds-coded servers,” IEEE Trans. Inf. Forensics Security, vol. 17, pp. 659–669, 2022.
  17. N. Prakash, G. M. Kamath, V. Lalitha, and P. V. Kumar, “Optimal linear codes with a local-error-correction property,” in 2012 IEEE Int. Symp. Inf. Theory Proc.   IEEE, 2012, pp. 2776–2780.
  18. C. Huang, H. Simitci, Y. Xu, A. Ogus, B. Calder, P. Gopalan, J. Li, and S. Yekhanin, “Erasure coding in windows azure storage,” in Presented as part of the 2012 USENIX Annual Technical Conference (USENIX ATC ’12), 2012, pp. 15–26.
  19. I. Tamo and A. Barg, “A family of optimal locally recoverable codes,” IEEE Trans. Inf. Theory, vol. 60, no. 8, pp. 4661–4676, 2014.
  20. H. Jeong, F. Ye, and P. Grover, “Locally recoverable coded matrix multiplication,” in 2018 56th Annual Allerton Conf. Commun., Control, Comput. (Allerton).   IEEE, 2018, pp. 715–722.
  21. M. Fahim and V. R. Cadambe, “Numerically stable polynomially coded computing,” IEEE Trans. Inf. Theory, vol. 67, no. 5, pp. 2758–2785, 2021.
  22. M. Rudow, K. V. Rashmi, and V. Guruswami, “A locality-based approach for coded computation,” arXiv preprint arXiv:2002.02440, 2020.
  23. F. P. Preparata and D. V. Sarwate, “Computational complexity of fourier transforms over finite fields,” Math. Comput., vol. 31, no. 139, pp. 740–751, 1977.
  24. D. Data, L. Song, and S. N. Diggavi, “Data encoding for byzantine-resilient distributed optimization,” IEEE Trans. Inf. Theory, vol. 67, no. 2, pp. 1117–1140, 2020.
  25. A. H. Bentbib and A. Kanber, “Block power method for svd decomposition,” Analele Ştiinţifice ale Universităţii ”Ovidius” Constanţa. Seria Matematică, vol. 23, no. 2, pp. 45–58, 2015.
  26. S. Wang, Q. Yan, J. Zhang, J. Wang, and L. Song, “Coded alternating least squares for straggler mitigation in distributed recommendations,” in Proc. 2021 IEEE Int. Symp. Inf. Theory (ISIT).   IEEE, 2021, pp. 1058–1063.
  27. M. Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transport,” in Adv. Neural Inf. Process. Syst., vol. 26, 2013.
  28. R. Freivalds, “Fast probabilistic algorithms,” in Proc. Int. Symp. Math. Found. Comput. Sci.   Springer, 1979, pp. 57–69.
  29. Q. Yu, S. Li, N. Raviv, S. M. M. Kalan, M. Soltanolkotabi, and S. A. Avestimehr, “Lagrange coded computing: Optimal design for resiliency, security, and privacy,” in Proc. 22nd Int. Conf. Artif. Intell. Stat. (AISTATS).   PMLR, 2019, pp. 1215–1225.
  30. K. S. Kedlaya and C. Umans, “Fast polynomial factorization and modular composition,” SIAM J. Comput., vol. 40, no. 6, pp. 1767–1802, 2011.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now