Myths and Legends in High-Performance Computing (2301.02432v3)

Abstract: In this thought-provoking article, we discuss certain myths and legends that are folklore among members of the high-performance computing community. We gathered these myths from conversations at conferences and meetings, product advertisements, papers, and other communications such as tweets, blogs, and news articles within and beyond our community. We believe they represent the zeitgeist of the current era of massive change, driven by the end of many scaling laws such as Dennard scaling and Moore's law. While some laws end, new directions are emerging, such as algorithmic scaling or novel architecture research. Nevertheless, these myths are rarely based on scientific facts, but rather on some evidence or argumentation. In fact, we believe that this is the very reason for the existence of many myths and why they cannot be answered clearly. While it feels like there should be clear answers for each, some may remain endless philosophical debates, such as whether Beethoven was better than Mozart. We would like to see our collection of myths as a discussion of possible new directions for research and industry investment.

• The paper debunks prevalent HPC myths by interrogating assumptions around quantum supremacy, deep learning efficiency, and hardware heterogeneity.
• It employs a myth-based analysis to contrast empirical performance data with traditional HPC paradigms, highlighting economic and technical trade-offs.
• The study advises balancing innovation with realistic expectations, urging strategic investment to enhance current HPC systems rather than chasing speculative technologies.

Analyzing "Myths and Legends in High-Performance Computing"

The paper "Myths and Legends in High-Performance Computing" by Matsuoka et al. examines prevalent notions within the high-performance computing (HPC) community that are not strictly grounded in scientific fact but influence ongoing discourse and research direction. Through a series of debated myths, the authors interrogate assumptions and expectations regarding the future trajectory of computing technologies, spanning quantum computing, AI integration, architectural specialization, and the shifting landscape of HPC within cloud environments.

Summary of Key Myths

The authors scrutinize twelve myths, some of which include:

1. Quantum Computing Supremacy: Although quantum computing promises exponential speedups, practical limitations in data input/output and algorithmic speedup have impeded its immediate adoption as a replacement for classical HPC. Quantum computing is posited to augment rather than supplant existing architectures, with its broader applicability and integration remaining a topic for extensive research.
2. Omnipresence of Deep Learning: Deep learning (DL) models promise transformative applications but are constrained by speed-accuracy trade-offs. There remains skepticism about deep learning's capability to replace traditional simulation methods fundamentally, particularly in scenarios necessitating precision and reliability.
3. Excessive Hardware Specialization: This myth argues for increased hardware heterogeneity, akin to smartphone SoCs, in supercomputers. However, the authors point out the economic and programming burdens this approach entails, advocating for a balance that maximizes weak scaling, as exhibited by effectively integrating GPUs.
4. Imminent Zettascale Computing: Expectations for achieving zettascale performance—or executing a zettaflop/s—are tempered by the challenges in improving power efficiency. Realistic timelines suggest a decadal horizon for zettaops systems, contingent upon advancements in energy efficiency and component integration.
5. Dominance of Low Precision Arithmetic: While AI and machine learning successfully employ low-precision arithmetic, the broader HPC domain remains more tentative about its adoption due to potential convergence and accuracy concerns. Mixed precision strategies offer some mitigation, demonstrating incremental rather than revolutionary potential.

The authors present each of these myths with closing questions, prompting the community to debate the future directions and investments they merit. The myths are contested with varying degrees of skepticism or optimism based on the empirical realities of current technological capabilities and historical trends in HPC system performance.

Implications and Future Directions

The discussion of these myths sheds light on the multifaceted challenges and decisions facing the HPC community. Practical considerations such as energy efficiency, cost feasibility, and the maturity of software ecosystems continue to influence the feasibility of emerging technologies. Furthermore, the divergence of computing architectures to meet specialized needs may not always align with the scale and demands of HPC tasks that benefit from homogeneity and standardized APIs.

The paper underscores the necessity for careful guidance in selecting investment areas, suggesting that an overemphasis on speculative technologies may divert resources from strengthening current systems' capacities. Notably, algorithmic innovation is highlighted as a critical component of advancing computational capabilities, albeit within limits defined by computational complexity and the continued scaling of silicon technologies.

In closing, the paper encourages readers to critically evaluate the myths within the context of evolving paradigms, technological developments, and the broader needs of scientific computation. This reflective stance will inform not only the design of future HPC systems and infrastructures but also the shaping of research agendas that will navigate the complexities and realities of next-generation computing environments.