--- title: High-Performance Cloud Computing url: https://www.emergentmind.com/topics/high-performance-cloud type: topic --- # High-Performance Cloud Computing Searching arXiv for recent and foundational papers on high-performance cloud computing. High-performance cloud denotes the use of cloud computing infrastructures as platforms for high-performance computing workloads, including tightly coupled scientific simulation, data-intensive analytics, accelerator-driven inference, and domain-specific large-scale services. Across the literature, it is not treated as a single architecture but as a family of designs that combine virtualization, elastic provisioning, high-bandwidth interconnects, specialized storage systems, and software environments intended to approximate or complement traditional HPC clusters. The central technical question is whether cloud systems can provide performance, scalability, and operational usability comparable to conventional on-premises clusters while adding portability, reproducibility, elasticity, and service-oriented deployment models [0901.0029]. ## 1. Historical emergence and scope Early work framed cloud computing as a possible alternative to in-house clusters and supercomputing resources for scientific users. In that formulation, cloud infrastructure offered on-demand access to “reliable, high performance clusters and storage” without the capital and administrative burden of owning dedicated hardware, while virtualization allowed scientific codes and dependencies to be pre-installed and preserved inside reusable machine images [0901.0029]. This early view emphasized organizational elasticity as much as raw speed: users could start as many instances as needed and scale resources to current demand. A second early strand defined high-performance cloud around wide-area, high-bandwidth networking rather than only local datacenter virtualization. In that view, persistent distributed storage and computation should be designed to exploit 10 Gbps-class inter-site networks, so that large data could remain distributed while computation moved to the data when possible [0808.1802]. Related work on the Open Cloud Testbed extended this argument by treating the network as a first-class experimental resource and by benchmarking cloud middleware across geographically distributed sites connected by a dedicated 10 Gb/s wide-area network [0907.4810]. Subsequent research narrowed the question from generic scientific feasibility to application-specific suitability. Materials simulations and quantum chemistry were used to test whether public IaaS clouds could support not only loosely coupled workloads but also MPI- and ScaLAPACK-based workloads requiring fast inter-node communication [1110.0543]. Later studies broadened the scope further to include exhaustive protein-protein docking on Azure, power-system electromagnetic transients simulation with heterogeneous CPU/GPU execution, serverless RDMA-accelerated remote execution, confidential HPC in public clouds, and cloud-native architectures for cross-provider acceleration and container portability [2006.08905], [1903.01081], [2106.13859], [2212.02378], [2401.04552]. This trajectory suggests that “high-performance cloud” is best understood as a convergence area between classic HPC and cloud computing rather than as a simple migration of batch jobs into virtual machines. A plausible implication is that the term covers multiple submodels: virtual HPC clusters on IaaS, data-intensive WAN-optimized clouds, accelerator-backed domain services, cloud-native disaggregated systems, and serverless or container-based execution substrates. ## 2. Architectural principles and system models The foundational IaaS model uses virtual machines, machine images, and cloud object storage to create reusable scientific environments. Early Amazon EC2 studies used Fedora-based Amazon Machine Images with compilers, numerical libraries, MPI, and domain applications, together with scripts for cluster launch, configuration, monitoring, and teardown [0901.0029], [1110.0543]. In these systems, a local workstation often served as control plane, while instantiated cloud nodes formed the compute plane. A recurring architectural principle is the transformation of cloud instances into a Beowulf-style or master/worker virtual cluster. The SCC platform for materials simulations automated node discovery, generation of `/etc/hosts` and MPI hostfiles, passwordless ssh setup, NFS-shared working directories, and cluster metadata management, so that an EC2 allocation behaved like a local UNIX cluster [1110.0543]. Similar ideas appear in Azure-based docking environments, where multiple VMs plus storage were deployed by Azure CLI and then used as an MPI master-worker system [2006.08905]. Networking is consistently treated as decisive. In wide-area cloud designs such as Sector/Sphere and the Open Cloud Testbed, the network is a distributed backplane rather than a mere transport substrate. These systems assume high-capacity links, use WAN-aware transport such as UDT, and combine storage and compute designs with monitoring, provisioning, and protocol choices intended to reduce the “wide area penalty” [0808.1802], [0907.4810]. In public-cloud HPC studies, the corresponding architectural distinction is between ordinary instances and HPC-oriented instances with dedicated 10 Gbps Ethernet, RDMA, or EFA-like transport [1110.0543], [2011.00656]. Storage architectures also divide the field. Some systems rely on conventional VM-attached or shared filesystems, while others are explicitly storage-disaggregated. CloudPSS packages each simulation task into a Docker-based Virtual Simulation Engine and dispatches it to heterogeneous cloud resources [1903.01081]. OceanBase Bacchus instead uses a cloud-native shared-storage architecture with object storage, a Shared Block Cache Service, and a shared PALF log service so that compute, cache, and storage can scale independently [2602.23571]. XaaS generalizes these patterns into a container-centered model where provider-maintained accelerated APIs expose tuned communication, I/O, and compute libraries inside performance-portable containers [2401.04552]. This suggests that high-performance cloud architecture is defined less by any one abstraction—VM, container, or function—and more by whether the architecture preserves low-overhead access to communication, storage, and accelerators while retaining cloud properties such as elasticity, isolation, and service-managed deployment. ## 3. Performance characteristics: compute, memory, communication, and storage Early EC2 experiments with FEFF showed that cloud CPU performance could be competitive with local workstation-class systems and that compiler/toolchain choice could outweigh nominal processor class. In particular, a PGI-built optimized FEFF84 executable on a 2.6 GHz AMD Opteron “small” instance outperformed a 2.33 GHz Intel Xeon “medium” instance, leading to the recommendation that optimized compilers and numerical libraries should be included in scientific AMIs [0901.0029]. The strongest early result on communication-sensitive workloads came from the SCC study. FEFF9 and WIEN2k k-point parallelism scaled similarly on local InfiniBand clusters, regular EC2 instances, and EC2 Cluster Instances because these workloads were loosely coupled. By contrast, a tightly coupled WIEN2k Hamiltonian diagonalization stalled on regular EC2 once it extended beyond one 8-core instance, so severely that “the speedup drops to zero,” whereas EC2 HPC instances with dedicated 10 Gbps Ethernet matched the speedup of a local 20 Gbps InfiniBand cluster [1110.0543]. This established a central distinction that remains common in later work: virtualization itself is often not the dominant bottleneck; inter-node network capability is. A more recent multi-level comparison of AWS and NERSC Cori found that high-end cloud instances can now deliver HPC-competitive performance not only for compute-intensive applications but also for memory- and communication-intensive applications at modest scales. AWS R5 was the fastest platform for both an N-Body proxy and a communication-heavy FFT proxy, while AWS C5 performed poorly on `MPI_Alltoall`, showing that cloud competitiveness depends strongly on the specific instance/network pairing rather than on “cloud” as an undifferentiated category [2011.00656]. Data-intensive cloud platforms highlight a different performance regime. Sector achieved long-distance transfer performance at roughly 60% to 98% of local-area performance as measured by the LLPR metric, and Sphere outperformed Hadoop by roughly 2–3× on the reported TeraSort benchmark [0808.1802]. On the Open Cloud Testbed, Sector incurred only a 4.7% wide-area penalty in a distributed benchmark where Hadoop suffered 31–34%, reinforcing the importance of WAN-aware transport and load balancing [0907.4810]. Storage-centric cloud performance is illustrated by mdx II, an OpenStack-based data-science cloud. A 16-vCPU mdx II VM reached 1344 GFLOPS in LINPACK versus 656 GFLOPS on an AWS `c7i.4xlarge`, 164 GB/s versus 97 GB/s in BabelStream, and substantially higher block and Lustre throughput; by contrast, a 224-vCPU VM occupying a whole host showed minimal compute virtualization overhead but a 22% reduction in memory throughput on memory-intensive workloads [2502.10820]. This suggests that cloud systems can expose near-bare-metal compute performance while still paying nontrivial penalties in memory- and NUMA-sensitive cases. ## 4. Workload suitability and application classes The literature consistently rejects the idea that all HPC workloads are equally cloud-ready. The most successful classes are loosely coupled, task-parallel, or throughput-oriented workloads. FEFF energy-point parallelism, WIEN2k k-point decomposition, and exhaustive protein-protein docking with independent pairwise tasks all map well to public cloud because they require little interprocess communication and can scale across many instances without needing high-end interconnects [1110.0543], [2006.08905]. Exhaustive docking on Azure is a particularly clear example. MEGADOCK reached a strong-scaling value of 0.931 on 100 `H16` instances and 0.89 on 20 `NC24` GPU instances, while RDMA-enabled variants brought little benefit because the application was dominated by independent FFT-based docking tasks [2006.08905]. This supports a broader conclusion already visible in earlier FEFF studies: task decomposition and modest communication intensity are major predictors of cloud suitability. Tightly coupled distributed linear algebra and communication-heavy iterative methods represent the opposite end of the spectrum. The SCC study showed that ScaLAPACK-based Hamiltonian diagonalization becomes viable in cloud only on specialized HPC instances with dedicated high-bandwidth networking [1110.0543]. The 2025 usability study reached a similar conclusion using a much broader cross-platform set of proxy applications: weak-scaling AMG2023 and some moderate-scale LAMMPS runs were credible cloud candidates, but Laghos, with strong-scaling sensitivity to latency and interconnect, was described as infeasible on the tested clouds [2506.02709]. Domain-specific cloud services can expand the feasible set by restructuring the application around the cloud rather than simply porting legacy executables. CloudPSS uses automatic code generation from a computation graph, heterogeneous CPU/GPU execution, and different parallelization strategies for large single cases versus multi-scenario studies, reaching up to 16× speedup for a large-scale EMT case and up to 80× speedup for 1000 scenarios on a P100 GPU [1903.01081]. ThunderServe does something analogous for LLM inference: by separating prefill and decode, mapping them to different GPU classes, and explicitly optimizing cross-replica KV communication, it achieves up to a 2.1× increase in throughput and up to a 2.5× reduction in latency deadlines at the same price budget [2502.09334]. A plausible synthesis is that high-performance cloud works best when the workload is either structurally communication-light or has been refactored so that its communication, storage, and accelerator use are made explicit in the cloud deployment plan. ## 5. Usability, reproducibility, elasticity, and economics Cloud adoption in HPC is driven not only by speed but by operational properties. Early work stressed reproducibility and ease of deployment through machine images containing optimized software stacks [0901.0029]. Later studies elevated portability, reproducibility, dynamism, and automation to first-class requirements of scientific workflows, arguing that “portability is a new dimension of performance” [2506.02709]. Usability studies show that these benefits are real but costly to realize. The 2025 cross-platform study required 220 unique container builds, of which 74 were actually used, and documented substantial differences among AWS, Azure, and Google Cloud in quota acquisition, placement behavior, driver management, and orchestration stability [2506.02709]. Azure environments, for example, were hampered by especially large hookup times and custom InfiniBand-driver work, whereas Google GKE appeared comparatively smoother in deployment. This suggests that cloud usability for HPC depends as much on environment engineering and platform maturity as on hardware. Economics are similarly workload-dependent. The SCC paper already noted that a realistic production environment would be needed to assess the “overall economic feasibility” of scientific cloud computing [0901.0029]. More recent papers provide concrete examples. For Azure docking, the largest measured GPU run on 40 `NC24` instances completed in 448 s at a total fee of 21.5 USD, compared with 39.9 USD for the best CPU-only alternative on 100 `H16` instances, so the GPU cloud run was both faster and cheaper [2006.08905]. CloudPSS likewise reports lower total job cost on a P100 than on an Intel E5-2682 despite the higher weekly rental price of the GPU resource [1903.01081]. At the architectural level, object storage is repeatedly identified as a major cost lever. OceanBase Bacchus reports storage-cost reductions of 59% in OLTP scenarios and 89% in OLAP scenarios while preserving high availability and competitive performance [2602.23571]. This suggests that, in storage-disaggregated cloud systems, cost-performance is shaped not only by accelerator or CPU selection but also by whether hot, warm, and cold data can be placed across cache, local persistent storage, and object stores in a way that preserves transactional or analytical performance. ## 6. Open problems, security, and future directions Several controversies and unresolved issues recur across the literature. One is the extent to which cloud can substitute for traditional HPC at scale. Modern results support modest-scale competitiveness for compute-, memory-, and even communication-intensive applications when specialized cloud interconnects are available [2011.00656], but larger-scale tightly coupled workloads remain much more sensitive to collective communication, placement, and network contention [2506.02709]. This suggests that the historical blanket statement that “cloud is not suitable for HPC” is too strong, but the opposite blanket statement is also unsupported. Another unresolved issue is confidentiality. Public-cloud HPC requires users to export data and computation to partially untrusted infrastructure, and confidential computing based on TEEs is proposed as the most practical near-term approach. Yet the hardest problems are not only baseline TEE overhead but side channels, legacy HPC software support, distributed execution, workflow reproducibility, and multi-node scalability [2212.02378]. A plausible implication is that confidential high-performance cloud will require redesign at the level of runtime systems, workflows, and metadata services rather than enclave wrapping alone. Serverless and cloud-native execution models provide a different future path. rFaaS shows that serverless can approach HPC-style invocation costs by replacing centralized scheduling and HTTP/TCP transport with leases and RDMA, achieving 326 ns average hot-invocation overhead over pure RDMA and 4.67 μs warm overhead [2106.13859]. XaaS extends this line into a broader architecture based on performance-portable containers, provider-maintained accelerated APIs, and high-performance communication and I/O, though it remains a vision paper rather than a quantitative validation [2401.04552]. Finally, the field is moving toward converged infrastructures that mix cloud elasticity with HPC communication and storage semantics. Recent usability work frames this explicitly as “converged computing” and evaluates clouds not only against on-prem clusters but as environments that may be operationally preferable for some scientific workflows because of portability and automation [2506.02709]. At the same time, storage-disaggregated systems such as Bacchus and performance-evaluated research clouds such as mdx II show that high-performance cloud increasingly includes specialized shared services, object-storage-aware data structures, and academic or national infrastructures that are neither classic public clouds nor classic supercomputers [2602.23571], [2502.10820]. These developments suggest that the future of high-performance cloud lies less in making clouds imitate yesterday’s batch clusters and more in designing systems that combine cloud elasticity, containerized software portability, accelerator heterogeneity, advanced caching and logging services, and application-aware scheduling into new execution substrates for scientific and data-intensive computing. Source: https://www.emergentmind.com/topics/high-performance-cloud