Serverless HPC: Elastic Cloud Computing
- Serverless HPC is a computing paradigm that applies serverless execution models to HPC, AI, and big data, enabling dynamic scaling and fine-grained billing.
- It externalizes state and decouples task orchestration from traditional HPC schedulers, using architectures like DAG-based functions and hybrid-cloud placements.
- Real-world demonstrations report up to 250x speedup and significant cost reductions, underscoring both its transformative potential and inherent limitations.
Serverless HPC designates a family of approaches that apply serverless execution models—most prominently Function-as-a-Service, but also related serverless container and cloud-native orchestration mechanisms—to compute-intensive applications in HPC, AI, and big data. In this literature, the common objective is to obtain elasticity, fine-grained billing, and reduced operational burden without abandoning HPC-like performance, scalability, or QoS. A systematic literature review identified 122 research articles published between 2018 and early 2025 and proposed a taxonomy comprising eight primary research directions and nine targeted use case domains (Besozzi et al., 14 Jan 2026). Early concrete demonstrations already showed that serverless computing could provide “low cost access to hundreds of CPUs, on demand” for scientific workloads: an all-against-all comparison of 20,336 unique human proteins was completed in approximately 2 minutes, at a cost of less than $1, using AWS Lambda (Kumanov et al., 2018).
1. Historical emergence and conceptual scope
In the strict Function-as-a-Service sense, serverless computing means that the user does not provision or manage servers or VMs, but instead deploys functions to a provider platform that allocates CPU, memory, and container instances dynamically, with billing per invocation and per duration (Kumanov et al., 2018). This definition established a direct contrast with traditional HPC clusters, which rely on fixed nodes, long-lived jobs, and schedulers such as SLURM or PBS, and with cloud IaaS, where the user provisions and manages instances explicitly (Kumanov et al., 2018).
The topic quickly broadened beyond pure FaaS. Clowdr, described as “a serverless tool for platform-agnostic computational experiment management,” did not implement FaaS, but made HPC and cloud execution feel serverless at the experiment-management layer by avoiding an active server, using Boutiques descriptors and invocations, supporting Docker and Singularity, and launching tasks on local machines, HPC clusters, or cloud resources while collecting a provenance bundle and an HTML summary page (Kiar et al., 2018). This suggests that, in the literature, “serverless HPC” often includes both infrastructure-level serverlessness and user-level abstractions that hide scheduler, platform, and environment heterogeneity.
By 2025, the concept had become entangled with cloud–HPC convergence. The AI Factories position paper framed “Serverless HPC” as the “cloud challenges of HPC,” arguing for a dual-stack architecture in which supercomputers combine an HPC stack with a cloud-native stack, including Kubernetes, object storage, FaaS, and SaaS front-ends (Lopez et al., 16 Sep 2025). In that framing, serverless HPC is not merely a cloud outsourcing strategy; it is also a redesign agenda for supercomputing systems that need to support AI inference, agentic applications, and multi-tenant, public-facing services (Lopez et al., 16 Sep 2025).
2. Architectural patterns and execution models
A recurring architectural pattern is stateless compute plus externalized state. In the biomedical AWS Lambda case, each function read two protein subsets from Amazon S3, ran the Striped Smith-Waterman executable, wrote similarity scores back to S3, and signaled completion through Amazon SQS, while a Python client on an EC2 instance or laptop partitioned work into 861 tasks and monitored queue completion (Kumanov et al., 2018). The arrangement behaved like an ephemeral elastic compute cluster: hundreds of short-lived workers, shared storage via object storage, and orchestration from a lightweight client rather than from a resident batch scheduler (Kumanov et al., 2018).
Later work generalized this into structured programming models. On OpenFaaS, a farm skeleton was implemented with an emitter function, a worker pool, a collector function, Redis queues for both data and control paths, and an external orchestrator that managed the degree of parallelism (Li et al., 6 Feb 2026). The state exposed to the controller was a 9-element vector,
which made queue lengths, active workers, processing times, arrival rate, and QoS explicit control variables rather than opaque platform artifacts (Li et al., 6 Feb 2026).
Hybrid-cloud orchestration introduced another pattern: representing serverless applications as DAGs of functions and deciding placement across private and public infrastructures. The Hybrid Cloud Scheduler modeled batch-processing applications as pipelines of functions, represented as a Directed Acyclic Graph, and combined a policy-level hybrid placement decision with local bin-packing on the edge cluster (Peri et al., 2024). In a different domain, GraphFlash adopted a BSP-like coordinator/worker design over serverless infrastructure, with a coordinator function, stateless worker functions, and external “Memory-as-a-Service” for control metadata, partitioned graph data, and messages, thereby recovering superstep synchronization on top of serverless storage primitives (Zhao et al., 12 May 2026).
These systems differ in surface form, but they share a small set of invariants: short-lived compute elements, explicit partitioning of the workload, externalized state and messaging, and an orchestration layer that substitutes for the node-oriented assumptions of traditional HPC.
3. Workload classes and algorithmic mappings
The strongest serverless-HPC results appear in workloads that are embarrassingly parallel or can be decomposed into bounded subproblems. The protein-comparison case study partitioned 20,336 sequences into 41 subsets of roughly 500 sequences each, mapped the distinct subset pairs to 861 Lambda invocations, and thereby converted a global similarity computation into a tiled decomposition that fit the time and memory bounds of AWS Lambda (Kumanov et al., 2018). This is the canonical serverless-HPC pattern: a large scientific kernel recast as many small, independent tiles.
A more elaborate example is the serverless variant-calling pipeline migrated from a single-node HPC workflow to AWS Lambda. FASTA and FASTQ inputs were partitioned, alignment tasks were generated as the Cartesian product of 20 FASTA chunks and 60 FASTQ chunks, producing 1200 tasks, and the reduce stage required 112 reduce tasks to perform variant calling over grouped mpileup ranges (Arjona et al., 2023). The port forced the redesign of blocking HPC-style dependencies into an asynchronous, fully S3-based architecture, which the authors described as necessary to avoid deadlock under Lambda concurrency limits (Arjona et al., 2023).
Other domains exhibit the same mapping logic with different computational kernels. CloudVVUQ extended EasyVVUQ so that VVUQ studies with tens of thousands of independent model runs could be launched on AWS Lambda, Google Cloud Functions, or Google Cloud Run, which was practical for models such as a lower-limb haemodynamics model with runtime about 150 ms per run and a Ten Tusscher cardiac model with runtime about 4 minutes per run (Kica et al., 2023). In high-energy physics, the ROOT-based distributed serverless engine partitioned datasets into non-overlapping ranges of entries, packaged the mapper logic and Kerberos ticket into Lambda payloads, and reduced partial results on the client side, thereby treating event-level physics analysis as a stateless MapReduce workload over TTree clusters (Kuśnierz et al., 2022).
Not all serverless-HPC workloads are embarrassingly parallel. FSD-Inference treated large sparse DNN inference as a distributed sparse linear algebra problem on Lambda, replacing direct inter-process communication with SNS+SQS or S3-based message exchange and using hypergraph partitioning to minimize cross-worker communication (Oakley et al., 2024). GraphFlash did something analogous for iterative graph analytics by moving from vertex-centric messaging to a subgraph-centric model with partition-aware key aggregation, intra-function partition co-location, and superstep-aware activation (Zhao et al., 12 May 2026). These cases show that serverless HPC is not restricted to trivial parallelism, but the engineering burden increases sharply as communication intensity rises.
4. Performance and cost evidence
The literature repeatedly reports substantial reductions in wall-clock time when a workload fits the serverless execution model. In the 2018 biomedical demonstration, the laptop-only execution of all 861 tasks took 8.7 hours, whereas the same workflow completed in approximately 2 minutes on Lambda, with the paper highlighting a 250x speedup and total estimated AWS costs of $0.86823 with an EC2 client or $0.81651 with a laptop client (Kumanov et al., 2018). The authors also reported total Lambda duration of 542.14 CPU-minutes in the EC2-client case, consistent with “low cost access to hundreds of CPUs, on demand” (Kumanov et al., 2018).
The serverless variant-calling pipeline reported a Trypanosome run reduced from 106:08.21 on a Dell PowerEdge R440 to 2:50.64 on AWS Lambda, and a human-genome hg19 experiment reduced from about 14 hours to 7 minutes 16.57 seconds, with Lambda cost $10.68, S3 SELECT cost $0.0644, and total cost about $10.75 (Arjona et al., 2023). For VVUQ, CloudVVUQ reported approximately 983 concurrent simulations for the lower-limb haemodynamics model, reducing about 2.6 hours of aggregate simulation time to about 48 seconds, and up to 1536 active containers in parallel for the Ten Tusscher model on Cloud Run, yielding a 723× improvement in sample processing time compared to sequential processing (Kica et al., 2023).
HEP analysis on AWS Lambda showed near-linear speedup up to about 128 concurrent Lambdas, with CPU utilization consistently above 90% for the CPU-bound benchmark and a maximum delay of about 4 seconds between first and last start and about 10 seconds between first and last end in a 128-Lambda run (Kuśnierz et al., 2022). In graph processing, GraphFlash outperformed existing serverless-compatible systems by up to 127x in execution time, reduced resource consumption by up to 98% under higher-resource configurations, and on AWS Lambda achieved up to 48x speedup and 99.97% cost reduction over prior serverless solutions (Zhao et al., 12 May 2026).
Hybrid scheduling contributes a different kind of evidence. The Hybrid Cloud Scheduler kept CPU utilization above 90% in the edge cluster and achieved cost reduction up to 87% compared to scheduling all workloads in the public cloud, while meeting nearly all deadlines given moderate slack (Peri et al., 2024). CAI, for astronomy foundation-model inference, reported 28 s for a 12.6 GB dataset on AWS Lambda, versus 140.8 s on an HPC GPU and 1,793 s on an HPC CPU, with 25 MB partitions reaching 18.04B bps throughput and full-dataset runs costing on the order of a few tenths of a dollar per configuration (Staylor et al., 8 Jan 2025). These results do not imply uniform superiority over specialized HPC systems, but they do show that serverless platforms can deliver HPC-scale throughput or competitive end-to-end latency for appropriately structured workloads.
5. Hybridization with traditional HPC and data-centric infrastructures
A major strand of the literature does not replace HPC schedulers but layers serverless semantics on top of them. “Chat AI” placed a web service on a cloud VM while keeping the scalable inference backend on HPC systems managed by Slurm, using a scheduler script, a routing table, and SSH ForceCommand as a circuit breaker so that the public-facing server could not become a general shell on the cluster (Doosthosseini et al., 2024). The system was deployed in production, ran side by side with regular Slurm workloads, and was able to utilize gaps in the schedule created by Slurm (Doosthosseini et al., 2024). This suggests that one meaning of serverless HPC is “HPC as a serverless backend,” even when the physical substrate remains a batch-managed cluster.
Software resource disaggregation pushes the same idea further inside the supercomputer. Using rFaaS, the authors showed how fine-grained function execution could exploit unallocated and underutilized nodes, idle cores, memory, and accelerators while retaining near-native performance (Copik et al., 2024). Median idle nodes on Piz Daint were about 252 at any instant, 70–80% of idle nodes were available for less than 10 minutes, node memory utilization was often as low as 24%, and co-location improved utilization up to 52% in the reported scenarios (Copik et al., 2024). Here the serverless layer functions as a software mechanism for dynamic, partial-node harvesting rather than as a public-cloud replacement.
Other systems emphasize data locality across federated infrastructures. In SRCNet, serverless functions were embedded into the SKA Regional Centre Network so that functions such as Gaussian convolution could run directly at sites where data replicas were stored, thereby reducing latency and minimizing transfers (Parra-Royón et al., 12 Jan 2026). The paper described SKAO as producing about 700 PB of data products annually and used GateKeeper, IAM, permissions, Site-Capabilities, DataLink, Kubernetes, and GitOps to make functions portable, policy-aware, and data-proximate across regional centers (Parra-Royón et al., 12 Jan 2026). Hybrid cloud orchestration reaches a similar conclusion from the opposite direction: HCS places DAG nodes across a private edge cluster and public serverless backends to preserve deadlines while minimizing cloud expenditure (Peri et al., 2024).
These hybrid and federated designs indicate that serverless HPC is often most compelling when it is treated as an augmentation layer: serverless for burstability, orchestration, or data-proximate analytics; traditional HPC for long-running, tightly coupled, or accelerator-heavy kernels.
6. Limits, controversies, and research directions
The literature is unusually explicit that serverless HPC is not a universal replacement for traditional HPC. AWS Lambda constraints in the 2018 protein study included 500 MB of ephemeral disk, 5 minutes maximum runtime, 3 GB RAM, up to 2 vCPUs, and a default concurrency limit of 1,000 concurrent requests per account (Kumanov et al., 2018). The VVUQ study likewise restricted the viability of FaaS to cases where each single run is relatively lightweight, while emphasizing that tightly coupled MPI applications, large memory footprints, and strong data locality requirements remain better aligned with conventional HPC (Kica et al., 2023). The hybrid scheduler paper made the same point more bluntly, noting that tightly coupled parallelism, large memory footprints, and high inter-node communication are only partially addressed or missing in its design (Peri et al., 2024).
Control-plane and platform effects are another recurrent limitation. OpenFaaS farm management reported scale-up latency of about 5–8 seconds per worker, which reactive analytical controllers struggled to accommodate, whereas RL-based policies maintained much higher QoS under the same platform constraints (Li et al., 6 Feb 2026). In production Slurm-native LLM serving, cold starts for 70B models could take up to about 10 minutes, and the system could not gracefully scale a service to zero and then re-create it on demand without substantial redesign (Doosthosseini et al., 2024). HEP analysis on Lambda encountered the documented limit of 10 new synchronous invocations per second, which stretched the startup of large runs and became visible in the distribution of function start times (Kuśnierz et al., 2022).
Data movement and workflow composition remain contested design points. The astronomy serverless-function paper for SRCNet identified architectural complexity, workflow composition, transient intermediate data, and heterogeneity across regional centres as open challenges (Parra-Royón et al., 12 Jan 2026). FSD-Inference showed that even for highly optimized serverless communication, the choice between SNS+SQS and S3 is workload-dependent and cost-dependent, because pub-sub/queueing is cheaper for moderate messages and high parallelism, whereas object storage becomes preferable for very large payloads (Oakley et al., 2024). GraphFlash demonstrated that serverless graph processing can become practical only after substantial system-level optimization—partition-aware key aggregation, intra-function partition co-location, binary serialization, compression, and superstep-aware activation (Zhao et al., 12 May 2026). This suggests that serverless HPC frequently trades operational simplicity at deployment time for algorithmic and systems complexity during implementation.
The forward-looking agenda is correspondingly broad. The AI Factories position paper argues for dual-stack supercomputers that integrate Slurm, parallel file systems, and MPI with Kubernetes, object storage, FaaS, and SaaS front-ends (Lopez et al., 16 Sep 2025). The reinforcement-learning farm paper proposes extending structured serverless control to map, pipeline, divide-and-conquer, and multi-tenant deployments (Li et al., 6 Feb 2026). The systematic literature review positions all of this within a growing interdisciplinary field at the convergence of cloud and HPC infrastructures (Besozzi et al., 14 Jan 2026). A plausible implication is that “serverless HPC” will remain plural rather than singular: pure FaaS for bursty microtasks, serverless containers and workflow services for data-intensive pipelines, and scheduler-integrated or federated serverless layers for supercomputing environments that seek elasticity without abandoning established HPC substrates.