RADICAL-Pilot: HPC Pilot-Enabled Runtime
- RADICAL-Pilot is a Python-based pilot-enabled runtime system that decouples workload specification, resource acquisition, and task execution on HPC platforms.
- It implements a pilot abstraction by using job placeholders to dynamically bind tasks to available resources, reducing scheduler overhead and adapting to heterogeneous workloads.
- Performance studies demonstrate that RADICAL-Pilot scales to thousands of nodes with mixed workloads, integrating with various HPC schedulers and workflow systems.
RADICAL-Pilot (RP) is a pilot-enabled runtime system for executing many-task, heterogeneous workloads on high-performance computing (HPC) platforms. It implements the Pilot abstraction by submitting job placeholders, or pilots, through a platform scheduler and then late-binding tasks onto the resources acquired by those pilots, thereby decoupling workload specification, resource acquisition, and task execution. Across its design and performance papers, RP is described as portable, modular, extensible, and Python-based; it can be used stand-alone or as a runtime for higher-level workflow systems, and it has been characterized on leadership-class platforms including Titan, Summit, Frontera, Frontier, and other production HPC systems (Merzky et al., 2015, Merzky et al., 2021).
1. Pilot abstraction and execution semantics
Pilot systems use job placeholders to obtain a chunk of resources through the system scheduler, then schedule application tasks directly on those resources once the pilot becomes active. In RP, this separates three concerns: workload specification, which describes tasks and their resource needs independently of any particular machine or queue; resource selection, which chooses and submits pilots to one or more target machines and queues; and task execution, which binds tasks to pilots and to cores or nodes at runtime. Late-binding means tasks are mapped to pilots only when the pilot is active, allowing adaptation to resource availability, mitigation of queue latency, and scheduling of heterogeneous task sizes (Merzky et al., 2015).
On leadership-class machines, RP operationalizes this model as multi-level scheduling. A pilot can be large—for example, 1024 Summit nodes—and once active the Agent inspects the pilot’s resource topology, such as 42 CPU cores and 6 GPUs per Summit node, and schedules tasks in waves, or generations, that fit the available resources. This organization reduces per-task scheduler overhead and supports mixed workloads comprising MPI and non-MPI executables, Python functions, CPU tasks, and GPU tasks (Merzky et al., 2021).
A recurrent misconception is to treat RP as a replacement for the native batch scheduler. The architectural descriptions instead show that RP works through the platform scheduler, fitting HPC usage policies while transferring task-level placement and launch decisions to an application-facing runtime after pilot activation (Merzky et al., 2015).
2. Distributed architecture and state models
Architectural descriptions across the RP literature center on a distributed runtime with client-side management, a resource-side Agent, and a persistent coordination layer. Earlier descriptions emphasize Session, PilotManager, UnitManager, and an Agent connected through MongoDB and lightweight messaging; later descriptions condense the design into four modules—PilotManager, TaskManager, Agent, and DB—communicating over TCP/IP, a ZeroMQ mesh, RADICAL-SAGA, and MongoDB (Merzky et al., 2015, Merzky et al., 2021).
| Component | Role |
|---|---|
| Session / PilotManager | Lifetime management and pilot submission |
| UnitManager / TaskManager | Task ingestion, queueing, and pilot assignment |
| Agent | Scheduling, launching, staging, monitoring on pilot resources |
| DB / Messaging | MongoDB queueing; ZeroMQ control-plane communication |
Within the Agent, the core subcomponents are the Scheduler, Executors or Launchers, and Stagers. The Scheduler discovers the pilot’s cores and GPUs, partitions them across nodes, and assigns resources to tasks. Executors derive placement and launch commands and spawn processes via methods such as /bin/sh, Popen, ORTE, or PRRTE-backed mechanisms. Stagers implement optional input and output movement through local filesystem operations and RADICAL-SAGA-mediated transfers, including scp, sftp, GSI-enabled variants, and Globus Online (Merzky et al., 2021).
RP also models pilots and tasks as stateful entities. The pilot state model includes NEW, PM_LAUNCH, P_ACTIVE, and DONE. The task or unit state model includes NEW, UM_SCHEDULING, A_SCHEDULING, and A_EXECUTING, with optional staging states such as UM_STAGING_IN, A_STAGING_IN, A_STAGING_OUT, and U_STAGING_OUT. Transitions are sequential; any transition may fail or be canceled; and the transition to P_ACTIVE is determined by the resource manager’s queue policies (Merzky et al., 2015).
3. Workload model, schedulers, and platform integration
RP supports heterogeneous tasks along multiple dimensions. The 2021 design paper states five dimensions of heterogeneity: task type, including executables, Python functions, and class methods; parallelism, including scalar, MPI, OpenMP, and multi-process or multithread patterns; compute support, including CPU and GPU; size, from one hardware thread to thousands of nodes; and duration, from seconds to hours. Tasks are treated as black boxes with explicit resource envelopes, including cores, threads, GPUs, ranks, and environment specifications, enabling precise placement and binding (Merzky et al., 2021, Alsaadi et al., 2021).
The scheduling subsystem combines generality with specialization. In early Cray-focused descriptions, RP distinguishes a general-purpose scheduler for heterogeneous workloads from specialized schedulers for homogeneous bags of tasks. Later leadership-class descriptions add three placement policies: Continuous, which places tasks contiguously across nodes; Torus, for BG/Q-like topologies; and Tagged, which pins tasks to specific nodes. GPU tasks are bound to explicit GPU indices, multithreaded tasks are packed onto single nodes, and MPI tasks are aligned on topologically close nodes (Merzky et al., 2015, Merzky et al., 2021).
Launch integration is a central differentiator. On Cray systems, RP supports ALPS/aprun, CCM, ORTE-CLI, and ORTE-LIB, with the latter using CFFI calls into ORTE libraries to reduce per-launch filesystem and socket overhead. The broader launcher inventory later expands to 15 methods, including aprun, ccmrun/mpirun_ccmrun, jsrun, dplace/mpirun_dplace, runjob, POE, srun, ibrun, generic ORTE and PRRTE launchers, mpirun/mpiexec variants, ssh/rsh, and fork (Merzky et al., 2015, Merzky et al., 2021).
Data movement remains optional and per task. RP stages input and output via local shell commands and RADICAL-SAGA, with targets including shared scratch on the resource, shared or local filesystems, and paths accessible from the user workstation. This I/O model is part of the task lifecycle rather than a separate external workflow concern (Merzky et al., 2015).
4. Performance characterization and scaling results
RP’s original performance characterization reports that it can spawn more than 100 tasks per second and supports the steady-state execution of up to 16,000 concurrent tasks. The same study formalizes throughput as , concurrency as the number of units in a given state at time , and resource utilization as . Microbenchmarks show scheduler assignment rates of about 150 units/s initially on large pilots, stabilizing around 50 units/s as the free-core pool fills; ORTE-LIB single-Executor throughput is reported as units/s at 1024 cores, at 2048, and at 4096, with around 33 units/s for larger pilots (Merzky et al., 2015).
On Titan, RP was characterized with homogeneous, MPI-like workloads emulating GROMACS BPTI via Synapse. Weak scaling covered 32 to 4096 tasks on 1024 to 131,072 cores, with each task using 32 cores; average time to execution was s up to 4096 cores, corresponding to about 11% overhead over the 828 s ideal, increasing to 18% at 8192 cores and 160% at 131,072 cores. Strong scaling fixed 16,384 tasks and varied pilot size from 16,384 to 65,536 cores, yielding observed time-to-execution values of s, s, and s. The default scheduler became a bottleneck at large pilot sizes, while a special-purpose scheduler for homogeneous multi-node MPI tasks improved scheduling throughput from about 7 tasks/s to about 70 tasks/s (Merzky et al., 2018).
Summit and Frontera exposed a different performance regime. On Summit, the improved scheduler reached about 300 tasks/s, compared with 6 tasks/s previously with an older scheduler. For a 1024-node Summit run with 3098 heterogeneous tasks and 4 PRRTE DVMs, resource utilization was about 77% and overhead about 7%; at 4097 nodes with 12,276 tasks and 16 DVMs, resource utilization was about 41% and overhead about 9%, with the dominant bottleneck identified as the distributed filesystem under high-concurrency PRRTE launch. On Frontera, RP plus RAPTOR executed 126,471,524 Python function calls on 7000 nodes and 392,000 cores; resource utilization reached about 90% quickly and maintained about 98% after roughly 300 s for about 3000 s, while task throughput averaged about 37,000 tasks/s with peaks around 40,000 tasks/s (Merzky et al., 2021).
A Summit-specific comparison of JSM/jsrun and PRRTE established lower bounds for RP on CPU-only, homogeneous 900 s single-core workloads. JSM was constrained by a 4096 open-file limit on the batch node, capping concurrency at about 967 tasks, whereas RP+PRRTE with multiple executors on compute nodes and a raised open-file limit reached 16,384 tasks over 401–410 nodes. With optimized PRRTE configuration, 4 concurrent sub-agents, and a reduced RP-to-PRRTE delay of 0.01 s, mean time to execution fell from 3236 s to 1296 s and workload Exec Cmd rose to 63.557% on 404 nodes (Turilli et al., 2019).
5. Ecosystem role and later extensions
RP has consistently been positioned both as a stand-alone runtime and as a backend for higher-level workflow systems. Across the papers, reported integrations include Ensemble Toolkit, Replica Exchange frameworks, domain workflows, EnTK, RepEx, ExTASY, Parsl, Swift/T, PanDA, QCFractal, Airflow, and BV-BRC. The same literature emphasizes that RP exposes a uniform API for pilots and tasks while preserving portability across batch systems such as Slurm, PBSPro, Torque, LGI, Cobalt, LSF, and LoadLeveler (Merzky et al., 2015, Merzky et al., 2018, Merzky et al., 2021, Merzky et al., 17 Mar 2025).
The Parsl integration paper introduces RPEX, a Parsl executor that uses RP as its workload manager. RPEX preserves the Parsl API and dataflow model, adds per-task resource specifications required by RP, implements a Task Translator that maps Parsl tasks one-to-one onto RP tasks, and relies on a new RP-side executor for MPI Python functions. This combination enables concurrent single-node Python functions, multi-node MPI executables, and MPI Python functions using both CPUs and GPUs. On Frontera, the Colmena workflow showed approximately linear strong scaling from 32 to 128 nodes before Ibrun became the bottleneck at 256 nodes, while the Ice Wedge Polygons workflow showed low RP and RPEX overheads—on the order of single-digit seconds—through 16 GPU nodes (Alsaadi et al., 2021).
The 2025 service-based extension broadens RP from task execution to service-oriented AI-out-HPC coupling. It adds a Service Task abstraction, a ServiceManager, a DataManager, extended scheduler semantics for service-before-client priority relations, and ZeroMQ-based communication channels between services and tasks. In the prototype, Ollama hosts Meta Llama 3 8B as local or remote inference services. Preliminary results report service bootstrap characterization on Frontier with up to 640 concurrent service instances, local inter-node latency of 0 ms 1 ms on Delta, remote latency of 2 ms 3 ms between Delta and an R3 cloud server, and the observation that for real LLM inference the inference time dominates overall response time (Merzky et al., 17 Mar 2025).
A subsequent hybrid-runtime study extends the same building-block logic to concurrent integration with Flux and Dragon on Frontier. In that design, RP routes executables and tightly coupled jobs to Flux, which provides hierarchical scheduling and fine-grained placement, and routes short-lived Python function tasks to Dragon, which uses lightweight dispatch and shared-memory queues. Quantitatively, RP+Flux sustains up to 930 tasks/s, RP+Flux+Dragon exceeds 1500 tasks/s with at least 99.6% utilization, and the Slurm srun baseline peaks at 152 tasks/s with utilization below 50% under the reported conditions (Merzky et al., 25 Sep 2025).
6. Limitations, optimization strategies, and significance
The RP literature repeatedly identifies launch-layer and scheduler-layer limits rather than a single intrinsic ceiling. The general-purpose scheduler’s throughput decreases with pilot size and, in steady state, lock contention further reduces throughput. On Cray systems, ALPS/aprun limits concurrent applications and struggles to share nodes; CCM availability varies by site; ORTE-CLI encounters filesystem interactions, per-task sockets, race conditions, and system limits above about 16,000 concurrent tasks; and applications linked against Cray MPI could not be run via ORTE at the time of the 2015 study. On Summit, JSM was capped by the open-file limit on the batch node, and PRRTE performance at scale was constrained by shared-filesystem behavior and, under extreme load, by robustness issues (Merzky et al., 2015, Turilli et al., 2019, Merzky et al., 2021).
The papers also document a consistent set of mitigations. For homogeneous multi-node workloads, a special-purpose scheduler partitions pilot cores into equal-sized blocks on first scheduling, achieving constant-time assignment per task and substantially reducing lock contention. On Cray systems, ORTE-LIB reduces per-launch filesystem and socket overhead compared to ORTE-CLI, and multiple Executor instances increase throughput until ORTE-layer limits are reached. On Summit, RP partitions pilots across multiple PRRTE DVMs, with reported configurations using at most 256 nodes per DVM, and the broader leadership-class guidance recommends multiple 1024-node pilots over a single pilot larger than 4000 nodes when launcher and filesystem contention dominate. Later work proposes deeper partitioning of the Agent into a Metascheduler coordinating multiple Schedulers and Executors, and the Flux/Dragon integration extends the same principle to backend-specific partitions with modest startup costs of about 20 s per Flux instance and about 9 s per Dragon instance (Merzky et al., 2015, Merzky et al., 2021, Merzky et al., 25 Sep 2025).
Within HPC runtime research, RP’s significance lies in its combination of portability, modularity, extensibility, and introspection. The 2015 characterization emphasized portability and interoperability across diverse HPC platforms and launch methods, including challenging Cray environments; a clean Pilot-API in Python; modular architecture; and profiling and analytics support. That work also reported Titan-scale runs consuming about 25 million core-hours. Later papers preserve the same architectural core while extending it toward heterogeneous CPU/GPU workloads, high-throughput function execution, workflow-system interoperability, ML service hosting, and hybrid AI-HPC orchestration (Merzky et al., 2015, Merzky et al., 2021).
RADICAL-Pilot is available as open-source software at https://github.com/radical-cybertools/radical.pilot; related analytics and experiment artifacts are available at https://github.com/radical-cybertools/radical.analytics and https://github.com/radical-experiments/jsspp18 (Merzky et al., 2015).
