StickyInvoc: Efficient LLM Inference on HPC

• StickyInvoc is a task model that decouples persistent state creation from inference execution, enabling significant amortization of multi-gigabyte model loading costs.
• It leverages a sticky task to load and hold model state and invocation tasks to execute inference, achieving up to a 3.6× speedup in high-throughput workflows.
• The design integrates with HPC schedulers using StateManager processes and peer-to-peer transfers to handle fault tolerance and mitigate thundering-herd filesystem effects.

Searching arXiv for the specified StickyInvoc paper and closely related workflow systems. StickyInvoc is a pair of task models for high-throughput workflows that targets LLM inference on HPC clusters by decoupling state creation from goodput execution. In the traditional create–destroy model, each inference task must reconstruct its computational state and destroy it upon completion, which repeatedly transfers multi-gigabyte model parameters from long-term storage to local disk, CPU memory, and GPU memory. StickyInvoc replaces that pattern with a symbiotic relationship between a persistent sticky task and subsequent invocation tasks that inherit the loaded state, allowing model-loading cost to be paid once and amortized over many inferences (Phung et al., 20 Jun 2026).

1. Problem setting and cost model

The motivating workload is LLM inference embedded in high-throughput workflows. In this setting, the dominant cold-start cost is the multi-stage transfer of model parameters. The paper decomposes model loading into three components: $$T_{\mathrm{load}} = T_{\mathrm{FS\to disk}} + T_{\mathrm{disk\to CPU}} + T_{\mathrm{CPU\to GPU}}.$$ A fixed session-startup cost,

$T_{\mathrm{startup}},$

covers Python interpreter startup, framework initialization, and JIT compilation. The per-task overhead of the conventional model is therefore

$$O_{\mathrm{create\_destroy}} = T_{\mathrm{load}} + T_{\mathrm{startup}}.$$

The reported typical values are $T_{\mathrm{load}} \approx 3$ s, $T_{\mathrm{startup}} \approx 0.5$ s, and hence $O_{\mathrm{create\_destroy}} \approx 3.5$ s. Since most single-query inferences take $0.3$–$0.5$ s on the GPU, model loading dominates, accounting for approximately $88\%$–$92\%$ of the total in the reported measurements (Phung et al., 20 Jun 2026).

This overhead becomes especially consequential at workflow scale. The paper states that workflows often comprise $T_{\mathrm{startup}},$0–$T_{\mathrm{startup}},$1 independent inference tasks, so a per-task startup cost of roughly 3 s scales to hours of overhead. The situation is further exacerbated by thundering-herd bursts of concurrent loads that saturate the shared filesystem, and by opportunistic HPC resources that are both heterogeneous and preemptible without warning. In the reported environment, GPUs span eight major models from 2016 to 2023. Larger batches prolong runtime, which increases preemption risk and can force another full cold start.

2. Sticky and invocation task semantics

StickyInvoc introduces two distinct task types. A sticky task materializes a persistent state on a single node from a user-provided template but does not execute any goodput computation by itself. An invocation task inherits that state and performs the actual computation without paying the state-creation cost again and without destroying the state on exit (Phung et al., 20 Jun 2026).

The persistent state is denoted

$T_{\mathrm{startup}},$2

where $T_{\mathrm{startup}},$3 denotes model weights, $T_{\mathrm{startup}},$4 the tokenizer, and $T_{\mathrm{startup}},$5 compiled CUDA kernels. Each invocation then performs

$T_{\mathrm{startup}},$6

Operationally, the sticky task loads and holds the model state once per node, persists that state on the node, and remains alive, for example as a StateManager process. The invocation task obtains a handle to the persistent state, retrieves the model from that state, and executes inference on the input batch. The separation is therefore between state materialization and useful work, rather than between different phases of a single monolithic task.

The principal analytical consequence is amortization. If one sticky task incurs the cold-start cost once and $T_{\mathrm{startup}},$7 invocation tasks share it, then the amortized overhead per inference is

$T_{\mathrm{startup}},$8

or, omitting the small startup term,

$T_{\mathrm{startup}},$9

Because $$O_{\mathrm{create\_destroy}} = T_{\mathrm{load}} + T_{\mathrm{startup}}.$$0 s and $$O_{\mathrm{create\_destroy}} = T_{\mathrm{load}} + T_{\mathrm{startup}}.$$1 s, the paper notes that once $$O_{\mathrm{create\_destroy}} = T_{\mathrm{load}} + T_{\mathrm{startup}}.$$2,

$$O_{\mathrm{create\_destroy}} = T_{\mathrm{load}} + T_{\mathrm{startup}}.$$3

3. Scheduler integration, data movement, and fault handling

StickyInvoc is integrated with an HPC scheduler through a division of responsibilities between sticky tasks, invocation tasks, the scheduler, and a StateManager. Sticky tasks are submitted as pilot jobs pinned to 1 GPU each. They allocate local disk, CPU memory, and GPU memory, load the model, and register a StateManager that listens for invocations. Invocation tasks carry a pointer to the template. The TaskVine scheduler matches each invocation to any worker hosting that template, dispatches it immediately, and the StateManager executes the work in place (Phung et al., 20 Jun 2026).

A notable systems detail is the use of peer-to-peer transfers to bootstrap new nodes from existing ones, thereby avoiding filesystem spikes. This mechanism directly addresses the thundering-herd behavior identified in the motivation, where simultaneous cold starts can saturate the shared filesystem.

Fault tolerance is handled at the task-model level. If a worker or its StateManager crashes or is preempted, the scheduler detects the failure and re-queues the sticky task on another node; template data is fetched from a peer. Invocation tasks that were executing re-queue and resume on any remaining node that holds the state. Upon workflow completion, or under resource pressure, the scheduler may terminate idle sticky tasks to reclaim GPUs. The design therefore couples persistence with explicit requeue semantics rather than assuming that persistent state is permanently available.

4. Experimental configuration

The reported evaluation uses both a full local cluster and a stable testbed. The local HPC cluster comprises 567 GPUs across eight major models and a Panasas distributed filesystem with 94 k IOPS read. For stable measurements, the paper uses 20 preallocated GPUs: 10 $$O_{\mathrm{create\_destroy}} = T_{\mathrm{load}} + T_{\mathrm{startup}}.$$4 A10 and 10 $$O_{\mathrm{create\_destroy}} = T_{\mathrm{load}} + T_{\mathrm{startup}}.$$5 TITAN X Pascal (Phung et al., 20 Jun 2026).

The workflow is PromptVerify on 145 449 FEVER claims, using SmolLM2 with 1.7 B parameters, for a total of 150 000 inferences. The Parsl-TaskVine configuration assigns 2 cores, 10 GB RAM, 20 GB disk, and 1 GPU per task.

These details matter because the claimed benefit is not merely microbenchmark improvement in isolated inference latency. The evaluation targets a high-throughput, heterogeneous, and preemptible execution regime in which startup overhead, filesystem contention, and resource churn can dominate end-to-end runtime.

5. Reported performance and scaling behavior

The end-to-end runtime comparison on the stable testbed reports the following values (Phung et al., 20 Jun 2026):

Implementation	Runtime (s)	Speedup vs Create-Destroy
Create-Destroy	10 400	1.0×
StickyI/O	5 300	1.96×
StickyInvoc	2 900	3.59×

The speedup is defined as

$$O_{\mathrm{create\_destroy}} = T_{\mathrm{load}} + T_{\mathrm{startup}}.$$6

At 20 GPUs, StickyInvoc completes 150 k inferences in 2 900 s, corresponding to approximately $$O_{\mathrm{create\_destroy}} = T_{\mathrm{load}} + T_{\mathrm{startup}}.$$7 inf/s. When dynamically scaling out to 186 GPUs, or $$O_{\mathrm{create\_destroy}} = T_{\mathrm{load}} + T_{\mathrm{startup}}.$$8 of the cluster, the workflow finishes in 784 s, corresponding to approximately $$O_{\mathrm{create\_destroy}} = T_{\mathrm{load}} + T_{\mathrm{startup}}.$$9 inf/s. The paper states that the scaling curve demonstrates near-linear speedup up to 186 nodes.

The abstract summarizes these results by stating that, when rewritten in the StickyInvoc paradigm, a claim verification workflow consisting of 150k inferences achieves a 3.6x speedup on a stable testbed with 20 GPUs, and completes in just 784 seconds by incrementally scaling out to 186 otherwise idle GPUs. Within the scope of the reported experiments, the improvement is therefore an end-to-end workflow effect, not only a per-task latency reduction.

6. Adoption guidance, limitations, and open questions

The paper gives several best practices for adopting StickyInvoc. Model initialization should be factored into a lightweight template function. Invocation batch size should balance per-task overhead; the reported guidance notes that StickyInvoc is robust even at batch 1. Peer transfers should be enabled to avoid thundering-herd effects on the shared filesystem. State managers should be monitored for memory growth, with eviction or restart if necessary (Phung et al., 20 Jun 2026).

The limitations are explicit. StickyInvoc only applies to models that fit entirely in one GPU’s memory. The persistent state manager introduces a small runtime and resource overhead, including monitoring and heartbeat traffic. Extending the approach to multi-GPU models or very large ensembles remains future work. Automatic eviction policies and global coordination among multiple templates can be refined.

These constraints delimit the current scope of the model. StickyInvoc is not presented as a universal replacement for all task abstractions in distributed inference. Rather, it addresses the specific case in which repeated state construction is the dominant systems bottleneck and in which a persistent per-node model state can be safely reused across many invocation tasks. Within that regime, the central contribution is the decoupling of the costly create–destroy cycle from goodput, so that model loading is amortized across many invocations in a heterogeneous, preemptible HPC environment.