StickyInvoc: Rethinking Task Models for High-throughput Workflows in the LLM Era

Abstract: The integration of LLMs into high-throughput workflows is creating a new class of workloads on HPC clusters that promises to accelerate advances in scientific discovery with unprecedented generative capabilities. However, the traditional task model imposes a prohibitive overhead in this new domain: each task must create its computational state from scratch and destroy it upon completion. For each LLM inference task, this "create-destroy" model forces the repeated and costly transfer of multi-gigabyte model parameters from a long-term, reliable storage to a compute node's local disk, its CPU memory, and finally its GPU memory. This overhead, compounded by the inherently high startup cost of LLM inference, the typical scale of thousands of tasks in high-throughput workflows, and the heterogeneous and preemptible nature of high-throughput resources, presents a significant performance barrier. To overcome this barrier, this paper presents StickyInvoc: a symbiotic relationship between two new task models for high-throughput workflows. Specifically, a "sticky" task creates a persistent state on a compute node from a user-provided template, but doesn't execute any goodput computation by itself. Instead, this state is then inherited by subsequent "invocation" tasks, which perform the actual computation without incurring the state creation overhead or destroying the state upon exit. StickyInvoc thus allows the decoupling of the creation and destruction of computational states, allowing the computational state of LLM models to be created once per sticky task and its cost amortized over many subsequent invocation tasks. Our evaluation shows that when rewritten in the StickyInvoc paradigm, a claim verification workflow consisting of 150k inferences achieves a 3.6x speedup on a stable testbed with 20 GPUs, and completes in just 784 seconds by incrementally scaling out to 186 otherwise idle GPUs.

• The paper introduces StickyInvoc, a paradigm that decouples model initialization from task execution to dramatically reduce redundant multi-GB data transfers.
• It utilizes persistent sticky tasks and transient invocation tasks to reuse GPU-resident model states, achieving up to a 3.6x speedup and improved runtime stability.
• Quantitative evaluations demonstrate robust performance under preemption and varied batch sizes, enabling scalable and efficient LLM workflows on volatile HPC resources.

StickyInvoc: Decoupling Computational State from Task Execution for High-Throughput LLM Workflows

Motivation and Problem Statement

The integration of LLMs into high-throughput workflows on HPC clusters exposes a fundamental mismatch between modern inference workloads and traditional stateless task models. The conventional "create-destroy" paradigm enforces task isolation and fault tolerance by requiring each task to initialize and destroy all computational state from scratch. For LLM inference, this means repeated and prohibitively expensive transfer of multi-gigabyte model parameters across storage, memory, and GPU for every task instance. In workflows spanning thousands of inference tasks, this behavior leads to severe performance bottlenecks and distributed storage contention, as evidenced by analysis showing model loading times dominating total inference runtimes (84.6%–92.2%) even on optimized local setups.

Resource allocation strategies exacerbate these inefficiencies. Static allocations, while stable, often underutilize heterogeneous clusters or experience long queue delays. Preemptible allocations leverage otherwise idle resources but are highly volatile and subject to unpredictable preemption, negating the reliability of batch sizing heuristics for amortizing initialization costs.

StickyInvoc Paradigm: Design and Implementation

StickyInvoc introduces a fundamental rethinking of task models by decoupling computational state creation from actual computation. It leverages two symbiotic task types:

• Sticky tasks: Materialize persistent and inheritable computational state (e.g., LLM model in GPU memory) from a user template but perform no useful computation. The state is retained on the compute node for subsequent reuse.
• Invocation tasks: Bind to persistent states generated by sticky tasks, execute inference workloads leveraging inherited state, and exit without destroying the state.

The workflow manager orchestrates the distribution and inheritance of state across opportunistic resources. State templates are peer-transferred among nodes to mitigate distributed filesystem "thundering herd" phenomena. Upon preemption, invocation tasks are rapidly rescheduled to nodes hosting compatible sticky states, transparently avoiding repeated model initialization. The transformation from stateless to persistent task models is enabled within the Parsl–TaskVine stack, allowing seamless Pythonic abstraction and dynamic task scheduling while maintaining global workflow consistency and resilience to node churn.

Quantitative Evaluation and Claims

StickyInvoc was evaluated within PromptVerify, a claim verification workflow utilizing the FEVER dataset (145,449 claims) and a 1.7B SmolLM2 model, executed across an 18-model, 567-GPU cluster. The following version comparisons were performed:

• Create-destroy: Traditional model, fresh LLM state per inference task.
• StickyI/O: Caches model parameters and dependencies locally, but requires fresh GPU state per task.
• StickyInvoc: Full persistence of model state in GPU, inherited by many invocation tasks.

Key numerical results:

• End-to-end execution time reduction: StickyInvoc yields a 3.6x speedup relative to create-destroy (10,408s → 2,904s), and accelerates further to 784s when scaling opportunistically to 186 GPUs (32.8% cluster utilization).
• Inference task runtime stability: StickyInvoc significantly decreases variance (σ: 147.61 → 4.35), with nearly eliminated model loading cost.
• Batch size robustness: Execution time is decoupled from inference batch size choice; StickyInvoc incurs only a minor penalty (400s, ~13.6%) for suboptimal sizing, unlike StickyI/O, which may degrade by two orders of magnitude.
• Resilience to aggressive preemption: Under repeated GPU evictions, StickyInvoc maintains smooth inference throughput and completes 16.9k more inferences than StickyI/O at similar preemption rates.

Practical and Theoretical Implications

StickyInvoc fundamentally alleviates the performance barrier imposed by repeated model initialization, making it possible for LLM-integrated workflows to execute efficiently on volatile, heterogeneous, and preemptible HPC resources. The paradigm enables scalable utilization of opportunistic cluster capacity without imposing cognitive or operational burdens on end users, directly addressing both workflow throughput and usability.

The decoupling of computational state from task execution not only amortizes expensive initialization costs but also paves the way for architectural innovations in workflow orchestration, such as persistent state managers, peer transfers, and non-destructive task completion. The approach integrates seamlessly with modern Pythonic workflow systems and generalizes to all LLMs that fit within per-node resources.

Theoretical implications include a reevaluation of fault tolerance guarantees—persistent task models require careful state management and may impose new classes of recovery logic. StickyInvoc's applicability is limited to LLMs whose resource footprint allows per-node persistence, and its efficacy relies on minimized management overhead compared to repeated cold starts.

Relations to Existing Work and Future Directions in AI

StickyInvoc distinguishes itself from prior techniques such as progress checkpointing and autoscaling, which are ineffective in HPC contexts lacking preemption warnings and fine-grained resource control. Related works in speculative decoding and KV cache management optimize inference speed but do not directly address initialization cost amortization at workflow scale.

The approach complements research on elastic resources (spot/preemptible instances (2606.22175)), memory management, and workflow system abstractions, and points toward future AI infrastructure developments that prioritize state sharing, persistent resource utilization, and failure-resilient orchestration models. There is potential for extension to distributed-state LLMs, improved recovery semantics, and integration with emerging AI cloud scheduling strategies.

Conclusion

StickyInvoc demonstrates that rethinking task models—by separating state creation from computation and introducing persistent, inheritable states—enables efficient, scalable, and robust execution of high-throughput LLM workflows on modern HPC clusters. The paradigm consistently delivers strong empirical speedups, robust throughput under preemption, and lowers operational complexity for users, representing an impactful advancement in task model design for scientific computing with LLMs (2606.22175).