GPU Offload in Rust: Portable, Safe, and Fast

Abstract: High-performance GPU programming has traditionally forced a compromise between execution efficiency and memory safety. While Rust guarantees compile-time memory safety for host CPUs via its strict ownership model, applying these constraints to massively parallel GPU execution environments has previously mandated either vendor-locked Domain-Specific Languages (DSLs) or escaping to explicit unsafe raw pointers. This paper presents a zero-overhead, multi-vendor GPU compilation framework built natively into the Rust compiler (rustc) and LLVM backends. We leverage Rust's rich type system, ownership system, and strict aliasing guarantees (noalias) to efficiently manage and optimize data transfers through LLVM's Offload infrastructure. We expose the technical challenges of cross-vendor ABI lowering mismatches between Host and Device targets and introduce a two-pass compilation pipeline capable of safely handling both manual and compiler-generated memory movements. Evaluating our framework on RAJAPerf demonstrates that our rustc-based solution can generate competitive LLVM IR for GPU kernels, achieving a solid kernel performance against native, hand-optimized CUDA and HIP C++ baselines.

• The paper presents a compiler-integrated Rust and LLVM framework that supports NVIDIA and AMD GPUs, preserves ownership and aliasing guarantees, and produces kernels generally close to CUDA and HIP performance on a RAJAPerf subset.
• The paper shows that type-directed automatic data movement can cause severe end-to-end costs, including a slowdown exceeding 400 times on an MI250X, while explicit staging keeps data resident and enables predictable multi-kernel execution.
• The paper demonstrates safe abstractions for disjoint global-memory access but leaves shared-memory safety, heterogeneous ABI validation, asynchronous transfer optimization, and broader performance validation as important open problems.

The paper presents a compiler-integrated framework for GPU offload in Rust that targets portability across NVIDIA and AMD accelerators while preserving Rust’s ownership, aliasing, and memory-safety properties. Its central claim is deliberately qualified: safe abstractions need not impose a material kernel-execution penalty, but automatic data movement can impose severe end-to-end costs unless compiler analyses or explicit staging eliminate redundant transfers. The implementation is based on modifications to rustc, LLVM, and the LLVM Offload infrastructure rather than on a separate Rust-specific GPU DSL, thereby placing GPU compilation within the existing Rust and LLVM toolchain (2608.13759).

Research problem and design objectives

Conventional GPU programming models expose high performance through CUDA, HIP, OpenMP, or SYCL, but they generally require explicit data mapping, device allocation, pointer manipulation, or vendor-specific APIs. In C, C++, and Fortran, the absence of language-level ownership and aliasing guarantees means that correctness depends heavily on programmer discipline. Rust offers a different baseline: ownership, lifetimes, borrowing, and data-race freedom are checked statically, while references are lowered to LLVM pointers carrying noalias information in most safe contexts. These guarantees can provide optimization information unavailable without manually written restrict annotations.

GPU execution nevertheless conflicts with the surface semantics of ordinary Rust references. A mutable reference is unique in sequential Rust, whereas a GPU kernel commonly performs many concurrent writes into disjoint elements of one logical array. Existing Rust GPU systems resolve this tension in different ways. rust-gpu is constrained by SPIR-V’s pointer model, rust-cuda relies extensively on unsafe raw pointers and targets NVIDIA, and cuda-oxide provides safe abstractions but is vendor-specific. The paper therefore targets three properties simultaneously:

• Portability across major accelerator vendors.
• Safety for ordinary parallel memory-access patterns.
• Performance competitive with native CUDA and HIP implementations.

The resulting framework is not a single programming interface. It is a family of interfaces that expose progressively more control over kernel execution and memory residency.

Three offloading interfaces

Interface A is the compiler-managed path. A programmer writes a Rust kernel using references and invokes it through an offload macro. Immutable references are interpreted as read-only inputs, while mutable references are treated as potentially modified outputs. The host-side lowering allocates device storage, transfers arguments, launches the kernel, synchronizes, and transfers modified values back according to the argument types.

This interface minimizes programmer effort but makes data movement sensitive to host-side uses. If one kernel writes an output and a later host operation reads it before another kernel consumes it, the runtime must perform a device-to-host transfer followed by a host-to-device transfer. Logging, debugging, or any other intervening host access can therefore introduce synchronization that is not explicit in the source. The paper reports that a naïve implementation of this model can be more than 400 times slower than explicit data management on the MI250X benchmark configuration. This result establishes that type-directed automatic mapping is not, by itself, sufficient for performance-sensitive GPU pipelines.

Interface B provides a common offload boundary for host-launched vendor libraries such as cuBLAS and rocBLAS. Since the compiler cannot inspect the library’s internal kernels, it cannot optimize their device computation directly. It can nevertheless infer the transfer requirements of the arguments and materialize device pointers through the same mapping machinery used for Rust kernels. This gives Rust code, vendor libraries, and Rust-written device kernels a shared operational interface. The design supports incremental replacement of CPU operations with vendor implementations and creates a common point for runtime profiling and diagnostics.

Interface C makes device residency explicit through staged types such as Preload and PreloadMut. These types retain the lifetime and aliasing constraints of the original host borrow through PhantomData, while storing raw host pointers only as runtime identifiers for mapped allocations. A PreloadMut value keeps the host object mutably borrowed until the staged value is dropped; dropping it transfers the potentially modified data back to the host. Multiple kernels can consequently operate on resident device data without implicit transfers at every launch.

The distinction between Preload and PreloadMut is important. Immutable staging permits multiple read-only handles, with runtime reference counting for the associated device allocation. Mutable staging is non-aliased by Rust’s borrowing rules and represents exclusive device-side ownership. Interface C therefore exposes synchronization at a statically controlled program point rather than allowing incidental host operations to trigger it. The trade-off is reduced convenience in exchange for predictable residency and transfer behavior.

Safe parallel kernels

The paper’s safe-kernel frontend separates the specification of parallel memory access from the kernel’s computational body. Its primary abstraction is Region, parameterized by a PartitioningStrategy. A strategy determines the memory subset assigned to each thread and provides views that are intended to be disjoint across concurrently executing threads. This permits kernels such as vector addition to operate on ordinary slices while avoiding user-written unsafe blocks for common disjoint-access patterns.

The safety argument is localized in the implementation of the partitioning strategy. The PartitioningStrategy trait itself is unsafe because the compiler cannot verify that its pointer arithmetic, bounds assumptions, and returned views satisfy the required disjointness invariants. However, strategy implementations can be written in Rust and packaged behind safe APIs. This shifts the unsafe proof obligation from every kernel author to a small, reusable abstraction layer.

The design differs from cuda-oxide in three respects emphasized by the authors. First, partitioning strategies are extensible by users rather than confined to one project. Second, the abstraction supports disjoint chunks rather than only scalar-oriented access. Third, indexing is computed by the strategy itself instead of exposing a specialized thread-index object to the kernel. The paper does not establish that either frontend dominates the other; it argues instead that their access-pattern abstractions could potentially converge while retaining different compiler and portability architectures.

Shared memory is treated more conservatively. The framework exposes workgroup-local memory through raw pointers, requiring unsafe code. The authors identify four hazards: exceeding the dynamically requested allocation, using incorrect offsets among multiple objects, constructing ordinary Rust references despite concurrent aliasing, and violating alignment requirements through casts. This is a significant boundary in the safety model: global-memory access can be encapsulated through partitioning, but cooperative shared-memory algorithms intrinsically require synchronization and aliasing patterns that the presented interface does not yet encode safely.

Two-pass compilation and ABI handling

The toolchain uses separate host and device compiler passes. The device pass compiles selected, monomorphized kernel instances for targets such as nvptx64-nvidia-cuda and amdgcn-amd-amdhsa, packages the resulting device image, and makes it available to the host pass. The host pass lowers Rust offload intrinsics to LLVM OpenMP target or Offload runtime calls, embeds the device image into host LLVM IR, and links the required runtime components. LLVM fat-binary support permits multiple device images to be embedded in one executable.

The two-pass architecture is motivated by Rust’s target-dependent compilation semantics. A single-pass design would simplify cross-boundary monomorphization and communication, but it would force cfg evaluation and target-specific source decisions to occur under one compilation target. This can cause CPU-specific implementations, intrinsics, or inline assembly to become reachable from device kernels. Independent host and device frontend passes avoid that problem, at the cost of explicit communication between the compilations.

Cross-pass monomorphization is handled by serializing kernel definition identifiers and concrete generic substitutions during the host phase. The device phase imports this metadata to seed its monomorphization collection. The authors expect the additional serialization cost to be negligible because it integrates with incremental compilation, although the supplied evaluation does not provide a detailed compile-time study.

The separation exposes ABI risks. Host and device targets may lower the same Rust type differently. The paper gives primitive slices as a concrete example: the x86-64 host and AMD GPU backends represent a slice as pointer-plus-length scalars, while the NVIDIA PTX backend lowers it as a fixed-size array. This discrepancy prevents the framework from treating arbitrary host-device types as automatically compatible. Automated ABI validation is therefore a prerequisite for supporting richer structs and other cross-boundary values.

Type-driven memory mapping and compiler optimization

The framework derives data-directionality information from Rust types and mutability. Immutable references and constant raw pointers are lowered to device-only input mappings; mutable references and mutable raw pointers receive bidirectional mappings; small scalar values are passed by value. This design replaces much of the explicit mapping metadata required by OpenMP and related systems with information already present in Rust MIR and type layouts.

The compiler also decomposes the convenient offload operation into explicit host-to-device transfer, kernel launch, and device-to-host transfer operations. This exposes sufficient structure for optimization passes to move transfers, eliminate redundant operations, and overlap transfers with computation. The proposed transformations include asynchronous prefetching, loop-invariant code motion, and cancellation of transfers across repeated kernel launches or unrolled sequences.

These optimizations are essential rather than ancillary. Interface A’s semantics force the compiler to preserve host visibility at uses that may appear unrelated to GPU execution. By separating transfers from launches, the optimizer can delay synchronization until a value is actually consumed or hoist invariant transfers outside loops. The paper reports that, if effective, these transformations could make the convenient interface approach the performance of explicit staging on the evaluated workloads. This conclusion remains provisional because the optimizations are described as prototypes and are not presented as a complete, systematically validated evaluation.

Performance evaluation

The evaluation ports a subset of RAJAPerf to Rust and compares it with RAJA-based CUDA and HIP implementations on an AMD MI250X, an NVIDIA H100, and an NVIDIA RTX A2000. The reported measurements cover kernel execution, transfer volume, total benchmark runtime, floating-point optimization modes, and register usage.

At the kernel level, Rust is generally close to the RAJA implementations on the MI250X and H100. The principal exceptions are FIR and LTIMES, both small microbenchmarks whose performance is sensitive to loop unrolling and compiler heuristics. The implication is that the Rust-to-LLVM lowering path can generate competitive device code for representative loop kernels, but it does not yet reproduce the optimization decisions of native CUDA or HIP in all instruction-light cases.

On the H100, Rust performs fewer and smaller transfers than RAJA: 53 host-to-device transfers totaling 423 MB, compared with 55 transfers totaling 468 MB; device-to-host traffic is 69 MB for Rust versus 99 MB for RAJA. However, lower transfer volume does not produce lower transfer time: Rust’s transfers reportedly take more time than RAJA’s approximately 16 ms total. The authors attribute this discrepancy to differences in memory kinds and asynchronous-transfer behavior. Thus, transfer-size reduction is not a reliable proxy for transfer-performance improvement in the current runtime.

Whole-benchmark runtime shows a mixed result:

Platform and comparison	Rust result
AMD MI250X versus base HIP/C++	32% faster to 43% slower
NVIDIA H100 versus base CUDA/C++	11% faster to 46% slower
Naïve automatic Interface A on MI250X	More than 400× slower than optimized Rust

The largest H100 deficits occur in FIR and LTIMES, where Rust is 44% and 46% slower than base CUDA. At the same time, Rust is 15% and 32% faster than base HIP on those two workloads. These contradictory outcomes indicate that the observed gap is strongly compiler- and backend-dependent rather than a uniform penalty from Rust’s safety mechanisms.

Algebraic floating-point operations provide a further optimization point. On the RTX A2000, they produce a 2× speedup for FIR and approximately 20% improvements for DEL_DOT_VEC_2D, VOL3D, and MATVEC3D. Rust does not expose the full C++-style fast-math contract because assumptions such as no NaN and no infinity could conflict with safe-language semantics. The algebraic operations recover much of the relevant optimization space without introducing those assumptions. Their effect is workload- and architecture-dependent: the MI250X experiments show no significant improvement outside the reported RTX A2000 results.

Register usage is modestly higher in Rust, averaging 33 registers versus 28 for RAJA-CUDA across 13 kernels on an RTX 2070. The paper suggests that additional bounds checks may contribute, although it reports no measured runtime impact from those checks in the evaluated kernels. This result leaves open whether the register difference matters for occupancy or becomes more consequential in kernels with higher register pressure.

Limitations and open questions

The implementation remains a prototype with several explicit limitations. Full Rust standard-library support on GPUs is not provided, and richer host-device types require ABI validation that has not yet been completed. Shared-memory programming remains unsafe, so the framework does not provide a uniformly safe programming model for all important GPU memory spaces. The evaluated safe partitioning abstractions cover selected disjoint-access patterns rather than the full range of irregular, collective, or synchronization-intensive kernels.

The performance study is also limited in scope. It uses a subset of RAJAPerf, does not provide a systematic compile-time or code-size evaluation, and does not fully quantify the cost and benefits of the proposed optimization passes. The reported whole-program slowdowns are substantially affected by synchronization and transfer behavior, while the strongest kernel-level claims concern relatively regular loop kernels. Consequently, the evidence supports competitive generated kernels, but not yet a general equivalence between automatic Rust offload and manually optimized CUDA or HIP across broader HPC workloads.

The paper leaves several technically precise questions open: whether automated ABI validation can support complex aggregates across heterogeneous targets; how safely to encode shared-memory aliasing and synchronization; whether asynchronous transfer and launch transformations remain correct and profitable under realistic control flow; and how the interface should represent multiple devices and peer-to-peer data movement.

Conclusion

The paper establishes a coherent architecture for Rust GPU offload based on rustc, LLVM Offload, type-directed mapping, and explicit separation between host and device compilation. Its strongest result is that Rust kernels can achieve near-parity with native GPU kernels on substantial portions of the RAJAPerf subset while supporting NVIDIA and AMD targets. Its most important qualification is equally clear: automatic per-launch data movement can be catastrophically expensive, with a measured slowdown exceeding 400×, unless staging or compiler optimization preserves device residency. The framework therefore demonstrates a credible path to portable and substantially safe GPU programming in Rust, while leaving ABI generality, shared-memory safety, runtime synchronization, and broader performance validation unresolved (2608.13759).