OOM-Free Alpamayo via CPU-GPU Memory Swapping for Vision-Language-Action Models

Abstract: End-to-end Vision-Language-Action (VLA) models for autonomous driving unify perception, reasoning, and control in a single neural network, achieving strong driving performance but requiring 20-60GB of GPU memory-far exceeding the 12-16GB available on commodity GPUs. We present a framework, which enables memory-efficient VLA inference on VRAM-constrained GPUs through system-level optimization alone, without model modification. Our work proceeds in three stages: (1) Sequential Demand Layering reduces VRAM usage from model-level to layer-level granularity; (2) Pipelined Demand Layering hides parameter transfer time within layer execution time via transfer--compute overlap; and (3) a GPU-Resident Layer Decision Policy, informed by per-module residency benefit analysis, eliminates the residual transfer overhead that pipelining cannot hide. We further propose a performance prediction model that determines the optimal configuration-both the number and placement of resident layers-from a single profiling run with less than 1.3% prediction error across all configurations. Applied to NVIDIA's Alpamayo-R1-10B (21.52GB) on an RTX 5070Ti (16GB), our work achieves up to 3.55x speedup over Accelerate offloading while maintaining full BF16 precision.

• The paper introduces a system-level CPU-GPU memory swapping framework that enables OOM-free inference for large VLA models with a 3.55× speedup.
• It employs sequential and pipelined demand layering techniques to optimize VRAM usage by balancing DMA and execution costs on commodity GPUs.
• A lightweight prediction model with an interleaved placement policy accurately determines optimal GPU-resident layers while preserving bit-exact outputs.

System-Level CPU-GPU Memory Swapping for Commodity VLA Inference: A Technical Assessment of "OOM-Free Alpamayo via CPU-GPU Memory Swapping for Vision-Language-Action Models" (2605.11678)

Introduction and Motivation

End-to-end Vision-Language-Action (VLA) models, epitomized by architectures such as Alpamayo-R1-10B, combine perception, reasoning, and control within an integrated neural framework for autonomous driving. These models exhibit strong driving performance but exhibit prohibitive GPU memory footprints (20–60 GB), far exceeding the 12–16 GB VRAM available in commodity GPUs. This memory gap presents a concrete bottleneck impeding practical experimentation and deployment on non-premium hardware.

The paper proposes an optimized CPU-GPU memory swapping framework—targeted expressly at inference for VLA models under VRAM constraints—achieving efficient execution solely through system-level optimizations, with no modification to model architecture or weights. The authors advance a three-stage optimization pipeline, culminating in a policy-driven selection of GPU-resident layers, buttressed by performance modeling and empirical validation.

Architectural Challenges and Model Profiling

For context, Alpamayo-R1-10B's inference pipeline is decomposed into five major functional components:

1. ViT Encoder: Qwen3-VL Vision Transformer for raw image embedding.
2. Patch Merger: Dimensionality projection of vision tokens.
3. VLM Prefill: Multi-layer VL backbone for preliminary context generation.
4. VLM Decode: Autoregressive language decoding, accounting for nearly 80% of end-to-end latency.
5. Diffusion-Based Action Generation: Multi-step non-autoregressive waypoint prediction via a 36-layer expert decoder.

Profiling reveals that model parameters constitute approximately 93% of VRAM requirements, identifying parameter storage/transfer as the critical optimization target, whereas activations and KV cache make up a minor residual.

Sequential and Pipelined Demand Layering

Prior solutions—such as Hugging Face Accelerate offloading—transfer individual parameter tensors synchronously, resulting in fragmented memory transactions, serial bottlenecks, and suboptimal overlap between PCIe DMA (host-to-device transfer) and CUDA kernel execution. As observed, this yields an order-of-magnitude increase in latency relative to theoretical bounds, especially as VRAM shrinks (see discussion after Figure 1).

As a baseline improvement, Sequential Demand Layering loads only the current layer's parameters into VRAM immediately before execution, tightly constraining VRAM usage to layer granularity. This significantly conserves memory but cannot overlap transfer and compute.

Pipelined Demand Layering, built atop a double-buffered memory architecture (Double Flat Buffer, DFB), leverages the concurrency of CUDA execution and copy engines. This design achieves maximal overlap between DMA and compute streams where possible, especially for EXE-intensive layers (where execution time surpasses transfer time, $C_\text{EXE} > C_\text{DMA}$). For such modules, transfer cost is fully hidden, limiting the overhead to the initial transfer per inference.

Figure 2: Average per-layer DMA and execution times show which Alpamayo components are EXE- or DMA-intensive on RTX 5070 Ti.

However, modules such as VLM decode (~21 tokens autoregressively decoded, per-inference) and the diffusion expert (10 steps) are DMA-intensive ($C_\text{DMA} \gg C_\text{EXE}$); their transfer cost dominates due to high frequency of invocation and cannot be hidden by pipelining alone.

Figure 3: When execution dominates (EXE-intensive), residency confers no benefit beyond the first layer due to pipelined overlap.

Figure 4: For DMA-intensive modules, each offloaded layer incurs irreducible DMA overhead not masked by pipelining.

GPU-Resident Layer Decision and Performance Modeling

The critical insight is that, for DMA-intensive regimes, only keeping layers permanently in VRAM eliminates transfer overhead. The authors introduce a closed-form residency benefit metric computed as latency reduction normalized to VRAM cost, parametrized by repeated invocation counts (e.g., token/diffusion step):

$B_\ell = \frac{R_\ell \cdot \Delta C_\ell}{M_\ell}$

where $R_\ell$ is repetition count, $\Delta C_\ell$ the transfer time saved, and $M_\ell$ the resident footprint.

Empirical evaluation confirms that the benefit is heavily skewed toward VLM decoder layers due to high repetition. Analytical and measured results show that prioritizing residency of VLM layers dominates; ViT and diffusion expert layers confer far lower benefit per unit VRAM.

Figure 5: Residency benefit explodes for VLM layers at typical output token counts, surpassing all other modules.

Figure 6: Output token distribution confirms $N \ge 11$ for all real driving cases, justifying VLM prioritization.

Notably, determining the “optimal” resident configuration avoids an exponential search: since VLM decoder layers are homogeneous, only the number of resident layers (not their identities) matters, reducing the space to 37 configurations (for 36 layers). The authors introduce an interleaved placement policy to avoid consecutive-residency saturation.

Figure 7: Empirically, keeping ViT and Diffusion offloaded and maximizing VLM layer residency achieves lowest latency.

A lightweight prediction model—requiring only static profiling and one timed measurement—predicts inference latency across the full VRAM-latency spectrum with $<$1.3% absolute error, enabling VRAM-to-latency planning without brute-force trials.

Evaluation and Comparative Results

Progressive optimization shows compound benefit: sequential Demand Layering shaves latency and slashes VRAM; pipelining confers further gains; but the introduction of dynamic VLM decoder residency yields the largest impact, enabling overall $3.55\times$ speedup over Accelerate offloading at matched VRAM and preserving full BF16 precision.

Figure 8: Each optimization stage materially reduces latency, with VLM residency dominating gains.

Figure 9: For all VRAM budgets, the proposed system achieves a persistent 3.5–3.6$\times$ inference speedup over Accelerate.

Further scalability testing on RTX 3080 Ti (PCIe Gen3) and on OpenVLA-7B demonstrates that bandwidth-dependent module behaviors (i.e., the shift from EXE- to DMA-intensive) and residency optimization generalize. On lower-bandwidth hardware, the fractional gains decrease due to baseline transfer sensitivity, but the method still outperforms previous baselines.

Figure 10: VRAM-latency tradeoff curve for Alpamayo operating on lower-bandwidth/VRAM hardware.

The framework preserves numerical exactness: all configurations produce bit-exact outputs versus traditional full-GPU preloading, confirming the absence of side-effect bugs or numerical drift.

Implications and Future Prospects

Practically, this framework renders state-of-the-art VLA models usable on commodity workstations without quantization or intrusive rewrites, streamlining development and testing cycles for large-scale perception-and-control models. For the broader AI system community, it reifies the principle that substantial real-world performance can be unlocked through precise system-level intervention, provided detailed module profiling is available.

Theoretically, the work sharpens the trade-off landscape: even with optimal residency/swapping, current VLA models are fundamentally bottlenecked by compute throughput. Approaching closed-loop (<100 ms) inference for real-time autonomous driving will require new DNN architectures and further leaps in consumer hardware, as even unconstrained preloading on RTX 5090 yields only ~1 s latency. Thus, system-level VRAM optimizations, while necessary, are insufficient for grand real-time deployment, but remain central for research and partial-integration workflows.

Future research may explore tighter integration with hardware-aware compilers, dynamic compression alongside residency, and multi-GPU/NUMA-aware extension for both inference and training-time use cases.

Conclusion

This work formalizes and solves the problem of efficient, OOM-free inference for large VLA models on commodity GPUs, obtaining up to $C_\text{DMA} \gg C_\text{EXE}$0 lower latency than industry-standard offloading without quality trade-offs. It delivers a unified methodology for demand-layered pipelining, GPU-resident layer selection, and efficient residency configuration search. While the absolute gap to hard real-time autonomy persists, the contributions here will enable new levels of accessibility and empirical scaling for vision-language-action researchers and practitioners (2605.11678).