Efficient, VRAM-Constrained xLM Inference on Clients

Abstract: To usher in the next round of client AI innovation, there is an urgent need to enable efficient, lossless inference of high-accuracy LLMs and vision LLMs (VLMs), jointly referred to as xLMs, on client systems. To address this, we present pipelined sharding, a novel, benchmark-profile-guided CPU-GPU hybrid scheduling technique to achieve efficient, VRAM-constrained inference for both dense and mixture-of-experts (MoE) LLMs. Using a combination of model sharding at the sub-layer level, CPU offloading, pipelined copy-compute, and prioritized tensor placement in VRAM, it optimizes both time-to-first-token (TTFT) and tokens per second (TPS) metrics, while flexibly adapting to system and inference conditions. For efficient, high-accuracy VLM inference, we combine pipelined sharding with a llama$.$cpp implementation of three well-understood prior ideas (jointly called VLMOpt), namely, vision tensor CPU offloading, flash attention, and vision and LLM VRAM overlap avoidance. These enhancements are targeted at improving client xLM inference in future releases of two important NVIDIA products - the In-Game Inferencing software development kit (IGI SDK) and the Cosmos-Reason1 (CR1) physical AI reasoning VLM. Highlights from our rigorous evaluation spanning multiple models and client systems include: for interactive use, TTFT improves by up to 6.7x and TPS by up to 30x for LLMs, and CR1 inference's VRAM demand is down by 10x, while in batched mode, throughput improves by up to 8.2x, all compared to their respective aggressive baselines. This paper is accepted at the 9th MLSys Conference (Industry Track), 2026. Code and artifact available at: https://github.com/deepshnv/pipeshard-mlsys26-ae

• The paper introduces pipelined sharding, a profile-guided hybrid scheduling framework that enables lossless inference for both dense and MoE xLMs on client devices with limited VRAM.
• The paper demonstrates up to 6.7x TTFT and 30x TPS speedups along with a 10x VRAM reduction compared to static partitioning, validated across various models and configurations.
• The paper details complementary VLMOpt techniques, including CPU-pinned offloading and Q-tiling, to optimize high-resolution VLM inference within stringent VRAM constraints.

Efficient, VRAM-Constrained xLM Inference on Clients: An Analytical Essay

Introduction and Motivation

Transformer-based LLMs and vision-LLMs (VLMs)—collectively termed xLMs—are foundational to a growing set of interactive and physical AI applications. Despite pervasive cloud-based inference, there are critical demands for efficient on-device xLM inference driven by privacy, cost, and the necessity for real-time, context-aware operation. However, client-class GPUs impose stringent constraints on available video RAM (VRAM), making it non-trivial to deploy accurate, high-parameter xLMs locally without degradation via quantization, pruning, or other lossy techniques.

This paper introduces pipelined sharding, a profile-guided, phase-adaptive CPU-GPU hybrid scheduling methodology, for efficient, lossless xLM inference under arbitrary VRAM budgets on client systems (2604.26334). The approach is shown to generalize to both dense and Mixture-of-Experts (MoE) LLMs, as well as state-of-the-art VLMs, such as Cosmos-Reason1 (CR1).

Pipelined Sharding: Hybrid Scheduling with Profile-Guided Optimization

Summary of Approach

Pipelined sharding architecturally divides the xLM computation graph at sub-layer granularity. This sharding facilitates flexible assignment of each shard to either the CPU or GPU, leveraging asynchronous copy-compute overlap and prioritized VRAM allocation. A three-phase pipeline underpins deployment: install-time hardware profiling, planning phase schedule generation, and dynamic inference phase schedule selection. At inference time, the scheduler dynamically selects a plan from pre-profiled options based on the real token workload and system conditions.

Figure 1: Pipelined sharding's three-phase pipeline: install-time profiling, profile-guided planning, and dynamic inference-phase schedule execution.

The profiling phase collects system-representative FLOPS metrics and bandwidth characteristics under contextually relevant loads (e.g., with PCIe contention). During planning, token "tiers" are introduced—representing batch-wide 'new token' increments—to guide optimal plan selection. Three canonical plans are evaluated: (1) all shards on GPU with weights streamed as needed, (2) statically partitioned execution between CPU and GPU to minimize PCIe transfers, and (3) hybrid schemes with double-buffered weight streaming and interleaved CPU execution to exploit available bandwidth and thread resources.

The scheduler's flexibility is enabled by (i) sharding at the sub-layer (e.g., attention/FFN boundary) for arithmetic homogeneity, (ii) prioritized VRAM allocation (e.g., attention and KV cache over FFN), and (iii) robust roofline-based cost modeling for compute/memory tradeoff across device boundaries.

Comparison to Prior Art and Claims

Contrasting with prior hybrid CPU-GPU inference techniques (e.g., TwinPilots, HeteGen, EdgeMoE), pipelined sharding uniquely adapts to both batch size and execution phase (context vs. decode), system characteristics (CPU core count, PCIe BW, VRAM size), and model subtype (MoE or dense). Previous approaches typically make static heuristic partitioning decisions or focus only on throughput but not TTFT, or vice versa.

The implementation specifically surpasses llama.cpp's static layer partitioning and manual scheduling options, especially in high-context (large KV cache) or low VRAM scenarios, quantitatively achieving up to 6.7x TTFT and 30x TPS speedups for LLMs, and 10x VRAM reduction in CR1 VLM inference relative to baseline strategies (see Figure 2 and Table results). The approach also enables successful batch and interactive inference for models with disk footprints far exceeding device VRAM (e.g., Qwen235B at 77 GB on disk running at 2 GB VRAM).

(Figure 2)

Figure 2: Speedups in TTFT and TPS across context sizes and VRAM budgets relative to static partitioning baselines.

VLMOpt: Vision-Specific VRAM Optimization

For VLMs, pipelined sharding is complemented by VLMOpt, a set of engineering optimizations: CPU-pinned VRAM-offloaded vision tensors, VRAM allocation decoupling between vision and language subgraphs, and vision FlashAttention with Q-tiling. Collectively, these enable high-resolution image processing (up to 1440p) on limited VRAM budgets by drastically reducing intermediate tensor allocations.

Evaluation: Numerical Results and Ablations

Model Coverage and Methodology

The solution is evaluated over a spectrum of state-of-the-art models, including both MoE and dense LLMs (Mistral, Qwen3, Nemo) and VLMs (Cosmos-Reason1). Performance is measured by TTFT, TPS, and E2EL, across batch sizes, context lengths, VRAM budgets (2G–32G+), and system heterogeneity (client laptops to desktops).

Key Results

• Interactive performance at minimal VRAM: Qwen235B delivers >5 TPS with only 2G VRAM, a 39× VRAM reduction over its disk size.
• Batch and context adaptivity: The scheduler reliably selects the optimal of three profiling-driven plans, matching or exceeding a hand-tuned oracle in all test configurations. Speedups scale with batch size and available CPU threads.
• Robust VLM inference: Cosmos-Reason1's VRAM requirement is reduced by up to 10× at high resolutions while maintaining lossless accuracy and temporal performance, enabling inference on previously intractable client-grade devices.
• Pareto-optimal resource sharing: When deployed concurrently with a AAA video game, pipelined sharding finds a "sweet spot" in VRAM allocation, maximizing both FPS and TPS.

(Figure 3)

Figure 3: Schedule choice adaptation across context size, VRAM budget, and CPU thread count. Color encodes optimal plan (GPU-only, CPU-only, or hybrid).

Implications and Future Directions

Practical Deployment and Generalization

By eliminating manual placement heuristics of current inference frameworks, pipelined sharding democratizes high-accuracy xLM inference on commodity GPUs—enabling real-time, in-game local models and physical AI applications. The profile-guided, phase-adaptive approach accommodates emerging hardware trends (e.g., increasing CPU core counts, CPU NPUs, multi-process VRAM sharing) with minimal changes. The technique is immediately relevant for commercial SDKs such as NVIDIA's IGI and Cosmos-Reason1.

Theoretical Implications

Token-tier-based schedule selection establishes a systematic approach to balancing compute and memory resources under multi-dimensional constraints. The decoupling of profiling, schedule planning, and inference phases offers a reusable paradigm for future hybrid hardware designs beyond discrete CPU-GPU architectures, including NPU accelerators or tightly coupled heterogeneous systems.

Prospects for Extension

The extensibility of the pipeline—profile-driven schedule estimation over a configurable space—enables straightforward incorporation of new operation types, hardware resources, or stricter online adaptation (e.g., under thermal constraints or dynamic power states).

Conclusion

Pipelined sharding and the complementary VLMOpt collectively establish a formal, extensible, and lossless solution for VRAM-constrained xLM inference on client systems. The methodology robustly adapts to model, system, and workload heterogeneity, yielding strong empirical speedups over state-of-the-art baselines and enabling new practical deployments that would have previously been infeasible on consumer hardware. This framework is poised to facilitate the next wave of real-time, locally running AI agents in both digital and embodied environments.