Virtual Compute Cores Overview

• Virtual Compute Cores (VCCs) are abstraction layers that virtualize GPU streaming multiprocessors and cloud CPU cores into independent, fine-grained execution units.
• On GPUs, VCCs enable asynchronous micro-operation scheduling that reduces latency by up to 23% and simplifies kernel programming.
• In cloud systems, VCCs support dynamic vCPU oversubscription with adaptive, risk-aware policies that optimize resource utilization while preventing hot nodes.

Virtual Compute Cores (VCCs) are increasingly central abstractions for both modern GPU runtime decoupling and adaptive cloud CPU provisioning. In the context of GPUs, VCCs form a software-managed “virtual SM” (streaming multiprocessor) layer that exposes asynchronous compute resources as independent execution engines, thereby aligning software orchestration with modern hardware. In cloud infrastructure, VCCs (typically as “vCPUs”) represent fractional, virtualized shares of physical CPU cores, enabling fine-grained overprovisioning strategies for maximizing physical resource utilization while striving to minimize risk. This entry presents a comprehensive view of VCCs across these domains, their theoretical models, scheduling strategies, empirical performance, and limitations (He et al., 4 May 2026, Wang et al., 2024).

1. Virtual Compute Cores: Definitions and System Models

On GPUs, a Virtual Compute Core (VCC) in the VDCores system is a lightweight, software-managed logical entity exposing asynchronous hardware units, such as CUDA-core pipelines, Tensor Cores, and WGMMAs, as independent, fine-grained virtual engines. Each VCC implements its own register file, shared-memory allocator, and per-core queue of compute micro-operations (μ-ops). Physical SMs are virtualized into sets of VCCs, each dynamically fetching and executing μ-ops as dependencies and hardware resources permit. This decomposition allows asynchronous compute resources to be orchestrated independently, overcoming inefficiencies of the monolithic kernel model (He et al., 4 May 2026).

In cloud infrastructure, VCCs correspond to “virtual compute cores” (vCPUs)—fractional logical compute units assigned to virtual machines (VMs) running on physical machines (PMs). Each physical node $j$ with $p_j$ CPU cores hosts a collection of VMs $i$ entitled to $v_i(t)$ vCPUs at decision step $t$, subject to node-specific allocation constraints and possible oversubscription (Wang et al., 2024). Utilization of VCCs in this context is defined as $U_i(t) = \frac{u_i(t)}{v_i(t)}$, where $u_i(t)$ is the instantaneous CPU usage of VM $i$.

2. Resource Decoupling and Asynchronous Execution on GPUs

The VDCores programming and runtime model introduces VCCs as the fundamental units to decouple resource allocation from operation scheduling in GPUs. The traditional static fusion of asynchronous hardware pipelines and monolithic kernel launches is replaced by a persistent kernel hosting $N$ VCCs per SM (e.g., 2 VCCs/SM). Each VCC independently interacts with the GPU’s underlying compute engines, fetching and scheduling μ-ops from its queue according to a local dependency and resource readiness model. The μ-ops, categorized as memory, compute, and control operations, are organized in a DAG $G = (U, E)$, with scheduling constraints that ensure operations only issue once dependencies have been satisfied.

Dependencies within and between μ-ops are explicitly encoded using small depIds and send/recv flags. Each VCC uses these fields to check readiness—specifically, $p_j$0—which eliminates the need for a global dependency scoreboard and enables robust, fine-grained scheduling (He et al., 4 May 2026).

At runtime, the instruction window (μ-op queue) of each VCC is divided into “virtual flows”—in-order chains of μ-ops. Within each flow, μ-ops issue in program order, but cross-flow μ-ops may bypass one another, enabling automatic overlap of independent memory-bound and compute-bound operations. The VCC scheduler issues a μ-op $p_j$1 when both $p_j$2 and resource conditions $p_j$3 are fulfilled. This design delivers software-pipeline efficiency without explicit user-managed pipelining or warp specialization.

3. Dynamic Oversubscription and Scheduling in Cloud Platforms

In cloud environments, VCCs are fundamental to oversubscription strategies, wherein the sum of vCPUs assigned to all VMs on node $p_j$4 may exceed the number of physical cores ($p_j$5), i.e., $p_j$6. Selecting $p_j$7 enables recovery of stranded compute capacity but incurs risk of overload (“hot nodes”) if aggregate usage peaks exceed physical availability.

Risk-aware adaptive oversubscription, as implemented by ProtoHAIL, uses imitation learning over expert trajectories to derive flexible, interpretable policies for allocating VCCs. ProtoHAIL encodes utilization and allocation histories into trajectory embeddings $p_j$8, develops representative usage prototypes $p_j$9, and learns quadratic policies on similarity to those prototypes. The full loss function combines representativeness, diversity, interpretability, and imitation objectives. Human-in-the-loop feedback refines policies by re-weighting loss terms via an advice potential gate, incorporating up/down-votes and merge/split operations for prototypes (Wang et al., 2024).

The adaptive policy adjusts vCPU allocations per VM dynamically by mapping features and state history to an allocation ratio $i$0 at each decision step, thereby precisely tuning oversubscription in line with recent system load patterns.

4. Practical Benefits and Performance Results

On GPUs, VCC-based orchestration in VDCores yields substantial improvements in throughput and development productivity. Benchmarks on four LLM inference workloads (Qwen 1.7B, Qwen 8B, Llama3 1B, Llama3 8B) across NVIDIA GH200, H100, and RTX 6000 Pro GPUs demonstrate a geometric-mean speedup of 1.31× (23% latency reduction) over leading operator-per-kernel and megakernel baselines. Under dynamic input scenarios (e.g., mixed LoRA adapters, uneven context lengths), speedup reaches 1.77×. Microbenchmarks show VDCores sustaining up to 82% of peak FP32 FLOPS and 93% of peak DRAM bandwidth (He et al., 4 May 2026).

Programming effort is dramatically reduced: a full Llama-1B pipeline is implemented using only 741 lines of CUDA C++ and 6 μ-ops, as opposed to 2,339 lines and 8 fused kernels required by hand-optimized megakernels, amounting to a ~90% reduction in kernel programming labor.

For cloud platforms, ProtoHAIL achieves orders-of-magnitude reductions in risk and increases in available vCPUs. In internal Microsoft cloud data, the protoHAIL policy reduces hot-node rate (risk) to 0% while saving an average of 8,161 stranded cores—outperforming grid-search, RL, and classical imitation learning methods. Human-in-the-loop refinement required an average of only 6 queries in the vCPU setting. On an analogous airline overbooking domain, the ProtoHAIL policy achieves lowest “compensation cost” (risk) with highest “extra profit” (benefit).

Summary of vCPU Oversubscription Results ((Wang et al., 2024), Table):

Method	Risk (Hot-node %)	Cores saved
Grid-search	0.00%	7,450
Moving Average	1.39%	7,628
DDPG (RL)	1.47%	5,030
Behavior Cloning	1.19%	7,870
GAIL	1.20%	6,980
DAgger	0.96%	7,938
ProtoHAIL w/o HITL	0.00%	8,153
ProtoHAIL (full)	0.00%	8,161

5. Software Architecture, Programming Model, and Usability

The decoupled programming model of VDCores eliminates static orchestration from the user codebase. Programmers implement only primitive μ-ops—such as tile-level matmul (COP_MATMUL), tiled attention (COP_ATTENTION), load.dep, and store. The high-level scheduler composes these into flows and automatically embeds message-queue hand-offs, relieving user code of explicit buffering, fusion order tuning, or launch schedule specialization. All orchestration logic reduces to μ-op composition and scheduling, resulting in kernel generality and maintainability (He et al., 4 May 2026).

In the cloud context, the ProtoHAIL system drives policy learning toward interpretable, prototype-based solutions. Prototypes automatically correspond to real-world workload patterns (“work-hour patterns,” “batch-job patterns,” “social-media peaks”). All major architectural components—trajectory encoders (LSTM/Transformer), policy heads, and feedback mechanisms—are as specified in the empirical evaluations.

6. Limitations, Head-of-Line Blocking, and Future Directions

While VCC-based abstractions demonstrate significant performance and productivity advantages, several challenges and limitations remain. In VDCores, when the number of concurrent virtual flows exceeds available hardware slots (CFU/EU), head-of-line blocking can occur—slow μ-ops in a physical warp can delay peers. Although virtual-flow assignment mitigates this, it does not fully eliminate the effect. Fine-grained μ-op scheduling adds runtime overhead (3% aggregate), necessitating careful pipelining; e.g., a two-stage CFU→LDU pipeline achieves a 4.2× improvement over naïve loops (He et al., 4 May 2026).

Looking ahead, extensions planned include support for new asynchronous GPU subdomains such as in-SM tensor-memory accelerators and cross-SM shared-memory units, each to be exposed as distinct virtual core types (e.g., “Virtual TMA Core”). Additional research targets multi-GPU and disaggregated memory orchestration, using VCC abstractions to span inter-device messaging and dependency management, as exemplified by the prospective “Virtual Memory Core” handling remote memory endpoints.

In the cloud domain, continued focus is on scaling adaptive decision-making to larger multi-tenant clusters, reducing human-in-the-loop interactions further, and handling increasingly nonstationary workload patterns in oversubscription risk control (Wang et al., 2024).

7. Cross-Domain Synthesis and Theoretical Implications

VCCs provide a unifying paradigm of resource decoupling and virtualization, applicable across heterogeneous compute substrates. On GPUs, VCCs reveal the unconstrained potential of asynchronous hardware; in cloud systems, they enable policy-driven dynamic allocation with robust risk controls. The resource–slack tradeoff in oversubscription models and the μ-op DAG in GPU scheduling both underscore the centrality of dependency-aware, fine-grained decision models. A plausible implication is that further convergence between hardware-aware μ-op scheduling and adaptive, history-based policy learning may yield fully autonomous, context-sensitive resource allocation systems with minimal need for hand tuning.

References:

• "VDCores: Resource Decoupled Programming and Execution for Asynchronous GPU" (He et al., 4 May 2026)
• "Risk-aware Adaptive Virtual CPU Oversubscription in Microsoft Cloud via Prototypical Human-in-the-loop Imitation Learning" (Wang et al., 2024)