GPU-Virt-Bench: Virtualization Benchmark

• GPU-Virt-Bench is a benchmarking framework for software GPU virtualization, defining 56 metrics across 10 categories including overhead, isolation, and LLM sensitivity.
• It compares software approaches like HAMi-core and BUD-FCSP against a simulated MIG-Ideal baseline, emphasizing performance, fairness, and quota enforcement.
• Designed for multi-tenant cloud and container environments, it provides actionable insights for optimizing GPU sharing and deploying efficient LLM inference systems.

GPU-Virt-Bench is a comprehensive benchmarking framework for software-based GPU virtualization systems, introduced to provide a standardized methodology for evaluating GPU sharing layers in cloud and container environments where demand is driven increasingly by artificial intelligence and LLM inference workloads. The framework is designed to compare software virtualization approaches such as HAMi-core and BUD-FCSP against an idealized Multi-Instance GPU baseline, and it does so through 56 performance metrics organized into 10 categories spanning overhead, isolation, LLM behavior, memory bandwidth, cache behavior, PCIe throughput, multi-GPU communication, scheduling, memory fragmentation, and error recovery (VG et al., 26 Nov 2025).

1. Scope, motivation, and system model

GPU-Virt-Bench addresses a specific operational problem: GPUs in Kubernetes and cloud environments are often allocated in an all-or-nothing manner, so a single container receives exclusive access to a full device even when it requires only a fraction of its resources. The framework is motivated by the resulting poor utilization, higher cost, inflexible scheduling, and weak support for multi-tenancy. Within this context, the paper distinguishes three GPU-sharing models: hardware partitioning through MIG, time-slicing, and software virtualization implemented through middleware that intercepts CUDA and NVML calls to enforce limits in software (VG et al., 26 Nov 2025).

The benchmark is explicitly concerned with software-based virtualization rather than general-purpose GPU benchmarking. Its central question is not merely how fast a GPU runs a workload, but how a virtualization layer behaves under multi-tenant pressure: how much overhead it adds, how accurately it enforces memory and compute quotas, how stable performance remains across tenants, how sensitive transformer-style workloads are to the added software path, and how the system responds to fragmentation and faults. This positioning differentiates GPU-Virt-Bench from broader suites that focus on modern GPGPU workloads, Unified Virtual Memory, or cloud 3D rendering rather than virtualization middleware itself (VG et al., 26 Nov 2025).

A recurring point in the framework is the use of an idealized reference. The benchmark supports four execution modes—Native, HAMi-core, BUD-FCSP, and MIG-Ideal—and the paper is explicit that MIG-Ideal is a simulated ideal baseline derived from NVIDIA specifications and published benchmarks, not a measurement on actual MIG hardware. This is important because many comparative statements in GPU-Virt-Bench are normalized against that idealized target rather than against a physically instantiated MIG partition (VG et al., 26 Nov 2025).

2. Internal architecture and execution workflow

GPU-Virt-Bench is modular. Its command-line entry point is main.c, which parses options such as --system native, --system hami, and --system fcsp, then runs the selected tests. The measurement logic is distributed across CUDA-based modules in src/metrics/*.cu, specifically overhead.cu, isolation.cu, LLM.cu, bandwidth.cu, cache.cu, pcie.cu, nccl.cu, scheduling.cu, fragmentation.cu, and error.cu. Supporting utilities include timing.c for high-precision timing through clock_gettime and CUDA events, statistics.c for mean, median, percentiles, and coefficient of variation, report.c for JSON/CSV/TXT output, and process.c for multi-process orchestration in isolation tests (VG et al., 26 Nov 2025).

This structure reflects the framework’s operational assumptions. Several metrics require multiple tenants or multiple processes to be active concurrently, so process orchestration is a first-class part of the design rather than an external harness. Likewise, output generation is not limited to a single textual report; the benchmark is intended to support repeated comparative runs and machine-readable regression analysis. This suggests a benchmarking workflow oriented toward infrastructure validation rather than one-off microbenchmarking.

The evaluated virtualization systems embody different implementation strategies. HAMi-core is described as a software virtualization layer using CUDA and NVML interception through dlsym hooks, shared accounting regions, and rate limiting. BUD-FCSP is presented as an enhanced software virtualization system with finer SM control, reduced hook overhead, improved token bucket behavior, and better fairness through weighted fair queuing. Native mode provides the bare-metal baseline, while MIG-Ideal provides the normalized target (VG et al., 26 Nov 2025).

3. Metric taxonomy and representative measurements

The 56 metrics are grouped into 10 categories. The category structure is itself one of the main contributions, because it defines what the paper considers a complete virtualization evaluation.

Category	Metrics	Focus
Overhead	10	Interception and runtime cost
Isolation	10	Quota enforcement and interference
LLM	10	Transformer-sensitive behavior
Memory Bandwidth	4	Bandwidth preservation under sharing
Cache	4	L2 sharing and contention
PCIe	4	Host-device transfer behavior
NCCL/P2P	4	Multi-GPU communication
Scheduling	4	Context switching and preemption
Fragmentation	3	Long-running allocation behavior
Error Recovery	3	Resilience and post-fault behavior

The overhead group includes launch, allocation, and context-creation costs, along with interception and accounting overheads. For example, kernel launch latency is defined as

$t_{launch} = t_{kernel\_start} - t_{API\_call},$

and API interception overhead is modeled as

$$t_{intercept} = t_{hook\_resolution} + t_{function\_lookup}.$$

The benchmark also formalizes rate-limiter behavior through a token bucket:

$$tokens_{available} = \min(bucket_{max},\ tokens + rate \times \Delta t).$$

These formulas make explicit that GPU-Virt-Bench treats virtualization overhead as a composition of software events rather than an opaque slowdown (VG et al., 26 Nov 2025).

The isolation group measures how close enforced behavior is to configured behavior and how much tenant interference remains. Memory limit accuracy is

$$accuracy = \frac{\min(actual,\ configured)}{\max(actual,\ configured)} \times 100\%,$$

while SM utilization accuracy is

$$accuracy = \max\left(0,\ 1 - \frac{|util_{target} - util_{actual}|}{util_{target}}\right).$$

Cross-tenant compute isolation is measured by

$isolation = \frac{perf_{contended}}{perf_{solo}},$

QoS consistency uses the coefficient of variation,

$CV = \frac{\sigma_{perf}}{\mu_{perf}},$

and fairness uses Jain’s index,

$$J(x_1,\ldots,x_n) = \frac{\left(\sum_{i=1}^{n}x_i\right)^2}{n\cdot \sum_{i=1}^{n}x_i^2}.$$

Together, these choices show that the framework treats isolation as a quantitative spectrum rather than a Boolean property (VG et al., 26 Nov 2025).

At the lower system layers, the framework includes memory-bandwidth, cache, PCIe, NCCL/P2P, scheduling, fragmentation, and error-recovery metrics. Representative definitions include memory bandwidth isolation,

$$isolation = \frac{BW_{contended}}{BW_{solo}} \times 100\%,$$

the memory-bus saturation point,

$$saturation = \min_n \{n : BW_n \geq 0.95 \times BW_{max}\},$$

L2 hit rate,

$$t_{intercept} = t_{hook\_resolution} + t_{function\_lookup}.$$0

PCIe transfer bandwidth,

$$t_{intercept} = t_{hook\_resolution} + t_{function\_lookup}.$$1

and the fragmentation index,

$$t_{intercept} = t_{hook\_resolution} + t_{function\_lookup}.$$2

The error-recovery section culminates in a graceful degradation score,

$$t_{intercept} = t_{hook\_resolution} + t_{function\_lookup}.$$3

with weights $$t_{intercept} = t_{hook\_resolution} + t_{function\_lookup}.$$4, $$t_{intercept} = t_{hook\_resolution} + t_{function\_lookup}.$$5, and $$t_{intercept} = t_{hook\_resolution} + t_{function\_lookup}.$$6 (VG et al., 26 Nov 2025).

4. LLM-oriented benchmarking within the framework

A notable feature of GPU-Virt-Bench is that it assigns a full category to LLM-sensitive behavior rather than treating transformer inference as just another CUDA workload. The LLM group includes attention kernel throughput, KV-cache allocation speed, batch-size scaling, token generation latency, memory-pool efficiency, multi-stream performance, large tensor allocation, mixed-precision support, dynamic batching impact, and multi-GPU scaling. This design reflects the paper’s argument that virtualization must be evaluated against modern AI serving patterns, not only against traditional throughput kernels (VG et al., 26 Nov 2025).

The framework uses a simplified TFLOPS proxy for attention:

$$t_{intercept} = t_{hook\_resolution} + t_{function\_lookup}.$$7

where $$t_{intercept} = t_{hook\_resolution} + t_{function\_lookup}.$$8 is batch size, $$t_{intercept} = t_{hook\_resolution} + t_{function\_lookup}.$$9 is sequence length, $$tokens_{available} = \min(bucket_{max},\ tokens + rate \times \Delta t).$$0 is embedding dimension, and $$tokens_{available} = \min(bucket_{max},\ tokens + rate \times \Delta t).$$1 is attention runtime. The $$tokens_{available} = \min(bucket_{max},\ tokens + rate \times \Delta t).$$2 dependence is intended to capture the quadratic scaling of attention. Batch scaling is measured by

$$tokens_{available} = \min(bucket_{max},\ tokens + rate \times \Delta t).$$3

and token-latency behavior is decomposed into time-to-first-token

$$tokens_{available} = \min(bucket_{max},\ tokens + rate \times \Delta t).$$4

and mean inter-token latency

$$tokens_{available} = \min(bucket_{max},\ tokens + rate \times \Delta t).$$5

These are serving-oriented metrics, and their inclusion makes the framework particularly relevant for inference platforms rather than only training or HPC environments (VG et al., 26 Nov 2025).

The LLM metrics also expose memory-management pathologies that are especially consequential in virtualized environments. KV-cache allocation speed is measured in allocations per second, large tensor allocation is modeled as a function of size and fragmentation, and memory-pool efficiency compares pooled allocation with direct allocation:

$$tokens_{available} = \min(bucket_{max},\ tokens + rate \times \Delta t).$$6

A plausible implication is that the framework is engineered to capture the interaction between virtualization and allocator behavior, which is frequently a dominant issue in long-context or high-concurrency inference workloads. The paper’s own interpretation is narrower: it states that KV cache and allocation behavior are major bottlenecks for LLM inference under software virtualization (VG et al., 26 Nov 2025).

5. Scoring, normalization, and comparative semantics

GPU-Virt-Bench does not stop at reporting raw measurements. It converts metrics into normalized scores relative to the MIG-Ideal baseline. For lower-is-better metrics, the normalization rule is

$$tokens_{available} = \min(bucket_{max},\ tokens + rate \times \Delta t).$$7

while for higher-is-better metrics it is

$$tokens_{available} = \min(bucket_{max},\ tokens + rate \times \Delta t).$$8

Category scores are arithmetic means,

$$tokens_{available} = \min(bucket_{max},\ tokens + rate \times \Delta t).$$9

and the overall score is a weighted sum,

$$accuracy = \frac{\min(actual,\ configured)}{\max(actual,\ configured)} \times 100\%,$$0

The default weights are 0.15 for Overhead, 0.20 for Isolation, 0.20 for LLM, 0.10 for Memory Bandwidth, 0.08 for Cache, 0.07 for PCIe, 0.05 for NCCL/P2P, 0.07 for Scheduling, 0.04 for Fragmentation, and 0.04 for Error Recovery (VG et al., 26 Nov 2025).

These weights encode the framework’s operational priorities. Isolation and LLM performance receive the largest emphasis, indicating that the benchmark is designed primarily for production multi-tenant AI deployments rather than for purely synthetic conformance testing. The inclusion of fragmentation and error recovery with smaller weights suggests that these dimensions are treated as important but secondary.

A common misconception is that the overall score constitutes an absolute measure of virtualization quality. The paper supports a more limited interpretation. Because scores are normalized against MIG-Ideal, they indicate proximity to a chosen target model rather than universal performance quality. Likewise, the 100% scores for MIG-Ideal and Native do not mean that a native system is literally equivalent to hardware partitioning; they mean that within this scoring system both define the upper reference point for their respective comparisons (VG et al., 26 Nov 2025).

6. Experimental evaluation and observed behavior

The paper evaluates the framework on an NVIDIA A100-40GB PCIe GPU with an AMD EPYC 7742 CPU, 512 GB of memory, Ubuntu 22.04, CUDA 12.0, and driver 525.105.17. On this platform, software virtualization incurs measurable overhead, particularly for memory operations. Kernel launch latency is reported as 4.2 µs for Native, 15.3 µs for HAMi, and 8.7 µs for FCSP. Memory allocation latency is 12.5 µs for Native, 45.2 µs for HAMi, and 28.3 µs for FCSP; memory free latency is 8.1 µs, 32.4 µs, and 18.6 µs, respectively; and context creation overhead is 125 µs, 312 µs, and 198 µs. Total throughput degradation is 18.5% for HAMi and 9.2% for FCSP (VG et al., 26 Nov 2025).

Isolation results are reported for four concurrent tenants. Memory limit accuracy reaches 98.2% for HAMi and 99.1% for FCSP; SM utilization accuracy is 85.4% for HAMi and 92.7% for FCSP; the fairness index is 0.87 for HAMi and 0.94 for FCSP; and noisy-neighbor impact is 24.3% for HAMi versus 12.1% for FCSP. The framework also reports that both systems achieved memory isolation and fault isolation, while SM enforcement remained approximate rather than exact. This is a central empirical point: software virtualization can enforce limits usefully, but it does not reproduce the hard isolation of hardware partitioning (VG et al., 26 Nov 2025).

The LLM-focused results are especially prominent. Using synthetic transformer-like benchmarks, the paper reports attention throughput at 82.3% of native for HAMi and 91.5% for FCSP, KV-cache allocation speed at 76.4% and 88.2%, TTFT at 45.2 ms and 28.7 ms, ITL at 12.8 ms and 8.4 ms, and batch scaling at 0.78 and 0.89. These numbers are used to support the paper’s conclusion that LLM workloads are particularly sensitive to virtualization overhead, especially through allocator and KV-cache behavior (VG et al., 26 Nov 2025).

The overall normalized scores summarize the comparison: MIG-Ideal scores 100%, Native scores 100%, BUD-FCSP scores 85.2% and receives Grade B+, and HAMi-core scores 72.0% and receives Grade C. The paper interprets this as evidence that software virtualization is viable but still materially below ideal hardware partitioning, and that FCSP closes much of the gap relative to HAMi-core (VG et al., 26 Nov 2025).

7. Position in the benchmarking landscape, interpretation, and practical use

GPU-Virt-Bench occupies a distinct place among GPU benchmarking frameworks. Altis modernizes general GPGPU benchmarking with modern workloads and CUDA runtime features (Hu et al., 2019). UVMBench is specialized for Unified Virtual Memory and oversubscription behavior (Gu et al., 2020). Pictor provides a benchmark and measurement infrastructure for cloud 3D applications, including settings with virtualized GPUs and containers (Liu et al., 2020). BAT 2.0 focuses on tunable GPU kernels for autotuning research (Tørring et al., 2023). GPU-Virt-Bench differs from all of these by centering the software GPU virtualization layer itself as the object of measurement, with explicit attention to quota enforcement, tenant interference, fragmentation, and LLM inference behavior (VG et al., 26 Nov 2025).

The framework also clarifies several interpretive points that are easy to miss. It is not a benchmark of hardware MIG implementations, because MIG-Ideal is simulated. It is not a generic AI benchmark, because its LLM tests are synthetic transformer-like benchmarks rather than end-to-end serving traces. It is not purely a microbenchmark suite, because it mixes low-level mechanisms such as PCIe and cache measurements with operational metrics such as TTFT, fairness, and graceful degradation. This suggests that GPU-Virt-Bench is best understood as a systems benchmark for production deployment decisions.

The paper’s practical guidance follows directly from this framing. It recommends using MIG when available for strict hardware-enforced isolation, preferring BUD-FCSP over HAMi-core for LLM inference or where fairness matters, tuning memory limits conservatively to reduce enforcement overhead and fragmentation, and running only the relevant subset of metrics when the deployment question is narrow—for example, only the LLM metrics when evaluating inference service quality (VG et al., 26 Nov 2025).

In broader terms, GPU-Virt-Bench formalizes an evaluation vocabulary for software-based GPU sharing. Its contribution is not only the set of 56 metrics, but also the claim that virtualization quality must be decomposed across overhead, isolation, LLM sensitivity, memory-system behavior, interconnect behavior, scheduler response, allocator health, and fault handling. This suggests a shift away from single-number throughput comparisons toward a multidimensional assessment of whether a GPU sharing system is suitable for real multi-tenant workloads.