An Online Fragmentation-Aware Scheduler for Managing GPU-Sharing Workloads on Multi-Instance GPUs

Abstract: Modern GPU workloads increasingly demand efficient resource sharing, as many jobs do not require the full capacity of a GPU. Among sharing techniques, NVIDIA's Multi-Instance GPU (MIG) offers strong resource isolation by enabling hardware-level GPU partitioning. However, leveraging MIG effectively introduces new challenges. First, resource contention persists due to shared components such as PCIe bandwidth. Second, GPU fragmentation becomes a critical issue, which is different from prior fine-grained GPU sharing work due to MIG's limited number of valid MIG configurations. Fragmentation arises not only from spatial discontinuity but also from rigid profile placement constraints, especially after job arrivals and terminations. To address these issues, we propose an online scheduling framework that integrates conditional load balancing, dynamic partitioning, and job migration. Our approach dynamically adapts job placement to minimize contention and reorganizes GPU allocations to combat both internal and external fragmentation. Experimental results show that our method significantly improves system efficiency. When all techniques are applied, the makespan improves by up to 35%.