KIScaler: GPU-Aware Autoscaling for Kubernetes

• KIScaler is a reinforcement-learning component that uses PPO to dynamically scale CPU and GPU replicas for latency-sensitive inference services.
• It integrates a rich observability model from Prometheus to balance P95 latency, GPU utilization, and replica overhead in a heterogeneous environment.
• Trained entirely in simulation with KISim, KIScaler is deployed directly on Kubernetes to effectively handle dynamic and bursty traffic patterns.

KIScaler is the learning-based autoscaling component of the KIS-S framework, introduced as a reinforcement-learning autoscaler for Kubernetes inference services, implemented with Proximal Policy Optimization (PPO), and designed specifically for GPU-aware scaling of containerized deep learning inference workloads. Its purpose is to decide how many CPU-backed and GPU-backed inference replicas should run, and with what preference for placement, so that latency-sensitive AI serving can react better to changing demand than default Kubernetes autoscaling. The paper frames its objective as jointly minimizing end-to-end latency—especially P95 latency—improving resource efficiency, maximizing GPU utilization, and avoiding excessive scaling overhead, and emphasizes a sim-to-real workflow in which KIScaler is trained entirely in simulation using KISim and then deployed directly without retraining or fine-tuning (Zhang et al., 10 Jul 2025).

1. Position within the KIS-S architecture

KIScaler is one of two main ideas in KIS-S: KISim, the GPU-aware Kubernetes inference simulator, and KIScaler, the controller. KISim provides the environment in which the RL agent can be trained safely and repeatedly; KIScaler is the PPO agent that interacts with that environment, learns from it, and is then deployed against a real Kubernetes cluster. The simulator exists because RL requires many interactions, and training directly on a live GPU cluster would be risky, expensive, and hardware-constrained (Zhang et al., 10 Jul 2025).

Operationally, KIScaler sits outside the default Kubernetes control plane as an external autoscaling controller. The standard Kubernetes API server, scheduler, and controller manager remain intact. The paper is explicit that it does not modify the scheduler and focuses on autoscaling rather than placement internals. This design choice confines KIScaler to the control of replica counts and a workload placement preference signal, rather than to low-level scheduling policy.

The deployment setting is GPU-accelerated inference on Kubernetes under dynamic, bursty, latency-sensitive traffic. In the reported testbed, the cluster runs Triton inference pods serving MobileNetV4 models. In the single-node setup, the worker node is logically partitioned into CPU and GPU partitions: three CPU inference pods can run, and three GPU pods are defined, but only one GPU-enabled pod can be active at a time because there is a single RTX 3080 GPU. This hardware asymmetry is central to the problem formulation, because the controller must reason over heterogeneous CPU and GPU serving pools rather than over a homogeneous replica population.

2. Observability, control loop, and Kubernetes integration

At each control step, KIScaler queries Prometheus for current system metrics, forms an observation vector, runs the PPO policy to choose an action, and then updates Deployment replica counts through Kubernetes API calls. Metrics are gathered from Prometheus, which scrapes system and GPU signals, including those exported by NVIDIA’s DCGM Exporter (Zhang et al., 10 Jul 2025).

The observability model differs from default Kubernetes autoscaling by integrating GPU-level and application-level signals. Inputs include request throughput, P95 latency, GPU utilization, CPU usage, memory usage, replica counts, and trend features for latency and throughput; the system design section also says Prometheus collects GPU usage and memory metrics and pod-level resource statistics. The state description suggests that KIScaler incorporates both node- and container-level GPU utilization. This suggests an observability model aligned with inference bottlenecks rather than with CPU utilization alone.

The paper presents default Kubernetes autoscaling, especially HPA, as inadequate in this setting for four reasons. First, HPA is reactive and threshold-based. Second, HPA is mainly driven by CPU and memory metrics, while GPU utilization is not a native first-class scaling signal. Third, HPA is stateless and does not learn from prior traffic behavior. Fourth, it does not distinguish well between CPU and GPU resource types in a heterogeneous inference deployment. KIScaler is intended to address exactly these weaknesses: it is GPU-aware, latency-aware, heterogeneity-aware, and adaptive through RL.

The control interval is not numerically specified. The paper states that after an action is applied, the agent waits for a “short stabilization period” before collecting metrics and computing reward, and that each episode lasts 300 seconds. It does not give the exact step duration or Prometheus scrape interval used by the RL loop, and it does not describe explicit handling of pod cold-start delays beyond noting that bursty demand and delayed scaling responses are a challenge.

3. Reinforcement-learning formulation

The environment state at time $t$ is a 10-dimensional vector,

$s_t \in \mathbb{R}^{10},$

whose components are described textually as the number of active replicas of the inference service $n_{\text{replicas}}$; GPU utilization $u_{\text{GPU}}$ at node and container levels; 95th percentile latency $l_{\text{p95}}$; request throughput $\theta_{\text{req}}$; normalized CPU usage $u_{\text{CPU}}$; normalized memory usage $u_{\text{mem}}$; first-order trend in latency $\Delta l$; first-order trend in throughput $\Delta \theta$; normalized episode progress $s_t \in \mathbb{R}^{10},$0; and a categorical load-pattern identifier $s_t \in \mathbb{R}^{10},$1. The paper states that all features are normalized to $s_t \in \mathbb{R}^{10},$2 before being fed to the RL agent (Zhang et al., 10 Jul 2025).

The action space is multi-discrete: $s_t \in \mathbb{R}^{10},$3 with

$s_t \in \mathbb{R}^{10},$4

The policy therefore chooses how to change the number of GPU-backed replicas, how to change the number of CPU-backed replicas, and a workload placement preference bit indicating CPU-first or GPU-first bias. The finite action values implicitly clip the step size per decision epoch, but the paper does not provide more detailed hard constraints such as cooldown timers, clipping rules beyond the action set itself, or min/max replica equations.

The reward function is defined as

$s_t \in \mathbb{R}^{10},$5

Here, $s_t \in \mathbb{R}^{10},$6 is the measured P95 latency at time $s_t \in \mathbb{R}^{10},$7, $s_t \in \mathbb{R}^{10},$8 is average GPU utilization, and $s_t \in \mathbb{R}^{10},$9 penalizes over-provisioning when active replicas exceed workload demand. The coefficients $n_{\text{replicas}}$0, $n_{\text{replicas}}$1, and $n_{\text{replicas}}$2 are tunable weights controlling the tradeoff between low latency, efficient GPU use, and low scaling/provisioning cost. The paper also describes the reward in words as balancing P95 latency minimization, GPU utilization maximization, and scaling penalty reduction; in the experiments section it additionally mentions throughput and scaling smoothness, but no exact augmented reward formula is provided.

The PPO implementation is described only at a high level. The paper states that KIScaler uses PPO, that it has an actor-critic architecture, that PPO was chosen for stable training and practical performance, and that the PPO agent has 137k parameters. It does not provide the clipped PPO surrogate objective, value loss, entropy term, or exact optimization hyperparameters such as discount factor, clipping coefficient, rollout horizon, learning rate, GAE parameter, entropy regularization strength, minibatch size, or optimizer choice.

4. Training in simulation and sim-to-real deployment

Training proceeds episodically. KIScaler interacts with KISim over fixed-length episodes, each corresponding to a simulated inference session under one of four synthetic traffic patterns: ramp, periodic, random, and spike. The paper states that training spans 100 episodes, each 300 seconds long, cycling through all load patterns, with evaluation every 20 episodes (Zhang et al., 10 Jul 2025).

At each step, the agent queries Prometheus, chooses an action, applies the scaling change through Kubernetes, waits for stabilization, measures resulting metrics, computes reward, and updates the PPO policy. Training stops when convergence is detected based on reward variance or average performance improvement; the learned model is then saved to disk. The paper also states that, due to the single-GPU hardware constraint, synthetic feedback from the simulator was used during training to enable safe, accelerated learning and fallback exploration.

The sim-to-real workflow is central to the contribution. KIScaler is trained entirely in simulation using KISim and then deployed directly to the real cluster without retraining. The justification given is that KISim is Kubernetes-native, uses real GPU hardware, integrates with Prometheus and Triton, and preserves realistic resource constraints and container orchestration behavior. A plausible implication is that the contribution is as much about a training pipeline as about a control policy: the simulator is the mechanism that makes repeated RL interaction operationally feasible.

The paper’s demonstrated generalization is across traffic dynamics within the same basic deployment setting. The main mechanism described is exposure to multiple synthetic traffic patterns, and the agent is also given the load-pattern identifier $n_{\text{replicas}}$3 as part of the state. The paper does not mention domain randomization in the conventional sim-to-real sense, nor meta-learning, nor explicit cross-workload transfer beyond MobileNetV4, and it does not say that the policy is trained over multiple clusters or hardware types.

5. Empirical results, strengths, and failure modes

During training, KIScaler’s moving average reward improves from $n_{\text{replicas}}$4 to $n_{\text{replicas}}$5 over 100 episodes, which the paper reports as a $n_{\text{replicas}}$6 increase, with a peak of $n_{\text{replicas}}$7. Pattern-specific learning curves show that periodic and ramp traffic yield the highest and most stable rewards, random is intermediate, and spike is the hardest (Zhang et al., 10 Jul 2025).

In deployment, KIScaler consistently reduces P95 latency versus static baselines under most traffic patterns. The reported table values are:

Traffic pattern	KIScaler P95 latency	Baseline comparison
Ramp	$n_{\text{replicas}}$8 ms	vs GPU $n_{\text{replicas}}$9 ms, CPU $u_{\text{GPU}}$0 ms
Periodic	$u_{\text{GPU}}$1 ms	vs GPU $u_{\text{GPU}}$2 ms, CPU $u_{\text{GPU}}$3 ms
Random	$u_{\text{GPU}}$4 ms	vs GPU $u_{\text{GPU}}$5 ms, CPU $u_{\text{GPU}}$6 ms
Spike	$u_{\text{GPU}}$7 ms	vs GPU $u_{\text{GPU}}$8 ms, CPU $u_{\text{GPU}}$9 ms

The corresponding speedups stated in the paper are $l_{\text{p95}}$0 and $l_{\text{p95}}$1 for ramp, $l_{\text{p95}}$2 for periodic, and $l_{\text{p95}}$3 and $l_{\text{p95}}$4 for random. Under spike traffic, however, KIScaler is worse in the reported table, with $l_{\text{p95}}$5 and $l_{\text{p95}}$6 relative “speedup,” meaning slowdown. The paper is explicit that spike traffic is the most difficult pattern because sudden bursts create delayed scaling responses.

The abstract additionally claims that KIScaler reacts $l_{\text{p95}}$7 faster to bursty traffic and improves average GPU utilization by $l_{\text{p95}}$8, but the detailed supporting table for those exact numbers is not included in the provided text. Those claims therefore remain paper-reported summary results rather than numerically unpacked findings.

Several limitations are also explicit. The prototype runs on a single-node cluster with one RTX 3080 GPU; only one GPU pod can actually be scheduled concurrently; this restricts the action space and introduces resource imbalance. Quantitative HPA comparisons are discussed in the abstract and introduction, but the detailed quantitative tables shown in the provided content report results mainly against GPU-only and CPU-only baselines. The paper also omits many PPO hyperparameters and operational safeguards. Taken together, these points indicate that the evidence for KIScaler is strongest for single-node, heterogeneous CPU/GPU inference under controlled synthetic traffic, and weakest for highly abrupt spike traffic and for broad infrastructure generalization.

6. Relation to autoscaling research and conceptual significance

KIScaler differs from traditional autoscaling in several fundamental ways. HPA is described as a reactive feedback controller centered on threshold violations, usually CPU and memory. KIScaler is a learned sequential decision policy that reasons over trends, episode progress, and accumulated reward, and explicitly includes GPU utilization and P95 latency in both state and reward. HPA usually scales a homogeneous replica count; KIScaler operates over heterogeneous CPU and GPU replica counts plus a placement preference signal (Zhang et al., 10 Jul 2025).

Within the broader autoscaling literature, KIScaler occupies a distinct point in the design space. DeepScaler is a holistic, dependency-aware, prediction-driven microservice autoscaler that jointly scales multiple interacting services using an adaptive graph and a spatiotemporal GNN, with reported average reductions of $l_{\text{p95}}$9 in SLA violations and $\theta_{\text{req}}$0 in resource spending. NeuroScaler is a predictive, MPC-based Kubernetes autoscaler that aggregates multi-tier telemetry and optimizes energy use under a P95 latency SLO, with a reported $\theta_{\text{req}}$1 reduction in energy consumption relative to HPA while maintaining target latency (Meng et al., 2023, Chaves et al., 9 Feb 2026).

Against those systems, KIScaler is narrower in scope but more specialized in resource heterogeneity. It is focused on GPU-accelerated inference, heterogeneous CPU/GPU replica control, Prometheus-observable GPU metrics, and sim-to-real PPO training. It is not presented as a multi-service dependency model like DeepScaler, nor as an energy-first MPC orchestrator like NeuroScaler. Instead, its central claim is that richer state representation, GPU-aware observability, learned latency/resource tradeoffs, and repeated simulation-based training together yield a practical alternative to CPU-threshold autoscaling for latency-sensitive inference services.

This suggests that KIScaler’s significance lies less in general autoscaling theory than in a specific orchestration problem: Kubernetes inference serving where GPU scarcity, heterogeneous replica pools, and latency objectives interact in ways that default threshold controllers do not capture well. The paper positions it as a step beyond reactive autoscaling toward intelligent orchestration for scalable GPU-accelerated environments, while simultaneously acknowledging that the present evidence is bounded by a constrained single-node, single-GPU testbed.