RLScale-Bench: DRL Autoscaling Benchmark

• RLScale-Bench is a reproducible benchmark and evaluation protocol for deep reinforcement learning in adaptive resource control via Kubernetes Horizontal Pod Autoscaling.
• It standardizes comparisons using matched architectures, training budgets, and multi-workload tests modeled as a discrete-action Markov Decision Process with cost and SLO constraints.
• Empirical results show that a well-calibrated HPA outperforms DRL algorithms on cost efficiency, with RL offering advantages in bursty traffic scenarios.

Searching arXiv for RLScale-Bench and closely related autoscaling benchmark work to ground the article. RLScale-Bench is a reproducible benchmark and evaluation protocol for deep reinforcement learning in adaptive resource control, instantiated on Kubernetes Horizontal Pod Autoscaling, and designed to answer a specific question under controlled and realistic conditions: when, if ever, do standard deep RL algorithms actually beat a properly calibrated rule-based autoscaler? In the benchmark, an agent allocates compute to a dynamic workload under cost and service-level constraints; the setting is autoscaling for a single microservice, where the control variable is the number of pod replicas, the objective is to minimize infrastructure cost, and the constraints are deployment-level latency and reliability thresholds. Its central empirical result is that a realistically calibrated HPA-like rule-based controller is more cost-efficient than all six tested DRL algorithms on every workload, while RL provides value only on certain bursty or flash-like regimes when constraint violations are much more costly than extra resources (Zhang et al., 26 May 2026).

1. Definition and benchmark scope

RLScale-Bench is simultaneously a Gym-style MDP environment modeling autoscaling or adaptive resource control for a microservice, a standardized comparison of six mainstream DRL algorithms under matched architectures and training budgets, a strong calibrated rule-based baseline built from Kubernetes HPA with realistic parameters, a multi-workload multi-seed evaluation protocol, and a distribution-shift test in which policies are trained on one workload and evaluated on several others (Zhang et al., 26 May 2026).

The benchmark studies the autoscaling problem as a sequential decision process. At each decision step, the controller must choose how many replicas to provision in response to a dynamic request stream. The trade-off is between infrastructure cost and service-level behavior, operationalized through latency and error-rate conditions. In this setting, RLScale-Bench is not merely an algorithm leaderboard; it is an evaluation framework structured to test whether apparent gains from DRL survive comparison against a properly calibrated production-faithful autoscaler.

A defining feature is its emphasis on equal experimental footing. PPO, DQN, A2C, SAC, TD3, and DDPG are evaluated under matched architectures, identical training budgets, and identical reward functions, while the comparison baseline is not intentionally weakened. This design directly targets a recurrent issue in autoscaling literature: comparisons in which learned controllers are measured against under-tuned rule-based heuristics. RLScale-Bench instead treats baseline calibration as part of the scientific question rather than as an experimental convenience.

2. Formalization as an MDP

The environment is modeled as a Markov Decision Process with discrete time, $t = 0, 1, \dots$, in which the agent observes state $s_t$, chooses action $a_t$, transitions to $s_{t+1}$ according to workload and system dynamics, and receives reward $r_t$ (Zhang et al., 26 May 2026).

The state is a 6-dimensional vector: $$s_t = \bigl[ \text{CPU}_t,\; \text{Mem}_t,\; \text{QPS}_t,\; p95_t,\; \text{ErrRate}_t,\; \text{Replicas}_t \bigr].$$ Its components are CPU utilization, memory usage, current requests per second, 95th-percentile latency, error rate, and current replica count. The details specify the following ranges where stated: $\text{CPU}_t \in [0,100]$, $\text{Mem}_t \in [0,512]$, $\text{ErrRate}_t \in [0,1]$, and $\text{Replicas}_t \in [1,10]$. The dynamics of CPU, latency, and errors are calibrated to measurements from a real Kubernetes cluster.

The underlying control variable is the change in replica count. The environment uses a discrete action space: $s_t$0 meaning scale down or up by at most two replicas at a step, with replica counts clamped to $s_t$1. For PPO, A2C, and DQN, this is the native Discrete(5) action space. For SAC, TD3, and DDPG, the benchmark uses a Box(-1, 1) action mapped to the five discrete actions through uniform bins with boundaries at $s_t$2. The paper identifies this DiscreteToBoxWrapper as the source of the action-space mismatch problem discussed in the empirical findings.

The objective is to minimize infrastructure cost subject to service-level constraints. The SLO conditions are:

• latency SLO: $s_t$3 ms
• reliability SLO: $s_t$4

An SLO violation occurs when either threshold is breached. The instantaneous cost is proportional to the number of replicas. In the calibrated environment, the cost per replica per 15-second decision step is $s_t$5 USD, which roughly matches $s_t$6 per vCPU-hour. Total cost over an episode of length $s_t$7 is: $s_t$8

Reward is a negative cost plus a penalty for SLO violations: $s_t$9 with $a_t$0. Rewards are normalized to $a_t$1 via min-max scaling for training stability. The benchmark does not explicitly solve a constrained MDP; instead, the constraints are folded into a scalar reward via the penalty term. This suggests that RLScale-Bench is designed to study practical DRL behavior under a common engineering formulation rather than to compare specialized CMDP algorithms.

3. Kubernetes instantiation, workloads, and protocol

RLScale-Bench is instantiated on Kubernetes Horizontal Pod Autoscaling. The controlled resource is the number of pod replicas for a single microservice, and metrics are generated from a calibrated simulator based on measurements from a real Online Boutique deployment on a Kind cluster with Prometheus monitoring (Zhang et al., 26 May 2026). The details provide conceptual models such as: $a_t$2 and

$a_t$3

with sharp degradation under overload.

The benchmark defines six synthetic but calibrated workload patterns spanning steady, periodic, non-stationary, bursty, ramping, and flash-like traffic.

Workload	Pattern description	Range
Constant	Flat rate + noise	$a_t$4 requests/min
Periodic	Sinusoidal, 10-min cycle	$a_t$5–$a_t$6 requests/min
Variable	Random walk	$a_t$7–$a_t$8 requests/min
Bursty	Baseline + random Poisson bursts	$a_t$9–$s_{t+1}$0 requests/min
Ramp	Linear ramp-up	$s_{t+1}$1 requests/min
Flash	Sudden 3× spike and drop	$s_{t+1}$2 requests/min

Training is online for 50,000 timesteps on the variable workload. The decision interval is 15 seconds, episode length is 240 steps or 60 minutes, the network is an MLP with two hidden layers of 256 units for all algorithms, and five random seeds are used: $s_{t+1}$3. After training, each policy is evaluated on all six workloads, each for 240 steps, and results are reported as mean $s_{t+1}$4 std over five seeds. The evaluation metrics are total cost in USD, number of SLO-violating steps, and mean replica count.

A central component is the distribution-shift generalization test. Training occurs only on the variable workload, while evaluation on constant, periodic, bursty, ramp, and flash workloads tests generalization under workload shift without retraining. Overall, the benchmark comprises 6 algorithms $s_{t+1}$5 5 seeds $s_{t+1}$6 6 workloads plus 2 baselines $s_{t+1}$7 5 seeds $s_{t+1}$8 6 workloads, approximately 240 runs.

4. Algorithms and baselines

RLScale-Bench evaluates six standard DRL algorithms implemented in Stable-Baselines3: PPO, A2C, DQN, SAC, TD3, and DDPG (Zhang et al., 26 May 2026). PPO and A2C are on-policy discrete methods; DQN is off-policy discrete; SAC, TD3, and DDPG are off-policy continuous methods. The common configuration is an MLP with layers $s_{t+1}$9, ReLU activations, a 50k-step training budget, and identical reward shaping and normalization. Algorithm-specific hyperparameters are mostly SB3 defaults.

The action representation differs sharply across these groups. PPO, A2C, and DQN operate directly in the native Discrete(5) action space. SAC, TD3, and DDPG instead produce a Box(-1,1) action that is wrapped to discrete actions via fixed bins. This wrapper is not an incidental implementation detail; it is the basis for one of the benchmark’s main findings about action-space mismatch in resource-allocation tasks.

The key non-learning baseline is a production-faithful Kubernetes Horizontal Pod Autoscaler. Its target CPU utilization is 70%, described as a typical production setting. The simplified scaling rule is: $r_t$0 followed by clamping to $r_t$1. Decisions are made every 15 seconds, matching the RL agents.

“Properly calibrated” is defined concretely in three ways: the target CPU utilization of 70% is chosen to provide a safety margin for bursts while maintaining cost-efficiency, bounds and time step match real deployments, and the baseline is tuned with at least as much care as RL hyperparameters. The benchmark also includes a random baseline that samples uniformly from $r_t$2, serving as a lower bound and sanity check.

This baseline construction is methodologically central. RLScale-Bench is not framed as comparing learned control to a straw-man heuristic; it is framed as comparing learned control to a rule-based controller that reflects realistic tuning practice.

5. Empirical findings

The principal finding is that HPA has the lowest cost on all six workloads. The cost numbers reported for a 60-minute run are: constant $r_t$3, periodic $r_t$4, variable $r_t$5, bursty $r_t$6, ramp $r_t$7, and flash $r_t$8 USD (Zhang et al., 26 May 2026). Example best RL costs are consistently higher: on constant, best RL is approximately $r_t$9; on periodic, approximately $$s_t = \bigl[ \text{CPU}_t,\; \text{Mem}_t,\; \text{QPS}_t,\; p95_t,\; \text{ErrRate}_t,\; \text{Replicas}_t \bigr].$$0; on variable, approximately $$s_t = \bigl[ \text{CPU}_t,\; \text{Mem}_t,\; \text{QPS}_t,\; p95_t,\; \text{ErrRate}_t,\; \text{Replicas}_t \bigr].$$1; on bursty, DQN is approximately $$s_t = \bigl[ \text{CPU}_t,\; \text{Mem}_t,\; \text{QPS}_t,\; p95_t,\; \text{ErrRate}_t,\; \text{Replicas}_t \bigr].$$2; on ramp, best RL is approximately $$s_t = \bigl[ \text{CPU}_t,\; \text{Mem}_t,\; \text{QPS}_t,\; p95_t,\; \text{ErrRate}_t,\; \text{Replicas}_t \bigr].$$3–$$s_t = \bigl[ \text{CPU}_t,\; \text{Mem}_t,\; \text{QPS}_t,\; p95_t,\; \text{ErrRate}_t,\; \text{Replicas}_t \bigr].$$4; on flash, best RL is approximately $$s_t = \bigl[ \text{CPU}_t,\; \text{Mem}_t,\; \text{QPS}_t,\; p95_t,\; \text{ErrRate}_t,\; \text{Replicas}_t \bigr].$$5. RL agents generally use more replicas on average, $$s_t = \bigl[ \text{CPU}_t,\; \text{Mem}_t,\; \text{QPS}_t,\; p95_t,\; \text{ErrRate}_t,\; \text{Replicas}_t \bigr].$$6–$$s_t = \bigl[ \text{CPU}_t,\; \text{Mem}_t,\; \text{QPS}_t,\; p95_t,\; \text{ErrRate}_t,\; \text{Replicas}_t \bigr].$$7, than HPA, $$s_t = \bigl[ \text{CPU}_t,\; \text{Mem}_t,\; \text{QPS}_t,\; p95_t,\; \text{ErrRate}_t,\; \text{Replicas}_t \bigr].$$8–$$s_t = \bigl[ \text{CPU}_t,\; \text{Mem}_t,\; \text{QPS}_t,\; p95_t,\; \text{ErrRate}_t,\; \text{Replicas}_t \bigr].$$9.

The SLO results are more differentiated. On steady or predictable workloads, RL can match but not beat HPA. On constant traffic, HPA has $\text{CPU}_t \in [0,100]$0 violations, and PPO, A2C, and SAC also have $\text{CPU}_t \in [0,100]$1. On periodic traffic, HPA has $\text{CPU}_t \in [0,100]$2, PPO, A2C, and SAC also have $\text{CPU}_t \in [0,100]$3, while DQN is $\text{CPU}_t \in [0,100]$4. On ramp traffic, HPA has $\text{CPU}_t \in [0,100]$5, and PPO and SAC also have $\text{CPU}_t \in [0,100]$6. On the training workload, variable, HPA records $\text{CPU}_t \in [0,100]$7 violations; PPO is $\text{CPU}_t \in [0,100]$8, A2C is $\text{CPU}_t \in [0,100]$9, DQN is $\text{Mem}_t \in [0,512]$0, and SAC is $\text{Mem}_t \in [0,512]$1, substantially fewer than HPA but at higher cost.

The bursty workload is the clearest case in which RL helps. HPA has $\text{Mem}_t \in [0,512]$2 violations, PPO has $\text{Mem}_t \in [0,512]$3, A2C has $\text{Mem}_t \in [0,512]$4, and SAC has $\text{Mem}_t \in [0,512]$5. PPO therefore reduces violations by approximately 54% relative to HPA, at approximately 24% higher cost. On flash traffic, however, RL does not clearly beat HPA: HPA has $\text{Mem}_t \in [0,512]$6 violations, PPO has $\text{Mem}_t \in [0,512]$7, and SAC has $\text{Mem}_t \in [0,512]$8, with rankings depending on seeds.

A second major finding concerns action representation. Discrete-action algorithms dramatically outperform continuous-action algorithms in SLO compliance, by one to two orders of magnitude. SAC is viable but costlier; TD3 exhibits extreme variance, with many seeds showing hundreds of violations, such as $\text{Mem}_t \in [0,512]$9–$\text{ErrRate}_t \in [0,1]$0 on periodic and ramp workloads; DDPG collapses to “always use one replica,” yielding cost around $\text{ErrRate}_t \in [0,1]$1 but $\text{ErrRate}_t \in [0,1]$2–$\text{ErrRate}_t \in [0,1]$3 violations per workload. The paper attributes this to action-space mismatch: the underlying control variable is integer replicas, continuous actors output real values that are snapped to discrete bins, reward is constant within a bin with respect to the continuous action, and small changes at bin edges cause discrete jumps. The result is aliased reward signal and disrupted learning.

A third major finding is that no single algorithm dominates across workloads. HPA is rank 1 on constant, periodic, and ramp by SLO violations but drops to about rank 5 on bursty traffic. PPO is typically among the top performers and relatively stable. DQN is usually the weakest among viable discrete methods. SAC is sometimes best, especially on variable workloads, but is consistently more expensive. TD3 and DDPG are effectively unusable in this setting. Rankings can shift by up to four positions between steady-state and bursty traffic.

6. Generalization, methodological lessons, and relation to adjacent benchmark design

The generalization experiment is deliberately simple: all RL agents are trained only on the variable random-walk workload and then evaluated on the other workload families without retraining (Zhang et al., 26 May 2026). PPO generalizes well to easier workloads, achieving $\text{ErrRate}_t \in [0,1]$4 SLO violations on constant, periodic, and ramp traffic despite not having seen those exact patterns during training. On bursty and flash workloads, performance deteriorates. SAC generalizes relatively better in low-violation terms on dynamic workloads, with $\text{ErrRate}_t \in [0,1]$5 on variable and $\text{ErrRate}_t \in [0,1]$6 on bursty, but uses more replicas and thus incurs higher cost. The authors describe the extra resource usage of robust policies as a “transfer tax”: algorithms that generalize more robustly under distribution shift tend to pay for robustness by allocating more capacity.

The broader methodological conclusion is that the bottleneck in RL-based resource control is not algorithm selection but baseline calibration, reward engineering, and realistic evaluation protocols. Baseline calibration matters because a carefully tuned HPA is difficult to beat on cost and performs perfectly on many steady-workload SLO metrics. Reward engineering matters because a single scalar penalty for violations may not capture operator preferences, and pathological policies can optimize cost while ignoring SLO behavior. Realistic evaluation matters because single-seed results are unreliable, and evaluating only on the training workload obscures ranking changes under distribution shift.

The practical guidelines stated in the paper are correspondingly specific: always start from and compare against a calibrated rule-based controller, prefer discrete-action RL methods such as PPO and A2C for replica-count control, and use RL mainly when workloads are highly unpredictable and the cost of violations is large relative to infrastructure cost. A plausible implication is that RLScale-Bench recasts autoscaling research from “which DRL algorithm wins” to “under what operational assumptions does learning outperform careful systems tuning.”

RLScale-Bench also sits in a broader benchmark-design conversation. BatchBench, a position paper on autoscaling policies in big-data batch processing, proposes an open benchmarking framework intended to place rule-based, learned, and agentic autoscaling policies on equal experimental footing, with workload taxonomy, a standardized policy interface, held-out workload splits, and first-class accounting for inference cost (Budigi et al., 12 May 2026). The data explicitly notes that RLScale-Bench would live in almost exactly the same problem space as BatchBench. The difference is that RLScale-Bench is an empirical benchmark instantiated on Kubernetes HPA for a single-microservice autoscaling problem, whereas BatchBench is a proposed framework for big-data batch processing. This suggests a convergence toward autoscaling benchmarks that emphasize workload diversity, stronger baselines, reproducibility, and out-of-distribution evaluation rather than narrow per-paper comparisons.