When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control

Abstract: A properly calibrated rule-based autoscaler can beat every one of six mainstream deep reinforcement learning (DRL) algorithms on cost across every workload we test - so when, if ever, does DRL actually help? We study this in RLScale-Bench, a reproducible benchmark and evaluation protocol for DRL on adaptive resource control, where an agent allocates compute to a dynamic workload under cost and service-level constraints. We evaluate PPO, DQN, A2C, SAC, TD3, and DDPG under matched architectures, training budgets, and reward functions against a calibrated rule-based baseline across six workload patterns and five seeds (240 runs), instantiate the benchmark on Kubernetes Horizontal Pod Autoscaling, and probe distribution-shift generalization. Three findings challenge common assumptions: (i) the calibrated controller achieves the lowest cost on all six workloads, though it trails the best RL agents on bursty and flash traffic; (ii) discrete-action algorithms outperform continuous-action ones by one to two orders of magnitude in constraint violations due to action-space mismatch; and (iii) no single algorithm dominates across workloads, with rankings shifting by up to four positions. The bottleneck in RL-based resource control is not algorithm selection but baseline calibration, reward engineering, and realistic evaluation protocols.

• The paper shows that calibrated HPA consistently achieves lower infrastructure costs and superior SLO compliance on steady workloads compared to DRL agents.
• It demonstrates that discrete-action DRL methods, particularly PPO, reduce SLO violations by 54% on bursty workloads, outperforming continuous-action methods.
• The study emphasizes that robust baseline calibration and thoughtful reward engineering are critical for reliable adaptive resource control under varying workload conditions.

Deep RL vs. Calibrated Baselines in Adaptive Resource Control: Analysis of RLScale-Bench

Benchmark Methodology and Experimental Setup

The paper presents RLScale-Bench, a rigorous benchmark protocol for evaluating deep reinforcement learning (DRL) and rule-based controllers on adaptive resource management, particularly within the context of Kubernetes Horizontal Pod Autoscaling (HPA). The evaluation protocol eliminates confounding factors by ensuring matched neural architectures, identical reward functions, and equal training budgets across six representative DRL algorithms (PPO, DQN, A2C, SAC, TD3, DDPG). Extensive reproducibility is achieved through 5-seed training and evaluation, 240 total runs, and deployment across six diverse traffic patterns: constant, periodic, variable, bursty, ramp, and flash loads.

The state space consists of a six-dimensional vector with relevant system observables, while the action space models indivisible resource allocation through a discretized five-action set, mapping to increasing or decreasing pod replicas. The reward function captures both infrastructure cost and SLO (Service Level Objective) compliance, with a high penalty for SLO violations.

Figure 1: RLScale-Bench benchmarking pipeline—comprising matched RL agents and calibrated baselines, multi-seed evaluation, workload shifts, and reporting of key empirical findings.

Algorithm-specific hyperparameters conform to Stable-Baselines3 defaults with controlled deviations, and the benchmark strictly enforces fair comparator conditions. Critically, the calibrated HPA baseline is tuned following production best practices (70% CPU target, realistic clamping), providing a non-trivial benchmark for DRL agents, unlike prior work with simplistic or uncalibrated baselines.

Main Empirical Findings

Baseline Competitiveness and Cost-Efficiency

A major result is that the properly calibrated HPA controller achieves the lowest infrastructure cost on every evaluated workload. On steady and mildly dynamic workloads, it maintains full SLO compliance, consistently achieving lower mean pod counts than all DRL agents. The cost-SLO trade-off continually favors the baseline, except on bursty or flash workloads:

Figure 3: Costs and SLO violations for viable algorithms—HPA achieves minimum cost across workloads; RL agents only outperform on SLO compliance in highly dynamic settings.

DRL Superiority on Unpredictable Workloads

When traffic is highly non-stationary (bursty or flash), DRL—particularly PPO—excels at preemptive scaling, cutting SLO violations by 54% relative to the calibrated baseline, albeit at a 24% higher cost. However, these gains are restricted to scenarios where violation costs far outweigh minor resource overspending.

Discrete vs. Continuous Action Algorithms

The performance difference between discrete and continuous DRL algorithms is quantitatively stark: continuous-action methods (SAC, TD3, DDPG) exhibit one to two orders of magnitude more SLO violations compared to discrete-action methods (PPO, DQN, A2C). This is traced to the action-space mismatch—continuous algorithms, when forced onto a discrete action manifold, suffer from reward aliasing and poor gradient signal, impairing sample efficiency and stability.

Figure 4: Discrete-action DRL algorithms (PPO, DQN, A2C) dramatically outperform continuous algorithms (SAC, TD3, DDPG) on SLO compliance—median SLO violations are >100× lower.

No Universally Dominant Algorithm

Algorithmic ranking shifts substantially with workload distribution. No RL or rule-based method is universally dominant. For instance, HPA is optimal for constant and periodic loads but performs worse in bursty scenarios; SAC is Pareto optimal on dynamic workloads for SLO compliance but overprovisions resources; PPO is most rank-stable, but never strictly top-performing across all regimes.

Figure 5: Algorithm ranking by SLO violations fluctuates widely across workload types; no method holds rank 1 universally, indicating context-dependent optimality.

Generalization Under Workload Shift

Distribution-shift generalization experiments reveal that DRL models trained on one workload typically underperform on others, highlighting overfitting and lack of robustness. The calibrated baseline, by contrast, is invariant to workload distribution, as it does not rely on specific traffic traces seen during training.

Cost-SLO Pareto Frontier and Reward Engineering

Pareto analysis quantitatively confirms that the most cost-effective strategies do not align with those minimizing SLO violations. RL agents can push the envelope on SLO adherence, yet only at increased infrastructure spend. The degenerate behavior of DDPG, which collapses to a minimal-replica policy with catastrophic SLO breaches, demonstrates the pitfalls of naïve cost minimization lacking hard constraints.

Figure 2: Cost-SLO Pareto front—HPA is optimal on cost, while SAC delivers best SLO on dynamic workloads. DDPG’s single-replica policy fails SLO catastrophically.

Implications for Practice and Benchmarking

These results have direct implications for both practitioners and RL algorithm designers:

• Calibrating Baselines is Critical: In many scenarios, realistic, well-tuned rule-based controllers are competitive or superior, especially given their robustness to distribution shift.
• Discrete Algorithms for Discrete Control: Tasks involving indivisible resource units mandate discrete-action RL; continuous-action methods (unless fundamentally re-architected) are unsuitable.
• RL’s role is SLO-centric: The added value of DRL manifests on workloads with unpredictable, high-variance demand, provided SLO adherence is mission-critical compared to moderate cost increases.
• Workload Diversity and Seed Variance: Proper benchmarking requires extensive coverage of workload scenarios and variance estimates across seeds; conclusions drawn from single seeds or limited patterns are misleading.

The findings also emphasize the importance of reward engineering and environment calibration in RL applications—failure to design domain-appropriate reward structures can result in misleading improvements or degenerate behavior.

Figure 6: Composite performance heatmap showing that HPA and SAC define the viable performance frontier, with DDPG consistently underperforming on dynamic workloads.

Theoretical and Future Directions

The study underscores that algorithmic selection is not the primary bottleneck for DRL in adaptive resource management. Advances will hinge on principled reward engineering, deeper modeling of system dynamics under workload shift, and perhaps hybrid approaches combining fast-reactive baselines with RL-driven anticipatory adjustments. The proposed extension to graph-structured, multi-service environments and integration of safe planning with world models represents a promising research direction.

Conclusion

The RLScale-Bench study robustly demonstrates that properly calibrated rule-based controllers remain formidable in adaptive resource control, limiting the real-world advantage of deep RL to highly volatile, SLO-critical regimes. Discrete-action RL algorithms outperform continuous counterparts due to action-space alignment. Algorithmic rankings are unstable under distribution shift, invalidating claims based solely on training scenarios. Progress in this domain now critically depends on strong baseline calibration, robust evaluation protocols, and judicious reward design. The benchmark and findings provide a foundation for advancing practical and theoretical research in resource management automation.