Sample-Efficient Pareto Front Modeling for Energy-Aware Reinforcement Learning Using Bayesian Optimization

Abstract: Industrial automation increasingly demands control strategies that balance operational performance with strict energy efficiency requirements. A common approach to solving this multi-objective problem, particularly within the framework of reinforcement learning (RL), is to formulate a single, scalar reward function that linearly combines the competing objectives. However, the manual weighting of these different objectives is heavily reliant on domain intuition, incredibly time-consuming, prone to human bias, and frequently fails to uncover optimal trade-off solutions. This work addresses the critical challenge of automating the weight selection process to systematically and efficiently discover the Pareto front of optimal trade-off policies. We formulate the weight selection process as a multi-objective Bayesian optimization (MOBO) problem and evaluate its sample efficiency against a standard uniform grid search baseline. Using a physical Quanser Aero 2 testbed configured for 1-DoF pitch control, our results demonstrate that the MOBO approach, utilizing the expected hypervolume improvement (qEHVI) acquisition function, consistently outperforms uniform grid sampling. MOBO achieves superior hypervolume and maximum spread, successfully identifying high-quality, diverse trade-off policies with a reduced evaluation budget, thereby enabling highly efficient energy-aware control in complex mechatronic systems.

• The paper demonstrates a sample-efficient Pareto front discovery using MOBO, reducing RL training runs by approximately 64%.
• It integrates a qEHVI acquisition function with PPO agents to optimize energy consumption and tracking performance, outperforming grid search.
• The study provides a scalable, automated trade-off framework for balancing energy use and performance in industrial RL applications.

Sample-Efficient Multi-Objective Bayesian Optimization for Energy-Aware RL in Industrial Mechatronics

Introduction

This work presents a paradigm for automating trade-off policy discovery in continuous control settings with energy and performance as conflicting objectives, specifically targeting the challenges inherent in industrial automation. By formulating the multi-objective reward weighting process as a multi-objective Bayesian optimization (MOBO) problem, the approach circumvents the inefficiencies of manual scalarization weighting and exhaustive grid search strategies. The validation is anchored in a physical Quanser Aero 2 mechatronic testbed operating in a single Degree-of-Freedom (DoF) control configuration, typical for industrial benchmarking. The empirical study quantifies the sample efficiency, dominated hypervolume, and front coverage realized by MOBO with a qEHVI acquisition function against uniform parameter grid search baselines.

Figure 1: The Quanser Aero 2 mechatronic testbed used in this study; evaluation focuses on 1-DoF pitch control.

Problem Formulation and System Architecture

The control task involves a PPO agent that must regulate the pitch of the Aero 2 system while minimizing electrical energy consumed by the actuators. The state space encapsulates the current pitch, angular velocity, and reference target, while the action space covers continuous DC motor voltages $\in [-24, 24]$V. The scalarized reward blends tracking error and normalized power consumption, controlled by a weight $\alpha \in [0, 1]$. Varying $\alpha$ modulates the trade-off from aggressive tracking to energy saving.

Manually tuning $\alpha$ to identify satisfactory operating regimes is computationally expensive and produces clusters of Pareto-suboptimal policies due to the highly nonlinear mapping from weight settings to realized performance objectives. Training RL agents on hardware further exacerbates sample inefficiency due to physical wear and operational cost, reinforcing the need for systematic, sample-efficient Pareto front discovery.

Bayesian Optimization for RL Reward Scalarization

MOBO is leveraged as an outer-loop controller, encapsulating RL agent learning as a black-box process parameterized by $\alpha$. The surrogate model, a Gaussian process, is updated in each trial with the actual tracking error and power consumption observed after RL convergence. The qEHVI acquisition function prioritizes sampling in regions that are expected to maximally increase the dominated hypervolume of the current Pareto approximation. This adaptivity is fundamentally superior to a-priori uniform exploration of the weight space.

The experimental protocol restricts all optimization strategies to an 11-sample budget per run. For grid search, $\alpha$ is sampled uniformly. For MOBO, 5 Sobol-initialized quasi-random samples are refined with 6 adaptive MOBO-guided selections. The optimization is orchestrated using the Ax/BoTorch API, tightly integrated with Stable Baselines3 PPO implementations.

Results

Pareto Front Discovery and Sample Efficiency

MOBO delivers robust improvements in all salient multi-objective metrics. The final hypervolume realized by MOBO was 769.48 units compared to the grid search's 762.78, indicating stronger non-dominated policy coverage. Importantly, MOBO achieves the grid search’s final hypervolume in just 4 evaluations—a reduction of approximately 64% in required RL training runs.

Figure 2: Comparison between grid search and MOBO-selected reward weights. MOBO covers broader and more diverse regions of the true Pareto front.

Figure 3: Hypervolume progression per trial reveals MOBO's rapid front expansion compared to grid search.

Figure 4: Maximum spread progression demonstrates that MOBO rapidly approaches the full objective range covered by grid search.

The non-uniformity of the $\alpha \rightarrow$ objective mapping is evident: grid search samples are redundant in parameter space, missing "knee" regions where optimal trade-offs lie. MOBO, in contrast, adaptively densifies sampling along the emerging Pareto frontier, resulting in lower spacing (11.22 vs 12.26, lower is better) and the same or slightly increased spread of solutions (51.33 vs 51.31).

Practical and Theoretical Implications

This methodology directly benefits physically-constrained RL applications, allowing practitioners to surface diverse energy/performance trade-offs with minimal system strain. By eliminating the trial-and-error manual tuning loop, MOBO makes the Pareto front accessible for real-world deployment and facilitates systematic benchmarking.

Theoretically, the automated discovery of optimal RL reward weightings via MOBO establishes a scalable protocol for black-box multi-objective control in robotics and related industrial domains. It further highlights the inadequacy of uniform parameter sweep methods where policy landscapes are highly nonlinear and high-fidelity sample acquisition is costly.

Limitations and Future Directions

Boundary regime exploration, especially for low $\alpha$ values favoring performance, frequently results in aggressive "bang-bang" controls that can physically damage hardware. Integrating formal safe RL constraints into the BO outer loop is essential to ensure exploration does not compromise physical assets during sample-efficient Pareto surface modeling. Extending evaluation to the coupled 2-DoF pitch/yaw system and transferring discovered weightings to novel platforms are promising directions, supporting fast policy initialization and domain adaptation.

Conclusion

Multi-objective Bayesian Optimization is demonstrated as a highly effective meta-controller for energy-aware RL reward weighting in industrial mechatronics, increasing both sample efficiency and Pareto frontier quality while controlling for physical system wear. The integration of qEHVI-driven MOBO with high-fidelity RL workflows defines a robust protocol for practical, automated trade-off discovery in complex real-world control tasks. This closes an essential methodological gap for industrial adoption of RL in energy-constrained environments and suggests a framework broadly applicable to similar high-stakes, multi-objective RL problems.

(2607.03140)