Combined ES–DRL Controller

• The combined ES–DRL controller integrates DRL for fast, high-dimensional control with bounded ES as a robust fallback during out-of-distribution events.
• A supervisory mechanism, using latent OOD detection and safety thresholds, arbitrates between the fast nominal DRL and the adaptive model-independent ES components.
• Applications in particle accelerator tuning and robotic manipulation show that the hybrid design effectively balances speed, safety, and robustness under dynamic conditions.

Searching arXiv for the primary paper and closely related hybrid ES–DRL control work. A combined ES–DRL controller is a hybrid control architecture that couples deep reinforcement learning (DRL) with bounded extremum seeking (ES) so that fast, high-dimensional control learned from data can be retained in nominal operation while model-independent adaptive control is available when the system departs from the training regime. In the formulation developed for nonlinear time-varying systems, the DRL component supplies rapid in-distribution actions, bounded ES supplies robust fallback behavior under distribution shift, and a supervisory mechanism determines which controller is active at test time (Saxena et al., 9 Jun 2026). Closely related formulations also appear in robotic manipulation under distribution shift, where DRL is used before contact and bounded ES after contact, and in time-varying accelerator control, where ES and DRL are combined to exploit complementary strengths in speed and robustness (Saxena et al., 1 Apr 2026, Saxena et al., 2 Oct 2025).

1. Definition and scope

The term “combined ES–DRL controller” denotes a controller in which a learned policy and an extremum-seeking law are not merely co-located, but are assigned distinct functional roles within a single supervisory architecture. In the main accelerator formulation, the architecture uses a deep RL policy for fast, high-dimensional control when the system operates near its training regime, and a bounded extremum seeking controller as a robust, model-independent fallback when the system is out-of-distribution (OOD) (Saxena et al., 9 Jun 2026). The switching decision is supervised by a variational autoencoder (VAE) and a Mahalanobis-distance–based latent OOD detector trained on in-distribution observations from the nominal operating regime (Saxena et al., 9 Jun 2026).

This family of designs is motivated by a recurring asymmetry between the two components. DRL can learn coordinated control policies in large continuous action spaces and deploy them at low latency, but its performance can degrade when deployment conditions differ from the training distribution. Bounded ES, by contrast, is model-free and robust to unknown and time-varying dynamics, but its local search can be slower in high-dimensional spaces and may converge to local optima (Saxena et al., 9 Jun 2026, Saxena et al., 2 Oct 2025). This suggests a division of labor in which DRL provides nominal performance and ES provides adaptive robustness.

The concept is not confined to one application domain. In particle accelerator control, the combined controller is used for tuning a low-energy beam transport section under time-varying lattice perturbations (Saxena et al., 9 Jun 2026). In robotic manipulation, a related hybrid switches from DDPG to bounded ES after contact has been established, so that contact-rich adaptation is performed using only a scalar performance signal (Saxena et al., 1 Apr 2026). In a broader time-varying-systems setting, a supervisor based on beam-envelope safety thresholds switches between a DDPG policy and bounded ES for tuning 22 quadrupole magnets in a linear accelerator (Saxena et al., 2 Oct 2025).

2. Architectural forms

Two principal architectural forms appear in the cited literature. The first is a hard-switching supervisory architecture. In the accelerator OOD-detection framework, the applied control is

$$u_t = \beta_t u_{\text{RL},t} + (1-\beta_t) u_{\text{ES},t}, \qquad \beta_t \in \{0,1\},$$

so the combination is a binary switch rather than a continuous blend (Saxena et al., 9 Jun 2026). The switch is driven by a latent-space OOD score computed from beam-profile observations encoded by a VAE (Saxena et al., 9 Jun 2026). In the manipulation formulation, the switch is event-based rather than statistical: RL controls until the first contact time $t_c$, after which ES takes over, with the ES action warm-started from the RL action at the switching instant (Saxena et al., 1 Apr 2026).

The second form is a supervisor defined by safety or performance margins. In the time-varying accelerator study, a state-dependent binary variable $\beta(o(t))$ is set according to beam-envelope safety thresholds, so that DRL is used while the beam remains well inside the pipe radius and ES takes over when the average envelopes approach the safety margin (Saxena et al., 2 Oct 2025). This formulation shares the same logic as the OOD-based switch, but the switching criterion is directly tied to physical beam geometry rather than latent statistical distance.

A broader control literature supplies neighboring hybrid patterns that clarify what a combined controller is not. Some systems use learned switching among multiple conventional planners rather than ES-based adaptation, as in a navigation stack in which a DRL meta-controller chooses between TEB and a DRL local planner (Kästner et al., 2021). Others use soft blending rather than hard switching, such as autonomous driving designs where DDPG, artificial potential fields, and path-tracking outputs are linearly combined (Xiong et al., 2016). These works are relevant as hybrid-control antecedents, but the combined ES–DRL controller is distinguished by the specific pairing of extremum seeking with deep RL and by the role of ES as a model-independent adaptive mechanism under shift.

3. DRL component

The DRL component is generally chosen for continuous control over many parameters. In both accelerator and manipulation settings, the cited works use Deep Deterministic Policy Gradient (DDPG) (Saxena et al., 9 Jun 2026, Saxena et al., 1 Apr 2026, Saxena et al., 2 Oct 2025). In the accelerator beamline problem, the observation is a sampled beam profile

$o_t = [X(z,t), Y(z,t), X'(z,t), Y'(z,t)],$

with each channel sampled at $N_z = 4000$ longitudinal locations, giving an observation in $\mathbb{R}^{4 \times 4000}$, while the action is the 22-dimensional vector of quadrupole magnet setpoints

$$Q_t = [Q_1(t), \dots, Q_{22}(t)]^\top \in \mathbb{R}^{22}$$

(Saxena et al., 9 Jun 2026). The actor implements a deterministic policy $\pi_\theta : o_t \mapsto u_{\text{RL},t}$, trained under fixed lattice geometry but randomized initial beam conditions (Saxena et al., 9 Jun 2026).

The reward in that setting is explicitly shaped around beam quality. Path-averaged envelope sizes and squared envelope slopes are penalized, terminal beam circularity and terminal slope are penalized, and the scalar reward is given by

$R_t = \frac{1}{1 + P_t} \in (0,1],$

with $P_t$ the sum of envelope, smoothness, and terminal penalties (Saxena et al., 9 Jun 2026). This construction makes the RL controller fast and effective on the training distribution, but its deployment regime remains limited by the assumption of fixed quadrupole geometry during training (Saxena et al., 9 Jun 2026).

The same basic logic appears in robotic manipulation. There the DDPG actor is trained on FetchPush and FetchPickAndPlace under nominal friction and static within-episode goals, with dense reward based on end-effector–object distance and object–goal distance. The policy is then frozen at deployment and combined with bounded ES under friction shifts and time-varying goals (Saxena et al., 1 Apr 2026). In the more general LEBT robustness study, DDPG again provides a fast nominal controller over 22 quadrupole settings, with replay buffer, target networks, and exploration noise during training, but no online learning during deployment (Saxena et al., 2 Oct 2025).

A plausible implication is that combined ES–DRL designs favor actor–critic methods with deterministic continuous outputs because such policies can directly produce high-dimensional actuator commands while remaining easy to hand over to a bounded adaptive layer.

4. Bounded extremum seeking component

Bounded ES is the adaptive half of the architecture. It is a model-free, gradient-free optimization method that updates control variables using a scalar measured objective rather than an explicit model of the plant (Saxena et al., 9 Jun 2026). In the accelerator controller, ES operates on the same 22-dimensional quadrupole vector as DRL and uses the same beam-quality objective as the RL reward, or an equivalent cost derived from it (Saxena et al., 9 Jun 2026). Its updates are bounded so that magnet settings remain within safe limits, which is essential in safety-critical accelerator operation (Saxena et al., 9 Jun 2026).

The accelerator paper summarizes the ES increment as

$t_c$0

with $t_c$1, $t_c$2, $t_c$3, $t_c$4, and the internal state $t_c$5 chosen to yield bounded, stable adaptation (Saxena et al., 9 Jun 2026). In the more general time-varying-systems treatment, bounded ES is analyzed through a dithered law whose averaged dynamics implement a gradient ascent on the reward $t_c$6 despite unknown and time-varying control directions, while maintaining bounded amplitudes and update rates (Saxena et al., 2 Oct 2025). In the manipulation formulation, after contact the first three Cartesian action components are updated by ES according to

$t_c$7

with the gripper command held fixed from the RL phase (Saxena et al., 1 Apr 2026).

Three properties are repeatedly emphasized. First, ES is model-independent: it uses only current performance measurements and does not require an explicit analytical model of the plant (Saxena et al., 9 Jun 2026, Saxena et al., 2 Oct 2025). Second, ES is local: it performs a neighborhood search around its current operating point and can slow down in large action spaces or settle in local minima (Saxena et al., 9 Jun 2026, Saxena et al., 2 Oct 2025). Third, bounded ES is safe by construction in the sense that updates are kept within prescribed bounds (Saxena et al., 9 Jun 2026, Saxena et al., 2 Oct 2025).

The role of warm-starting is central. In the accelerator OOD-switching architecture, ES is initialized from the current RL-suggested magnet setting when the switch occurs (Saxena et al., 9 Jun 2026). In the manipulation system, ES is initialized at contact from the RL action at that instant (Saxena et al., 1 Apr 2026). This suggests that the practical utility of ES in high-dimensional systems depends not only on its robustness but also on receiving a favorable initial point from the learned policy.

5. Supervisory mechanisms and controller arbitration

The key design question in a combined ES–DRL controller is not whether to combine the two controllers, but how to arbitrate between them. The most developed supervisory mechanism in the cited materials is latent OOD detection. In the accelerator study, a VAE is trained solely on in-distribution beam profiles generated around an expert-tuned reference quadrupole vector, yielding approximately 700,000 profiles split into 680,000 training, 10,000 validation, and 10,000 test samples (Saxena et al., 9 Jun 2026). The VAE encoder maps a beam profile $t_c$8 to latent mean $t_c$9 and variance, with latent dimension $\beta(o(t))$0 selected because $\beta(o(t))$1 produced poor reconstruction and a highly non-Gaussian latent distribution, whereas $\beta(o(t))$2 provided good reconstruction quality and a near-Gaussian latent distribution (Saxena et al., 9 Jun 2026).

At deployment, the current beam profile is encoded as $\beta(o(t))$3, and a latent Mahalanobis distance is computed:

$\beta(o(t))$4

In the main experiments, $\beta(o(t))$5 and $\beta(o(t))$6, so the score reduces approximately to $\beta(o(t))$7 (Saxena et al., 9 Jun 2026). The binary switch is then

$\beta(o(t))$8

so RL is used in-distribution and ES under OOD operation (Saxena et al., 9 Jun 2026).

Empirically, the latent-space plots show that in-distribution beam profiles form a compact cluster, while a moving-magnet OOD trajectory leaves this cluster as magnet displacement increases (Saxena et al., 9 Jun 2026). The Mahalanobis distance rises from roughly 2–4 for ID profiles to around 5–6 as the quadrupole motion becomes pronounced, and reconstruction error rises in parallel (Saxena et al., 9 Jun 2026). The paper interprets this as evidence that the latent representation carries physically meaningful information about beam degradation and magnet drift (Saxena et al., 9 Jun 2026).

A second supervisory pattern is event-triggered switching. In robotic manipulation, the switch occurs at contact time $\beta(o(t))$9: before contact, RL is responsible for reaching and grasp acquisition; after contact, ES is responsible for pushing or post-grasp transport under shifted conditions (Saxena et al., 1 Apr 2026). A third pattern is safety-threshold switching, where average beam envelopes relative to $o_t = [X(z,t), Y(z,t), X'(z,t), Y'(z,t)],$0 determine whether DRL or ES is active (Saxena et al., 2 Oct 2025).

A plausible implication is that combined ES–DRL control is less about a fixed mixture and more about state-contingent controller trust: the supervisor encodes when the learned policy remains reliable and when adaptation must take precedence.

6. Application domains and operating regimes

The most detailed application is particle accelerator control. The system is a low-energy beam transport section modeled by Kapchinskij–Vladimirskij envelope equations, with state channels $o_t = [X(z,t), Y(z,t), X'(z,t), Y'(z,t)],$1, $o_t = [X(z,t), Y(z,t), X'(z,t), Y'(z,t)],$2, $o_t = [X(z,t), Y(z,t), X'(z,t), Y'(z,t)],$3, and $o_t = [X(z,t), Y(z,t), X'(z,t), Y'(z,t)],$4, and 22 quadrupole magnets as actuators (Saxena et al., 9 Jun 2026). Time variation is introduced by spatial motion of a quadrupole magnet along the beamline, producing OOD beam profiles not seen during RL training (Saxena et al., 9 Jun 2026). In a related study, strong perturbations are applied through exogenous sinusoidal disturbances on two quadrupoles and time-varying spacing between magnets, showing that the hybrid controller maintains higher and more stable reward than DRL-only or ES-only baselines under severe drift (Saxena et al., 2 Oct 2025).

A second domain is robotic manipulation under distribution shift. In pushing and pick-and-place tasks, DDPG is trained under uniform friction and static within-episode goals, then deployed under spatially varying friction patches and time-varying goals (Saxena et al., 1 Apr 2026). RL-only policies lose contact or fail to track moving targets, whereas ES-only lacks the task-structuring ability needed before contact. The hybrid uses RL for task entry and ES for post-contact adaptation, which improves robustness to frictional variation and moving goals (Saxena et al., 1 Apr 2026).

A broader interpretation of combined-controller design appears in other hybrid DRL literature, though without ES. For controller parametrization of gain-scheduled systems, DRL is used to adapt B-spline geometry control points online, while the representation itself suggests a low-dimensional search space that could be optimized by ES (Rudolf et al., 2024). In local navigation, a DRL switch selects among conventional and learned planners based on sensor observations, demonstrating the utility of runtime controller arbitration (Kästner et al., 2021). In assisted reinforcement learning, a DQN-like switch chooses between a DDPG policy and a simple classical controller during training, with the classical controller serving as “training wheels” before being phased out (Xie et al., 2018).

These neighboring examples do not define the combined ES–DRL controller directly, but they situate it within a larger design space of hybrid learned/classical control. This suggests that the ES–DRL variant is one specific realization of a broader principle: retaining the sample efficiency and expressive power of DRL while embedding an explicit mechanism for robustness, safety, or adaptation.

7. Limitations, design tensions, and related interpretations

Several limitations recur across the cited work. First, switch design is delicate. The OOD-driven accelerator formulation uses a single threshold $o_t = [X(z,t), Y(z,t), X'(z,t), Y'(z,t)],$5 on latent Mahalanobis distance, and the paper does not explicitly introduce hysteresis or temporal smoothing (Saxena et al., 9 Jun 2026). This suggests threshold selection trades conservative early switching against delayed intervention. The same text notes that hysteresis or temporal filtering would be advisable in practice to avoid chattering near the threshold, but that remains an engineering extension rather than part of the reported design (Saxena et al., 9 Jun 2026).

Second, ES remains a local adaptive method. Both accelerator papers stress that convergence slows as the number of tuned parameters increases and that ES can become trapped in local minima (Saxena et al., 9 Jun 2026, Saxena et al., 2 Oct 2025). This is exactly why warm-starting from DRL is used. It also means that combined ES–DRL architectures implicitly assume that the learned policy can place ES near a region where local adaptation is useful.

Third, DRL generalization is the failure mode the architecture is built around. In the accelerator setting, the RL policy is trained under randomized initial beam conditions but fixed lattice geometry, so geometry changes at deployment are OOD by construction (Saxena et al., 9 Jun 2026). In manipulation, the policy is trained with static goals within episodes and uniform friction, so moving goals and friction patches induce test-time shift (Saxena et al., 1 Apr 2026). The combined architecture therefore does not remove the distributional limitation of DRL; it manages it.

Fourth, the exact form of ES–DRL coupling varies by domain. Some systems use a binary switch (Saxena et al., 9 Jun 2026, Saxena et al., 1 Apr 2026, Saxena et al., 2 Oct 2025). Other hybrid-control papers use convex blending or mixture-of-experts structures, such as demonstration-assisted DDPG for autonomous lateral control (Wu et al., 4 Jun 2025) or convex combinations of PPO policies for morphing-quadrotor control (Yang et al., 2024). These neighboring formulations indicate that “combined controller” can denote switching, blending, or demonstration-assisted coupling, but the strict ES–DRL usage in the primary papers refers to a supervisory hybrid in which ES acts as a robust fallback or post-contact adaptive controller.

Finally, sequential ES→DRL hybrids should be distinguished from runtime combined controllers. A recent study on ES pretraining for DRL found that ES initialization can help in simple environments such as Flappy Bird but provides minimal or no benefit in more complex tasks such as Breakout and MuJoCo Walker (Martínez et al., 31 Mar 2026). That work concerns ES as a pretraining stage rather than as an online adaptive fallback, so it addresses a different notion of combination. This suggests that “combined ES–DRL controller” has at least two meanings in the literature: sequential parameter-transfer hybrids and runtime supervisory hybrids. The runtime supervisory interpretation is the one developed most fully in the accelerator and manipulation papers (Saxena et al., 9 Jun 2026, Saxena et al., 1 Apr 2026, Saxena et al., 2 Oct 2025).

In summary, the combined ES–DRL controller is best understood as a hybrid supervisory controller for nonlinear systems under shift: DRL supplies nominal, fast, high-dimensional actuation based on historical training data, while bounded ES supplies model-independent online adaptation when the current operating regime is no longer trusted. The architecture is defined not only by the presence of both components, but by the explicit supervisory logic that determines when each should dominate (Saxena et al., 9 Jun 2026).