Surr-RLDE: Surrogate-Assisted RL for DE

• The paper demonstrates that integrating surrogate learning with RL for DE reduces evaluation costs while maintaining competitive optimization performance.
• Surr-RLDE employs a two-stage framework where a per-instance Kolmogorov-Arnold Network is pretrained with a relative-order-aware loss before guiding a Double-DQN-based dynamic operator configuration.
• Empirical tests on COCO-BBOB problems show that Surr-RLDE achieves robust ranking and performance, outperforming conventional DE and RLDE frameworks in several cases.

Surrogate-Assisted RL for Differential Evolution (Surr-RLDE) is a Meta-Black-Box Optimization (MetaBBO) framework that combines surrogate learning with reinforcement learning-aided Differential Evolution (DE) in order to reduce the intensive function evaluation cost of meta-policy training. Its defining structure is two-stage: a surrogate learning stage trains a per-instance Kolmogorov-Arnold Network (KAN) with a relative-order-aware loss, and a policy learning stage trains a Double-DQN controller that dynamically configures DE by interacting with those learned surrogates rather than with the original training objectives. In the broader taxonomy of black-box optimization, Surr-RLDE is categorized as a hybrid RL-enhanced evolutionary method for dynamic operator configuration in DE, with the surrogate supplying cheap fitness information and pseudo-rewards during policy training (Ma et al., 23 Mar 2025, Kimiaei et al., 29 Sep 2025).

1. Conceptual position in MetaBBO

Surr-RLDE is situated within MetaBBO, where a low-level optimizer solves black-box optimization problems and a meta-level policy learns how to configure that optimizer during search. The meta-policy is trained over a problem distribution $\mathbb{P}$, and the paper formulates the meta-objective as

$$\pi_\omega^* = \underset{\pi_\omega}{\arg\max}\left[\mathbb{E}_{f \in \mathbb{P}}\left[\sum_{t=1}^{T} r^t\right]\right].$$

Within this formulation, Surr-RLDE addresses a specific bottleneck: conventional MetaBBO training consumes a very large number of true objective evaluations because policy learning requires repeated optimizer rollouts over many problems and many learning steps (Ma et al., 23 Mar 2025).

The framework is therefore neither a classical surrogate-only optimizer nor a pure RL meta-optimizer. The survey literature classifies it as a hybrid method in which DE remains the search engine, RL performs dynamic operator configuration, and the surrogate reduces evaluation cost during policy learning. In that taxonomy, Surr-RLDE appears under RL-based dynamic operator configuration for black-box optimization, with the concise summary that “RL configures DE operators with surrogate pseudo-rewards” in the MetaBBO domain (Kimiaei et al., 29 Sep 2025).

The optimization setting is continuous single-objective black-box minimization. The reported experiments use COCO-BBOB functions with base dimension $D=10$, search range $[-5,5]^D$, a training/test split of 16 and 8 functions respectively, and out-of-distribution tests on shifted/rotated 10D problems and 30D variants. The survey further emphasizes that Surr-RLDE is framed for expensive-evaluation black-box optimization rather than specifically for noisy or constrained objectives (Ma et al., 23 Mar 2025, Kimiaei et al., 29 Sep 2025).

2. Two-stage architecture and design rationale

Surr-RLDE comprises two sequential learning stages.

In the Surrogate Learning Stage (SLS), a separate surrogate $F_\theta^{(f)}$ is trained for each training problem instance $f$. The training data are

$D_f = \{(x_i, f(x_i))\}_{i=1}^{N},$

with $N = 5 \times 10^4$ points per training function, generated by Latin Hypercube Sampling over $[-5,5]^{10}$. Since 16 training functions are used in the reported setup, surrogate construction consumes $16 \times 5\times10^4 = 8\times10^5$ true evaluations in total (Ma et al., 23 Mar 2025).

In the Policy Learning Stage (PLS), the learned surrogates replace the original objectives during RL training. DE episodes are executed on surrogate functions rather than on true functions, and surrogate outputs are used for population evaluation, state extraction, reward computation, and best-so-far tracking. The paper describes this replacement as the main mechanism by which Surr-RLDE “saves all function evaluations needed during training” after the surrogate datasets have been built (Ma et al., 23 Mar 2025).

This architecture is strictly one-way in the reported formulation. SLS first trains fixed surrogates, and PLS then trains the RL controller on those fixed surrogates. There is no online joint adaptation of the surrogate and policy during policy learning. The survey description is consistent with this view, but presents the workflow more generically as periodic surrogate refresh with new true data; that broader phrasing should be read as a high-level hybrid template rather than as the exact two-stage protocol of the original Surr-RLDE study (Kimiaei et al., 29 Sep 2025).

The design rationale is that MetaBBO policies do not require exact objective values so much as sufficiently faithful optimization trajectories. Surr-RLDE therefore prioritizes preservation of comparative structure, especially ranking consistency among candidate solutions, over exact value regression. That choice is reflected directly in the surrogate loss design (Ma et al., 23 Mar 2025).

3. Surrogate learning stage

The surrogate model class is a Kolmogorov-Arnold Network. The paper writes the underlying representation as

$$\pi_\omega^* = \underset{\pi_\omega}{\arg\max}\left[\mathbb{E}_{f \in \mathbb{P}}\left[\sum_{t=1}^{T} r^t\right]\right].$$0

and a multilayer KAN as

$$\pi_\omega^* = \underset{\pi_\omega}{\arg\max}\left[\mathbb{E}_{f \in \mathbb{P}}\left[\sum_{t=1}^{T} r^t\right]\right].$$1

Each trainable univariate transform is

$$\pi_\omega^* = \underset{\pi_\omega}{\arg\max}\left[\mathbb{E}_{f \in \mathbb{P}}\left[\sum_{t=1}^{T} r^t\right]\right].$$2

with

$$\pi_\omega^* = \underset{\pi_\omega}{\arg\max}\left[\mathbb{E}_{f \in \mathbb{P}}\left[\sum_{t=1}^{T} r^t\right]\right].$$3

and

$$\pi_\omega^* = \underset{\pi_\omega}{\arg\max}\left[\mathbb{E}_{f \in \mathbb{P}}\left[\sum_{t=1}^{T} r^t\right]\right].$$4

The reported implementation uses layer width $$\pi_\omega^* = \underset{\pi_\omega}{\arg\max}\left[\mathbb{E}_{f \in \mathbb{P}}\left[\sum_{t=1}^{T} r^t\right]\right].$$5, grid size $$\pi_\omega^* = \underset{\pi_\omega}{\arg\max}\left[\mathbb{E}_{f \in \mathbb{P}}\left[\sum_{t=1}^{T} r^t\right]\right].$$6, spline order $$\pi_\omega^* = \underset{\pi_\omega}{\arg\max}\left[\mathbb{E}_{f \in \mathbb{P}}\left[\sum_{t=1}^{T} r^t\right]\right].$$7, and the pykan implementation (Ma et al., 23 Mar 2025).

The surrogate is first pretrained by mean squared error: $$\pi_\omega^* = \underset{\pi_\omega}{\arg\max}\left[\mathbb{E}_{f \in \mathbb{P}}\left[\sum_{t=1}^{T} r^t\right]\right].$$8 This pretraining phase is then followed by fine-tuning with the Relative-Order-Aware (ROA) loss. For a minibatch of size $$\pi_\omega^* = \underset{\pi_\omega}{\arg\max}\left[\mathbb{E}_{f \in \mathbb{P}}\left[\sum_{t=1}^{T} r^t\right]\right].$$9, sorted by descending true objective value, the loss is

$D=10$0

where

$D=10$1

The stated purpose of the order-correction term is to encourage $D=10$2 to remain numerically between neighboring ground-truth objective values in the sorted batch, thereby improving local rank preservation (Ma et al., 23 Mar 2025).

The training schedule is explicitly staged. MSE pretraining runs for $D=10$3 epochs, followed by ROA fine-tuning for $D=10$4 epochs, with batch size $D=10$5 and learning rate $D=10$6. During ROA fine-tuning, $D=10$7 is initialized at 1 and then decays linearly (Ma et al., 23 Mar 2025).

The paper’s ablations distinguish value accuracy from order fidelity. RBF performs poorly on global landscape prediction for complex problems, MLP and KAN are similar in MSE accuracy, and KAN achieves better ROA accuracy. This is the empirical basis for choosing KAN rather than a standard MLP or RBF surrogate in Surr-RLDE (Ma et al., 23 Mar 2025).

A closely related but conceptually different surrogate literature studies empirical performance models for DE parameter configurations rather than decision-vector-to-objective surrogates. In that setting, the surrogate input is the DE configuration $D=10$8, and the output is empirical DE performance on a fixed problem instance. Gaussian Process and Random Forest models were found to be the strongest overall choices there (Li et al., 2019). Surr-RLDE does not use that surrogate target; its surrogate approximates the objective function itself.

4. Reinforcement-learning control of Differential Evolution

The low-level optimizer is vanilla DE, but the RL controller dynamically configures its mutation behavior. The state is a 9-dimensional representation borrowed from SYMBOL and divided into three groups: $D=10$9 describe distributional properties of the population, $[-5,5]^D$0 are convergence-related fitness-distance-correlation-style features, and $[-5,5]^D$1 are time-stamp or progress features (Ma et al., 23 Mar 2025).

The action space contains $[-5,5]^D$2 discrete actions, combining five mutation operators with three mutation strengths $[-5,5]^D$3. The five operators are:

• DE/rand1
• DE/best1
• DE/current-to-rand
• DE/current-to-pbest
• DE/current-to-best

Crossover is binary crossover with $[-5,5]^D$4, initialization is uniform random sampling, and the RL policy controls only the mutation configuration; it does not control crossover rate, population size, or survivor selection (Ma et al., 23 Mar 2025).

The DE dynamics remain standard. Mutation and crossover generate a new population under the selected action, and selection is greedy. The survey presents the standard DE backbone in the familiar form

$[-5,5]^D$5

$[-5,5]^D$6

and

$[-5,5]^D$7

In Surr-RLDE, these comparisons are made on surrogate outputs during policy learning (Kimiaei et al., 29 Sep 2025).

The RL algorithm is Double-DQN with an MLP of shape $[-5,5]^D$8, ReLU activations, discount factor $[-5,5]^D$9, learning rate $F_\theta^{(f)}$0, target-network update period $F_\theta^{(f)}$1, and maximum learning steps $F_\theta^{(f)}$2 (Ma et al., 23 Mar 2025).

The paper prints the reward as

$F_\theta^{(f)}$3

where $F_\theta^{(f)}$4 is the best-so-far surrogate objective value. The accompanying text says that “a positive reward signal is returned if a better solution is found.” Because the benchmark setting is minimization, this suggests that the inequality must be interpreted consistently with improvement under minimization, even though the printed direction is reversed (Ma et al., 23 Mar 2025).

The survey summarizes the same mechanism more abstractly as off-policy RL with pseudo-rewards derived from surrogate ranking consistency and occasional true rewards, but the original study reports a fixed two-stage setup in which policy learning proceeds entirely on surrogate objectives after surrogate construction (Kimiaei et al., 29 Sep 2025).

5. Empirical evidence and ablation findings

The main benchmark protocol uses COCO-BBOB with 16 training functions and 8 held-out test functions at $F_\theta^{(f)}$5, plus out-of-distribution evaluations on shifted/rotated 10D tests and 30D tests. Each test result is averaged over 51 independent runs, and the maximum number of function evaluations per optimization run is $F_\theta^{(f)}$6 (Ma et al., 23 Mar 2025).

On the 8-function in-distribution test set, Surr-RLDE achieves an average rank of 3.0. The reported comparison points are: Surr-RLDE-O 3.5, DE-DDQN 3.375, DEDQN 9.0, GLEET 2.75, and plain DE 3.625. Thus the surrogate-assisted version remains competitive with recent MetaBBO baselines and outperforms the same RLDE framework trained directly on the original objectives in this setup (Ma et al., 23 Mar 2025).

The mean best objective values reported for Surr-RLDE on the 8 held-out test functions are:

• Weierstrass: $F_\theta^{(f)}$7
• Schaffers: $F_\theta^{(f)}$8
• Schaffers_high_cond: $F_\theta^{(f)}$9
• Composite_Grie_rosen: $f$0
• Gallagher_101Peaks: $f$1
• Gallagher_21Peaks: $f$2
• Katsuura: $f$3
• Lunacek_bi_Rastrigin: $f$4

Surr-RLDE ranks first on Gallagher 101Peaks and Gallagher 21Peaks, two multimodal problems with weak global structure according to the paper’s discussion (Ma et al., 23 Mar 2025).

The ablation results are central. Comparing ROA-based surrogate training against MSE-only training, the paper reports large downstream differences on several functions, for example:

• Schaffers: $f$5 vs $f$6
• Schaffers high cond: $f$7 vs $f$8
• Gallagher 101Peaks: $f$9 vs $D_f = \{(x_i, f(x_i))\}_{i=1}^{N},$0
• Gallagher 21Peaks: $D_f = \{(x_i, f(x_i))\}_{i=1}^{N},$1 vs $D_f = \{(x_i, f(x_i))\}_{i=1}^{N},$2

These results support the claim that MSE-only surrogate fitting is insufficient for stable policy learning when the downstream optimizer depends on relative comparisons (Ma et al., 23 Mar 2025).

The comparison between Surr-RLDE and Surr-RLDE-O is also notable. Surr-RLDE-O uses true objectives during policy learning and obtains an average rank of 3.5, while Surr-RLDE obtains 3.0. The paper interprets this as evidence that surrogate-assisted policy learning does not merely preserve performance; in some cases it improves it. A plausible implication is that the surrogate may smooth complex training landscapes in a way that benefits meta-policy learning, although that mechanism is not formalized (Ma et al., 23 Mar 2025).

The survey’s higher-level synthesis is consistent: it states that extensive experiments show Surr-RLDE significantly reduces evaluation cost while maintaining competitive performance, and that it generalizes to higher-dimensional problems while matching or outperforming MetaBBO baselines such as DEDQN, GLEET, and SYMBOL (Kimiaei et al., 29 Sep 2025).

6. Relation to adjacent surrogate-assisted and RL-based DE research

Surr-RLDE occupies a specific intersection between two neighboring lines of work.

On one side are surrogate-assisted DE methods without RL. “Combining Lipschitz and RBF Surrogate Models for High-dimensional Computationally Expensive Problems” develops LSADE, which combines a global RBF surrogate, a Lipschitz underestimation surrogate, and a local RBF plus SQP refinement step inside a DE-based optimizer under a budget of 1000 true evaluations (Kudela et al., 2022). “Better call Surrogates” develops SteaDE, where RBF interpolation and GP-based Bayesian optimization guide DE mutation, but final survival decisions still use true objective evaluations (Biswas et al., 2020). These methods use surrogates to guide infill or candidate generation; Surr-RLDE instead uses surrogates to replace the objective during meta-policy training (Ma et al., 23 Mar 2025).

On the other side are RL-controlled DE methods without surrogates. RLDE-AFL uses learned attention-based population features and PPO to choose mutation and crossover operators plus parameter distributions, but it evaluates the true objective throughout training and test runs (Guo et al., 23 Mar 2025). RL-DAS uses PPO to switch dynamically among JDE21, MadDE, and NL-SHADE-RSP during a run, with a hand-crafted state and context restoration mechanism (Guo et al., 2024). The GECCO Companion study on adaptive metaheuristics uses PPO to control $D_f = \{(x_i, f(x_i))\}_{i=1}^{N},$3 and $D_f = \{(x_i, f(x_i))\}_{i=1}^{N},$4 generation by generation in DE/best/1/bin (Tessari et al., 2022). The automated-design framework rlDE uses Double DQN and ELA features to choose initialization, mutation, crossover, and parameter settings before optimization starts, rather than controlling DE online through surrogate rollouts (Yang et al., 22 Jan 2025). Surr-RLDE differs from all of these by placing the surrogate directly inside policy learning (Ma et al., 23 Mar 2025).

The survey explicitly places Surr-RLDE between these two research streams. It is not presented as a classical surrogate-only black-box optimizer such as Bayesian optimization, and it is not presented as a pure offline RL meta-optimizer such as Q-Mamba. Rather, it is a hybrid method in which DE provides the population dynamics, RL provides online operator control, and the surrogate provides cheap evaluation and pseudo-reward signals (Kimiaei et al., 29 Sep 2025).

This positioning is also consistent with empirical benchmarking outside MetaBBO. In a comparison of eleven surrogate-assisted evolutionary algorithms on two CFD problems, the strongest aggregate performers—LSADE, ESA, TS-DDEO, and IKAEA—all used DE as one of their optimization mechanisms (Kudela et al., 2024). That result does not validate Surr-RLDE specifically, but it supports DE as a strong backbone for surrogate-assisted expensive optimization.

7. Limitations, caveats, and open problems

Several limitations follow directly from the reported formulation.

First, Surr-RLDE trains one surrogate per training problem instance. This yields strong per-instance approximation quality but makes the surrogate layer non-amortized across tasks. A plausible implication is that scaling to a much larger or continuously parameterized training distribution would require a different surrogate strategy, such as a conditional or shared model (Ma et al., 23 Mar 2025).

Second, the method has no explicit uncertainty estimation, no active-learning correction during policy learning, and no trust mechanism. The survey states that there is no explicit uncertainty quantification, trust threshold, acquisition function, or fidelity scheduler formula for Surr-RLDE. The original paper likewise does not introduce predictive variance, ensemble disagreement, or periodic relabeling of policy-learning trajectories with true objective calls (Kimiaei et al., 29 Sep 2025, Ma et al., 23 Mar 2025).

Third, the framework depends strongly on rank fidelity. The surrogate is used not only for value approximation but also for DE survivor selection, best-so-far tracking, state extraction, and reward generation. If the surrogate preserves global shape but misorders locally competitive candidates, the entire policy-learning trajectory changes. The ROA loss is introduced precisely because MSE alone does not protect against this failure mode (Ma et al., 23 Mar 2025).

Fourth, some implementation details are delegated elsewhere. The exact formulas of the 9-dimensional state are inherited from SYMBOL and not reproduced in full. The printed reward inequality is inconsistent with the minimization semantics and must be interpreted from context. These issues do not alter the conceptual design, but they complicate strict reimplementation from the Surr-RLDE paper alone (Ma et al., 23 Mar 2025).

Finally, the survey notes that Surr-RLDE is not explicitly presented as a constrained optimizer, a noise-robust method, or a mixed-variable solver. Those roles are assigned to other methods in the same survey, such as LB-SGD for constrained optimization and RBO or CAS-MORE for robustness under noise (Kimiaei et al., 29 Sep 2025). Surr-RLDE is therefore best understood as a surrogate-assisted, RL-controlled DE framework for expensive continuous MetaBBO rather than as a general solution to all black-box optimization regimes.

In that sense, its distinctive contribution is narrow but clear: it shows that meta-policy training for DE can be moved from true objective space into surrogate space, provided that the surrogate preserves enough local order structure to sustain meaningful DE trajectories and RL feedback (Ma et al., 23 Mar 2025).