SuperDE: DRL Auto-Config for Constrained Optimization

• SuperDE is a foundation model for constrained evolutionary optimization that leverages DRL to adaptively select and configure DE components.
• It treats per-generation algorithm configuration as a sequential decision process using a 22-dimensional state and 28-action space.
• Empirical results show that its joint selection of mutation and constraint-handling techniques enhances convergence and generalization across diverse COP benchmarks.

Searching arXiv for the specified SuperDE paper and closely related work on DRL-assisted differential evolution for constrained optimization. arXiv search query: (Yang et al., 14 Sep 2025) OR "Deep Reinforcement Learning-Assisted Component Auto-Configuration of Differential Evolution Algorithm for Constrained Optimization: A Foundation Model" SuperDE is a foundation model for constrained evolutionary optimization that automates the per-generation configuration of Differential Evolution (DE) components through deep reinforcement learning. Introduced in "Deep Reinforcement Learning-Assisted Component Auto-Configuration of Differential Evolution Algorithm for Constrained Optimization: A Foundation Model" (Yang et al., 14 Sep 2025), it targets constrained optimization problems (COPs) of the form

$$\begin{aligned} & \text{minimize} \quad f(\mathbf{x}) \ & \text{subject to} \quad g_i(\mathbf{x}) \leq 0,\quad h_j(\mathbf{x}) = 0 , \end{aligned}$$

with constraint violation quantified as

$$cv(\mathbf{x}) = \sum_{i=1}^{m}\max(0,g_i(\mathbf{x})) + \sum_{j=1}^{p} \max(0,|h_j(\mathbf{x})| - \delta).$$

Its central premise is that, because COPs are heterogeneous and the no free lunch theorem precludes a universally best optimizer, DE should not rely on static mutation operators or hand-crafted constraint handling rules. Instead, SuperDE learns a reusable policy offline across diverse COPs and then applies that policy in a zero-shot manner to unseen problems (Yang et al., 14 Sep 2025).

1. Problem setting and motivation

SuperDE is motivated by the limitations of standard DE and related evolutionary algorithms on COPs. Real-world COPs are described as varied and dynamic, while conventional DE designs typically fix evolutionary operators and constraint handling techniques (CHTs), or adapt them through manually designed, problem-specific rules. The paper further notes that existing online adaptation methods often suffer from inefficiency, weak convergence, and limited generalization on constrained optimization problems (Yang et al., 14 Sep 2025).

Within this context, SuperDE treats algorithm configuration itself as a sequential decision problem. Rather than assuming that one mutation strategy or one feasibility-handling mechanism is broadly adequate, it models the optimizer as an agent that observes the evolving population and selects a configuration accordingly. The design objective is therefore not a better static DE variant, but an adaptive policy capable of identifying which combination of search operator and constraint-handling rule is appropriate at a given generation for a given problem instance (Yang et al., 14 Sep 2025).

This positioning is important for interpreting the phrase "foundation model" in the paper. Here, the term denotes a single policy trained across a wide variety of COPs and then reused zero-shot on unseen COPs, rather than a problem-specific tuner or an instance-wise online controller trained from scratch (Yang et al., 14 Sep 2025).

2. System architecture and optimization loop

At a high level, SuperDE operates as a DRL-driven policy agent coupled to a DE optimizer. The workflow has three stages: offline meta-training across a large corpus of varied COPs, per-generation decision-making during DE execution, and zero-shot inference on unseen problems without retraining, although the paper notes that further online fine-tuning is possible if desired (Yang et al., 14 Sep 2025).

During optimization, each generation begins by constructing a feature-rich state vector from the current population. The DDQN policy then maps that state to an action, where an action is a joint choice of mutation strategy and CHT. The selected components are applied to evolve the population, after which the next state and reward are produced. This establishes a closed loop in which configuration is conditioned on real-time evolutionary dynamics rather than fixed a priori (Yang et al., 14 Sep 2025).

The framework is therefore not merely a parameter tuner. It dynamically reconfigures two substantive parts of the DE pipeline: the search dynamics induced by the mutation operator and the feasibility-selection dynamics induced by the CHT. A plausible implication is that SuperDE targets both exploration/exploitation balance and feasibility progression simultaneously, which is consistent with the paper’s emphasis on joint component configuration (Yang et al., 14 Sep 2025).

3. Markov decision process formulation

SuperDE formalizes DE auto-configuration as a discrete-action Markov decision process. Its state space is a fixed-length 22-dimensional vector intended to capture feasibility, optimization progress, structural properties of the feasible region, and population diversity (Yang et al., 14 Sep 2025).

The reported state descriptors include feasible ratio $fsr$, best, worst, mean, and median fitness, statistics on objective values and constraint violations, structure of the feasible region through clusters, feasible boundary crossing ratio, Spearman correlation between objectives and CVs, diversity measures such as mean standard deviation of variables and distance to the best individual, active constraints proportion, and the number of function evaluations (Yang et al., 14 Sep 2025). The state representation is thus explicitly engineered to expose both quality and geometry of the evolving search distribution.

The action space is the Cartesian product of four mutation strategies and seven CHTs, yielding 28 discrete actions (Yang et al., 14 Sep 2025).

Component class	Elements	Count
Mutation strategies	DE/rand/1; DE/best/1; DE/cur-to-best/1; DE/rand-to-best/1	4
Constraint handling techniques	Death Penalty; Weighted Penalty; Feasibility Rule 1; Feasibility Rule 2; Tournament; Epsilon Feasibility Ranking; Stochastic Ranking	7

The reward is designed to reflect a two-stage optimization logic: first feasibility, then objective quality. If feasible solutions exist, the reward depends on best-fitness improvement together with the new feasible ratio; if no feasible solution exists, it depends on improvement in average CV and minimum CV (Yang et al., 14 Sep 2025). The paper states that this reward structure aligns the agent with the evolutionary goals: first feasibility, then solution quality. That alignment is a central modeling decision, because COP solvers often fail precisely when objective improvement is rewarded before feasible-region acquisition (Yang et al., 14 Sep 2025).

4. DDQN mechanism and meta-learning regime

The control policy in SuperDE is implemented with a Double Deep Q-Network. The Q-network $Q(s,a;\theta)$ takes the 22-dimensional state vector as input, uses a series of fully connected layers with 256 neurons and ReLU activations, and outputs one Q-value for each of the 28 actions. A target network $Q(s,a;\theta^-)$ with the same architecture is updated more slowly to stabilize target estimation (Yang et al., 14 Sep 2025).

For a transition $(s_g, a_g, r_g, s_{g+1}, done)$, the DDQN target is

$$y_g = r_g + (1 - done) \cdot \gamma^k \cdot Q(s_{g+1}, a^*; \theta^-), \qquad a^* = \arg\max_a Q(s_{g+1}, a; \theta),$$

and the learning objective is the weighted mean squared error

$$\mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s',done)\sim \text{EB}} \left[ w \cdot \left( Q(s,a;\theta) - y_g \right)^2 \right],$$

where $w$ is an importance sampling weight from prioritized replay. The target network is synchronized by

$\theta^- \leftarrow (1-\xi)\theta^- + \xi\theta .$

The implementation also uses an experience buffer for off-policy learning and variance reduction, together with an epsilon-greedy policy to balance exploitation and exploration (Yang et al., 14 Sep 2025).

Generalization is framed through a MetaBBO bi-level optimization scheme. At the outer level, diverse training COPs are episodically sampled from the CEC2010 suite; at the inner level, the DDQN policy is updated using per-instance experience. The target is a single policy

$$cv(\mathbf{x}) = \sum_{i=1}^{m}\max(0,g_i(\mathbf{x})) + \sum_{j=1}^{p} \max(0,|h_j(\mathbf{x})| - \delta).$$0

so that the learned controller performs well across the entire training distribution and can act intelligently out-of-the-box on new COPs (Yang et al., 14 Sep 2025). This meta-learning formulation is the basis for the paper’s zero-shot generalization claim.

5. Experimental protocol and empirical findings

The training set for SuperDE is CEC2010, consisting of 18 scalable COPs. Generalization is evaluated on CEC2010-extended, CEC2017 with 28 functions, G2000 with 13 classical problems, and BBOB2022 with 54 non-linear COPs. Reported baselines include DE + penalty, CMODE, CAMDE, IepsilonJADE, C2ODE, CW, and ablated variants of SuperDE (Yang et al., 14 Sep 2025).

Evaluation follows a feasibility-aware criterion. For problems where feasible solutions are found, the metric is the average best, i.e. lowest, feasible objective value. If no feasible solution is found, the metric is minimum average constraint violation (Yang et al., 14 Sep 2025). This choice mirrors the reward design and preserves comparability across cases where some methods fail to enter the feasible region.

The paper reports that on CEC2010E, SuperDE found feasible solutions or correct optima where other methods failed and achieved best or comparable objective or CV values across almost all test cases. On CEC2017, G2000, and BBOB2022, it generalized strongly, finding feasible solutions where others could not and outperforming competitors in both solution quality and CV in most cases. Its solutions were also reported to be more consistent, with lower variance (Yang et al., 14 Sep 2025). Statistical validation is provided through the Wilcoxon signed-rank test, under which SuperDE beats baselines in most cases and is equal or superior in the remainder. Convergence plots further show faster and more stable convergence toward feasibility and improvement in CV than all baselines (Yang et al., 14 Sep 2025).

These results support two distinct claims. First, SuperDE improves raw optimization performance on the tested suites. Second, and more specifically, it exhibits cross-benchmark generalization despite being trained offline on a different set of COPs. The second claim is what distinguishes it from conventional adaptive DE methods that are tuned or specialized for narrower problem families (Yang et al., 14 Sep 2025).

6. Ablation results, interpretation, and scope

The ablation study isolates the effect of joint component configuration. SuperDE1 configures only the CHT while keeping the mutation strategy fixed; SuperDE2 configures only the mutation strategy while keeping the CHT fixed; SuperDE3 uses random configuration. The full model significantly outperforms all three variants, and the paper interprets this as evidence of synergy, expressed informally as $$cv(\mathbf{x}) = \sum_{i=1}^{m}\max(0,g_i(\mathbf{x})) + \sum_{j=1}^{p} \max(0,|h_j(\mathbf{x})| - \delta).$$1 (Yang et al., 14 Sep 2025). In other words, mutation control and feasibility handling are not independent levers: their interaction materially affects performance.

That result also clarifies what SuperDE is not. It is not merely a mechanism for choosing among mutation operators, nor solely a learned replacement for classical CHT heuristics. Its empirical advantage is tied to the coordinated selection of both. This suggests that the policy learns contingent relationships between population state, search dynamics, and constraint-violation structure that are not recoverable when the two decision axes are decoupled (Yang et al., 14 Sep 2025).

The paper identifies several limitations. SuperDE does not yet support discrete or combinatorial optimization without adaptation. Its offline meta-training is computationally intensive and depends on the diversity of the training set. In addition, its state and action spaces are manually engineered, and the stated future direction is toward end-to-end and generative configuration spaces (Yang et al., 14 Sep 2025). These limitations place the method squarely within continuous constrained optimization and within a fixed library of DE operators and CHTs.

A terminological ambiguity is worth noting. The string "SuperDE" also appears in the data envelopment analysis literature as a shorthand associated with super-efficiency models implemented in the deaR package (Bolos et al., 4 Jun 2025). In the optimization literature of (Yang et al., 14 Sep 2025), however, SuperDE refers specifically to a DRL-assisted foundation model for per-generation DE component auto-configuration on constrained optimization problems.