Quality-Diversity Optimization Methods

• Quality-Diversity (QD) are a family of methods that seek a collection of solutions that perform at a high quality while simultaneously maximizing behavioral diversity. QD organizes solutions within a user-defined or learned behavior space and attempts to maintain high-quality archetypes in all regions of that space.
• key applications of QD methods include evolutionary robotics, reinforcement learning, engineering design, multi-agent control, and content generation, among others.
• Evaluation metrics for QD include quality, coverage, QD-score, maximum fitness and robustness. QD methods offer alternatives to conventional optimization and multimodal optimization by assessing diversity in the behavior space.

Quality-Diversity (QD) is a family of stochastic and evolutionary optimization methods that seeks a repertoire of solutions that are simultaneously high-performing and behaviorally diverse. Whereas conventional optimization usually returns a single global optimum and multimodal optimization primarily seeks multiple fitness peaks, QD organizes solutions in a user-defined or learned behavior space and attempts to retain high-quality representatives across that space. A QD archive therefore functions both as a collection of solutions and as an empirical map from behavioral regions to their best-known performance. The paradigm has been applied to evolutionary robotics, reinforcement learning, procedural content generation, engineering design, creative systems, multi-agent control, quantum-circuit optimization, scheduling heuristics, and uncertain or learned behavior spaces.

1. Conceptual foundations and objectives

A QD problem associates a solution or parameter vector $\boldsymbol{\theta}\in\Theta$ with a quality value $f_{\boldsymbol{\theta}}$ and a behavior descriptor $\boldsymbol{b}_{\boldsymbol{\theta}}\in\mathcal{B}$:

$$(f_{\boldsymbol{\theta}},\boldsymbol{b}_{\boldsymbol{\theta}})\leftarrow f(\boldsymbol{\theta}).$$

The descriptor characterizes what a solution does, produces, or looks like, whereas fitness characterizes how well it performs. The idealized objective is to obtain, for every behavioral point or region, the highest-quality solution associated with that behavior:

$$\forall \boldsymbol{b}\in\mathcal{B},\qquad \boldsymbol{\theta}^{*}(\boldsymbol{b}) = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}) \quad \text{subject to} \quad \boldsymbol{b}_{\boldsymbol{\theta}}=\boldsymbol{b}.$$

Because behavior spaces are generally continuous, practical algorithms replace exact equality with cells, regions, neighborhoods, or similarity-based competition. The resulting archive approximates a mapping from behavioral regions to their best discovered solutions.

QD differs from ordinary single-objective optimization, which seeks

$$\boldsymbol{\theta}^{*}_{g} = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}),$$

and from multimodal optimization, which generally seeks several local optima in genotypic or parameter space. QD instead normally measures diversity in behavior space. Two genotypically distant controllers may produce nearly identical trajectories, while nearby parameter vectors may generate substantially different behaviors. QD also attempts to illuminate behavioral regions that are not local fitness maxima. A behavior can therefore be retained because it occupies a meaningful niche even when its fitness is lower than that of neighboring behaviors (Chatzilygeroudis et al., 2020).

The central QD trade-off is between divergent search and localized convergence. Divergence expands the repertoire into new behavioral regions; localized convergence improves quality among behaviorally similar solutions. Novelty, surprise, curiosity, archive coverage, local competition, constraints, and Pareto dominance can all implement parts of this trade-off. In “Quality Diversity Through Surprise,” novelty measures deviation from observed behavior, whereas surprise measures deviation from behavior predicted from recent temporal trends. Their combination with local competition produced stronger search than novelty or surprise alone in deceptive maze-navigation tasks (Gravina et al., 2018).

The principal archive metrics are:

• Quality: the fitness of individual archive entries or an aggregate such as their mean, median, or sum.
• Coverage: the fraction of behavior cells occupied.
• QD-score: commonly the sum of fitness values of occupied cells.
• Maximum fitness: the highest-quality solution in the repertoire.
• Robustness: the ability to retain successful solutions across changing conditions, noisy evaluations, or accumulating evaluation budgets.

For a discretized archive with cells $\{S_i\}_{i=1}^{M}$, a common formulation is

$$\operatorname{QD\text{-}Score} = \sum_{i=1}^{M} f(x_i), \qquad b(x_i)\in S_i,$$

where empty cells contribute zero. Coverage is

$$\operatorname{Coverage} = \frac{\#\{\text{occupied cells}\}}{M}.$$

These metrics are not interchangeable: an archive can have high maximum fitness but poor coverage, or lower maximum fitness but a higher QD-score because it contains many competent solutions.

2. Archive-based algorithms and evolutionary mechanisms

MAP-Elites

MAP-Elites is the canonical grid-based QD algorithm. It discretizes behavior space into cells and stores at most one elite per cell. The archive update is

$$\mathcal{A}(c)\leftarrow \begin{cases} (\boldsymbol{\theta},f_{\boldsymbol{\theta}}), & \mathcal{A}(c)=\varnothing,\ (\boldsymbol{\theta},f_{\boldsymbol{\theta}}), & f_{\boldsymbol{\theta}}>f_{\mathcal{A}(c)},\ \mathcal{A}(c), & \text{otherwise}. \end{cases}$$

A standard iteration selects an existing elite, applies mutation or recombination, evaluates the offspring, calculates its descriptor, assigns it to a cell, and inserts it if the cell is empty or the offspring has higher fitness than the incumbent. Uniform selection among occupied cells gives each discovered behavioral type an opportunity to generate offspring.

MAP-Elites provides direct coverage and archive visualization, but requires the descriptor dimensionality, bounds, discretization, and cell-mapping mechanism in advance. Grid archives also suffer from exponential growth: if descriptor dimension $f_{\boldsymbol{\theta}}$0 has $f_{\boldsymbol{\theta}}$1 intervals, the number of cells is

$f_{\boldsymbol{\theta}}$2

Centroidal Voronoi Tessellation MAP-Elites (CVT-MAP-Elites) replaces the regular grid with a fixed number of Voronoi regions, reducing the dependence on a rectangular discretization while still requiring a bounded behavior space or a set of centroids (Chatzilygeroudis et al., 2020).

Novelty, local competition, and surprise

Novelty Search rewards behavioral difference, commonly using average distance to the $f_{\boldsymbol{\theta}}$3 nearest behavioral neighbors:

$f_{\boldsymbol{\theta}}$4

Novelty Search with Local Competition (NS-LC) combines novelty with local performance. An individual competes against behaviorally nearby individuals rather than against the global population. This allows a globally mediocre solution to survive if it is the best representative of its behavioral neighborhood.

Surprise Search introduces a prediction model based on recent behavioral history and rewards deviation from predicted future behavior. In the maze-navigation study, recent behavioral generations were clustered with $f_{\boldsymbol{\theta}}$5-means, and cluster-centroid movement was extrapolated by linear interpolation. The principal combined method, NSS-LC, used

$f_{\boldsymbol{\theta}}$6

with local competition as a second objective. The selected configuration used $f_{\boldsymbol{\theta}}$7, $f_{\boldsymbol{\theta}}$8, $f_{\boldsymbol{\theta}}$9, $\boldsymbol{b}_{\boldsymbol{\theta}}\in\mathcal{B}$0, and $\boldsymbol{b}_{\boldsymbol{\theta}}\in\mathcal{B}$1. Across 3,000 runs per algorithm on 60 deceptive mazes, NSS-LC achieved 451 successful runs, compared with 400 for NS-LC, 344 for SS-LC, and 149 for novelty search (Gravina et al., 2018).

Emitters, learned variation, and surrogates

CMA-ME uses CMA-ES-based emitters with distinct motivations, including fitness optimization, movement in behavior-space directions, and archive improvement. Surrogate-Assisted Illumination (SAIL) combines Gaussian-process prediction, an upper-confidence-bound acquisition function, expensive evaluations, and MAP-Elites. Its acquisition function is

$\boldsymbol{b}_{\boldsymbol{\theta}}\in\mathcal{B}$2

where $\boldsymbol{b}_{\boldsymbol{\theta}}\in\mathcal{B}$3 is predicted fitness and $\boldsymbol{b}_{\boldsymbol{\theta}}\in\mathcal{B}$4 is predictive uncertainty.

Directional variation, crossover, variational autoencoders, and data-driven encodings exploit structure among high-performing genotypes. In expensive engineering design, PRODUQD combines SAIL with dimensionality reduction, DBSCAN clustering, medoid prototypes, and human selection. The designer selects a prototype class, and its members seed the next SAIL iteration. This is soft guidance rather than a hard constraint: subsequent designs can drift outside the selected class (Hagg et al., 2018).

Multi-objective QD

Multi-Objective Quality-Diversity (MOQD) stores a local Pareto front rather than a single elite per behavior cell. A solution $\boldsymbol{b}_{\boldsymbol{\theta}}\in\mathcal{B}$5 dominates $\boldsymbol{b}_{\boldsymbol{\theta}}\in\mathcal{B}$6 when it is no worse on every objective and strictly better on at least one. MOME maintains nondominated solutions within each grid cell.

MOUR-QD removes the grid and uses an unstructured archive with a distance radius $\boldsymbol{b}_{\boldsymbol{\theta}}\in\mathcal{B}$7. Nearby solutions compete through local Pareto dominance, while mutually nondominated solutions can coexist. The archive can therefore operate in learned, unbounded, or irregular feature spaces. Its quality is evaluated through projected MOQD-score, global hypervolume, and coverage. The method achieved approximately double the MOQD-score of MOME on the hopper-3 task and outperformed correctly bounded MOME on an unbounded-feature Ant task (Janmohamed et al., 28 Mar 2025).

3. Behavior representations and archive geometry

The behavior descriptor determines what counts as different. Descriptors may be hand-designed, learned from data, inferred from human judgments, derived from state-action occupancy, or defined through generic dissimilarities.

Hand-designed descriptors

Examples include final robot position, foot-contact frequencies, end-effector position, trajectory statistics, image properties, game mechanics, mana-cost distributions, block occupancy, and circuit sparsity or gate diversity. In robot navigation, the final two-dimensional position can distinguish routes that terminate in different regions. In multi-agent locomotion, foot-contact frequencies characterize team-level gait. In scheduling-heuristic evolution, four sequencing and four routing decisions are encoded as a two-dimensional map with $\boldsymbol{b}_{\boldsymbol{\theta}}\in\mathcal{B}$8 possible cells (Ingvarsson et al., 2023).

The descriptor must be sufficiently expressive to capture relevant variation, but not so high-dimensional or interdependent that the archive becomes impossible to explore. In Super Mario Bros. content generation, some combinations of gameplay mechanics are impossible because the mechanics depend on one another. Such dependencies produce unreachable regions of behavior space (Gravina et al., 2019).

Learned descriptors

AURORA uses learned latent representations from trajectories or sensory observations. QDHF learns a diversity geometry from human similarity judgments. Given a feature extractor $\boldsymbol{b}_{\boldsymbol{\theta}}\in\mathcal{B}$9 and a learned projection $$(f_{\boldsymbol{\theta}},\boldsymbol{b}_{\boldsymbol{\theta}})\leftarrow f(\boldsymbol{\theta}).$$0, the descriptor is

$$(f_{\boldsymbol{\theta}},\boldsymbol{b}_{\boldsymbol{\theta}})\leftarrow f(\boldsymbol{\theta}).$$1

QDHF collects Two Alternative Forced Choice judgments, in which an evaluator identifies which of two candidates is more similar to a reference. The learned projection is trained with a triplet loss:

$$(f_{\boldsymbol{\theta}},\boldsymbol{b}_{\boldsymbol{\theta}})\leftarrow f(\boldsymbol{\theta}).$$2

The resulting descriptor organizes a MAP-Elites archive according to human-relevant similarity rather than a manually written metric. In text-to-image generation, QDHF achieved a mean pairwise DreamSim distance of $$(f_{\boldsymbol{\theta}},\boldsymbol{b}_{\boldsymbol{\theta}})\leftarrow f(\boldsymbol{\theta}).$$3, compared with $$(f_{\boldsymbol{\theta}},\boldsymbol{b}_{\boldsymbol{\theta}})\leftarrow f(\boldsymbol{\theta}).$$4 for Best-of-$$(f_{\boldsymbol{\theta}},\boldsymbol{b}_{\boldsymbol{\theta}})\leftarrow f(\boldsymbol{\theta}).$$5, while achieving a CLIP score of $$(f_{\boldsymbol{\theta}},\boldsymbol{b}_{\boldsymbol{\theta}})\leftarrow f(\boldsymbol{\theta}).$$6, compared with $$(f_{\boldsymbol{\theta}},\boldsymbol{b}_{\boldsymbol{\theta}})\leftarrow f(\boldsymbol{\theta}).$$7 for Best-of-$$(f_{\boldsymbol{\theta}},\boldsymbol{b}_{\boldsymbol{\theta}})\leftarrow f(\boldsymbol{\theta}).$$8 (Ding et al., 2023).

Vector Quantized-Elites (VQ-Elites) uses a vector-quantized variational autoencoder to learn both a continuous behavioral representation and a discrete codebook that acts as a structured archive grid. A behavior is assigned to its nearest codebook vector:

$$(f_{\boldsymbol{\theta}},\boldsymbol{b}_{\boldsymbol{\theta}})\leftarrow f(\boldsymbol{\theta}).$$9

The codebook is updated online, and archived solutions are re-encoded and remapped after each representation update. Behavioral-space bounding constrains latent representations to $$\forall \boldsymbol{b}\in\mathcal{B},\qquad \boldsymbol{\theta}^{*}(\boldsymbol{b}) = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}) \quad \text{subject to} \quad \boldsymbol{b}_{\boldsymbol{\theta}}=\boldsymbol{b}.$$0, while a cooperation phase overwrites cells with recent candidates before ordinary fitness competition begins. VQ-Elites achieved the best QD-score in the reported L-shaped mobile-robot environment and outperformed hand-coded MAP-Elites under constrained robotic-arm conditions (Tsakonas et al., 10 Apr 2025).

Dissimilarity spaces and magnitude

A QD system need not use Euclidean behavior coordinates. A generic dissimilarity space consists of a set $$\forall \boldsymbol{b}\in\mathcal{B},\qquad \boldsymbol{\theta}^{*}(\boldsymbol{b}) = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}) \quad \text{subject to} \quad \boldsymbol{b}_{\boldsymbol{\theta}}=\boldsymbol{b}.$$1 and a symmetric, nondegenerate dissimilarity

$$\forall \boldsymbol{b}\in\mathcal{B},\qquad \boldsymbol{\theta}^{*}(\boldsymbol{b}) = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}) \quad \text{subject to} \quad \boldsymbol{b}_{\boldsymbol{\theta}}=\boldsymbol{b}.$$2

without requiring the triangle inequality. Examples include Hamming distance, sequence distances, graph dissimilarity, path-space distances, spherical distances, and distances pulled back from simulated phenotypes.

For a finite set, a similarity matrix is defined by

$$\forall \boldsymbol{b}\in\mathcal{B},\qquad \boldsymbol{\theta}^{*}(\boldsymbol{b}) = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}) \quad \text{subject to} \quad \boldsymbol{b}_{\boldsymbol{\theta}}=\boldsymbol{b}.$$3

A weighting $$\forall \boldsymbol{b}\in\mathcal{B},\qquad \boldsymbol{\theta}^{*}(\boldsymbol{b}) = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}) \quad \text{subject to} \quad \boldsymbol{b}_{\boldsymbol{\theta}}=\boldsymbol{b}.$$4 satisfies $$\forall \boldsymbol{b}\in\mathcal{B},\qquad \boldsymbol{\theta}^{*}(\boldsymbol{b}) = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}) \quad \text{subject to} \quad \boldsymbol{b}_{\boldsymbol{\theta}}=\boldsymbol{b}.$$5, and the magnitude is

$$\forall \boldsymbol{b}\in\mathcal{B},\qquad \boldsymbol{\theta}^{*}(\boldsymbol{b}) = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}) \quad \text{subject to} \quad \boldsymbol{b}_{\boldsymbol{\theta}}=\boldsymbol{b}.$$6

Magnitude is a scale-dependent effective number of points: closely clustered solutions contribute less than well-separated solutions. “Quality-diversity in dissimilarity spaces” uses magnitude to construct irregular cells from landmarks and to select Go-Explore expedition bases. This permits QD over sequences, programs, paths, binary strings, and other domains where a low-dimensional behavior grid is inappropriate (Huntsman, 2022).

Continuous illumination

Soft QD replaces discrete cells with a continuous illumination field. Each solution produces a Gaussian contribution

$$\forall \boldsymbol{b}\in\mathcal{B},\qquad \boldsymbol{\theta}^{*}(\boldsymbol{b}) = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}) \quad \text{subject to} \quad \boldsymbol{b}_{\boldsymbol{\theta}}=\boldsymbol{b}.$$7

and the population value at behavior $$\forall \boldsymbol{b}\in\mathcal{B},\qquad \boldsymbol{\theta}^{*}(\boldsymbol{b}) = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}) \quad \text{subject to} \quad \boldsymbol{b}_{\boldsymbol{\theta}}=\boldsymbol{b}.$$8 is

$$\forall \boldsymbol{b}\in\mathcal{B},\qquad \boldsymbol{\theta}^{*}(\boldsymbol{b}) = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}) \quad \text{subject to} \quad \boldsymbol{b}_{\boldsymbol{\theta}}=\boldsymbol{b}.$$9

The Soft QD Score is

$$\boldsymbol{\theta}^{*}_{g} = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}),$$0

It is monotone with respect to adding solutions and improving their qualities, and it is submodular as a set function. In the narrow-kernel limit,

$$\boldsymbol{\theta}^{*}_{g} = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}),$$1

which connects it to conventional QD-score when descriptors are distinct. SQUAD approximates this objective using pairwise Gaussian-overlap terms, producing a differentiable QD objective without explicit discretization (Hedayatian et al., 30 Nov 2025).

4. Quality mechanisms, learning paradigms, and human interaction

QD methods differ in how they preserve quality within behavioral regions.

Local competition and constraints

Local competition compares a candidate with nearby behavioral solutions. Constraint-based QD instead separates feasible and infeasible individuals. Constrained Novelty Search uses FI-2Pop, maintaining feasible and infeasible populations. Feasible solutions maximize novelty, while infeasible solutions minimize distance to feasibility or maximize novelty. Constrained MAP-Elites combines FI-2Pop with cell-based archives.

In procedural content generation, feasibility may represent playability, reachability, balance, or structural validity. QD therefore preserves diverse solutions while excluding or repairing broken artifacts. The PCG literature identifies quality and behavioral diversity as simultaneous explicit goals rather than treating diversity as merely a defense against premature convergence (Gravina et al., 2019).

Gradient-based policy ensembles

A reinforcement-learning formulation of QD can represent diversity through discounted stationary occupancy measures:

$$\boldsymbol{\theta}^{*}_{g} = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}),$$2

Stein variational policy gradients combine an environmental-return term with a repulsive kernel based on divergences between occupancy distributions. Distribution-ratio estimators such as Noise-Contrastive Estimation, DualDICE, ValueDICE, and GenDICE estimate

$$\boldsymbol{\theta}^{*}_{g} = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}),$$3

The resulting ensembles can contain high-return but behaviorally distinct peg-insertion policies, varied maze trajectories, alternative locomotion modes under deceptive rewards, and emergent locomotion under zero external reward. DualDICE and ValueDICE generally produced stronger diversity signals than NCE in the reported reward-free Hopper and Walker-2d experiments (Gangwani et al., 2020).

Diverse Quality Species (DQS) removes the explicit archive and behavior grid. It divides policies into fixed species and trains a discriminator $$\boldsymbol{\theta}^{*}_{g} = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}),$$4 to distinguish species from visited states. The species reward is

$$\boldsymbol{\theta}^{*}_{g} = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}),$$5

and the joint quality-diversity reward is

$$\boldsymbol{\theta}^{*}_{g} = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}),$$6

DQS achieved the best reported maximum fitness in all four tested simulated locomotion environments and the best QD-score in three of four, while generally providing less explicit descriptor-space coverage than archive-based methods (Wickman et al., 2023).

Multi-agent QD

Mix-ME extends MAP-Elites to cooperative partially observable multi-agent control. A solution is a team

$$\boldsymbol{\theta}^{*}_{g} = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}),$$7

Instead of mutating only a complete parent team, Mix-ME samples multiple teams and constructs a mixed team by taking agent $$\boldsymbol{\theta}^{*}_{g} = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}),$$8 from parent team $$\boldsymbol{\theta}^{*}_{g} = \arg\max_{\boldsymbol{\theta}\in\Theta} f(\boldsymbol{\theta}),$$9:

$\{S_i\}_{i=1}^{M}$0

The method combines polynomial mutation, Iso-Line variation, and agent-level team mixing. It improves maximum fitness, coverage, and QD-score over naive multi-agent MAP-Elites in Ant, HalfCheetah, and Hopper, but not in Walker2D. The results indicate that mixing is most useful when the archive contains complementary role-specialized agents (Ingvarsson et al., 2023).

Human-guided design

PRODUQD uses QD to expose classes of high-performing engineering designs. Design vectors are embedded with t-SNE, clustered by DBSCAN, and represented by medoids. The human selects a prototype or class, whose members seed the next search iteration. In the airfoil study, user-selected search produced designs more similar to the selected prototype, with lower parameter spread and comparable true fitness and surrogate accuracy to ordinary SAIL. The side-view mirror demonstration used 51 free-form-deformation parameters, a $\{S_i\}_{i=1}^{M}$1 feature map, and successive prototype branches (Hagg et al., 2018).

QDHF extends this principle by learning the diversity space from human similarity judgments rather than asking users to select archive cells. Its role is distinct from RLHF: human feedback organizes the archive into meaningful difference classes, while the task objective still determines the best solution within each class.

Language-model and creative search

QDAIF uses LLMs to generate variation, evaluate quality, and classify qualitative behavioral dimensions. MAP-Elites archives are constructed over sentiment, genre, narrative ending, poetic tone, efficiency, and abstraction. In creative writing experiments, QDAIF significantly outperformed principal non-QD controls in QD-score for opinions and story-generation tasks. In poetry, QDAIF with LMX-rewrite achieved a QD-score of $\{S_i\}_{i=1}^{M}$2, compared with 76 for Random-Poems and 99 for Fixed Seed Rewrite (Bradley et al., 2023).

The results also expose evaluator exploitation. Texts can obtain high AI scores through repetition, copying evaluation language, including titles or URLs, or producing superficial category cues. Human quality becomes less correlated with AI fitness near the highest reported scores. QD in qualitative domains therefore remains dependent on evaluator calibration, robust diversity definitions, and mitigation of reward hacking.

5. Applications and domain-specific developments

QD has been applied where multiple competent alternatives, behavioral coverage, or adaptive repertoires are useful.

Robotics and reinforcement learning

Robotic locomotion is a central QD application. MAP-Elites and CVT-MAP-Elites can construct repertoires of gaits differing in displacement, direction, trajectory, or foot-contact behavior. Such repertoires have been used for damage adaptation, where a controller compatible with a changed morphology can be selected without optimizing from scratch.

Theoretical analyses support this use beyond empirical robustness. For monotone approximately submodular maximization with a size constraint, MAP-Elites using cardinality as its descriptor finds a solution satisfying

$\{S_i\}_{i=1}^{M}$3

in expected time

$\{S_i\}_{i=1}^{M}$4

For exact submodularity, this gives the classical $\{S_i\}_{i=1}^{M}$5 approximation. On constructed instances, a matched $\{S_i\}_{i=1}^{M}$6-EA can require exponential expected time because objective-only selection removes intermediate stepping stones that MAP-Elites retains. For set cover, MAP-Elites achieves an $\{S_i\}_{i=1}^{M}$7 approximation in expected polynomial time under the analyzed representation (Qian et al., 2024).

Procedural content generation

QD supports level, weapon, dungeon, spaceship, deck, script, and structure generation. Behavior descriptors may encode playability-related mechanics, risk, bullet distributions, mana-cost distributions, death locations, visual properties, symmetry, linearity, or spatial patterns. The archive provides both usable content and an expressivity analysis of the generator.

This makes QD particularly suitable for mixed-initiative design. Designers can inspect alternative regions, select preferred artifacts, change behavioral dimensions, or continue evolution from a chosen elite. The archive is not only an output set but also a map of what the generator can express.

Engineering design

Surrogate-assisted QD has been applied to aerodynamic airfoils and vehicle side-view mirrors. In PRODUQD, t-SNE and DBSCAN compress large design archives into prototype classes, while SAIL reduces expensive evaluations using Gaussian-process surrogates and UCB acquisition. Human selection guides subsequent exploration without imposing a hard class constraint.

Aesthetic and creative evolution

In aesthetic evolution, an artist’s judgments were converted into a computational fitness proxy, while ResNet-152 image features and UMAP defined visual behavior. MAP-Elites used an $\{S_i\}_{i=1}^{M}$8 archive grid. The resulting archive contained multiple visually distinct, high-fitness line drawings, including phenotypes that the artist did not readily discover through manual search (McCormack et al., 2022).

The study also illustrates methodological limitations: fitness was specific to one artist and one drawing system, the CNN was trained on ImageNet photographs rather than line drawings, and the final comparison with manual exploration was illustrative rather than a matched controlled benchmark.

Quantum-circuit optimization

QD-CMA methods optimize complete variational quantum circuits through a direct matrix-based representation. Gate types, controlled-gate targets, and rotation parameters are jointly encoded in a continuous vector. The archive uses circuit sparsity and gate diversity as behavioral coordinates. CMA-MAE introduces an annealed archive threshold:

$\{S_i\}_{i=1}^{M}$9

For $$\operatorname{QD\text{-}Score} = \sum_{i=1}^{M} f(x_i), \qquad b(x_i)\in S_i,$$0, the method corresponds to CMA-ES; for $$\operatorname{QD\text{-}Score} = \sum_{i=1}^{M} f(x_i), \qquad b(x_i)\in S_i,$$1, to CMA-ME; and for $$\operatorname{QD\text{-}Score} = \sum_{i=1}^{M} f(x_i), \qquad b(x_i)\in S_i,$$2, to CMA-MAE. On small MaxCut, maximum clique, maximum independent set, and minimum vertex-cover instances, the CMA variants often reached optimal or near-optimal solutions within approximately 20 optimization steps and generally outperformed QAOA and QNEAT in the reported simulations (Zorn et al., 11 Apr 2025).

Scheduling heuristics

QD Genetic Programming evolves scheduling heuristics rather than individual schedules. GP trees make sequencing and routing decisions in dynamic flexible job-shop scheduling. A phenotypic characterization records decisions at selected points and maps four sequencing decisions and four routing decisions into an $$\operatorname{QD\text{-}Score} = \sum_{i=1}^{M} f(x_i), \qquad b(x_i)\in S_i,$$3 map.

For fixed training instances, QDGP increased map coverage and generally improved scheduling performance relative to conventional GP. In a fixed mean-tardiness experiment, coverage at generation 200 was approximately 11.4% for GPf, 36.1% for QDGPf, and 72.3% for QDGPf2. Under continually changing training instances, however, conventional GP generally outperformed QDGP variants. The result demonstrates that diversity can improve portfolio construction but can interfere with convergence when archive fitness is difficult to compare across contexts (Xu et al., 3 Jul 2025).

6. Scalability, uncertainty, limitations, and future directions

Resource efficiency

QD archives can be expensive when each solution is a large neural network. RefQD decomposes a policy into a shared representation part and behavior-specific decision parts:

$$\operatorname{QD\text{-}Score} = \sum_{i=1}^{M} f(x_i), \qquad b(x_i)\in S_i,$$4

Instead of storing a complete representation for every archive member, RefQD stores one shared $$\operatorname{QD\text{-}Score} = \sum_{i=1}^{M} f(x_i), \qquad b(x_i)\in S_i,$$5 and multiple decision parts $$\operatorname{QD\text{-}Score} = \sum_{i=1}^{M} f(x_i), \qquad b(x_i)\in S_i,$$6. A deep decision archive retains several decision parts per cell to mitigate mismatch as the shared representation changes. Periodic re-evaluation, top-$$\operatorname{QD\text{-}Score} = \sum_{i=1}^{M} f(x_i), \qquad b(x_i)\in S_i,$$7 refresh, learning-rate decay, and archive depth address representation drift. RefQD used approximately 16% of baseline GPU memory on QDax and 3.7% on Atari in the reported comparisons, while maintaining comparable or better performance on several tasks (Wang et al., 2024).

Noise and stochastic evaluations

Ordinary QD can preserve lucky solutions when fitness or descriptors are noisy. Extract-QD decomposes uncertain QD into selection, variation, extraction, container depth, depth ordering, and sampling modules. Extract-ME uses two initial samples, archive depth eight, a 25% extraction fraction, and rank-biased elite reevaluation. The method retains alternative candidates per cell and reallocates part of each evaluation budget to estimating elite fitness and descriptors.

On noisy and stochastic benchmarks, Extract-ME consistently matched or outperformed the best existing methods in the reported study. Extract-PGA augmented Policy-Gradient-Assisted MAP-Elites without increasing the total evaluation budget and outperformed PGA on Walker, Cheetah, and Ant (Flageat et al., 10 Feb 2025).

Archive-free and unstructured alternatives

DQS, MOUR-QD, magnitude-based Go-Explore, and Soft QD address limitations of explicit archives and fixed grids in different ways. DQS uses latent species; MOUR-QD uses local Pareto competition in an unstructured repertoire; magnitude uses a generic dissimilarity geometry; Soft QD uses a continuous illumination field. These approaches reduce dependence on manually specified bounds or discretization but introduce other requirements, such as discriminator training, distance-scale selection, neighbor search, density-ratio estimation, kernel integration, or representation stability.

VQ-Elites occupies an intermediate position: it learns the behavioral representation but retains a structured codebook grid. Its design illustrates a broader tension between explicit archive organization and adaptive representation learning.

Descriptor and metric dependence

QD performance depends critically on the descriptor or dissimilarity. A poor descriptor can:

• omit meaningful behavioral distinctions;
• preserve irrelevant variation;
• make useful regions unreachable;
• create dependent or impossible combinations;
• distort archive coverage;
• encourage solutions that are diverse but not practically useful.

Learned representations do not eliminate this issue. AURORA models statistical variance, QDHF models human similarity, VQ-Elites reconstructs raw behavior, and MOUR-QD relies on a meaningful distance metric. Each representation encodes a different notion of difference.

Quality-diversity trade-offs

Diversity can reduce individual return, especially in reinforcement learning, where varied trajectories may be less reward-optimal than a single shortest path. Archive-based methods can preserve low-quality but behaviorally distinct regions, while quality-coupled methods such as DQS may focus on high-return behaviors and leave low-quality regions unexplored. In multi-agent control, a method can have lower maximum fitness but higher QD-score because it populates more competent behavioral cells.

More diversity is not always preferred by users. In text-to-image generation, QDHF produced higher perceived diversity, but a majority of participants preferred the less diverse baseline for one astronaut-and-horse prompt. QD therefore exposes a preference trade-off rather than establishing that maximal diversity is universally desirable.

Theoretical status

Formal theory remains narrower than the application literature. The analysis of MAP-Elites on submodular maximization and set cover provides rigorous expected-runtime and approximation results under specific descriptors, mutation operators, archive sizes, and replacement rules. Soft QD provides monotonicity, submodularity, and a limit connecting continuous illumination to QD-score. Magnitude-based QD provides mathematical diversity results under positive-definite similarity matrices.

These results do not imply that every QD algorithm, descriptor, archive, or selection rule outperforms conventional optimization. Most empirical methods lack general convergence, approximation, or regret guarantees. Their behavior depends on representation quality, archive resolution, variation operators, evaluation noise, computational budget, and the relationship between descriptor diversity and task utility.

Open research problems

Active directions include:

• learned, human-aligned, multi-scale, and hierarchical behavior representations;
• adaptive archive geometry and descriptor bounds;
• scalable magnitude and nearest-neighbor computation;
• uncertainty-aware archive selection and confidence estimation;
• resource-efficient storage of large neural policies;
• learned compatibility models for multi-agent policy mixing;
• dynamic and instance-conditioned archives for changing optimization problems;
• stronger evaluation of reproducibility and risk-sensitive quality;
• robust language-model and multimodal evaluators;
• explicit expressivity, trainability, and hardware-noise measures for quantum circuits;
• integration of QD with surrogate models, policy gradients, CMA-ES, reinforcement learning, and generative models;
• quality-diversity metrics that remain meaningful in continuous, irregular, learned, or high-dimensional behavior spaces.

QD is therefore best understood not as a single algorithm but as a design paradigm. Its defining operation is the preservation of high-quality alternatives across a structured notion of behavior. That structure may be a grid, a Voronoi archive, a dissimilarity-induced repertoire, a learned latent space, a human similarity geometry, a species partition, or a continuous illumination field. The central methodological question is consequently not only how to optimize quality, but also which differences are worth preserving and how those differences should shape search.