Soft QD Using Approximated Diversity (SQUAD)

• The paper introduces SQUAD—a gradient-based, continuous quality-diversity optimization method that leverages a differentiable lower bound surrogate to maximize aggregate illumination over behavior space.
• It employs kernel-based interactions and pairwise repulsion to balance quality rewards with diversity constraints, effectively scaling to high-dimensional and large-population problems.
• Empirical evaluations on tasks such as LP, IC, and LSI demonstrate that SQUAD outperforms several state-of-the-art QD benchmarks with superior metrics like QVS and QD-Score.

Soft QD Using Approximated Diversity (SQUAD) is a differentiable, population-based optimization algorithm that reframes Quality-Diversity (QD) as continuous attraction-repulsion in behavior space. SQUAD circumvents the need for explicit discretization of the behavior space, scaling efficiently to high dimensions and large populations while preserving or outperforming state-of-the-art QD benchmarks. The approach formalizes QD objectives as maximization of aggregate "illumination" from a set of solutions over an abstract behavior space, using kernel-based interactions and a tractable differentiable approximation amenable to gradient-based optimization (Hedayatian et al., 30 Nov 2025).

1. Soft QD Objective: Definition and Intuition

Let $\Theta = \{\theta_1, ..., \theta_N\}$ denote a population of parameter vectors, where $f(\theta) \in \mathbb{R}_+$ is a differentiable quality (objective) function and $$\mathrm{desc}(\theta) \in \mathcal{B} \subseteq \mathbb{R}^d$$ a differentiable behavior descriptor. With $f_n = f(\theta_n)$ and $b_n = \mathrm{desc}(\theta_n)$, each solution is treated as an isotropic Gaussian "light source" in behavior space, its "brightness" $f_n$ decaying by bandwidth $\sigma > 0$.

The induced behavior-value field is

$$v_{\Theta}(b) = \max_{1 \leq n \leq N} f_n \exp\left(-\frac{\|b - b_n\|^2}{2 \sigma^2}\right),$$

and the Soft QD Score is defined as the total illumination:

$$S(\Theta) = \int_{b \in \mathbb{R}^d} v_{\Theta}(b) \, db.$$

Direct optimization of $S(\Theta)$ is intractable, so SQUAD proceeds via a tractable lower bound. By applying inclusion-exclusion, truncating at pairwise terms, and bounding $f(\theta) \in \mathbb{R}_+$0 by $f(\theta) \in \mathbb{R}_+$1, one obtains

$f(\theta) \in \mathbb{R}_+$2

where $f(\theta) \in \mathbb{R}_+$3.

• The sum of $f(\theta) \in \mathbb{R}_+$4 rewards high-quality solutions.
• The pairwise repulsion term, exponentially decaying with behavioral distance and weighted by $f(\theta) \in \mathbb{R}_+$5, enforces diversity.

2. Derivation, Differentiability, and Limit Properties

The lower bound $f(\theta) \in \mathbb{R}_+$6 derives from the inclusion-exclusion form:

$f(\theta) \in \mathbb{R}_+$7

For $f(\theta) \in \mathbb{R}_+$8, the $f(\theta) \in \mathbb{R}_+$9 term is tightly upper bounded by the geometric mean, which admits Gaussian integral solutions.

The resulting objective, dropping constant factors and using $$\mathrm{desc}(\theta) \in \mathcal{B} \subseteq \mathbb{R}^d$$0:

$$\mathrm{desc}(\theta) \in \mathcal{B} \subseteq \mathbb{R}^d$$1

If $$\mathrm{desc}(\theta) \in \mathcal{B} \subseteq \mathbb{R}^d$$2 and $$\mathrm{desc}(\theta) \in \mathcal{B} \subseteq \mathbb{R}^d$$3 are differentiable, so is $$\mathrm{desc}(\theta) \in \mathcal{B} \subseteq \mathbb{R}^d$$4. The per-solution gradient is: $$\mathrm{desc}(\theta) \in \mathcal{B} \subseteq \mathbb{R}^d$$5

Appendix proofs confirm key theoretical properties:

• $$\mathrm{desc}(\theta) \in \mathcal{B} \subseteq \mathbb{R}^d$$6 is nondecreasing under addition of new solutions or increase in $$\mathrm{desc}(\theta) \in \mathcal{B} \subseteq \mathbb{R}^d$$7.
• $$\mathrm{desc}(\theta) \in \mathcal{B} \subseteq \mathbb{R}^d$$8 is submodular; marginal gains diminish as the population grows.
• In the limit $$\mathrm{desc}(\theta) \in \mathcal{B} \subseteq \mathbb{R}^d$$9 ($f_n = f(\theta_n)$0), $f_n = f(\theta_n)$1, i.e., soft QD recovers the canonical QD-Score on a fine grid.

3. SQUAD Algorithm and Optimization

The SQUAD algorithm optimizes the lower bound $f_n = f(\theta_n)$2 using mini-batch stochastic gradient ascent. The procedure is as follows:

• Inputs: population size $f_n = f(\theta_n)$3, batch size $f_n = f(\theta_n)$4, neighbor count $f_n = f(\theta_n)$5, iterations $f_n = f(\theta_n)$6, diversity bandwidth $f_n = f(\theta_n)$7, optimizer $f_n = f(\theta_n)$8 (e.g., Adam with learning rate $f_n = f(\theta_n)$9).
• Initialization: sample $b_n = \mathrm{desc}(\theta_n)$0; compute qualities $b_n = \mathrm{desc}(\theta_n)$1 and behaviors $b_n = \mathrm{desc}(\theta_n)$2; initialize optimizer state $b_n = \mathrm{desc}(\theta_n)$3.
• Loop: For $b_n = \mathrm{desc}(\theta_n)$4:
  • Choose batch $b_n = \mathrm{desc}(\theta_n)$5 of size $b_n = \mathrm{desc}(\theta_n)$6.
  • For each $b_n = \mathrm{desc}(\theta_n)$7:
  • Identify $b_n = \mathrm{desc}(\theta_n)$8 nearest neighbors $b_n = \mathrm{desc}(\theta_n)$9 in behavior space.
  • Compute

  $f_n$0 - Update $f_n$1 using $f_n$2 via $f_n$3; re-evaluate $f_n$4.

• Termination: Return $f_n$5 at final iteration.

Key hyperparameters: $f_n$6, and the Gaussian kernel width $f_n$7 if computing $f_n$8 directly.

4. Theoretical Properties and Scalability

SQUAD inherits the following properties:

• Monotonicity: $f_n$9 is nondecreasing as new solutions are added or $\sigma > 0$0 is increased.
• Submodularity: Diminishing returns property enables approximate optimality under cardinality constraints.
• Limiting Behavior: For $\sigma > 0$1, SQUAD converges to standard QD-Score maximization over a grid.
• Curse-of-Dimensionality Avoidance: Does not require discretization or archives, and uses continuous, kernel-based repulsion, resulting in memory requirements invariant to behavior-space dimension $\sigma > 0$2.
• Approximation Error: Contributions from neglected higher-order overlaps in the inclusion-exclusion expansion are bounded and decay as behavioral coverage increases.

These properties collectively underpin SQUAD’s ability to scale to high-dimensional behavior spaces and large solution populations.

5. Empirical Evaluation

SQUAD's performance was benchmarked on diverse tasks and compared to existing methods including CMA-MEGA, CMA-MAEGA, Sep-CMA-MAE, GA-ME, DNS, and DNS-G.

Tasks and Metrics

• LP: Linear-Projection Rastrigin, solution dim $\sigma > 0$3, behavior dims $\sigma > 0$4.
• IC: Image Composition (1024 circles, 5-d behavior).
• LSI: Latent-Space Illumination via StyleGAN2+CLIP, $\sigma > 0$5 and $\sigma > 0$6.

Metrics included QD-Score (sum of best-in-cell over CVT), coverage (number of occupied CVT cells), Vendi Score (VS, effective number of clusters), QVS (mean-quality × VS), mean and max objective.

Outcome Summary

• LP: At $\sigma > 0$7, CMA-MAEGA/CMA-MEGA slightly outperform SQUAD, but SQUAD surpasses all baselines in both QVS and QD-Score at $\sigma > 0$8 and $\sigma > 0$9 (for example, at $$v_{\Theta}(b) = \max_{1 \leq n \leq N} f_n \exp\left(-\frac{\|b - b_n\|^2}{2 \sigma^2}\right),$$0 QVS: SQUAD $$v_{\Theta}(b) = \max_{1 \leq n \leq N} f_n \exp\left(-\frac{\|b - b_n\|^2}{2 \sigma^2}\right),$$1, CMA-MAEGA $$v_{\Theta}(b) = \max_{1 \leq n \leq N} f_n \exp\left(-\frac{\|b - b_n\|^2}{2 \sigma^2}\right),$$2, CMA-MEGA $$v_{\Theta}(b) = \max_{1 \leq n \leq N} f_n \exp\left(-\frac{\|b - b_n\|^2}{2 \sigma^2}\right),$$3). Gradient-based methods, including SQUAD, outperform mutation-only approaches at higher dimensionalities.
• IC: SQUAD achieves highest mean objective ($$v_{\Theta}(b) = \max_{1 \leq n \leq N} f_n \exp\left(-\frac{\|b - b_n\|^2}{2 \sigma^2}\right),$$4), highest max objective ($$v_{\Theta}(b) = \max_{1 \leq n \leq N} f_n \exp\left(-\frac{\|b - b_n\|^2}{2 \sigma^2}\right),$$5), and best VS ($$v_{\Theta}(b) = \max_{1 \leq n \leq N} f_n \exp\left(-\frac{\|b - b_n\|^2}{2 \sigma^2}\right),$$6). Coverage is slightly below CMA-MAEGA (5.68 vs 5.85), but VS more accurately reflects true diversity.
• Quality-Diversity Trade-Off: Varying $$v_{\Theta}(b) = \max_{1 \leq n \leq N} f_n \exp\left(-\frac{\|b - b_n\|^2}{2 \sigma^2}\right),$$7 in $$v_{\Theta}(b) = \max_{1 \leq n \leq N} f_n \exp\left(-\frac{\|b - b_n\|^2}{2 \sigma^2}\right),$$8 shows a tunable trade-off—higher diversity (VS) for larger $$v_{\Theta}(b) = \max_{1 \leq n \leq N} f_n \exp\left(-\frac{\|b - b_n\|^2}{2 \sigma^2}\right),$$9 at the expense of mean objective.
• LSI: In base ($$S(\Theta) = \int_{b \in \mathbb{R}^d} v_{\Theta}(b) \, db.$$0), SQUAD QD-Score $$S(\Theta) = \int_{b \in \mathbb{R}^d} v_{\Theta}(b) \, db.$$1, QVS $$S(\Theta) = \int_{b \in \mathbb{R}^d} v_{\Theta}(b) \, db.$$2, surpassing CMA-MEGA ($$S(\Theta) = \int_{b \in \mathbb{R}^d} v_{\Theta}(b) \, db.$$3), CMA-MAEGA ($$S(\Theta) = \int_{b \in \mathbb{R}^d} v_{\Theta}(b) \, db.$$4). In hard ($$S(\Theta) = \int_{b \in \mathbb{R}^d} v_{\Theta}(b) \, db.$$5), SQUAD ($$S(\Theta) = \int_{b \in \mathbb{R}^d} v_{\Theta}(b) \, db.$$6) vs best baseline ($$S(\Theta) = \int_{b \in \mathbb{R}^d} v_{\Theta}(b) \, db.$$7). Other methods often experience failure (negative mean objectives).

6. Implementation and Practical Considerations

Default Hyperparameters

• IC/LP: $$S(\Theta) = \int_{b \in \mathbb{R}^d} v_{\Theta}(b) \, db.$$8, $$S(\Theta) = \int_{b \in \mathbb{R}^d} v_{\Theta}(b) \, db.$$9, $S(\Theta)$0, $S(\Theta)$1, $S(\Theta)$2 tuned to domain (IC: $S(\Theta)$3; LP: easy/medium/hard $S(\Theta)$4).
• LSI: $S(\Theta)$5, $S(\Theta)$6, $S(\Theta)$7, $S(\Theta)$8, $S(\Theta)$9.

Algorithmic Details

• If $f(\theta) \in \mathbb{R}_+$00, behaviors should be mapped via $f(\theta) \in \mathbb{R}_+$01 to $f(\theta) \in \mathbb{R}_+$02; ablation indicates this is critical.
• Each batch update computes $f(\theta) \in \mathbb{R}_+$03 pairwise and $f(\theta) \in \mathbb{R}_+$04 quality terms, with overall iteration cost $f(\theta) \in \mathbb{R}_+$05.
• Mini-batching and limited nearest neighbors optimize memory efficiency.

Implementation Tips

• Automatic differentiation frameworks (JAX, PyTorch, etc.) are recommended for both objective and descriptor.
• Precompute and cache k-NN structures in behavior space.
• Annealing $f(\theta) \in \mathbb{R}_+$06 or adapting $f(\theta) \in \mathbb{R}_+$07 to local density can improve performance.
• Monitor VS, coverage, and mean objective during optimization; early stopping is often effective (e.g., $f(\theta) \in \mathbb{R}_+$08200 iterations for IC/LSI).
• Ensure $f(\theta) \in \mathbb{R}_+$09 is nonnegative for meaningful QVS evaluation.

Computational Cost

• Simple tasks (LP) complete in under 1 minute.
• IC requires $f(\theta) \in \mathbb{R}_+$10190 minutes for 1000 iterations (RTX 4090), but high performance is typically attained in fewer than 200 iterations.
• LSI (base/hard): $f(\theta) \in \mathbb{R}_+$11730/1300 minutes, with convergence in substantially less than the full budget.

7. Significance in Quality-Diversity Optimization

SQUAD provides an alternative to archive-based QD: it offers a smooth, differentiable objective and adaptively balances quality and diversity through a tunable, analytically tractable surrogate. This formulation permits large-scale, high-dimensional QD optimization previously infeasible with grid-based methods. Empirical evidence demonstrates competitiveness and often superiority versus established QD algorithms on standard benchmarks, with additional robustness and scalability. These features make SQUAD a theoretically well-founded and practically effective approach for large-scale, high-diversity optimization tasks (Hedayatian et al., 30 Nov 2025).