---
title: Hyper-Ellipsoid Space Partitioning
url: https://www.emergentmind.com/topics/hyper-ellipsoid-space-partitioning-hesp
type: topic
---

# Hyper-Ellipsoid Space Partitioning

Hyper-ellipsoid space partitioning (HESP), in the sense used by "MOCA-HESP: Meta High-dimensional Bayesian Optimization for Combinatorial and Mixed Spaces via Hyper-ellipsoid Partitioning," is a CMA-ES-style, Gaussian-based hyper-ellipsoid partitioning of an encoded mixed space for high-dimensional Bayesian optimization over combinatorial and mixed variables [2508.06847]. It maintains a multivariate normal search distribution over the encoded domain, defines local search regions as confidence hyper-ellipsoids of that distribution, and uses those regions as optimizer-specific trust regions for standard BO, CASMOPOLITAN, and Bounce [2508.06847]. Related arXiv work employs ellipsoidal partitioning for supervised classification rather than Bayesian optimization, most notably SEP-C and the Ellipsoidal Separation Machine [2302.10487] [2507.20698].

## 1. Formal definition and geometric structure

In MOCA-HESP, the encoded search space is \(\mathcal{Z}_e \subset \mathbb{R}^d\), and HESP maintains at iteration \(t\) a Gaussian search distribution
\[
\mathcal{N}_{\mathcal{Z}_e}^{(t)} = \mathcal{N}_{\mathcal{Z}_e}(\mathbf{m}^{(t)}, \mathbf{\Sigma}^{(t)}),
\]
where \(\mathbf{m}^{(t)} \in \mathbb{R}^d\) is the mean vector and \(\mathbf{\Sigma}^{(t)} \in \mathbb{R}^{d \times d}\) is a positive-definite covariance matrix. The Mahalanobis distance of a point \(\mathbf{z}_e\) from that distribution is
\[
\mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr)
=
\sqrt{(\mathbf{z}_e-\mathbf{m}^{(t)})^\top \bigl(\mathbf{\Sigma}^{(t)}\bigr)^{-1}(\mathbf{z}_e-\mathbf{m}^{(t)})}.
\]
An \(\alpha\)-confidence hyper-ellipsoid is then defined as
\[
\mathcal{E}^{(t)}
=
\left\{
\mathbf{z}_e \in \mathbb{R}^d
\;\middle|\;
\mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr)
\le
\chi^2_{1-\alpha}(d)
\right\}.
\]
This is the basic hyper-ellipsoid region used by HESP [2508.06847].

The metric structure is mixed. Continuous or encoded coordinates are handled by Mahalanobis distance, whereas categorical similarity is measured after decoding by Hamming distance,
\[
\mathbb{H}(\mathbf{h}_1,\mathbf{h}_2)=\sum_{i=1}^{d_h}\mathbb{I}(h_{1,i}\neq h_{2,i}),
\]
when HESP is integrated with CASMOPOLITAN or Bounce. This dual geometry is central: the ellipsoid is defined in the encoded continuous space, but categorical proximity can still be enforced in the original discrete representation [2508.06847].

Given the eigendecomposition \(\mathbf{\Sigma}^{(t)}=\mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^\top\), with \(\mathbf{\Lambda}=\mathrm{diag}(\lambda_1,\dots,\lambda_d)\), the ellipsoid boundary can be written as
\[
\mathcal{E}_b^{(t)}
=
\left\{
\mathbf{m}^{(t)}+\mathbf{Q}\mathbf{\Lambda}^{1/2}\mathbf{u}
:\;
\|\mathbf{u}\|_2 \le \chi^2_{1-\alpha}(d)
\right\}.
\]
The paper notes that volume is not explicitly used in the algorithm, even though the standard ellipsoid volume formula is available [2508.06847].

## 2. Moving local regions and effective partitioning

In MOCA-HESP, HESP is used to define and adapt a *single* moving hyper-ellipsoid region at any time, but through restarts and iterative updates it effectively partitions and explores different promising subregions of the high-dimensional space [2508.06847]. This point is important because HESP is not presented as a static tessellation of the search domain. The base local region is
\[
\mathcal{E}_b^{(t)}
=
\left\{
\mathbf{z}_e \in \mathbb{R}^d
\;\middle|\;
\mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr)
\le
\chi^2_{1-\alpha}(d)
\right\}.
\]

For optimizer variants that distinguish continuous and combinatorial dimensions, the covariance is scaled by a diagonal vector \(\mathbf{L}\) with entries
\[
L_i=
\begin{cases}
L_x^2 & \text{if dim } i \text{ corresponds to a continuous variable},\\
1 & \text{if dim } i \text{ corresponds to a combinatorial variable}.
\end{cases}
\]
Using the elementwise scaling operator \(\psi(\mathbf{\Sigma},\mathbf{L})\), HESP forms a scaled Gaussian and hence a scaled Mahalanobis ellipsoid. For CASMOPOLITAN-like behavior, that ellipsoid is intersected with a Hamming-ball constraint in the original discrete space; for Bounce, an analogous construction is used in the low-dimensional encoded subspace \(\mathcal{V}\) [2508.06847].

The update mechanism follows CMA-ES-style search-distribution adaptation. From a batch of \(\lambda\) encoded points and their function values, CMA-ES updates the mean, covariance, and step size; the new Gaussian defines the next ellipsoid. The high-level MOCA-HESP loop consists of an outer restart structure and an inner batch loop. Each restart initializes encoder weights, chooses an encoder, draws an initial design, encodes it, initializes \(\mathbf{m}^{(0)}\) and \(\mathbf{\Sigma}^{(0)}\), and defines the initial ellipsoid. Each inner iteration then performs local optimization inside the current ellipsoid with a chosen BO optimizer, evaluates the batch, updates the search distribution, reconstructs the optimizer-specific local region, applies adaptive encoder selection, and checks restart criteria [2508.06847].

Candidate allocation is implemented by sampling from the current Gaussian and rejecting points outside the current ellipsoid. In CASMOPOLITAN and Bounce variants, candidates must also satisfy the decoded Hamming constraint; only points satisfying all constraints form the pool over which the acquisition function is optimized. This makes HESP a geometric filter around the optimizer’s existing proposal mechanism rather than a replacement for the surrogate or acquisition layer [2508.06847].

## 3. Encoded mixed spaces and optimizer coupling

The original mixed domain is
\[
\mathcal{Z}=\mathcal{X}\times\mathcal{H},
\]
with continuous variables \(\mathcal{X}\subset\mathbb{R}^{d_x}\) and combinatorial variables \(\mathcal{H}=\mathcal{W}\times\mathcal{Q}\), where \(\mathcal{W}\subset\mathbb{Z}^{d_w}\) is ordinal and \(\mathcal{Q}\) is categorical. MOCA-HESP treats all combinatorial variables as categorical. It defines encoders
\[
g_k:\mathcal{H}\rightarrow\mathcal{H}_e\subset\mathbb{R}^{d_h},
\]
so that the encoded mixed domain becomes
\[
\mathcal{Z}_e=\mathcal{X}\times\mathcal{H}_e\subset\mathbb{R}^d
\]
[2508.06847].

Two encoders are used as bandit arms. The ordinal encoder assigns distinct numeric codes to the categories of each variable and does not require training. The target encoder uses class-conditional means, counts, a global mean, and a smoothing weight \(m>0\), yielding an empirical Bayes shrinkage toward the global mean. Decoding is performed by nearest code:
\[
\tilde{g}(x)=u_{i^*},
\qquad
i^*=\arg\min_{i\in\{1,\dots,c\}} |g(u_i)-x|.
\]
Applied coordinatewise, this decoder maps back from \(\mathcal{H}_e\) to \(\mathcal{H}\), while leaving continuous coordinates unchanged [2508.06847].

HESP itself operates purely on the encoded continuous space \(\mathcal{Z}_e\) and uses standard continuous covariance and Mahalanobis distance. The underlying BO optimizer may nonetheless use its own modeling encoding and kernel, and those are independent from the HESP encoding used for ellipsoid construction. This independence is what allows MOCA-HESP to wrap an existing optimizer without changing its internal surrogate or kernel [2508.06847].

| Variant | Surrogate and acquisition | HESP-constrained region |
|---|---|---|
| MOCA-HESP-BO | Matern 5/2 ARD kernel; Thompson Sampling | candidates sampled within \(\mathcal{E}_b^{(t)}\) |
| MOCA-HESP-Casmo | Transformed Overlapped kernel; Thompson Sampling | Mahalanobis ellipsoid plus Hamming constraint |
| MOCA-HESP-Bounce | Overlapped kernel with one-hot encoding; Expected Improvement | low-dimensional \(\mathcal{V}\) with Mahalanobis and Hamming constraints |

The optimizer-specific trust-region parameters are inherited from the wrapped methods. For MOCA-HESP-Casmo, the paper uses \(L_x \in [2^{-7},1.6]\), \(L_h \in [1,d]\), and success/failure thresholds \(\tau_{\text{succ}}=3\), \(\tau_{\text{fail}}=40\). For MOCA-HESP-Bounce, it uses \(L_x \in [2^{-7},1.6]\), \(L_h \in [1,d_\mathcal{A}]\), and dynamic trust-region updates after each iteration [2508.06847].

## 4. Adaptive encoder selection by EXP3

A distinctive feature of MOCA-HESP is that the categorical encoder is selected adaptively by a multi-armed bandit. Each encoder \(k\in\{1,\dots,K\}\) is treated as an arm, and the paper employs EXP3 to adaptively choose the best encoder based on optimization progress [2508.06847].

If \(w_k(t)\) is the weight of encoder \(k\) at iteration \(t\), the selection probability is
\[
p_k(t)
=
(1-\eta)\frac{w_k(t)}{\sum_{j=1}^K w_j(t)}
+
\frac{\eta}{K},
\]
where \(\eta \in [0,1]\) controls exploration versus exploitation. After each batch of \(\lambda\) evaluations, the objective values observed so far are normalized by min-max scaling,
\[
\hat{y}_i
=
\frac{y_i-\max_j y_j}{\min_j y_j-\max_j y_j}
\in[0,1].
\]
The batch reward for minimization is then
\[
r(t)=\min\{\hat{y}_i \mid i=n-\lambda+1,\dots,n\},
\]
and the unbiased reward estimate is
\[
\hat{r}(t)=\frac{r(t)}{p_{k_t}(t)},
\]
where \(k_t\) is the encoder used at iteration \(t\). The selected encoder weight is updated as
\[
w_{k_t}(t+1)=w_{k_t}(t)\exp\!\left(\frac{\eta \hat{r}(t)}{K}\right),
\]
while the other weights remain unchanged [2508.06847].

The practical learning-rate prescription is
\[
\eta
=
\min\left\{
1,\sqrt{\frac{K\ln K}{(1-e)N}}
\right\},
\]
with \(K\) the number of encoders and \(N\) the maximum number of iterations. The paper states that this follows the original EXP3 analysis. It also states that the encoder adaptation adds negligible overhead. The theoretical justification is correspondingly modular: HESP inherits its search-distribution behavior from CMA-ES, while the encoder selector inherits adversarial regret guarantees from EXP3. The paper does not present new convergence theorems specific to HESP in combinatorial or mixed BO [2508.06847].

## 5. Complexity, practical settings, and empirical behavior

HESP’s primary overhead is maintaining and updating the Gaussian search distribution, which is the same complexity as CMA-ES. With encoded dimension \(d=d_x+d_h\), the covariance and mean update is \(O(d^2)\) per iteration, with occasional \(O(d^3)\) eigendecomposition depending on implementation; sampling from the multivariate Gaussian is \(O(d^2)\) per sample with full covariance; checking Mahalanobis distance is \(O(d^2)\) naively or \(O(d)\) if pre-whitened; and Hamming computations for CASMOPOLITAN or Bounce regions are \(O(d_h)\) per candidate. Storage is \(O(d)\) for the mean and \(O(d^2)\) for the covariance, with additional projection matrices \(\mathbf{Q}\) and \(\mathbf{P}\) for Bounce. The paper reports that surrogate model cost and acquisition optimization often dominate, and that HESP adds “minimal computation overhead” in practice [2508.06847].

The implementation details follow standard CMA-ES initialization. The population size is \(\lambda = 4 + \lfloor 3 \ln d \rfloor\); the initial mean is the best point among the initial design \(\mathcal{D}_0\); the initial covariance is the identity; the initial step size is \(\sigma^{(0)}=0.3(u-l)\) after scaling the domain to \([l,u]^d\); and a lower bound on the standard deviation in discrete dimensions is recommended to avoid collapse in combinatorial coordinates. The paper also recommends using both ordinal and target encoders, with target-encoder smoothing taken from a standard library default [2508.06847].

The empirical study covers 9 benchmarks: combinatorial Ackley20c, Shifted Ackley20c, Antibody Design, LABS, Shifted LABS, MaxSAT28, and MaxSAT125, together with mixed Ackley53m and Cellular Network. Dimensions range from 11 to 125. “Shifted” versions randomize categorical labels to remove special structural patterns. The baselines are standard BO, CASMOPOLITAN, Bounce, TPE, SMAC, CMA-ES with ordinal encoding, and random search. MOCA-HESP-BO consistently outperforms standard BO on all 9 benchmarks and reaches the global optimum on Ackley20c within the evaluation budget. MOCA-HESP-Casmo is better on 7 of 9 problems and at least on par on the others. MOCA-HESP-Bounce outperforms Bounce on 6 benchmarks and eventually finds better solutions on MaxSAT125 after similar early performance. All proposed MOCA-HESP variants generally dominate TPE, SMAC, CMA-ES, and random search across the reported tasks [2508.06847].

The ablation on encoder choice shows that no single encoder dominates: ordinal is strong on some problems such as LABS and MaxSAT125, whereas the target encoder is stronger on others such as Antibody Design. The adaptive EXP3 version matches or outperforms the best fixed-encoder variant on almost all benchmarks. The paper’s 2D visualizations on discretized Ackley and Shifted Ackley further show that the initial ellipsoid is broad and roughly placed, then moves toward the global minimum over iterations, while the ellipsoid radii remain bounded away from zero because of minimum-standard-deviation constraints in discrete CMA-ES, ensuring continued exploration [2508.06847].

## 6. Related ellipsoidal methods and scope of the term

The expression “hyper-ellipsoid space partitioning” is not tied to a single methodology across the literature. In supervised learning, SEP-C is a convex-optimization-based classifier that sequentially partitions a dataset into several ellipsoids, where each ellipsoid contains nearly all points of the same label; Bayes’ formula is then applied to calculate a trust score for the predicted label [2302.10487]. Its core machinery is different from MOCA-HESP: SEP-C uses minimum-volume ellipsoids, reduced convex hull splits, impurity control through \(n_{\text{Imp}}\), and explicit decision rules for points lying in one ellipsoid, in intersections, or in no ellipsoid at all [2302.10487].

A second related line is the Ellipsoidal Separation Machine. ESM separates two data sets using an ellipsoid, formulates training as a convex problem that can be written as a semidefinite program, and then uses a more scalable nonconvex formulation amenable to block-Gauss-Seidel updates alternating between a smaller SDP and a separable SOCP. A characteristic of ESM is that it necessarily defines indeterminate points, making it suitable for classification with rejection [2507.20698].

A plausible implication is that hyper-ellipsoid partitioning has developed along at least two technical lines: sequential or separating ellipsoids for supervised classification, and moving Gaussian confidence ellipsoids for high-dimensional Bayesian optimization. In that narrower and acronym-bearing sense, HESP in MOCA-HESP denotes a search-distribution-based local-region mechanism for combinatorial and mixed BO, not a general-purpose classifier [2508.06847].

Source: https://www.emergentmind.com/topics/hyper-ellipsoid-space-partitioning-hesp