Hyper-Ellipsoid Space Partitioning

• The paper introduces MOCA-HESP, which uses a CMA-ES-style Gaussian approach to create confidence hyper-ellipsoids that guide local search in combinatorial and mixed spaces.
• Hyper-Ellipsoid Space Partitioning defines search regions via Mahalanobis distance in an encoded continuous space while incorporating Hamming distance for categorical variables.
• Adaptive encoder selection via the EXP3 algorithm ensures robust performance by dynamically choosing between ordinal and target encoders during optimization.

Hyper-ellipsoid space partitioning (HESP), in the sense used by "MOCA-HESP: Meta High-dimensional Bayesian Optimization for Combinatorial and Mixed Spaces via Hyper-ellipsoid Partitioning," is a CMA-ES-style, Gaussian-based hyper-ellipsoid partitioning of an encoded mixed space for high-dimensional Bayesian optimization over combinatorial and mixed variables (Ngo et al., 9 Aug 2025). It maintains a multivariate normal search distribution over the encoded domain, defines local search regions as confidence hyper-ellipsoids of that distribution, and uses those regions as optimizer-specific trust regions for standard BO, CASMOPOLITAN, and Bounce (Ngo et al., 9 Aug 2025). Related arXiv work employs ellipsoidal partitioning for supervised classification rather than Bayesian optimization, most notably SEP-C and the Ellipsoidal Separation Machine (Niranjan et al., 2023, Frangioni et al., 28 Jul 2025).

1. Formal definition and geometric structure

In MOCA-HESP, the encoded search space is $\mathcal{Z}_e \subset \mathbb{R}^d$, and HESP maintains at iteration $t$ a Gaussian search distribution

$$\mathcal{N}_{\mathcal{Z}_e}^{(t)} = \mathcal{N}_{\mathcal{Z}_e}(\mathbf{m}^{(t)}, \mathbf{\Sigma}^{(t)}),$$

where $\mathbf{m}^{(t)} \in \mathbb{R}^d$ is the mean vector and $\mathbf{\Sigma}^{(t)} \in \mathbb{R}^{d \times d}$ is a positive-definite covariance matrix. The Mahalanobis distance of a point $\mathbf{z}_e$ from that distribution is

$$\mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr) = \sqrt{(\mathbf{z}_e-\mathbf{m}^{(t)})^\top \bigl(\mathbf{\Sigma}^{(t)}\bigr)^{-1}(\mathbf{z}_e-\mathbf{m}^{(t)})}.$$

An $\alpha$-confidence hyper-ellipsoid is then defined as

$$\mathcal{E}^{(t)} = \left\{ \mathbf{z}_e \in \mathbb{R}^d \;\middle|\; \mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr) \le \chi^2_{1-\alpha}(d) \right\}.$$

This is the basic hyper-ellipsoid region used by HESP (Ngo et al., 9 Aug 2025).

The metric structure is mixed. Continuous or encoded coordinates are handled by Mahalanobis distance, whereas categorical similarity is measured after decoding by Hamming distance,

$$\mathbb{H}(\mathbf{h}_1,\mathbf{h}_2)=\sum_{i=1}^{d_h}\mathbb{I}(h_{1,i}\neq h_{2,i}),$$

when HESP is integrated with CASMOPOLITAN or Bounce. This dual geometry is central: the ellipsoid is defined in the encoded continuous space, but categorical proximity can still be enforced in the original discrete representation (Ngo et al., 9 Aug 2025).

Given the eigendecomposition $t$0, with $t$1, the ellipsoid boundary can be written as

$t$2

The paper notes that volume is not explicitly used in the algorithm, even though the standard ellipsoid volume formula is available (Ngo et al., 9 Aug 2025).

2. Moving local regions and effective partitioning

In MOCA-HESP, HESP is used to define and adapt a single moving hyper-ellipsoid region at any time, but through restarts and iterative updates it effectively partitions and explores different promising subregions of the high-dimensional space (Ngo et al., 9 Aug 2025). This point is important because HESP is not presented as a static tessellation of the search domain. The base local region is

$t$3

For optimizer variants that distinguish continuous and combinatorial dimensions, the covariance is scaled by a diagonal vector $t$4 with entries

$t$5

Using the elementwise scaling operator $t$6, HESP forms a scaled Gaussian and hence a scaled Mahalanobis ellipsoid. For CASMOPOLITAN-like behavior, that ellipsoid is intersected with a Hamming-ball constraint in the original discrete space; for Bounce, an analogous construction is used in the low-dimensional encoded subspace $t$7 (Ngo et al., 9 Aug 2025).

The update mechanism follows CMA-ES-style search-distribution adaptation. From a batch of $t$8 encoded points and their function values, CMA-ES updates the mean, covariance, and step size; the new Gaussian defines the next ellipsoid. The high-level MOCA-HESP loop consists of an outer restart structure and an inner batch loop. Each restart initializes encoder weights, chooses an encoder, draws an initial design, encodes it, initializes $t$9 and $$\mathcal{N}_{\mathcal{Z}_e}^{(t)} = \mathcal{N}_{\mathcal{Z}_e}(\mathbf{m}^{(t)}, \mathbf{\Sigma}^{(t)}),$$0, and defines the initial ellipsoid. Each inner iteration then performs local optimization inside the current ellipsoid with a chosen BO optimizer, evaluates the batch, updates the search distribution, reconstructs the optimizer-specific local region, applies adaptive encoder selection, and checks restart criteria (Ngo et al., 9 Aug 2025).

Candidate allocation is implemented by sampling from the current Gaussian and rejecting points outside the current ellipsoid. In CASMOPOLITAN and Bounce variants, candidates must also satisfy the decoded Hamming constraint; only points satisfying all constraints form the pool over which the acquisition function is optimized. This makes HESP a geometric filter around the optimizer’s existing proposal mechanism rather than a replacement for the surrogate or acquisition layer (Ngo et al., 9 Aug 2025).

3. Encoded mixed spaces and optimizer coupling

The original mixed domain is

$$\mathcal{N}_{\mathcal{Z}_e}^{(t)} = \mathcal{N}_{\mathcal{Z}_e}(\mathbf{m}^{(t)}, \mathbf{\Sigma}^{(t)}),$$1

with continuous variables $$\mathcal{N}_{\mathcal{Z}_e}^{(t)} = \mathcal{N}_{\mathcal{Z}_e}(\mathbf{m}^{(t)}, \mathbf{\Sigma}^{(t)}),$$2 and combinatorial variables $$\mathcal{N}_{\mathcal{Z}_e}^{(t)} = \mathcal{N}_{\mathcal{Z}_e}(\mathbf{m}^{(t)}, \mathbf{\Sigma}^{(t)}),$$3, where $$\mathcal{N}_{\mathcal{Z}_e}^{(t)} = \mathcal{N}_{\mathcal{Z}_e}(\mathbf{m}^{(t)}, \mathbf{\Sigma}^{(t)}),$$4 is ordinal and $$\mathcal{N}_{\mathcal{Z}_e}^{(t)} = \mathcal{N}_{\mathcal{Z}_e}(\mathbf{m}^{(t)}, \mathbf{\Sigma}^{(t)}),$$5 is categorical. MOCA-HESP treats all combinatorial variables as categorical. It defines encoders

$$\mathcal{N}_{\mathcal{Z}_e}^{(t)} = \mathcal{N}_{\mathcal{Z}_e}(\mathbf{m}^{(t)}, \mathbf{\Sigma}^{(t)}),$$6

so that the encoded mixed domain becomes

$$\mathcal{N}_{\mathcal{Z}_e}^{(t)} = \mathcal{N}_{\mathcal{Z}_e}(\mathbf{m}^{(t)}, \mathbf{\Sigma}^{(t)}),$$7

(Ngo et al., 9 Aug 2025).

Two encoders are used as bandit arms. The ordinal encoder assigns distinct numeric codes to the categories of each variable and does not require training. The target encoder uses class-conditional means, counts, a global mean, and a smoothing weight $$\mathcal{N}_{\mathcal{Z}_e}^{(t)} = \mathcal{N}_{\mathcal{Z}_e}(\mathbf{m}^{(t)}, \mathbf{\Sigma}^{(t)}),$$8, yielding an empirical Bayes shrinkage toward the global mean. Decoding is performed by nearest code: $$\mathcal{N}_{\mathcal{Z}_e}^{(t)} = \mathcal{N}_{\mathcal{Z}_e}(\mathbf{m}^{(t)}, \mathbf{\Sigma}^{(t)}),$$9 Applied coordinatewise, this decoder maps back from $\mathbf{m}^{(t)} \in \mathbb{R}^d$0 to $\mathbf{m}^{(t)} \in \mathbb{R}^d$1, while leaving continuous coordinates unchanged (Ngo et al., 9 Aug 2025).

HESP itself operates purely on the encoded continuous space $\mathbf{m}^{(t)} \in \mathbb{R}^d$2 and uses standard continuous covariance and Mahalanobis distance. The underlying BO optimizer may nonetheless use its own modeling encoding and kernel, and those are independent from the HESP encoding used for ellipsoid construction. This independence is what allows MOCA-HESP to wrap an existing optimizer without changing its internal surrogate or kernel (Ngo et al., 9 Aug 2025).

Variant	Surrogate and acquisition	HESP-constrained region
MOCA-HESP-BO	Matern 5/2 ARD kernel; Thompson Sampling	candidates sampled within $\mathbf{m}^{(t)} \in \mathbb{R}^d$3
MOCA-HESP-Casmo	Transformed Overlapped kernel; Thompson Sampling	Mahalanobis ellipsoid plus Hamming constraint
MOCA-HESP-Bounce	Overlapped kernel with one-hot encoding; Expected Improvement	low-dimensional $\mathbf{m}^{(t)} \in \mathbb{R}^d$4 with Mahalanobis and Hamming constraints

The optimizer-specific trust-region parameters are inherited from the wrapped methods. For MOCA-HESP-Casmo, the paper uses $\mathbf{m}^{(t)} \in \mathbb{R}^d$5, $\mathbf{m}^{(t)} \in \mathbb{R}^d$6, and success/failure thresholds $\mathbf{m}^{(t)} \in \mathbb{R}^d$7, $\mathbf{m}^{(t)} \in \mathbb{R}^d$8. For MOCA-HESP-Bounce, it uses $\mathbf{m}^{(t)} \in \mathbb{R}^d$9, $\mathbf{\Sigma}^{(t)} \in \mathbb{R}^{d \times d}$0, and dynamic trust-region updates after each iteration (Ngo et al., 9 Aug 2025).

4. Adaptive encoder selection by EXP3

A distinctive feature of MOCA-HESP is that the categorical encoder is selected adaptively by a multi-armed bandit. Each encoder $\mathbf{\Sigma}^{(t)} \in \mathbb{R}^{d \times d}$1 is treated as an arm, and the paper employs EXP3 to adaptively choose the best encoder based on optimization progress (Ngo et al., 9 Aug 2025).

If $\mathbf{\Sigma}^{(t)} \in \mathbb{R}^{d \times d}$2 is the weight of encoder $\mathbf{\Sigma}^{(t)} \in \mathbb{R}^{d \times d}$3 at iteration $\mathbf{\Sigma}^{(t)} \in \mathbb{R}^{d \times d}$4, the selection probability is

$\mathbf{\Sigma}^{(t)} \in \mathbb{R}^{d \times d}$5

where $\mathbf{\Sigma}^{(t)} \in \mathbb{R}^{d \times d}$6 controls exploration versus exploitation. After each batch of $\mathbf{\Sigma}^{(t)} \in \mathbb{R}^{d \times d}$7 evaluations, the objective values observed so far are normalized by min-max scaling,

$\mathbf{\Sigma}^{(t)} \in \mathbb{R}^{d \times d}$8

The batch reward for minimization is then

$\mathbf{\Sigma}^{(t)} \in \mathbb{R}^{d \times d}$9

and the unbiased reward estimate is

$\mathbf{z}_e$0

where $\mathbf{z}_e$1 is the encoder used at iteration $\mathbf{z}_e$2. The selected encoder weight is updated as

$\mathbf{z}_e$3

while the other weights remain unchanged (Ngo et al., 9 Aug 2025).

The practical learning-rate prescription is

$\mathbf{z}_e$4

with $\mathbf{z}_e$5 the number of encoders and $\mathbf{z}_e$6 the maximum number of iterations. The paper states that this follows the original EXP3 analysis. It also states that the encoder adaptation adds negligible overhead. The theoretical justification is correspondingly modular: HESP inherits its search-distribution behavior from CMA-ES, while the encoder selector inherits adversarial regret guarantees from EXP3. The paper does not present new convergence theorems specific to HESP in combinatorial or mixed BO (Ngo et al., 9 Aug 2025).

5. Complexity, practical settings, and empirical behavior

HESP’s primary overhead is maintaining and updating the Gaussian search distribution, which is the same complexity as CMA-ES. With encoded dimension $\mathbf{z}_e$7, the covariance and mean update is $\mathbf{z}_e$8 per iteration, with occasional $\mathbf{z}_e$9 eigendecomposition depending on implementation; sampling from the multivariate Gaussian is $$\mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr) = \sqrt{(\mathbf{z}_e-\mathbf{m}^{(t)})^\top \bigl(\mathbf{\Sigma}^{(t)}\bigr)^{-1}(\mathbf{z}_e-\mathbf{m}^{(t)})}.$$0 per sample with full covariance; checking Mahalanobis distance is $$\mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr) = \sqrt{(\mathbf{z}_e-\mathbf{m}^{(t)})^\top \bigl(\mathbf{\Sigma}^{(t)}\bigr)^{-1}(\mathbf{z}_e-\mathbf{m}^{(t)})}.$$1 naively or $$\mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr) = \sqrt{(\mathbf{z}_e-\mathbf{m}^{(t)})^\top \bigl(\mathbf{\Sigma}^{(t)}\bigr)^{-1}(\mathbf{z}_e-\mathbf{m}^{(t)})}.$$2 if pre-whitened; and Hamming computations for CASMOPOLITAN or Bounce regions are $$\mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr) = \sqrt{(\mathbf{z}_e-\mathbf{m}^{(t)})^\top \bigl(\mathbf{\Sigma}^{(t)}\bigr)^{-1}(\mathbf{z}_e-\mathbf{m}^{(t)})}.$$3 per candidate. Storage is $$\mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr) = \sqrt{(\mathbf{z}_e-\mathbf{m}^{(t)})^\top \bigl(\mathbf{\Sigma}^{(t)}\bigr)^{-1}(\mathbf{z}_e-\mathbf{m}^{(t)})}.$$4 for the mean and $$\mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr) = \sqrt{(\mathbf{z}_e-\mathbf{m}^{(t)})^\top \bigl(\mathbf{\Sigma}^{(t)}\bigr)^{-1}(\mathbf{z}_e-\mathbf{m}^{(t)})}.$$5 for the covariance, with additional projection matrices $$\mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr) = \sqrt{(\mathbf{z}_e-\mathbf{m}^{(t)})^\top \bigl(\mathbf{\Sigma}^{(t)}\bigr)^{-1}(\mathbf{z}_e-\mathbf{m}^{(t)})}.$$6 and $$\mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr) = \sqrt{(\mathbf{z}_e-\mathbf{m}^{(t)})^\top \bigl(\mathbf{\Sigma}^{(t)}\bigr)^{-1}(\mathbf{z}_e-\mathbf{m}^{(t)})}.$$7 for Bounce. The paper reports that surrogate model cost and acquisition optimization often dominate, and that HESP adds “minimal computation overhead” in practice (Ngo et al., 9 Aug 2025).

The implementation details follow standard CMA-ES initialization. The population size is $$\mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr) = \sqrt{(\mathbf{z}_e-\mathbf{m}^{(t)})^\top \bigl(\mathbf{\Sigma}^{(t)}\bigr)^{-1}(\mathbf{z}_e-\mathbf{m}^{(t)})}.$$8; the initial mean is the best point among the initial design $$\mathbb{M}\bigl(\mathbf{z}_e,\mathcal{N}_{\mathcal{Z}_e}^{(t)}\bigr) = \sqrt{(\mathbf{z}_e-\mathbf{m}^{(t)})^\top \bigl(\mathbf{\Sigma}^{(t)}\bigr)^{-1}(\mathbf{z}_e-\mathbf{m}^{(t)})}.$$9; the initial covariance is the identity; the initial step size is $\alpha$0 after scaling the domain to $\alpha$1; and a lower bound on the standard deviation in discrete dimensions is recommended to avoid collapse in combinatorial coordinates. The paper also recommends using both ordinal and target encoders, with target-encoder smoothing taken from a standard library default (Ngo et al., 9 Aug 2025).

The empirical study covers 9 benchmarks: combinatorial Ackley20c, Shifted Ackley20c, Antibody Design, LABS, Shifted LABS, MaxSAT28, and MaxSAT125, together with mixed Ackley53m and Cellular Network. Dimensions range from 11 to 125. “Shifted” versions randomize categorical labels to remove special structural patterns. The baselines are standard BO, CASMOPOLITAN, Bounce, TPE, SMAC, CMA-ES with ordinal encoding, and random search. MOCA-HESP-BO consistently outperforms standard BO on all 9 benchmarks and reaches the global optimum on Ackley20c within the evaluation budget. MOCA-HESP-Casmo is better on 7 of 9 problems and at least on par on the others. MOCA-HESP-Bounce outperforms Bounce on 6 benchmarks and eventually finds better solutions on MaxSAT125 after similar early performance. All proposed MOCA-HESP variants generally dominate TPE, SMAC, CMA-ES, and random search across the reported tasks (Ngo et al., 9 Aug 2025).

The ablation on encoder choice shows that no single encoder dominates: ordinal is strong on some problems such as LABS and MaxSAT125, whereas the target encoder is stronger on others such as Antibody Design. The adaptive EXP3 version matches or outperforms the best fixed-encoder variant on almost all benchmarks. The paper’s 2D visualizations on discretized Ackley and Shifted Ackley further show that the initial ellipsoid is broad and roughly placed, then moves toward the global minimum over iterations, while the ellipsoid radii remain bounded away from zero because of minimum-standard-deviation constraints in discrete CMA-ES, ensuring continued exploration (Ngo et al., 9 Aug 2025).

6. Related ellipsoidal methods and scope of the term

The expression “hyper-ellipsoid space partitioning” is not tied to a single methodology across the literature. In supervised learning, SEP-C is a convex-optimization-based classifier that sequentially partitions a dataset into several ellipsoids, where each ellipsoid contains nearly all points of the same label; Bayes’ formula is then applied to calculate a trust score for the predicted label (Niranjan et al., 2023). Its core machinery is different from MOCA-HESP: SEP-C uses minimum-volume ellipsoids, reduced convex hull splits, impurity control through $\alpha$2, and explicit decision rules for points lying in one ellipsoid, in intersections, or in no ellipsoid at all (Niranjan et al., 2023).

A second related line is the Ellipsoidal Separation Machine. ESM separates two data sets using an ellipsoid, formulates training as a convex problem that can be written as a semidefinite program, and then uses a more scalable nonconvex formulation amenable to block-Gauss-Seidel updates alternating between a smaller SDP and a separable SOCP. A characteristic of ESM is that it necessarily defines indeterminate points, making it suitable for classification with rejection (Frangioni et al., 28 Jul 2025).

A plausible implication is that hyper-ellipsoid partitioning has developed along at least two technical lines: sequential or separating ellipsoids for supervised classification, and moving Gaussian confidence ellipsoids for high-dimensional Bayesian optimization. In that narrower and acronym-bearing sense, HESP in MOCA-HESP denotes a search-distribution-based local-region mechanism for combinatorial and mixed BO, not a general-purpose classifier (Ngo et al., 9 Aug 2025).