---
title: 'MOCA-HESP: Optimizing Mixed and Combinatorial Spaces'
url: https://www.emergentmind.com/topics/moca-hesp
type: topic
---

# MOCA-HESP: Optimizing Mixed and Combinatorial Spaces

MOCA-HESP is a high-dimensional Bayesian optimization method for combinatorial and mixed-variable search spaces. In "MOCA-HESP: Meta High-dimensional Bayesian Optimization for Combinatorial and Mixed Spaces via Hyper-ellipsoid Partitioning" [2508.06847], the name is expanded as **Meta-Algorithm for High-dimensional Ordinal, Categorical and Mixed Bayesian Optimization via Hyper-ellipsoid Space Partitioning**. It is formulated as a meta-algorithm: rather than replacing existing combinatorial or mixed-domain Bayesian optimizers, it wraps them with an encoded-space local search mechanism based on adaptive hyper-ellipsoids, together with an online procedure for selecting categorical encoders.

## 1. Definition and optimization setting

MOCA-HESP targets expensive black-box optimization problems of the form
$$
z^* \in \arg\min_{z \in Z} f(z),
$$
where the domain is mixed:
$$
Z = X \times H,
$$
with $X \subset \mathbb{R}^{d_x}$ continuous and $H$ combinatorial. The combinatorial component is further decomposed as
$$
H = W \times Q,
$$
where $W \subset \mathbb{Z}^{d_w}$ is ordinal and $Q$ contains $d_q$ categorical variables; the total dimension is $d = d_x + d_h$ [2508.06847].

The method is motivated by the observation that high-dimensional Bayesian optimization has been studied predominantly for continuous domains, whereas ordinal, categorical, and mixed domains remain difficult. The stated difficulties include the absence of intrinsic ordering for categorical variables, the mismatch between Euclidean continuous kernels and categorical structure, the inapplicability of gradient-based acquisition optimization in discrete spaces, and the dimensional blow-up induced by naive one-hot encodings. MOCA-HESP addresses these issues by combining three ingredients: a continuous embedding of combinatorial variables, a hyper-ellipsoid local region in the embedded space, and adaptive encoder selection through a multi-armed bandit [2508.06847].

The paper positions MOCA-HESP as a **locally focused, encoder-aware** BO framework. This suggests that its main purpose is not to redesign the surrogate model from first principles, but to improve the geometry in which existing BO methods search and to restrict that search to a moving, reshaping local region.

## 2. Hyper-ellipsoid space partitioning in an encoded domain

The central mechanism is **hyper-ellipsoid space partitioning** (HESP). MOCA-HESP maintains a multivariate normal search distribution in an encoded continuous space,
$$
\mathcal{N}_{Z_e} = \mathcal{N}(m_{Z_e}, \Sigma_{Z_e}),
$$
and defines a base local region as the $\alpha$-confidence hyper-ellipsoid
$$
E_b = \{ z_e \in Z_e \mid M(z_e; \mathcal{N}_{Z_e}) \le \chi^2_{1-\alpha}(d_{Z_e}) \},
$$
where $M(\cdot;\cdot)$ is the Mahalanobis distance and $\chi^2_{1-\alpha}(d_{Z_e})$ is the chi-squared critical value [2508.06847].

For combinatorial and mixed variables, MOCA-HESP first maps the combinatorial component into a continuous embedding. A categorical encoder
$$
g: H \to H_e \subset \mathbb{R}^{d_h}
$$
defines an encoded search space
$$
Z_e = X \times H_e.
$$
All HESP operations—Gaussian search distribution maintenance, ellipsoid definition, and candidate sampling—occur in this encoded space. Candidate points are then mapped back to the original domain through a decoder
$$
\tilde g: H_e \to H.
$$
The paper uses nearest-value decoding for scalar encoded categories:
$$
\tilde g(x) = \arg\min_{u_i} |g(u_i) - x|.
$$
This encode-search-decode loop is what allows a continuous hyper-ellipsoid mechanism to be applied to discrete and mixed problems [2508.06847].

The search region is adaptive. A batch of $\lambda$ points is proposed inside the current local region using a BO backend, evaluated in the original domain, and then used to update the Gaussian mean and covariance by a CMA-ES-style adaptation. As a result, the hyper-ellipsoid moves and reshapes toward promising parts of the search space. The algorithm also includes restart logic to escape stagnation [2508.06847].

A plausible implication is that MOCA-HESP uses the ellipsoid not merely as a trust region in the conventional sense, but as a geometry-aware filter that can be imposed on a broad class of base optimizers without changing their internal acquisition definitions.

## 3. Encoders and adaptive selection by bandits

A distinctive feature of MOCA-HESP is that it does not assume a single categorical encoding is universally best. The paper considers two encoders: an **ordinal encoder** and a **target encoder** [2508.06847].

For the ordinal encoder, each category $u_i$ is mapped to a distinct real value,
$$
g_{\mathrm{ord}}(u_i) = a_i.
$$
This is direct and cheap, and it naturally fits ordinal variables. For nominal categorical variables, however, it imposes an arbitrary geometry. The paper explicitly notes that such arbitrary geometry can distort distances and correlations in Gaussian-process modeling [2508.06847].

For the target encoder, the mapping depends on observed objective values. Given current data
$$
D = \{(h_k, y_k)\}_{k=1}^n,
$$
with category counts $n_i$, global mean $\bar y$, and per-category mean $\bar y_{u_i}$, the encoder is
$$
g_{\mathrm{tgt}}(u_i) = \frac{n_i \bar y_{u_i} + m \bar y}{n_i + m},
$$
where $m \ge 0$ is a smoothing parameter. The paper describes this as an empirical-Bayes shrinkage approximation to $\mathbb{E}[y \mid h = u_i]$ [2508.06847].

Because no single encoder dominates across all problems or across all phases of optimization, MOCA-HESP uses **EXP3**, an adversarial multi-armed bandit, to select the encoder online. If arm $k$ has weight $w_k(t)$ at iteration $t$, its selection probability is
$$
p_k(t) = (1 - \eta)\frac{w_k(t)}{\sum_{j=1}^K w_j(t)} + \frac{\eta}{K},
$$
with $\eta \in [0,1]$. Rewards are constructed from the last $\lambda$ evaluations after normalizing objective values to $[0,1]$, and the selected arm’s weight is updated by
$$
w_{k_t}(t+1) = w_{k_t}(t)\exp\!\left(\frac{\eta \hat r(t)}{K}\right),
$$
with other weights unchanged [2508.06847].

This adaptive encoder mechanism is one of the paper’s central claims. It allows the encoded geometry itself to change during optimization, with HESP operating in whichever embedding the bandit currently prefers.

## 4. Meta-algorithm structure and instantiated variants

MOCA-HESP is presented as a **meta-algorithm** that can wrap existing optimizers for combinatorial and mixed spaces. The paper develops three concrete instantiations [2508.06847].

| Variant | Wrapped optimizer | Role of MOCA-HESP |
|---|---|---|
| MOCA-HESP-BO | standard BO | Adds encoded-space hyper-ellipsoid local region |
| MOCA-HESP-Casmo | CASMOPOLITAN | Replaces two trust regions with one encoded-space hyper-ellipsoid plus Hamming control |
| MOCA-HESP-Bounce | Bounce | Builds a projected hyper-ellipsoid in Bounce’s current target subspace |

In **MOCA-HESP-BO**, the local region is simply the base ellipsoid $E_b$ in the encoded space. The underlying optimizer retains its GP surrogate and acquisition function, while candidate generation is constrained to the ellipsoid [2508.06847].

In **MOCA-HESP-Casmo**, the method preserves CASMOPOLITAN’s kernel and Thompson-sampling-based acquisition logic, but replaces its two trust regions by a single encoded-space local region that combines a scaled Mahalanobis constraint with a Hamming-distance constraint after decoding. With a scaling vector $L$, the paper defines
$$
E_{\mathrm{Casmo}} = \left\{ z_e \mid M(z_e; \mathcal{N}_{Z_e,L}) \le \chi^2_{1-\alpha}(d_{Z_e}) \ \text{and} \ H(\tilde g(z_e), \tilde g(m_{Z_e})) \le L_h \right\},
$$
where $H(\cdot,\cdot)$ is Hamming distance [2508.06847].

In **MOCA-HESP-Bounce**, HESP is applied inside Bounce’s current low-dimensional target subspace $V$. If $Q: V \to Z_e$ is Bounce’s current projection and $P$ its Moore–Penrose inverse, then the Gaussian search distribution is projected into $V$, and the local region becomes
$$
E_{\mathrm{Bounce}} = \left\{ v \in V \mid M(v; \mathcal{N}_{V,L}) \le \chi^2_{1-\alpha}(d_V) \ \text{and} \ H(\tilde h(v), \tilde h(m_V)) \le L_h \right\}.
$$
This preserves Bounce’s subspace machinery while adding a hyper-ellipsoid restriction in that subspace [2508.06847].

The paper emphasizes that MOCA-HESP is orthogonal to prior work such as COMBO, CoCaBO, CASMOPOLITAN, Bounce, and BODi. In that framing, MOCA-HESP is not another surrogate model family, but an add-on layer that contributes local geometric focusing and encoder adaptation [2508.06847].

## 5. Algorithmic workflow and mathematical formulation

The algorithm begins with $n_0$ initial evaluations in the original domain, encoder training, and initialization of the Gaussian search distribution. The mean is set to the current incumbent in the encoded space, the covariance is initialized to the identity, and the step size is chosen following CMA-ES heuristics. The paper also specifies lower bounds on standard deviations for discrete dimensions so that the candidate set does not collapse in combinatorial spaces [2508.06847].

Each iteration then proceeds as follows. A candidate pool is sampled from the current Gaussian or scaled Gaussian distribution, filtered by the relevant local-region definition, decoded if necessary, and then scored by the wrapped optimizer’s acquisition function. The selected batch of $\lambda$ points is evaluated, after which the Gaussian mean and covariance are updated by a CMA-ES-style rule. EXP3 is then used to update encoder-selection probabilities, and the encoder may be retrained on all accumulated data. Restart criteria are applied when stagnation is detected [2508.06847].

The acquisition-optimization step remains backend-specific. For standard BO, the method can use continuous-kernel GP models such as Matérn-$5/2$ ARD and acquisitions such as Thompson sampling, expected improvement, or upper confidence bound over the filtered candidate set. For CASMOPOLITAN and Bounce, MOCA-HESP preserves their existing surrogate and search machinery while constraining proposals to the current local region [2508.06847].

The paper characterizes the computational overhead of HESP as modest. The added cost comes from Gaussian sampling and covariance updates, whereas GP fitting and acquisition optimization remain the dominant per-iteration costs. This suggests that the method is designed for expensive-function regimes where search-efficiency gains outweigh the modest additional internal overhead.

## 6. Empirical performance

The experimental study uses **nine tasks** with dimensions ranging from **11 to 125**, including synthetic combinatorial, synthetic mixed, and real-world combinatorial and mixed benchmarks. The listed tasks are Ackley20c-20D, Shifted Ackley20c-20D, Ackley53m-53D, Antibody Design-11D, LABS-50D, MaxSAT28-28D, MaxSAT125-125D, Shifted LABS-50D, and Cellular Network-30D. All methods use **20 initial points** and the same budgets [2508.06847].

The reported comparative findings are specific. **MOCA-HESP-BO** strictly outperforms standard BO on **all 9 problems** and reaches the global optimum on Ackley20c within budget. **MOCA-HESP-Casmo** outperforms CASMOPOLITAN on **7/9 tasks**. **MOCA-HESP-Bounce** outperforms Bounce on **6/9 tasks**, with improvements that increase over iterations on MaxSAT125. The paper also states that all three MOCA-HESP variants outperform **TPE, SMAC, CMA-ES, and Random Search** across all tasks [2508.06847].

The encoder-selection ablation is equally central. The adaptive EXP3 version is reported to be **better or on par** with fixed ordinal or fixed target encoding on all tasks. The paper notes that ordinal encoding can work well on LABS and MaxSAT125, whereas target encoding excels on Antibody Design; the adaptive mechanism is intended to switch appropriately over time [2508.06847].

The paper also includes a trajectory illustration on discretized two-dimensional Shifted Ackley, showing the ellipsoid moving from a random start toward the global minimum. This is used to support the claim that the minimum standard deviations maintained on discrete dimensions prevent premature shrinkage of the local region [2508.06847].

## 7. Scope, limitations, and terminology

The paper identifies several limitations. Encoder geometry can still be misleading if the embedding poorly reflects the true structure of the problem; ordinal encoding can distort nominal categories, and target encoding can be noisy early when data are sparse. Nearest-value decoding introduces discontinuities, and HESP’s Gaussian model assumes that the encoded space provides a useful continuous geometry. The method can also be sensitive to the confidence level $\alpha$, the scaling factors $L_x$ and $L_h$, and the lower bounds imposed on discrete-dimension standard deviations [2508.06847].

The authors recommend MOCA-HESP for **high-dimensional combinatorial or mixed BO under tight evaluation budgets**, especially in settings where CASMOPOLITAN, Bounce, or standard BO with encodings would otherwise be considered. This suggests that MOCA-HESP is best understood as a general-purpose upgrade layer for expensive black-box optimization in noncontinuous spaces rather than a specialized optimizer for a single benchmark family [2508.06847].

A common source of confusion is the acronym itself. The term **MOCA-HESP** in the Bayesian optimization literature refers specifically to the meta-algorithm described above [2508.06847]. This is distinct from unrelated uses of **MoCA** in other fields, including Monte Carlo Comptonisation in astrophysics [1908.10373; 1808.07399], the Multi-modal Cross-masked Autoencoder for digital health [2506.02260], and the Montreal Cognitive Assessment in dementia screening [2505.07246]. It is also distinct from **HESP** as the Hanle Echelle Spectrograph in astronomical instrumentation [2006.01453; 1701.02451]. The shared acronyms are purely terminological; the paper on MOCA-HESP does not connect the BO method to those domains [2508.06847].

In sum, MOCA-HESP is a geometry- and encoding-aware framework for high-dimensional Bayesian optimization over ordinal, categorical, and mixed search spaces. Its technical contribution lies in combining encoded continuous representations, adaptive hyper-ellipsoid local regions, and bandit-based encoder selection inside a meta-algorithmic wrapper that can improve several existing optimizers on difficult high-dimensional discrete and mixed benchmarks [2508.06847].

Source: https://www.emergentmind.com/topics/moca-hesp