---
title: 'HybridCORELS: Certifiably Optimal Hybrid Models'
url: https://www.emergentmind.com/topics/hybridcorels
type: topic
---

# HybridCORELS: Certifiably Optimal Hybrid Models

HybridCORELS is an algorithmic framework for constructing hybrid models that combine interpretable rule-list classifiers with complex black-box models, delivering certifiably optimal solutions under explicit transparency constraints. Through its extension of the CORELS branch-and-bound approach, HybridCORELS enables precise control over the division of input space between interpretable and black-box components while providing strong theoretical and empirical guarantees [2303.04437].

## 1. Model Composition and Gating

A HybridCORELS classifier addresses binary classification tasks on input space $\mathcal{X}$ with outputs in $\{0,1\}$. It consists of a triplet $(h_s, h_c, \Omega)$, where $h_s:\mathcal{X}\to\{0,1\}$ is an interpretable rule-list (the simple model), $h_c:\mathcal{X}\to\{0,1\}$ is a pre-trained black-box classifier, and $\Omega\subseteq\mathcal{X}$ is the region of input space covered by the rule list. The inference-time prediction is governed by a gating function $g(x) = \mathbf{1}[x\in\Omega]$, which routes $x$ to $h_s$ if $x\in\Omega$ and otherwise to $h_c$. The transparency of the hybrid model, $C_\Omega$, is the probability $x\in\Omega$ under the data distribution. In practice, the empirical transparency $\widehat{C}_\Omega$ is enforced as $\widehat{C}_\Omega \geq \tau$ for a user-chosen transparency level $\tau\in[0,1]$.

## 2. Optimization Objective

The HybridCORELS optimization problem seeks a rule list $r$ satisfying the transparency constraint while minimizing overall misclassification:

\[
\min_{r\,:\,\widehat{C}_{\Omega_r}\geq\tau} \quad
R(r; S\cap\Omega_r) + R(h_c; S\setminus\Omega_r) + \lambda\,\ell(r) + \beta\, (1 - \widehat{C}_{\Omega_r})
\]

where $R(f;S)$ is the empirical 0-1 loss on $S$, $\ell(r)$ is the length of the rule list, and $\beta$ is a small penalty (used only to break ties) since $\widehat{C}_{\Omega_r}\geq\tau$ is a hard constraint. The objective thus balances empirical error across both model regions, rule-list complexity, and coverage, with formal prioritization of strict transparency thresholds.

HybridCORELS has two major training paradigms:
- **Post–black-box**: The black-box $h_c$ is pre-trained and fixed, and $r$ is optimized with $h_c$ held constant.
- **Pre–black-box**: The rule-list is fit first, with regions not covered handled by a black-box trained subsequently.

## 3. Algorithmic Structure: CORELS Extension

HybridCORELS extends the CORELS (Certifiably Optimal RulE ListS) algorithm, originally formulated for pure rule-lists, by modifying the objective and update steps. CORELS uses a prefix-tree representing partial rule-lists, a priority queue sorted by valid lower bounds, and a branch-and-bound scheme to search for provably optimal solutions. HybridCORELS integrates the new hybrid objective and transparency constraint. The lower bound remains valid because black-box error can only decrease with rule-list extension and transparency penalties can be eliminated by additional coverage.

If the queue empties, HybridCORELS guarantees that the found rule list is globally optimal under the given transparency constraint.

| Component           | CORELS                     | HybridCORELS                                          |
|---------------------|---------------------------|-------------------------------------------------------|
| Objective           | $R(r;S) + \lambda\ell(r)$ | Mixed hybrid loss + complexity + coverage penalty     |
| Constraint          | None                      | Hard transparency $\widehat{C}_{\Omega_r} \geq \tau$  |
| Search Structure    | Prefix-tree, PQ           | Prefix-tree, PQ with transparency updates             |

## 4. Theoretical Foundation and Generalization

Under mild finiteness assumptions on both the interpretable and complex hypothesis spaces, the class of HybridCORELS models is PAC-learnable. For finite rule-list ($\mathcal{H}_s$) and black-box ($\mathcal{H}_c$) spaces, with an “oracle” hybrid model of true risk zero, the PAC bound for excess risk $\epsilon$ satisfies:

\[
\Pr\bigl[\exists\,\triplet\;\text{ERM on data}:\;\poploss(\triplet)>\epsilon\bigr]
\leq \sum_{\Omega\in\mathcal{P}} B(\epsilon, C_\Omega, |\mathcal{H}_c|, |\mathcal{H}_s|, M)
\]

where the bound $B(\epsilon, C, |\mathcal{H}_c|, |\mathcal{H}_s|, M)$ captures contributions from both model parts and $M$ is the sample size. For a fixed oracle region $\Omega^*$, the union disappears. The minimizer $C^{**}$ of the PAC bound as a function of $C$ quantifies an optimal “sweet-spot” transparency: the empirical risk can be better for a hybrid than either component alone, reflecting a fundamental hybrid regularization benefit.

## 5. Empirical Performance

HybridCORELS was evaluated on three benchmarks: COMPAS recidivism prediction ($\sim$6k examples), UCI Adult Income ($\sim$49k), and ACS Employment ($\sim$200k). Black-box baselines included Random Forests, AdaBoost, and Gradient-Boosted Trees (scikit-learn, cross-validated). Comparisons across Hybrid Rule Set (HyRS), Companion Rule List (CRL), and both HybridCORELS variants revealed:

- Transparency under HybridCORELS increases monotonically with $\tau$, displaying negligible run-to-run variance. In contrast, HyRS/CRL exhibit substantial stochasticity at given transparency levels.
- On Adult and ACS, transparency of approximately 0.7–0.8 can be achieved with no loss in overall accuracy relative to pure black-box models.
- On COMPAS, HybridCORELSPre surpasses black-box accuracy at intermediate transparency (0.5–0.6), by up to 2 percentage-points. This aligns with the “sweet-spot self-regularization” predicted by theory.
- HybridCORELS consistently matches or exceeds HyRS/CRL performance at all tested transparencies.

As an example, on ACS with AdaBoost, the pure black-box achieved approximately 74% accuracy; HybridCORELSPre matched this at 75% transparency, while HyRS/CRL trailed by 1–2 percentage points at comparable transparency [2303.04437].

## 6. Strengths, Limitations, and Future Directions

HybridCORELS provides:
- **Certifiably optimal hybridization**: Retains the CORELS global optimality guarantee under hard transparency.
- **Black-box agnosticism**: Compatible with any pre-trained $h_c$ supporting (optionally) instance weighting.
- **User-controllable transparency**: Enforcement of hard constraints on $\tau$ yields empirically realizable, user-specified transparency; stochastic instability common to HyRS/CRL is eliminated.
- **Empirical regularization sweet-spot**: Demonstrated capacity to attain accuracy superior to either component alone at intermediate transparency levels.

Limitations and open challenges include:
- **Scalability to large antecedent pools**: The prefix-tree search can become computationally expensive as the rule base grows, or when extreme transparency is required.
- **Dependency on quality of pre-mined antecedents**: The algorithm's success hinges on effective antecedent mining, an external, potentially non-trivial pipeline step.
- **Lack of end-to-end training**: Current methods require a fixed two-stage pipeline (pre–/post–black-box). Joint optimization of $h_s$ and $h_c$ remains unaddressed.
- **Generality beyond rule lists**: While extending to other interpretable classes is in principle straightforward, it requires new analysis and PAC bounds.

Future work includes exploring end-to-end hybrid learning with guarantees, adaptive (data-dependent) transparency, multiclass extensions, and investigating the empirical-theoretical gap in PAC bounds and observed sweet-spots [2303.04437].

Source: https://www.emergentmind.com/topics/hybridcorels