---
title: 'GMJMCMC: Bayesian Adaptive Feature Selection'
url: https://www.emergentmind.com/topics/genetically-modified-mjmcmc-gmjmcmc
type: topic
---

# GMJMCMC: Bayesian Adaptive Feature Selection

Searching arXiv for recent and foundational papers on GMJMCMC to ground the article.
Genetically Modified MJMCMC (GMJMCMC) is a Bayesian model-space exploration framework that combines Mode Jumping Markov Chain Monte Carlo (MJMCMC) with a genetic programming layer to perform Bayesian variable selection and Bayesian model averaging in combinatorially large and multimodal model spaces. It is used particularly when candidate predictors are not limited to a fixed set of covariates, but instead include structured, derived, or nonlinear features constructed from the original variables. In this setting, GMJMCMC maintains an evolving population of candidate features and runs MJMCMC within the subspace spanned by the current population, thereby coupling mode-jumping exploration with adaptive feature generation. The method has been developed for Bayesian logic regression, Bayesian generalized linear models (BGLM), and Bayesian generalized nonlinear models (BGNLM), and later extended through reversible variants and subsampling-based marginal likelihood estimation for tall data [1705.07616] [2110.05316] [2312.16997].

## 1. Conceptual definition and problem setting

GMJMCMC extends MJMCMC to settings in which the search space is not merely the set of subsets of a fixed design matrix, but a much larger structured space induced by transformations, interactions, projections, or logical compositions of covariates. MJMCMC itself is a Metropolis–Hastings sampler designed for multimodal posterior landscapes, using occasional large “mode-jumping” proposals to traverse low-posterior valleys separating high-probability basins. GMJMCMC adds a genetic framework that evolves the candidate feature set over successive populations, so that the algorithm can search over feature spaces that are exponentially large in the depth or complexity of feature construction [2312.16997].

In Bayesian model selection, the target is the posterior over models,
\[
p(M\mid y)\propto p(y\mid M)p(M),
\]
with marginal likelihood
\[
p(y\mid M)=\int p(y\mid \theta,M)p(\theta\mid M)\,d\theta.
\]
GMJMCMC uses this posterior score to compare models while adaptively modifying the set of available features. In BGLM, variable selection concerns inclusion or exclusion of columns of \(X\), whereas in BGNLM the predictor is built from nonlinear basis functions or constructed features \(F_j(x;\alpha_j)\), with inclusion indicators \(\gamma_j\in\{0,1\}\) [2312.16997].

A central motivation is that local proposals alone are inadequate in model spaces that are both multimodal and highly structured. In logic regression, the feasible predictors are Boolean expressions of binary covariates, and in BGNLM the candidate features are recursively generated compositions of algebraic and nonlinear transformations. In both cases, the model space is prohibitively large for classical MCMC methods, and many models can fit the data similarly well, which makes Bayesian model averaging essential for robust inference and prediction [1705.07616] [2312.16997].

## 2. Model classes and Bayesian formulation

GMJMCMC has been applied in at least three closely related model classes: Bayesian logic regression, BGLM, and BGNLM.

In Bayesian logic regression, covariates are binary and logic terms \(L_j(X)\) are built using Boolean operators \(\wedge\), \(\vee\), and \(\neg\). The model may be Gaussian,
\[
y_i = \beta_0 + \sum_{j=1}^{k} \beta_j L_j(X_i) + \varepsilon_i,\quad \varepsilon_i \sim \mathcal{N}(0,\sigma^2),
\]
or logistic,
\[
\Pr(y_i = 1 \mid X_i) = \text{logit}^{-1}\!\left(\beta_0 + \sum_{j=1}^{k} \beta_j L_j(X_i)\right).
\]
A model \(M\) is represented by a binary inclusion vector over candidate logic trees, subject to structural constraints such as maximum number of trees per model \(k_{\max}\) and maximum number of leaves per tree \(C_{\max}\) [1705.07616].

In BGLM, the predictor has the standard form \(\eta_i = x_i^\top \beta\) with link \(g(\mu_i)=\eta_i\), and the exponential-family log-likelihood is
\[
\ell(\beta) = \sum_{i=1}^n [y_i \theta_i - b(\theta_i)]/a(\phi) + c(y_i,\phi).
\]
In BGNLM, the linear predictor is replaced by a nonlinear composition of features:
\[
h(\mu_i) = \beta_0 + \sum_{j=1}^q \gamma_j \beta_j F_j(x_i;\alpha_j)\equiv \eta_i = f(x_i;\gamma,\beta),
\]
where the \(F_j\) are constructed recursively from covariates using algebraic operations and nonlinear transforms, under hard constraints on depth \(D\), width \(L\), and total number of features per model \(Q\) [2312.16997].

A recurring prior construction penalizes structural complexity. In BGNLM, with complexity measure \(c(F_j(\cdot,\alpha_j))\) taken as operations count \(oc\), the model prior is
\[
p(M)\propto \prod_j u^{\gamma_j c(F_j(\cdot,\alpha_j))},\qquad u\in(0,1).
\]
In logic regression, the prior takes the form
\[
p(M) \propto \mathbb{I}(|M| \le k_{\max}) \prod_{j=1}^{q} \rho(\gamma_j), \quad \rho(\gamma_j) = a^{\gamma_j c(L_j)},\; 0<a<1,
\]
with a data-agnostic complexity measure \(c(L_j)\) penalizing larger or more complex trees [1705.07616] [2312.16997].

This prior structure is closely tied to the interpretability objective of the method. GMJMCMC is designed to search over rich nonlinear or logical feature spaces while retaining a posterior preference for parsimonious models. A plausible implication is that the method is meant not only to locate high-scoring models, but also to maintain structural regularization directly at the model-space level rather than solely through coefficient shrinkage.

## 3. Core algorithmic architecture

GMJMCMC operates on two levels: an inner MJMCMC search over models within a current feature population, and an outer genetic update of that population.

At generation \(t\), the algorithm maintains a population \(S_t\) of candidate features, initially often consisting of bare covariates. A model \(M\) is then a binary vector \(\gamma\) over \(S_t\), subject to any model-size constraints. Within this population, MJMCMC explores the induced subspace using local add/drop/swap moves and occasional mode-jumping proposals that modify multiple components simultaneously. The Metropolis–Hastings acceptance probability is
\[
\alpha = \min\left\{1, \frac{p(y\mid M') p(M') q(M\mid M')}{p(y\mid M) p(M) q(M'\mid M)} \right\},
\]
where \(q(\cdot\mid\cdot)\) is the proposal kernel on model space [2312.16997].

Between populations, genetic operators modify the search space itself. The main operators described across the literature are:

- **Selection or filtration**: features with low posterior support or low marginal inclusion probability are filtered out, while high-PIP features are retained.
- **Mutation**: new features are generated by modifying existing ones or by reintroducing original covariates.
- **Crossover or interaction construction**: promising parent features are combined to form new compound features.
- **Modification or transformation**: nonlinear transforms are applied to existing features.
- **Projection**: derived features are constructed via nonlinear projections.
- **Survival or diversification**: some low-PIP features and random seeds are retained to avoid premature convergence [2110.05316] [2312.16997].

In the BGNLM formulation, the high-level procedure is: initialize \(S_0\) with bare covariates; for each population, run MJMCMC for \(K\) iterations on models spanned by \(S_t\); compute interim posterior inclusion probabilities for features; retain the strongest features; generate replacements by allowed transformations subject to \(D\), \(L\), and \(Q\); and finally aggregate posterior model weights over all visited models to compute PIPs and Bayesian model averaged predictions [2312.16997].

In Bayesian logic regression, the same architecture is specialized to logic trees. Initialization may begin with main effects only; inclusion probabilities of singletons are used to seed an initial population; and low-fitness trees are replaced by crossover or mutation, with reduction used to prune oversized trees back under \(C_{\max}\). Multiple independent GMJMCMC chains can be run in parallel and combined through weighted aggregation over visited models [1705.07616].

The key distinction from classical MJMCMC is that MJMCMC assumes a fixed feature set, whereas GMJMCMC adapts the candidate set itself. The key distinction from RJMCMC, as explicitly stated in the source material, is that RJMCMC handles trans-dimensional moves in parameter spaces but does not provide the genetic construction and adaptive curation of nonlinear features [2110.05316] [2312.16997].

## 4. Marginal likelihoods, posterior scoring, and model averaging

GMJMCMC relies on marginal likelihoods \(p(y\mid M)\) for posterior scoring, either exactly or via approximations depending on the model class.

For Gaussian linear models, closed-form expressions are available under Zellner’s \(g\)-prior. For example, one common form is
\[
p(y\mid M)\propto (1+g)^{-k_M/2}\ \bigl(1+g(1-R_M^2)\bigr)^{-(n-1)/2},
\]
and the FBMS framework gives the corresponding Gaussian marginal log-likelihood as
\[
\log p(Y|m) = \text{const} + \frac{n - k}2 \log(1+g) -  \frac{n - 1}2 \log\big(1 + g (1 - R_m^2)\big).
\]
For GLMs and nonlinear models, Laplace or BIC approximations are used, such as
\[
\log p(y\mid M)\approx \log L(\hat\theta_M)-\frac{k_M}{2}\log n + C
\]
or, in the BGNLM tall-data setting,
\[
\log p(y\mid M) \approx \log p(y\mid\hat\theta_M, M) + \log p(\hat\theta_M\mid M) + (d_M/2)\log(2\pi) - \tfrac12 \log |H(\hat\theta_M)|.
\]
Here \(H(\hat\theta_M)\) is the negative Hessian of the log-posterior at the estimated mode [2110.05316] [2312.16997] [2509.00753].

For BGLM, IRLS provides the mode-finding step through the penalized weighted least squares update
\[
\beta^{(t+1)} = (X^\top W^{(t)} X + \Lambda)^{-1} X^\top W^{(t)} z^{(t)},
\]
with the usual definitions of \(\mu_i^{(t)}\), \(z_i^{(t)}\), \(W^{(t)}\), and prior-induced regularization \(\Lambda\). This gives a convenient local approximation both for optimization and for Laplace-based scoring [2312.16997].

Posterior inclusion probabilities and Bayesian model averaging are then computed over visited models. With weights
\[
w(M)\propto \hat p(y\mid M)p(M),
\]
the PIP of feature \(j\) is approximated by
\[
\mathrm{PIP}(j)\approx \sum_{\text{visited}} 1\{j\in M\}\, w(M) \big/ \sum_{\text{visited}} w(M),
\]
and predictive expectations are averaged similarly across models [2312.16997].

Because GMJMCMC typically explores only a subset \(\mathcal{M}^*\) of the full model space, non-reversible implementations estimate posterior probabilities by renormalization over visited models:
\[
\widehat{p}(M\mid y)=\frac{p(M)\,p(y\mid M)}{\sum_{M'\in \mathcal{M}^*}p(M')\,p(y\mid M')}\mathbf{1}\{M\in \mathcal{M}^*\}.
\]
This is exact only insofar as the visited set captures the relevant posterior mass. The reversible formulation was introduced precisely to address this limitation [2110.05316].

## 5. Reversible GMJMCMC and theoretical status

A major development in the literature is Reversible Genetically Modified Mode Jumping MCMC (RGMJMCMC), introduced as a proper reversible MCMC algorithm for posterior model probabilities in extremely large model spaces [2110.05316].

Earlier GMJMCMC improved scalability by working within evolving populations \(\mathcal{S}_t\), but it was non-reversible and did not have a stationary distribution equal to the target posterior over the full explored space. Posterior model probabilities therefore depended on renormalization over visited models and, in principle, could be biased when coverage was incomplete [2110.05316].

RGMJMCMC augments the proposal with auxiliary forward and backward populations and local optimization paths. The chain’s primary state is a model \(m\), but proposals also involve a forward population \(\mathcal{S}'\), a forward local optimization path yielding \(m'_k\), a randomization step producing \(m'\), and corresponding backward auxiliary constructions. With this design, the acceptance ratio simplifies to
\[
a_{mh}=\frac{p(m'\mid y)\,q_r(m\mid \mathcal{S}, m_k)}{p(m\mid y)\,q_r(m'\mid \mathcal{S}', m'_k)},
\]
so that detailed balance holds for the marginal chain on models [2110.05316].

The paper states that, under mild regularity assumptions such as \(s\ge Q\), positive-probability generation of needed features by the genetic operators, and population constructions that include all features of the current model, the chain is irreducible in the explored model space and ergodic with stationary distribution \(p(m\mid y)\) [2110.05316].

Delayed acceptance is also introduced to reduce computational cost. The first-stage acceptance compares posterior probabilities,
\[
\min\left\{1,\frac{p(m'\mid y)}{p(m\mid y)}\right\},
\]
and only if this passes does the algorithm generate the backward auxiliary proposal and apply the second-stage correction involving the randomization kernel. This preserves invariance while avoiding unnecessary reverse-search computations for poor proposals [2110.05316].

The existence of both non-reversible and reversible variants is central to the literature on GMJMCMC. A common misconception is that “GMJMCMC” denotes a single fixed algorithm. In fact, the term covers a family of related procedures sharing the same population-evolution idea but differing in whether the overall search is a proper reversible MCMC or a posterior-guided heuristic with renormalization over visited models.

## 6. Scalability, subsampling, software, and empirical behavior

The 2023 tall-data work combines GMJMCMC with S-IRLS-SGD, a subsampling-based optimizer for estimating posterior modes and curvatures and thereby approximating marginal likelihoods through Laplace approximation [2312.16997]. This extension targets the case where exact likelihood evaluations and IRLS updates become expensive because \(n\) is large.

The S-IRLS-SGD procedure uses subsamples \(S_t\subset\{1,\dots,n\}\) of size \(s\), a few IRLS iterations on the subsample for initialization, and stochastic gradient refinement,
\[
\beta_{k+1} = \beta_k + \eta_k \widehat{\nabla}\ell_s(\beta_k),
\]
with step sizes satisfying Robbins–Monro conditions. Curvature is approximated via a subsampled Hessian or Fisher information, and the resulting mode and curvature are used in the Laplace approximation for \(p(y\mid M)\). Stabilization mechanisms include preconditioning, averaging over mini-batches, and appropriate scaling of gradient and Hessian estimates, particularly with canonical links in GLMs [2312.16997].

The source material reports several empirical results for GMJMCMC and its variants.

| Setting | Configuration | Reported outcome |
|---|---|---|
| Kepler’s third law, BGNLM | 64-thread GMJMCMC | true feature recovered 95% of the time, FDR 0.13 |
| Tall-data simulation | 0.75% subsample, 1000 MJMCMC iterations per population | Power 0.42/0.67/0.76 and FDR 0.89/0.78/0.74 for 1/32/128 threads |
| Abalone predictions | 10% subsample, 32 threads | RMSE \(\approx 1.95\) vs \(1.96\); MAE \(\approx 1.44\) vs \(1.45\); CORR \(\approx 0.785\) vs \(0.783\) for BGNLM\_SUB vs full-sample BGNLM |
| Planetary mass law, reversible variant | \(T=16\) threads | Power \(=0.94\), FP \(=0.29\), FDR \(=0.18\) |
| 3rd Kepler’s law, reversible variant | \(T=64\) threads | Power \(=1.00\), FP \(=0.04\), FDR \(=0.02\) |
| Logic regression simulations | logistic, \(n=1000, p=50\) | GMJMCMC shown to identify three-way and even four-way interactions with relatively large power |

These results support several conclusions stated directly in the underlying papers. First, increased parallelization improves exploration, often raising power and reducing FDR in multimodal spaces [2110.05316] [2312.16997]. Second, subsampled marginal likelihoods can be sufficiently accurate to guide model search and preserve predictive performance on tall data [2312.16997]. Third, in logic regression GMJMCMC can recover higher-order interactions that previous implementations struggled to detect [1705.07616].

Software implementations are explicitly reported. The 2023 work gives the R package **GMJMCMC** and the **S-IRLS-SGD** implementation on GitHub, while the 2025 FBMS package includes MJMCMC and GMJMCMC for Gaussian regression, BGNLMs, logic regression, and more complex settings such as mixed-effects and custom likelihood-based models [2312.16997] [2509.00753]. In FBMS, GMJMCMC is embedded in a broader framework with configurable transforms, operator probabilities, feature-depth limits, population-size constraints, and alternative marginal likelihood engines.

## 7. Limitations, practical constraints, and methodological significance

Several limitations recur across the literature. Nonlinear and logical feature spaces exhibit identifiability issues, high collinearity, and multimodality. Distinct feature constructions may be nearly equivalent in predictive effect, producing posterior dilution across many similar models. In the Kepler examples, multiple near-equivalent variants can fit comparably well; in logic regression, exact-tree recovery can be conservative because logically equivalent decompositions often co-occur [1705.07616] [2312.16997].

Overfitting control depends critically on structural constraints and complexity-penalizing priors. The source material repeatedly emphasizes constraints on depth \(D\), width \(L\), maximum model size \(Q\) or \(k_{\max}\), and complexity penalties such as \(u^{\gamma_j oc(F_j)}\) or priors inversely related to the number of logic trees of a given width [1705.07616] [2312.16997] [2509.00753].

Subsampled curvature estimation may be noisy in ill-conditioned designs, and approximate marginal likelihoods inherit bias if optimization fails to converge. The tall-data paper therefore recommends monitoring parameter changes \(\|\beta_{k+1}-\beta_k\|\), stability of \(\hat H\), and proposal acceptance-rate stability, and validating top models with larger subsamples when feasible [2312.16997]. RGMJMCMC adds another practical constraint: proposals are rejected if the backward population does not include all features in the current model, so careful construction of reverse proposals is necessary [2110.05316].

The overall methodological significance of GMJMCMC lies in the fact that it treats feature construction and model exploration as a coupled Bayesian search problem. Rather than assuming a predetermined design matrix, it adaptively curates a tractable working population of features, scores models with marginal likelihoods, and averages inference across visited posterior-supported structures. This distinguishes it from single-model optimization approaches and from purely shrinkage-based procedures. The 2025 FBMS presentation makes this perspective explicit by placing GMJMCMC alongside BMA, posterior inclusion probabilities, and custom marginal-likelihood machinery for increasingly complex regression settings [2509.00753].

A plausible implication is that GMJMCMC should be understood less as a single estimator than as an extensible algorithmic paradigm for posterior exploration in structured model spaces. Across logic regression, BGNLMs, reversible formulations, and tall-data subsampling, the common principle is the same: mode-jumping MCMC provides posterior traversal within a manageable subspace, while genetic evolution continually redesigns that subspace so that the search remains computationally feasible without abandoning Bayesian model scoring [1705.07616] [2110.05316] [2312.16997].

Source: https://www.emergentmind.com/topics/genetically-modified-mjmcmc-gmjmcmc