Slithering Through Gaps: Capturing Discrete Isolated Modes via Logistic Bridging

Abstract: High-dimensional and complex discrete distributions often exhibit multimodal behavior due to inherent discontinuities, posing significant challenges for sampling. Gradient-based discrete samplers, while effective, frequently become trapped in local modes when confronted with rugged or disconnected energy landscapes. This limits their ability to achieve adequate mixing and convergence in high-dimensional multimodal discrete spaces. To address these challenges, we propose \emph{Hyperbolic Secant-squared Gibbs-Sampling (HiSS)}, a novel family of sampling algorithms that integrates a \emph{Metropolis-within-Gibbs} framework to enhance mixing efficiency. HiSS leverages a logistic convolution kernel to couple the discrete sampling variable with the continuous auxiliary variable in a joint distribution. This design allows the auxiliary variable to encapsulate the true target distribution while facilitating easy transitions between distant and disconnected modes. We provide theoretical guarantees of convergence and demonstrate empirically that HiSS outperforms many popular alternatives on a wide variety of tasks, including Ising models, binary neural networks, and combinatorial optimization.

• The paper introduces HiSS, a novel sampling method that bridges separated discrete modes using a logistic auxiliary variable within a Metropolis-within-Gibbs framework.
• It employs a multi-step procedure that combines noising, denoising, and gradient-based refinements to enhance both global exploration and local mixing.
• Empirical evaluations show that HiSS significantly improves mode coverage, mixing rates, and accuracy across synthetic models, combinatorial optimization, and Bayesian neural networks.

Hyperbolic Secant-squared Gibbs Sampling for Discrete Multimodal Distributions

Motivation and Background

Sampling from high-dimensional discrete distributions exhibiting multimodal structure with isolated modes presents significant computational challenges. Standard gradient-based discrete MCMC methods (e.g., DMALA, GWG) leverage differentiable functional extensions but frequently become trapped in local modes due to the rugged, discontinuous nature of the underlying energy landscape. This issue is particularly pronounced when modes are separated by low-density or near-zero probability barriers that defeat global exploration strategies.

The paper introduces Hyperbolic Secant-squared Gibbs Sampling (HiSS) as a novel Metropolis-within-Gibbs sampling framework for discrete distributions, addressing the above challenge. HiSS incorporates a continuous auxiliary variable coupled via a logistic kernel, facilitating transitions across disconnected modes and enhancing mixing efficiency in scenarios where traditional discrete samplers struggle.

Figure 1: A visualization of the multimodal cost landscape in the Traveling Salesman Problem, showcasing solutions separated by low energy barriers.

HiSS Algorithmic Design

Joint Model and Kernel Bridging

HiSS defines a joint distribution over discrete primary variable $\bm{\theta}$ and continuous auxiliary variable $\bm{\theta}_a$, with energy:

$$U(\widetilde{\bm{\theta}}) = U(\bm{\theta}) - 2\ln\left(\cosh\left(\frac{\bm{\theta}_a - \bm{\theta}}{2\eta}\right)\right)$$

The logistic convolutional kernel ($\text{sech}^2$) enables the auxiliary variable to interpolate and bridge probability mass across otherwise isolated modes, retaining non-negligible density paths between them.

Empirical ablations show the logistic kernel's superiority in mode bridging compared to the Gaussian kernel. Its heavier tails distribute intermediate mass more robustly over low-density regions, supporting broader state-space exploration and greater robustness to hyperparameter selection.

Figure 2: Gaussian Kernel vs Logistic Kernel – the logistic kernel retains significantly more mass in the intermediate region, enabling efficient mode bridging.

Gibbs-like Procedure

HiSS employs a multi-step procedure per Gibbs sweep:

1. Noising: Perturbs the current discrete state by sampling $\bm{\theta}_a$ via the logistic kernel to encourage inter-modal exploration.
2. Denoising: Maps $\bm{\theta}_a$ back to a discrete candidate using a softmax over logistic energies for each coordinate.
3. Metropolis-Hastings Correction: Ensures detailed balance w.r.t. the marginal discrete distribution.
4. Gradient-based Refinement: Optionally applies a discrete gradient-based sampler (e.g., DMALA) for local intra-mode mixing.

Ablations confirm the critical impact of both the logistic kernel and the MH correction for proper marginal convergence; omitting the MH step substantially degrades coverage and convergence.

Figure 3: MH vs no-MH — skipping the Metropolis-Hastings step yields rapid but poor convergence to the marginal.

Theoretical Guarantees

The paper establishes detailed balance and uniform ergodicity for HiSS under mild regularity assumptions (locally log-concave energies, $M$-Lipschitz gradients, strongly concave regions) and provides explicit non-asymptotic lower bounds for the mixing rate:

$$\|P^k(x, \cdot) - \pi\|_{TV} \leq (1 - \epsilon_\alpha)^k$$

where $\epsilon_\alpha$ depends analytically on the kernel scale $\eta$, number of Gibbs sweeps $\bm{\theta}_a$0, refinement steps $\bm{\theta}_a$1, and energy landscape geometry (diameter and local gradient). Notably, increasing $\bm{\theta}_a$2 improves mode bridging at the cost of lower acceptance rates, and larger $\bm{\theta}_a$3, $\bm{\theta}_a$4 facilitate faster mixing, but with greater computational cost.

Experimental Evaluation

Synthetic Bernoulli and Ising Models

In a 4D joint Bernoulli example and a $\bm{\theta}_a$5 Ising model with sparse interactions, HiSS achieves the lowest logMAE and total variation distance to the target distribution among all baselines, including parallel tempering. Its ability to cover more isolated modes is quantitatively superior, confirming broader state-space exploration.

Figure 4: 4D Joint Bernoulli — HiSS rapidly achieves low error; competing methods show slower convergence and poor coverage.

Figure 5: Coverage Analysis for 4D Bernoulli — HiSS explores more modes across the state space than gradient or tempering alternatives.

Figure 6: Ising Model — HiSS reaches the lowest TVD, indicating robust mixing and rapid convergence for moderate lattice size.

Figure 7: Coverage Analysis for Ising Model — HiSS saturates the theoretical coverage limit rapidly, while baselines stall in local basins.

Combinatorial Optimization (TSP)

For the 14-city Traveling Salesman Problem, HiSS attains the best mean and variance for cost, highest unique solution count, maximal solution diversity (PMC, Jaccard), and efficiently navigates the exponentially large solution space. Parallel tempering, despite extreme computational overhead, underperforms in both optimality and exploration.

Binary Bayesian Neural Networks

On multiple UCI datasets, HiSS consistently outperforms GWG, DMALA, ACS, and PT in test log-likelihood and RMSE, confirming its capacity to sample from highly multimodal posteriors in binary BNNs, particularly under sparsity-inducing priors.

Practical and Theoretical Implications

HiSS demonstrates a robust framework for sampling discrete distributions with pronounced multimodality and disconnected modes, with rigorous convergence guarantees. The algorithmic decoupling via the auxiliary logistic kernel allows targeted global exploration, while gradient-based refinements focus on local mixing inside modes. Computational analysis reveals that HiSS achieves efficient mode hopping with only a modest increase in energy evaluations compared to gradient samplers, and drastically less than parallel tempering.

This methodology is immediately applicable to spin glasses, Bayesian neural networks, combinatorial optimization, and discrete graphical models. Its kernel-centric approach suggests avenues for further research into asymmetric proposals or landscape-aware bridging kernels to enhance acceptance rates and runtime efficiency.

Figure 8: Impact of scale of logistic noise on solution quality — as $\bm{\theta}_a$6 increases, HiSS produces more diverse, lower-cost solutions.

Conclusion

HiSS delivers an effective and rigorous mechanism for sampling disconnected multimodal discrete distributions, supporting broad coverage and rapid mixing. Its kernel-based auxiliary variable strategy, coupled with MH correction, provides strong theoretical and empirical results—advancing practical probabilistic inference and optimization in challenging discrete domains. Future work may generalize the framework to hybrid continuous-discrete spaces, incorporate landscape scouting for intelligent proposal generation, or analyze convergence properties with alternative refinement samplers.