Hybrid Variational Message Passing

• Hybrid variational message passing is a unified inference framework that blends exact marginalization, moment matching, and factorization constraints to optimize a Bethe free-energy objective.
• It generalizes classical belief propagation, expectation propagation, and variational message passing by permitting heterogeneous local update rules and scheduling within a unified variational framework.
• The approach is applicable to diverse models—from hybrid discrete–continuous systems to neural and reactive extensions—yielding improved computational efficiency and accurate inference.

Hybrid variational message passing denotes a family of inference procedures in which a variational objective is optimized through local message computations, while different parts of a model may obey different local approximations or consistency constraints. A common formalization treats belief propagation (BP), expectation propagation (EP), and variational message passing (VMP) as stationary algorithms for the same Bethe free-energy minimization problem under different constraints, so that hybrid algorithms arise by mixing exact marginalization, moment matching, and factorization constraints across factors, variable blocks, or edges (Zhang et al., 2017). In narrower usages, the term also refers to explicit VMP–EP hybrids, such as vector or matrix message passing schemes derived by variational free-energy minimization and designed to yield tractable Gaussian messages (Jiang et al., 2024).

1. Conceptual scope and relation to neighboring methods

A precise starting point is the constrained-Bethe view. For a factorization

$f(\mathbf{x})=\prod_a f_a(\mathbf{x}_a),$

the Bethe approximation introduces factor beliefs $b_a(\mathbf{x}_a)$ and variable beliefs $b_i(x_i)$, together with marginalization consistency constraints. Within this view, BP is recovered when exact marginal consistency is enforced, EP when consistency is relaxed to moment matching, and VMP when additional factorization constraints are imposed on factor beliefs (Zhang et al., 2017). Reactive Message Passing adopts the same constrained Bethe free-energy foundation and makes the hybrid point explicit by stating that hybrid message passing algorithms can include BP, VMP, EP, and expectation maximisation update rules within one factor-graph engine (Bagaev et al., 2021).

The term is therefore broader than classical mean-field VMP. In region-based variational inference, the Cluster Variational Method (CVM) starts from a free-energy approximation over overlapping regions and then expresses stationarity conditions through local fields. The resulting generalized belief propagation is variational in origin but message-passing in execution, which places it on the same conceptual boundary as hybrid variational message passing even when the local objects are region beliefs rather than factorized variational marginals (Dominguez et al., 2016). This suggests that the topic is best understood as a design space of variationally justified local-update algorithms, rather than a single fixed algorithm.

2. Constrained Bethe free energy and explicit hybrid rules

In the constrained-Bethe framework, the basic optimization problem is

$$\min_{\{b_a\},\{b_i\}} F_{\mathrm B}(\{b_a\},\{b_i\})$$

with

$$F_{\mathrm{B}}(\{b_a\},\{b_i\}) = \sum_a \int b_a(\mathbf{x}_a)\ln\frac{b_a(\mathbf{x}_a)}{f_a(\mathbf{x}_a)}\,d\mathbf{x}_a -\sum_i (A_i-1)\int b_i(x_i)\ln b_i(x_i)\,dx_i,$$

subject to local consistency constraints (Zhang et al., 2017). Exact consistency,

$$\int b_a(\mathbf{x}_a)\,d\mathbf{x}_{a\backslash i}=b_i(x_i),$$

yields BP. Moment matching,

$$\mathrm{E}_{b_a}[\mathbf t(x_i)] = \mathrm{E}_{b_i}[\mathbf t(x_i)],$$

yields EP. Factorization constraints,

$$b_a(\mathbf{x}_a)=\prod_{i\in\mathcal I_a} b_a(x_i),$$

yield VMP. Partial factorization,

$$b_a(\mathbf{x}_a)=\prod_v b_{a,v}(\mathbf{x}_{a,v}),$$

gives a more general hybrid construction in which some dependencies are preserved inside blocks and broken across blocks; the resulting factor-to-variable messages contain a VMP-generated effective factor inside a BP- or EP-style local marginalization (Zhang et al., 2017).

An explicit HVMP rule appears in generalized bilinear factorization. There the factor-to-variable update is

$$m_{f \rightarrow \boldsymbol{v}}(\boldsymbol{v}) = \frac{1}{m_{\boldsymbol{v} \rightarrow f}(\boldsymbol{v})} \operatorname{proj}\!\left[ m_{\boldsymbol{v} \rightarrow f}(\boldsymbol{v}) \times e^{\int_{\mathcal{V}_f\setminus\boldsymbol{v}} \prod_{\boldsymbol{v}^\prime\in\mathcal{V}_f\setminus \boldsymbol{v}} m_{f \rightarrow \boldsymbol{v}^\prime}(\boldsymbol{v}^\prime) m_{\boldsymbol{v}^\prime \rightarrow f}(\boldsymbol{v}^\prime) \ln f(\mathcal{V}_f)} \right].$$

The exponential of an expected log-factor is VMP-like, while the projection and division by the incoming message are EP-like, and the paper states that this rule “can be interpreted as a hybrid of two conventional message passing algorithms, namely, expectation propagation (EP) and variational message passing (VMP)” (Jiang et al., 2024). In that formulation, EP and VMP appear as limiting cases obtained by changing the auxiliary-factor factorization assumptions and the projection structure.

3. Region-based, mixed-operator, and gauge-free generalizations

Hybridization need not be restricted to mean-field blocks. In gauge-free CVM, messages arrive on a region from all its ancestors rather than only from direct parents, and moment matching assigns each moment constraint to the smallest region in which it first becomes local. The key rule is that a message $b_a(\mathbf{x}_a)$0 contains a field associated with subset $b_a(\mathbf{x}_a)$1 if and only if $b_a(\mathbf{x}_a)$2 is the smallest region among all those containing the variables in $b_a(\mathbf{x}_a)$3. The resulting generalized belief propagation is gauge free because the number of fields exactly matches the number of independent consistency equations, avoiding redundant message parameterizations (Dominguez et al., 2016). This is a region-based counterpart to hybrid VMP in which the global variational meaning is preserved while local updates are expressed through moment-consistency fields.

A different kind of hybridization arises in marginal MAP. The exact dual representation is

$b_a(\mathbf{x}_a)$4

so the variational objective retains entropy only for marginalized variables and removes it for maximized variables (Liu et al., 2013). The corresponding mixed-product message passing combines sum-product, max-product, and the novel argmax-product update

$b_a(\mathbf{x}_a)$5

for messages from max nodes to sum nodes (Liu et al., 2013). This shows that “hybrid” can also refer to mixed operators inside one variational message-passing scheme, not only to mixed approximating families.

The same variational-to-message pipeline extends to quantum region methods. In the quantum Bethe and Kikuchi formulations, the local beliefs are density operators, the consistency constraints are partial traces, and the Lagrange multipliers become message-like local fields. At the Bethe level, the stationary conditions are rewritten as BP-like equations over link-to-spin cavity fields; at the plaquette level they become GBP-like equations over plaquette, link, and site regions (Dominguez et al., 2017). This suggests that hybrid variational message passing is best seen as a general methodology for converting constrained free-energy approximations into local iterative solvers.

4. Hybrid discrete–continuous models and lifted formulations

A conceptually adjacent line of work treats hybrid factor graphs

$b_a(\mathbf{x}_a)$6

where each variable may be discrete or continuous and the potentials $b_a(\mathbf{x}_a)$7 are generic nonnegative functions (Chen et al., 2020). The approximation family is a mixture of fully factorized components,

$b_a(\mathbf{x}_a)$8

with Gaussian $b_a(\mathbf{x}_a)$9 for continuous variables and categorical $b_i(x_i)$0 for discrete variables. Two objectives are proposed. The first, BVI, replaces the intractable mixture entropy by the Bethe entropy,

$b_i(x_i)$1

yielding a restricted Bethe free-energy minimization. The second, NPVI, uses the Jensen lower bound

$b_i(x_i)$2

Both are optimized by direct gradient descent, with Gaussian quadrature for factor expectations and Adam in experiments, so they are not classical VMP updates (Chen et al., 2020).

The paper is explicit that its main bridge to message passing is the Bethe free energy: “message-passing algorithms such as belief propagation (BP) can also be used to find local optima of the BFE,” but the proposed methods instead use global gradient-based optimization (Chen et al., 2020). It is therefore best characterized as lifted, hybrid, mixture-based variational free-energy minimization rather than standard hybrid VMP. Even so, it contributes directly to the hybrid-VMP literature by showing how discrete and continuous variables can be handled jointly without Gaussianity or conjugacy assumptions, how multimodality can be represented through $b_i(x_i)$3, and how approximate symmetries can be enforced through color passing, tied variational parameters, and coarse-to-fine clustering of continuous evidence.

The lifting component is central. Exact symmetries are detected by color passing and represented as super-variables and super-factors with multiplicities, while approximate symmetry under continuous evidence is restored by a coarse-to-fine scheme that clusters evidence values by $b_i(x_i)$4-means and temporarily replaces them by Gaussian surrogates

$b_i(x_i)$5

Theoretical support is given by a sufficient condition under which the Bethe approximation is lower bounded over the marginal polytope: if

$b_i(x_i)$6

for some product of univariate densities, then

$b_i(x_i)$7

Empirically, BVI and lifted BVI tend to give better KL divergence on marginals, NPVI and lifted NPVI often do better on MAP or mode estimation, lifting gives roughly $b_i(x_i)$8–$b_i(x_i)$9 speedup on the toy hybrid MLN, and on tree-structured relational Kalman filters the reported runtime drops from around $$\min_{\{b_a\},\{b_i\}} F_{\mathrm B}(\{b_a\},\{b_i\})$$0s for BVI to around $$\min_{\{b_a\},\{b_i\}} F_{\mathrm B}(\{b_a\},\{b_i\})$$1s for L-BVI with essentially no loss in accuracy (Chen et al., 2020).

5. Recognition-guided, reactive, and neural extensions

One algorithmic extension preserves ordinary VMP or EP updates but changes the message schedule. Consensus Message Passing learns predictors

$$\min_{\{b_a\},\{b_i\}} F_{\mathrm B}(\{b_a\},\{b_i\})$$2

that map contextual messages from a lower layer to consensus messages sent to influential variables in large, loopy, layered generative models (Jampani et al., 2014). These messages do not change the factor definitions or the ordinary VMP or EP update equations; they augment the schedule, are given maximum priority within a layer, are sent bottom-up, and in experiments sending them only in the first iteration was sufficient and effective (Jampani et al., 2014). The resulting procedure is hybrid in an algorithmic sense: bottom-up discriminative prediction is inserted as message-level guidance, while top-down probabilistic inference remains standard VMP or EP.

A more formal neural hybrid is the structured inference-network approach for deep structured latent-variable models. The variational family is

$$\min_{\{b_a\},\{b_i\}} F_{\mathrm B}(\{b_a\},\{b_i\})$$3

which combines amortized neural factors with a graph-structured PGM factor (Lin et al., 2018). Efficient amortized inference requires tractable evaluation and differentiation of $$\min_{\{b_a\},\{b_i\}} F_{\mathrm B}(\{b_a\},\{b_i\})$$4 and the ability to sample from the structured posterior. Global variables are then updated by a natural-gradient/VMP step,

$$\min_{\{b_a\},\{b_i\}} F_{\mathrm B}(\{b_a\},\{b_i\})$$5

This is a direct hybridization of amortized neural evidence and graphical-model message passing (Lin et al., 2018).

Reactive Message Passing generalizes the execution model rather than the local algebra. It treats messages, marginals, and local variational posteriors as reactive observables and states that hybrid message passing algorithms including BP, VMP, EP, and EM can be executed through constrained Bethe free-energy minimization without a fixed global schedule (Bagaev et al., 2021). The hybrid choice is local: one factor can react to messages, another to marginals, and another to moment constraints, while the underlying score remains the Bethe free energy. This suggests that hybrid variational message passing is also an implementation paradigm in which locality and scheduling are first-class objects.

A broader architectural usage appears in neural Ising solvers. Message Passing Variational Autoregressive Networks optimize the variational free energy

$$\min_{\{b_a\},\{b_i\}} F_{\mathrm B}(\{b_a\},\{b_i\})$$6

for an autoregressive $$\min_{\{b_a\},\{b_i\}} F_{\mathrm B}(\{b_a\},\{b_i\})$$7, while embedding graph message passing inside the network (Ma et al., 2024). The paper does not present classical VMP, BP, or EP messages, so this is not standard HVMP. It nonetheless shows that the phrase can extend to settings in which a variational objective and a graph-structured message mechanism are combined within one learned approximation architecture.

6. Applications, reusable modules, and boundaries of the term

The most explicit application-specific HVMP construction in the surveyed literature is generalized bilinear factorization. There, vector or matrix variables replace scalar variables, the graph becomes a loop-free Bayesian network over $$\min_{\{b_a\},\{b_i\}} F_{\mathrm B}(\{b_a\},\{b_i\})$$8 and $$\min_{\{b_a\},\{b_i\}} F_{\mathrm B}(\{b_a\},\{b_i\})$$9, and tractable Gaussian matrix-normal messages are obtained by combining VMP-style expected-log factors with EP-style Gaussian projection (Jiang et al., 2024). In the reported experiments, the resulting GBF-HVMP algorithm significantly outperforms AEM-MP, BiG-AMP, and Pro-BiG-AMP in NMSE and runtime; for $$F_{\mathrm{B}}(\{b_a\},\{b_i\}) = \sum_a \int b_a(\mathbf{x}_a)\ln\frac{b_a(\mathbf{x}_a)}{f_a(\mathbf{x}_a)}\,d\mathbf{x}_a -\sum_i (A_i-1)\int b_i(x_i)\ln b_i(x_i)\,dx_i,$$0 and $$F_{\mathrm{B}}(\{b_a\},\{b_i\}) = \sum_a \int b_a(\mathbf{x}_a)\ln\frac{b_a(\mathbf{x}_a)}{f_a(\mathbf{x}_a)}\,d\mathbf{x}_a -\sum_i (A_i-1)\int b_i(x_i)\ln b_i(x_i)\,dx_i,$$1, it recovers $$F_{\mathrm{B}}(\{b_a\},\{b_i\}) = \sum_a \int b_a(\mathbf{x}_a)\ln\frac{b_a(\mathbf{x}_a)}{f_a(\mathbf{x}_a)}\,d\mathbf{x}_a -\sum_i (A_i-1)\int b_i(x_i)\ln b_i(x_i)\,dx_i,$$2 with $$F_{\mathrm{B}}(\{b_a\},\{b_i\}) = \sum_a \int b_a(\mathbf{x}_a)\ln\frac{b_a(\mathbf{x}_a)}{f_a(\mathbf{x}_a)}\,d\mathbf{x}_a -\sum_i (A_i-1)\int b_i(x_i)\ln b_i(x_i)\,dx_i,$$3 dB when $$F_{\mathrm{B}}(\{b_a\},\{b_i\}) = \sum_a \int b_a(\mathbf{x}_a)\ln\frac{b_a(\mathbf{x}_a)}{f_a(\mathbf{x}_a)}\,d\mathbf{x}_a -\sum_i (A_i-1)\int b_i(x_i)\ln b_i(x_i)\,dx_i,$$4, while the baselines fail to achieve $$F_{\mathrm{B}}(\{b_a\},\{b_i\}) = \sum_a \int b_a(\mathbf{x}_a)\ln\frac{b_a(\mathbf{x}_a)}{f_a(\mathbf{x}_a)}\,d\mathbf{x}_a -\sum_i (A_i-1)\int b_i(x_i)\ln b_i(x_i)\,dx_i,$$5 dB across the tested sparsity range, and the algorithm does so without adaptive damping (Jiang et al., 2024).

A second application class uses non-conjugate regression models but keeps message passing modular. For scalar linear predictors $$F_{\mathrm{B}}(\{b_a\},\{b_i\}) = \sum_a \int b_a(\mathbf{x}_a)\ln\frac{b_a(\mathbf{x}_a)}{f_a(\mathbf{x}_a)}\,d\mathbf{x}_a -\sum_i (A_i-1)\int b_i(x_i)\ln b_i(x_i)\,dx_i,$$6 and Gaussian $$F_{\mathrm{B}}(\{b_a\},\{b_i\}) = \sum_a \int b_a(\mathbf{x}_a)\ln\frac{b_a(\mathbf{x}_a)}{f_a(\mathbf{x}_a)}\,d\mathbf{x}_a -\sum_i (A_i-1)\int b_i(x_i)\ln b_i(x_i)\,dx_i,$$7, the non-conjugate observation factors are reduced to univariate Gaussian expectations

$$F_{\mathrm{B}}(\{b_a\},\{b_i\}) = \sum_a \int b_a(\mathbf{x}_a)\ln\frac{b_a(\mathbf{x}_a)}{f_a(\mathbf{x}_a)}\,d\mathbf{x}_a -\sum_i (A_i-1)\int b_i(x_i)\ln b_i(x_i)\,dx_i,$$8

so that the Gaussian variational block is updated by

$$F_{\mathrm{B}}(\{b_a\},\{b_i\}) = \sum_a \int b_a(\mathbf{x}_a)\ln\frac{b_a(\mathbf{x}_a)}{f_a(\mathbf{x}_a)}\,d\mathbf{x}_a -\sum_i (A_i-1)\int b_i(x_i)\ln b_i(x_i)\,dx_i,$$9

The paper’s point is that only univariate numerical integrals are required when analytical solutions are unavailable, and that no differentiability, conjugacy, or model-specific data-augmentation strategy is required (Castiglione et al., 2022). In hybrid-VMP terms, this is a template in which conjugate priors use standard VMP while non-conjugate likelihood factors emit custom Gaussian-compatible messages.

At the level of reusable software fragments, the Inverse G-Wishart family provides exact VMP modules for covariance and variance parameters. The sufficient statistics are

$$\int b_a(\mathbf{x}_a)\,d\mathbf{x}_{a\backslash i}=b_i(x_i),$$0

and both the inverse G-Wishart prior fragment and the iterated inverse G-Wishart fragment emit closed-form natural-parameter messages for $$\int b_a(\mathbf{x}_a)\,d\mathbf{x}_{a\backslash i}=b_i(x_i),$$1 and $$\int b_a(\mathbf{x}_a)\,d\mathbf{x}_{a\backslash i}=b_i(x_i),$$2 cases (Maestrini et al., 2020). These fragments are not themselves hybrid algorithms, but they are exact covariance-side modules that can be embedded in larger HVMP systems.

The boundary of the term is visible in adjacent work. Online polynomial NARMAX identification uses standard mean-field VMP on a Forney-style factor graph with Gaussian and Gamma variational factors, recursive posterior-as-prior updates, and no explicit mixing with BP, EP, or other local rules (Kouw et al., 2022). Exact message passing for model integration with logical constraints is similarly adjacent rather than variational: it computes piecewise-polynomial messages exactly on suitable primal graphs, but it does not optimize an ELBO or free energy (Zeng et al., 2019). This suggests that “hybrid variational message passing” is best reserved for methods that keep a variational semantics while allowing heterogeneity in local update rules, message parameterizations, or optimization schedules, even though the surrounding literature includes several closely related variational alternatives and message-passing precursors.