---
title: Log-Supermodular Graphical Models
url: https://www.emergentmind.com/topics/log-supermodular-graphical-models
type: topic
---

# Log-Supermodular Graphical Models

Searching arXiv for the cited papers to ground the article in the primary sources.
arXiv Search Query: id:1502.06531 OR id:1202.6035 OR id:1309.6859 OR id:1608.05258
Log-supermodular graphical models are probabilistic models on binary variables whose negative log-densities are submodular set-functions. In the Gibbs form, they define distributions \(P(S)=\frac{1}{Z}\exp(-F(S))\) over subsets \(S\subseteq V\), where \(F:2^V\to\mathbb{R}\) is submodular and \(Z=\sum_{T\subseteq V}\exp(-F(T))\) is the partition function. This class contains attractive pairwise Markov random fields (MRFs) with binary variables and also admits higher-order interactions that are not naturally handled by standard approximate inference procedures such as belief propagation and mean field. Research on the topic has developed a variational program known as L-FIELD, a minimum-norm characterization of its solution, lower bounds for the Bethe partition function in binary log-supermodular models, and parameter-learning methods based on tractable upper bounds on the log-partition function [1502.06531], [1202.6035], [1608.05258].

## 1. Definition and structural properties

Let \(V\) be a ground set with \(|V|\) binary variables \(X_i\in\{0,1\}\), identified equivalently with subsets \(S\subseteq V\). A set-function \(F:2^V\to\mathbb{R}\) is submodular if for all \(A\subseteq B\subseteq V\) and \(x\notin B\),
\[
F(A\cup\{x\})-F(A)\ge F(B\cup\{x\})-F(B),
\]
with the normalization \(F(\emptyset)=0\). In coordinate form, if \(x,y\in\{0,1\}^D\), submodularity is equivalently
\[
F(x)+F(y)\ge F(x\wedge y)+F(x\vee y),
\]
and the discrete partial differences \(x\mapsto F(x+e_i)-F(x)\) are nonincreasing in \(x\) [1608.05258].

A log-supermodular model, also called “attractive,” is the Gibbs distribution
\[
P(S)=\frac{1}{Z}\exp(-F(S)).
\]
In the equivalent multiplicative formulation, a strictly positive function \(f:\{0,1\}^n\to\mathbb{R}_{>0}\) is log-supermodular if
\[
f(x\wedge y)f(x\vee y)\ge f(x)f(y)
\]
for all \(x,y\in\{0,1\}^n\); equivalently, \(\log f\) is a supermodular set-function on the Boolean lattice [1202.6035].

The class admits decompositions into local submodular terms. A common form is
\[
F(S)=\sum_{i=1}^R F_i(S\cap V_i),
\]
where each \(F_i:2^{V_i}\to\mathbb{R}\) is submodular on its own ground set \(V_i\subseteq V\) [1502.06531]. Two canonical examples are explicitly identified. For a pairwise MRF, one may take \(V_i=\{p,p'\}\) and
\[
F_i(S\cap V_i)=w_{p,p'}\cdot 1_{|S\cap\{p,p'\}|=1}.
\]
For a higher-order potential on a superpixel \(P\), one may take a concave \(\phi:[0,1]\to\mathbb{R}\) and define
\[
F_i(S\cap P)=\phi(|S\cap P|/|P|).
\]
The attractive pairwise condition in the factorized viewpoint is
\[
\psi_{ij}(0,0)\psi_{ij}(1,1)\ge \psi_{ij}(0,1)\psi_{ij}(1,0),
\]
and the higher-order extension requires each factor to be log-supermodular on its local Boolean cube [1202.6035].

A central geometric object is the modular lower bound \(s(S)=\sum_{i\in S}s_i\), together with the polyhedra
\[
P(F)=\{s\in\mathbb{R}^V:\forall A\subseteq V,\ s(A)\le F(A)\},
\]
and
\[
B(F)=P(F)\cap\{s:s(V)=F(V)\},
\]
the base polytope [1502.06531]. These objects underlie both variational inference and optimization-based learning.

## 2. L-FIELD and variational inference

A major development for approximate Bayesian inference in this class is L-FIELD. If \(s\in P(F)\), then \(s(S)\le F(S)\) for all \(S\subseteq V\), which yields
\[
Z=\sum_S e^{-F(S)}\le \sum_S e^{-s(S)}=\prod_{i=1}^n (1+e^{-s_i}).
\]
Therefore
\[
\log Z \le L(s):=\sum_{i=1}^n \log(1+e^{-s_i}),
\]
and the variational problem is to minimize \(L(s)\) over \(s\in B(F)\) [1502.06531]. Because the objective is separable while the feasible set is the base polytope, the program converts a globally intractable partition-function computation into a convex optimization problem over a combinatorial polytope.

The striking structural result is that the unique minimizer of \(L(s)\) over \(B(F)\) is also the minimum-norm point of \(B(F)\), equivalently the solution of
\[
\min_{s\in B(F)} \|s\|_2^2.
\]
This equivalence reduces L-FIELD to the classical minimum-norm-point convex program and makes it possible to use standard submodular-polytope oracles and minimum-norm-point algorithms [1502.06531]. By Fujishige’s theorem, thresholding the minimum-norm solution \(s^\*\) at \(0\), or equivalently thresholding marginals at \(1/2\), yields the exact MAP minimizers.

The same program has an information-theoretic interpretation. Let \(Q_q\) be the fully factorized Gibbs distribution with modular \(q\in\mathbb{R}^V\),
\[
Q_q(S)\propto \exp(-q(S)),\qquad q(S)=\sum_{i\in S}q_i.
\]
The infinite-order Rényi divergence is
\[
D_\infty(P\|Q)=\log \sup_{S\subseteq V}\frac{P(S)}{Q(S)}.
\]
Using
\[
D_\infty(P\|Q)=\log Z_Q-\log Z_P+\sup_S[q(S)-F(S)],
\]
and restricting to \(q\in P(F)\), one obtains exactly the L-FIELD objective since \(\sup_S[q(S)-F(S)]=0\) and \(\log Z_Q=\sum_i \log(1+e^{-q_i})\) [1502.06531]. Thus L-FIELD finds the factorized \(Q\) minimizing \(D_\infty(P\|Q)\). This characterization explains why the approximation is conservative in a worst-case multiplicative sense rather than calibrated to the KL-type objectives more commonly associated with mean field.

## 3. Parallel inference as message passing

When the energy decomposes as \(F(S)=\sum_{i=1}^R F_i(S\cap V_i)\), inference can be implemented by parallel message passing on a factor graph \(G=(V\cup\{\text{factors}\},E)\), with edges \((v,F_i)\) whenever \(v\in V_i\) [1502.06531]. For \(x\in\mathbb{R}^S\), the analysis introduces the norms
\[
\|x_S\|_G^2=\sum_{v\in S}\frac{1}{|\delta(v)|}x_v^2,\qquad
\|x_S\|_{G^\*}^2=\sum_{v\in S}|\delta(v)|\,x_v^2,
\]
where \(\delta(v)\) denotes the neighboring factors of \(v\).

The algorithm maintains, for each factor \(F_i\), a vector \(q_i\in B(F_i)\), together with real-valued messages \(\mu_{v\to F_i}\) and \(\mu_{F_i\to v}\). The parallel variable-to-factor update is
\[
\mu_{v\to F_i}^{t+1}=\frac{1}{|\delta(v)|}\sum_{F_j\in\delta(v)}\mu_{F_j\to v}^t,
\]
so each variable averages incoming factor-to-variable messages. The factor update gathers \(m_i^t=(\mu_{v\to F_i}^t)_{v\in V_i}\) and solves
\[
q_i^{t+1}=\arg\min_{q_i\in B(F_i)} \|q_i-(q_i^t-m_i^t)\|_{G^\*}^2,
\]
then sends \(\mu_{F_i\to v}^{t+1}=q_i^{t+1,v}\). A global iterate can be recovered as
\[
s_v^t=\sum_{F_i\in\delta(v)}\mu_{F_i\to v}^t.
\]

This formulation avoids summing over all factor configurations, which is the major obstacle for many high-order methods. The convergence guarantee is explicit: if every variable has exactly \(\Delta_V\) incident factors, then
\[
\|q^t-q^\*\|\le 2\|q^0-q^\*\|_\infty \sqrt{\Delta_V\cdot |E|}\cdot \left(1-\frac{1}{|V|^2\Delta_V^2}\right)^t,
\]
which is a global linear, i.e. geometric, rate [1502.06531]. A plausible implication is that the factor-graph decomposition is not merely an implementation convenience; it supplies the regularity conditions under which scalable parallel inference becomes analyzable.

## 4. Bethe approximation, belief propagation, and partition-function bounds

For a factor graph \(G=(V,A)\) with unary potentials \(\phi_i(x_i)\) and higher-order potentials \(\psi_a(x_a)\), the true partition function is
\[
Z=\sum_{x\in\{0,1\}^n}\prod_{i\in V}\phi_i(x_i)\prod_{a\in A}\psi_a(x_a).
\]
The Bethe approximation defines a variational estimate of \(\log Z\) over pseudomarginals \(\{q_i(x_i),q_a(x_a)\}\) that satisfy normalization and marginal-consistency constraints, and the Bethe free energy is
\[
F_{\rm Bethe}(q)
=
-\sum_{i\in V}\sum_{x_i}q_i(x_i)\ln\phi_i(x_i)
-\sum_{a\in A}\sum_{x_a}q_a(x_a)\ln\psi_a(x_a)
+\sum_{i\in V}H(q_i)
-\sum_{a\in A}(|a|-1)H(q_a),
\]
where \(H(q)=-\sum q\ln q\) is the Shannon entropy [1202.6035]. A stationary point \(q^\*\) of \(F_{\rm Bethe}\) in the interior of the local polytope corresponds exactly to a fixed point of loopy belief propagation.

For arbitrary graphical models, the Bethe estimate \(Z_{\rm Bethe}=\exp(-F_{\rm Bethe}(q^\*))\) has no universal lower- or upper-bound guarantee. In binary log-supermodular models, however, the situation is different: every BP fixed point satisfies
\[
Z_{\rm Bethe}\le Z.
\]
This theorem applies to any factor graph over binary variables whose every factor is log-supermodular, not merely to pairwise attractive models [1202.6035].

The proof uses graph covers and a specialized correlation inequality. Vontobel’s theorem expresses the Bethe partition function through true partition functions on \(k\)-covers, and it therefore suffices to show that for every \(k\)-cover \(H\),
\[
Z(H)\le (Z(G))^k.
\]
Ruozzi’s argument establishes this using a new variant of the “four functions” theorem adapted to log-supermodular functions [1202.6035]. The result resolves, in the affirmative, a conjecture that the Bethe approximation corresponding to any fixed point of belief propagation over an attractive, pairwise binary graphical model provides a lower bound on the true partition function, and simultaneously extends the guarantee to higher-order log-supermodular factors.

Several limitations are explicit. The variables must be binary, the proof relies heavily on Boolean lattice structure, and no upper bound is claimed. In the non-supermodular case, such as antiferromagnetic models, the same technique does not apply [1202.6035]. This directly corrects a common misconception that “attractive” alone is sufficient for broad Bethe guarantees without domain restrictions.

## 5. Parameterization and learning

A standard exponential-family parameterization writes
\[
p_\theta(x)=\exp[-F_\theta(x)-A(\theta)],
\]
where \(A(\theta)=\log\sum_{x\in\{0,1\}^D}\exp[-F_\theta(x)]\) is the log-partition function [1608.05258]. One parameterization of the submodular energy is
\[
F_\theta(x)=\sum_{k=1}^K \alpha_k f_k(x)-t^Tx,
\]
with \(\alpha_k\ge 0\), \(t\in\mathbb{R}^D\), and each \(f_k\) submodular. Exact evaluation of \(A(\theta)\) is \(\#P\)-hard, so learning relies on tractable upper bounds.

Two upper bounds are identified. The separable L-field bound is
\[
A_L(\theta)=\min_{s\in B(F)}\sum_{d=1}^D \log(1+e^{-s_d}),
\]
obtained from the Lovász-extension representation \(F(x)=\max_{s\in B(F)} s^T x\) [1608.05258]. The perturb-and-MAP or logistic bound is
\[
A_{\log}(\theta)\equiv E_z\left[\max_y \{z^T y-F(y)\}\right],
\]
where \(z\) has i.i.d. logistic entries, derived from replacing a collection of \(2^D\) Gumbel perturbations by a factored collection of \(D\) pairs of i.i.d. Gumbels [1608.05258]. The dominance relation is
\[
A(\theta)\le A_{\log}(\theta)\le A_L(\theta).
\]
Accordingly, the bound based on separable optimization on the base polytope is always inferior to a bound based on perturb-and-MAP ideas.

Given \(N\) i.i.d. observations, replacing \(A(\theta)\) by \(A_{\log}(\theta)\) yields a convex surrogate for negative log-likelihood, and parameter learning proceeds by projected stochastic subgradient descent. If
\[
\Psi(\theta;z)=\max_{y\in\{0,1\}^D}\left\{z^T y+t^Ty-\sum_k \alpha_k f_k(y)\right\},
\]
and \(y^\*(z;\theta)\) is any maximizer, then the subgradients are
\[
\partial_t L(\theta)=-\langle x\rangle_{\rm emp}+E_z[y^\*(z;\theta)],
\qquad
\partial_{\alpha_k}L(\theta)=\langle f_k(x)\rangle_{\rm emp}-E_z[f_k(y^\*(z;\theta))],
\]
plus any \(\partial\)-terms from a convex regularizer \(\Omega(\theta)\) [1608.05258]. The algorithm initializes \(\alpha\leftarrow 0\), \(t\leftarrow 0\), samples \(z\sim \text{Logistic}^D\), solves the inner maximization by submodular minimization, and uses step sizes \(\eta_h=C/\sqrt{h}\). Because the surrogate objective is convex and Lipschitz continuous, the averaged iterate converges to the global minimizer at \(O(1/\sqrt{H})\) in function value.

The same perturb-and-MAP surrogate extends to conditional maximum likelihood. For noisy observations \(z\) of latent clean images \(x\), the posterior remains log-supermodular with energy
\[
F_{\theta,z}(x)=F_\theta(x)-x^T(u-2(u\odot z)),
\]
and one may optimize supervised, partially observed, or fully unsupervised objectives by stochastic subgradient descent of the corresponding approximate likelihoods [1608.05258]. This suggests that tractable MAP subroutines, rather than tractable marginalization, are the decisive computational primitive for learning in this family.

## 6. Extensions and adjacent model classes

The partition-function lower-bound theory motivates a broader set of constructions in which the original model is not itself binary or log-supermodular, but can be reformulated as a log-supermodular model on an expanded domain. One example is the ferromagnetic Potts model with a uniform external field. Using the random-cluster representation,
\[
Z_{\rm Potts}(G;q,J)=\sum_{A\subseteq E} q^{k_G(A)}\prod_{(i,j)\in A}(e^{J_{ij}}-1)=:Z_{\rm RC}(G;q,p),
\]
and since \(k_G(A)\) is supermodular in the indicator-vector of \(A\), \(q^{k_G(A)}\) is log-supermodular in that vector [1309.6859]. The same cover-based argument then yields
\[
Z_{\rm Bethe}(G;q,J,h)\le Z^+_{\rm Potts}(G;q,J,h)
\]
for the uniform-field Potts model.

A second example is a special class of weighted graph homomorphism problems. For
\[
Z_{\rm hom}(G;w,T)=\sum_{\sigma:V\to\{1..n\}}
\left[\prod_{i\in V}w_{\sigma_i}\right]
\left[\prod_{(i,j)\in E}T_{\sigma_i,\sigma_j}\right],
\]
if
\[
T=aa^T+bb^T,\qquad a,b\in\mathbb{R}_+^n,
\]
then the model can be rewritten as an edge-coloring model whose local factors are log-supermodular in the indicator-vector of an edge subset \(A\). This again implies
\[
Z_{\rm Bethe}(G;w,T)\le Z_{\rm hom}(G;w,T)
\]
whenever \(T=aa^T+bb^T\) [1309.6859].

These constructions do not redefine such models as log-supermodular graphical models in their native parameterization. Rather, they show that log-supermodularity functions as a transferable proof device: if an auxiliary representation is log-supermodular, then cover-based partition-function bounds can persist beyond the original binary setting.

## 7. Empirical behavior and application domains

The most detailed application in scalable variational inference is image segmentation. The reported dataset comprises 36 natural images with pixel-level ground truth, with image sizes up to \(427\times 640\) pixels [1502.06531]. The model is
\[
F(S)=\alpha\cdot m(S)+\beta\cdot F_{\rm cut}(S)+\gamma\cdot \sum_{P\in \text{superpixels}}\phi(|S\cap P|/|P|),
\qquad
\phi(z)=z(1-z).
\]
The comparisons include unary only (independent), Pairwise BP, Mean-Field, Fractional BP (libDAI), L-FIELD/Douglas–Rachford with pairwise only, and L-FIELD with higher-order potentials. Reported runtimes are DR (pairwise) \(<1\)s, HOP \(<12\)s, and BP/MF \(45\)s–\(3\)min. Accuracy, measured as AUC over the full image and over trimaps near boundaries, is reported as
\[
\text{HOP}\approx 0.964 > \text{DR}\approx 0.960 > \text{FBP}\approx 0.950 > \text{MF}\approx 0.949 > \text{unary}\approx 0.948 > \text{BP}\approx 0.944.
\]
Qualitatively, BP and FBP tend to be over-confident, L-FIELD yields more conservative marginals, and incorporating higher-order potentials yields sharper boundary marginals and higher AUC near edges [1502.06531].

A distinct learning-focused experiment concerns binary image denoising on binary horse silhouettes of size \(50\times 50\) pixels, with \(N_{\rm train}=100\) and \(N_{\rm test}=100\), and independent pixel flips with probabilities \(\pi\in\{1\%,5\%,10\%,20\%\}\) [1608.05258]. The model uses two submodular cut functions, corresponding to horizontal and vertical 4-neighbor boundary costs, plus a per-pixel modular bias \(t\). In the supervised setting, the reported mean-marginals error is approximately \(0.4\%\) at \(\pi=1\%\), versus \(0.6\%\) for a structured-SVM baseline; at \(\pi=20\%\), approximately \(4.1\%\) versus \(6.0\%\). In the unsupervised setting, when only the noisy image is observed but \(\pi\) is known, performance degrades only slightly: at \(\pi=5\%\), mean-marginals error is approximately \(1.0\%\), versus \(1.1\%\) in the supervised case. When \(\pi\) is also learned, errors rise, for example to approximately \(3.6\%\) at true \(\pi=5\%\), but the model still outperforms a trivial “no-change” decoder for moderate noise levels [1608.05258].

Taken together, these results show that log-supermodular graphical models support two complementary computational regimes. For inference, high-order attractive structure can be handled by base-polytope optimization and parallel message passing without explicit enumeration of factor states. For learning, tractable MAP subproblems combined with perturb-and-MAP surrogates enable approximate maximum likelihood even with missing or latent data.

Source: https://www.emergentmind.com/topics/log-supermodular-graphical-models