Log-Supermodular Graphical Models

• Log-supermodular graphical models are probabilistic models on binary variables whose negative log-densities are submodular functions, enabling both pairwise and higher-order interactions.
• The L-FIELD variational framework reformulates partition function computation as a convex optimization problem over the base polytope, ensuring efficient and scalable inference via parallel message passing.
• Extensions to models like the Potts model and graph homomorphisms, along with empirical successes in image segmentation and denoising, highlight the method’s versatility and practical impact.

Searching arXiv for the cited papers to ground the article in the primary sources. arXiv Search Query: id:(Djolonga et al., 2015) OR id:(Ruozzi, 2012) OR id:(Ruozzi, 2013) OR id:(Shpakova et al., 2016) Log-supermodular graphical models are probabilistic models on binary variables whose negative log-densities are submodular set-functions. In the Gibbs form, they define distributions $P(S)=\frac{1}{Z}\exp(-F(S))$ over subsets $S\subseteq V$, where $F:2^V\to\mathbb{R}$ is submodular and $Z=\sum_{T\subseteq V}\exp(-F(T))$ is the partition function. This class contains attractive pairwise Markov random fields (MRFs) with binary variables and also admits higher-order interactions that are not naturally handled by standard approximate inference procedures such as belief propagation and mean field. Research on the topic has developed a variational program known as L-FIELD, a minimum-norm characterization of its solution, lower bounds for the Bethe partition function in binary log-supermodular models, and parameter-learning methods based on tractable upper bounds on the log-partition function (Djolonga et al., 2015, Ruozzi, 2012, Shpakova et al., 2016).

1. Definition and structural properties

Let $V$ be a ground set with $|V|$ binary variables $X_i\in\{0,1\}$, identified equivalently with subsets $S\subseteq V$. A set-function $F:2^V\to\mathbb{R}$ is submodular if for all $A\subseteq B\subseteq V$ and $S\subseteq V$0,

$S\subseteq V$1

with the normalization $S\subseteq V$2. In coordinate form, if $S\subseteq V$3, submodularity is equivalently

$S\subseteq V$4

and the discrete partial differences $S\subseteq V$5 are nonincreasing in $S\subseteq V$6 (Shpakova et al., 2016).

A log-supermodular model, also called “attractive,” is the Gibbs distribution

$S\subseteq V$7

In the equivalent multiplicative formulation, a strictly positive function $S\subseteq V$8 is log-supermodular if

$S\subseteq V$9

for all $F:2^V\to\mathbb{R}$0; equivalently, $F:2^V\to\mathbb{R}$1 is a supermodular set-function on the Boolean lattice (Ruozzi, 2012).

The class admits decompositions into local submodular terms. A common form is

$F:2^V\to\mathbb{R}$2

where each $F:2^V\to\mathbb{R}$3 is submodular on its own ground set $F:2^V\to\mathbb{R}$4 (Djolonga et al., 2015). Two canonical examples are explicitly identified. For a pairwise MRF, one may take $F:2^V\to\mathbb{R}$5 and

$F:2^V\to\mathbb{R}$6

For a higher-order potential on a superpixel $F:2^V\to\mathbb{R}$7, one may take a concave $F:2^V\to\mathbb{R}$8 and define

$F:2^V\to\mathbb{R}$9

The attractive pairwise condition in the factorized viewpoint is

$Z=\sum_{T\subseteq V}\exp(-F(T))$0

and the higher-order extension requires each factor to be log-supermodular on its local Boolean cube (Ruozzi, 2012).

A central geometric object is the modular lower bound $Z=\sum_{T\subseteq V}\exp(-F(T))$1, together with the polyhedra

$Z=\sum_{T\subseteq V}\exp(-F(T))$2

and

$Z=\sum_{T\subseteq V}\exp(-F(T))$3

the base polytope (Djolonga et al., 2015). These objects underlie both variational inference and optimization-based learning.

2. L-FIELD and variational inference

A major development for approximate Bayesian inference in this class is L-FIELD. If $Z=\sum_{T\subseteq V}\exp(-F(T))$4, then $Z=\sum_{T\subseteq V}\exp(-F(T))$5 for all $Z=\sum_{T\subseteq V}\exp(-F(T))$6, which yields

$Z=\sum_{T\subseteq V}\exp(-F(T))$7

Therefore

$Z=\sum_{T\subseteq V}\exp(-F(T))$8

and the variational problem is to minimize $Z=\sum_{T\subseteq V}\exp(-F(T))$9 over $V$0 (Djolonga et al., 2015). Because the objective is separable while the feasible set is the base polytope, the program converts a globally intractable partition-function computation into a convex optimization problem over a combinatorial polytope.

The striking structural result is that the unique minimizer of $V$1 over $V$2 is also the minimum-norm point of $V$3, equivalently the solution of

$V$4

This equivalence reduces L-FIELD to the classical minimum-norm-point convex program and makes it possible to use standard submodular-polytope oracles and minimum-norm-point algorithms (Djolonga et al., 2015). By Fujishige’s theorem, thresholding the minimum-norm solution $V$5 at $V$6, or equivalently thresholding marginals at $V$7, yields the exact MAP minimizers.

The same program has an information-theoretic interpretation. Let $V$8 be the fully factorized Gibbs distribution with modular $V$9,

$|V|$0

The infinite-order Rényi divergence is

$|V|$1

Using

$|V|$2

and restricting to $|V|$3, one obtains exactly the L-FIELD objective since $|V|$4 and $|V|$5 (Djolonga et al., 2015). Thus L-FIELD finds the factorized $|V|$6 minimizing $|V|$7. This characterization explains why the approximation is conservative in a worst-case multiplicative sense rather than calibrated to the KL-type objectives more commonly associated with mean field.

3. Parallel inference as message passing

When the energy decomposes as $|V|$8, inference can be implemented by parallel message passing on a factor graph $|V|$9, with edges $X_i\in\{0,1\}$0 whenever $X_i\in\{0,1\}$1 (Djolonga et al., 2015). For $X_i\in\{0,1\}$2, the analysis introduces the norms

$X_i\in\{0,1\}$3

where $X_i\in\{0,1\}$4 denotes the neighboring factors of $X_i\in\{0,1\}$5.

The algorithm maintains, for each factor $X_i\in\{0,1\}$6, a vector $X_i\in\{0,1\}$7, together with real-valued messages $X_i\in\{0,1\}$8 and $X_i\in\{0,1\}$9. The parallel variable-to-factor update is

$S\subseteq V$0

so each variable averages incoming factor-to-variable messages. The factor update gathers $S\subseteq V$1 and solves

$S\subseteq V$2

then sends $S\subseteq V$3. A global iterate can be recovered as

$S\subseteq V$4

This formulation avoids summing over all factor configurations, which is the major obstacle for many high-order methods. The convergence guarantee is explicit: if every variable has exactly $S\subseteq V$5 incident factors, then

$S\subseteq V$6

which is a global linear, i.e. geometric, rate (Djolonga et al., 2015). A plausible implication is that the factor-graph decomposition is not merely an implementation convenience; it supplies the regularity conditions under which scalable parallel inference becomes analyzable.

4. Bethe approximation, belief propagation, and partition-function bounds

For a factor graph $S\subseteq V$7 with unary potentials $S\subseteq V$8 and higher-order potentials $S\subseteq V$9, the true partition function is

$F:2^V\to\mathbb{R}$0

The Bethe approximation defines a variational estimate of $F:2^V\to\mathbb{R}$1 over pseudomarginals $F:2^V\to\mathbb{R}$2 that satisfy normalization and marginal-consistency constraints, and the Bethe free energy is

$F:2^V\to\mathbb{R}$3

where $F:2^V\to\mathbb{R}$4 is the Shannon entropy (Ruozzi, 2012). A stationary point $F:2^V\to\mathbb{R}$5 of $F:2^V\to\mathbb{R}$6 in the interior of the local polytope corresponds exactly to a fixed point of loopy belief propagation.

For arbitrary graphical models, the Bethe estimate $F:2^V\to\mathbb{R}$7 has no universal lower- or upper-bound guarantee. In binary log-supermodular models, however, the situation is different: every BP fixed point satisfies

$F:2^V\to\mathbb{R}$8

This theorem applies to any factor graph over binary variables whose every factor is log-supermodular, not merely to pairwise attractive models (Ruozzi, 2012).

The proof uses graph covers and a specialized correlation inequality. Vontobel’s theorem expresses the Bethe partition function through true partition functions on $F:2^V\to\mathbb{R}$9-covers, and it therefore suffices to show that for every $A\subseteq B\subseteq V$0-cover $A\subseteq B\subseteq V$1,

$A\subseteq B\subseteq V$2

Ruozzi’s argument establishes this using a new variant of the “four functions” theorem adapted to log-supermodular functions (Ruozzi, 2012). The result resolves, in the affirmative, a conjecture that the Bethe approximation corresponding to any fixed point of belief propagation over an attractive, pairwise binary graphical model provides a lower bound on the true partition function, and simultaneously extends the guarantee to higher-order log-supermodular factors.

Several limitations are explicit. The variables must be binary, the proof relies heavily on Boolean lattice structure, and no upper bound is claimed. In the non-supermodular case, such as antiferromagnetic models, the same technique does not apply (Ruozzi, 2012). This directly corrects a common misconception that “attractive” alone is sufficient for broad Bethe guarantees without domain restrictions.

5. Parameterization and learning

A standard exponential-family parameterization writes

$A\subseteq B\subseteq V$3

where $A\subseteq B\subseteq V$4 is the log-partition function (Shpakova et al., 2016). One parameterization of the submodular energy is

$A\subseteq B\subseteq V$5

with $A\subseteq B\subseteq V$6, $A\subseteq B\subseteq V$7, and each $A\subseteq B\subseteq V$8 submodular. Exact evaluation of $A\subseteq B\subseteq V$9 is $S\subseteq V$00-hard, so learning relies on tractable upper bounds.

Two upper bounds are identified. The separable L-field bound is

$S\subseteq V$01

obtained from the Lovász-extension representation $S\subseteq V$02 (Shpakova et al., 2016). The perturb-and-MAP or logistic bound is

$S\subseteq V$03

where $S\subseteq V$04 has i.i.d. logistic entries, derived from replacing a collection of $S\subseteq V$05 Gumbel perturbations by a factored collection of $S\subseteq V$06 pairs of i.i.d. Gumbels (Shpakova et al., 2016). The dominance relation is

$S\subseteq V$07

Accordingly, the bound based on separable optimization on the base polytope is always inferior to a bound based on perturb-and-MAP ideas.

Given $S\subseteq V$08 i.i.d. observations, replacing $S\subseteq V$09 by $S\subseteq V$10 yields a convex surrogate for negative log-likelihood, and parameter learning proceeds by projected stochastic subgradient descent. If

$S\subseteq V$11

and $S\subseteq V$12 is any maximizer, then the subgradients are

$S\subseteq V$13

plus any $S\subseteq V$14-terms from a convex regularizer $S\subseteq V$15 (Shpakova et al., 2016). The algorithm initializes $S\subseteq V$16, $S\subseteq V$17, samples $S\subseteq V$18, solves the inner maximization by submodular minimization, and uses step sizes $S\subseteq V$19. Because the surrogate objective is convex and Lipschitz continuous, the averaged iterate converges to the global minimizer at $S\subseteq V$20 in function value.

The same perturb-and-MAP surrogate extends to conditional maximum likelihood. For noisy observations $S\subseteq V$21 of latent clean images $S\subseteq V$22, the posterior remains log-supermodular with energy

$S\subseteq V$23

and one may optimize supervised, partially observed, or fully unsupervised objectives by stochastic subgradient descent of the corresponding approximate likelihoods (Shpakova et al., 2016). This suggests that tractable MAP subroutines, rather than tractable marginalization, are the decisive computational primitive for learning in this family.

6. Extensions and adjacent model classes

The partition-function lower-bound theory motivates a broader set of constructions in which the original model is not itself binary or log-supermodular, but can be reformulated as a log-supermodular model on an expanded domain. One example is the ferromagnetic Potts model with a uniform external field. Using the random-cluster representation,

$S\subseteq V$24

and since $S\subseteq V$25 is supermodular in the indicator-vector of $S\subseteq V$26, $S\subseteq V$27 is log-supermodular in that vector (Ruozzi, 2013). The same cover-based argument then yields

$S\subseteq V$28

for the uniform-field Potts model.

A second example is a special class of weighted graph homomorphism problems. For

$S\subseteq V$29

if

$S\subseteq V$30

then the model can be rewritten as an edge-coloring model whose local factors are log-supermodular in the indicator-vector of an edge subset $S\subseteq V$31. This again implies

$S\subseteq V$32

whenever $S\subseteq V$33 (Ruozzi, 2013).

These constructions do not redefine such models as log-supermodular graphical models in their native parameterization. Rather, they show that log-supermodularity functions as a transferable proof device: if an auxiliary representation is log-supermodular, then cover-based partition-function bounds can persist beyond the original binary setting.

7. Empirical behavior and application domains

The most detailed application in scalable variational inference is image segmentation. The reported dataset comprises 36 natural images with pixel-level ground truth, with image sizes up to $S\subseteq V$34 pixels (Djolonga et al., 2015). The model is

$S\subseteq V$35

The comparisons include unary only (independent), Pairwise BP, Mean-Field, Fractional BP (libDAI), L-FIELD/Douglas–Rachford with pairwise only, and L-FIELD with higher-order potentials. Reported runtimes are DR (pairwise) $S\subseteq V$36s, HOP $S\subseteq V$37s, and BP/MF $S\subseteq V$38s–$S\subseteq V$39min. Accuracy, measured as AUC over the full image and over trimaps near boundaries, is reported as

$S\subseteq V$40

Qualitatively, BP and FBP tend to be over-confident, L-FIELD yields more conservative marginals, and incorporating higher-order potentials yields sharper boundary marginals and higher AUC near edges (Djolonga et al., 2015).

A distinct learning-focused experiment concerns binary image denoising on binary horse silhouettes of size $S\subseteq V$41 pixels, with $S\subseteq V$42 and $S\subseteq V$43, and independent pixel flips with probabilities $S\subseteq V$44 (Shpakova et al., 2016). The model uses two submodular cut functions, corresponding to horizontal and vertical 4-neighbor boundary costs, plus a per-pixel modular bias $S\subseteq V$45. In the supervised setting, the reported mean-marginals error is approximately $S\subseteq V$46 at $S\subseteq V$47, versus $S\subseteq V$48 for a structured-SVM baseline; at $S\subseteq V$49, approximately $S\subseteq V$50 versus $S\subseteq V$51. In the unsupervised setting, when only the noisy image is observed but $S\subseteq V$52 is known, performance degrades only slightly: at $S\subseteq V$53, mean-marginals error is approximately $S\subseteq V$54, versus $S\subseteq V$55 in the supervised case. When $S\subseteq V$56 is also learned, errors rise, for example to approximately $S\subseteq V$57 at true $S\subseteq V$58, but the model still outperforms a trivial “no-change” decoder for moderate noise levels (Shpakova et al., 2016).

Taken together, these results show that log-supermodular graphical models support two complementary computational regimes. For inference, high-order attractive structure can be handled by base-polytope optimization and parallel message passing without explicit enumeration of factor states. For learning, tractable MAP subproblems combined with perturb-and-MAP surrogates enable approximate maximum likelihood even with missing or latent data.