Reconsidering Dependency Networks from an Information Geometry Perspective

Abstract: Dependency networks (Heckerman et al., 2000) provide a flexible framework for modeling complex systems with many variables by combining independently learned local conditional distributions through pseudo-Gibbs sampling. Despite their computational advantages over Bayesian and Markov networks, the theoretical foundations of dependency networks remain incomplete, primarily because their model distributions -- defined as stationary distributions of pseudo-Gibbs sampling -- lack closed-form expressions. This paper develops an information-geometric analysis of pseudo-Gibbs sampling, interpreting each sampling step as an m-projection onto a full conditional manifold. Building on this interpretation, we introduce the full conditional divergence and derive an upper bound that characterizes the location of the stationary distribution in the space of probability distributions. We then reformulate both structure and parameter learning as optimization problems that decompose into independent subproblems for each node, and prove that the learned model distribution converges to the true underlying distribution as the number of training samples grows to infinity. Experiments confirm that the proposed upper bound is tight in practice.

• The paper introduces an information-geometric framework that interprets pseudo-Gibbs sampling as m-projections onto conditional manifolds.
• It presents novel full conditional divergence measures and decomposable cost functions for principled dependency network learning.
• The convergence analysis proves that with sufficient data, the DN's stationary distribution converges to the true joint distribution.

Information-Geometric Analysis of Dependency Networks

Overview

The paper "Reconsidering Dependency Networks from an Information Geometry Perspective" (2604.01117) delivers a comprehensive theoretical framework for understanding dependency networks (DNs) through the lens of information geometry. The core contribution is to rigorously connect the operation of DNs—specifically, pseudo-Gibbs sampling—to m-projections onto full conditional manifolds. The resulting analysis yields novel divergence measures, new cost functions amenable to decomposable learning, and asymptotic convergence results. Together, these insights bridge the gap left by the lack of closed-form joint distributions in DNs and enable principled learning and inference within this flexible family of graphical models.

Figure 1: Schematic of a node in a dependency network, containing a variable, information source, and a conditional distribution table.

Dependency Networks and Pseudo-Gibbs Sampling

Dependency networks [Heckerman et al 2000] model collections of conditional distributions by fitting, at each node, a regression (conditional probability table) of the variable given its Markov blanket (information source). Unlike Bayesian or Markov networks, DNs do not necessarily guarantee global consistency, i.e., their set of conditionals may not correspond to any joint distribution. Instead, the stationary distribution of the associated pseudo-Gibbs sampler is used as the model.

The crucial obstacle for theoretical analysis is that this stationary distribution has no closed form, rendering standard maximum likelihood or direct divergence-based learning infeasible.

Information Geometry of Pseudo-Gibbs Sampling

The paper's key insight is an information-geometric view of DN learning and sampling. Each update in pseudo-Gibbs sampling is interpreted as an m-projection (minimizing KL divergence) onto the manifold of distributions with a fixed conditional (for the currently updated node). The set of such manifolds for all nodes need not intersect, and the pseudo-Gibbs sampler cycles m-projections through these manifolds in sequence (sequential scan) or at random (random scan).

Figure 2: The m-projection of a point onto an e-flat manifold representing fixed conditionals.

For genuine Gibbs sampling, these manifolds intersect at a unique point that is the stationary distribution—recovering the classical case where global consistency allows reconstruction of the joint. In the general DN case, the orbits of $p^t$ under the Markov chain are confined to a neighborhood defined by the divergence between the current point and the closest point(s) on the conditional manifolds.

Figure 3: Evolution of $p^t$ under sequential scan in genuine Gibbs sampling, where conditional manifolds overlap.

Figure 4: Evolution of $p^t$ under sequential scan in pseudo-Gibbs sampling, where manifolds do not necessarily intersect.

Full Conditional Divergence and the FC-limit

A central technical accomplishment is the introduction of full conditional divergence (FC), which measures the average KL divergence from a candidate distribution to the specified full conditional manifolds (weighted by scan probabilities):

$FC(p\|q) = \sum_i c_i\, KL(p\|E_i(q))$

This divergence underpins a tight upper bound on the divergence between any target distribution $p$ and the stationary distribution $\pi$ of the pseudo-Gibbs sampler ("FC-limit"). Explicitly,

$FC(p\|\pi) \le \sum_i c_i KL(p\|E(\theta_i))$

This inequality supplies a rigorous justification for decomposable learning criteria: minimizing the set of per-node conditional KL divergences with respect to the empirical data leads the stationary distribution of the DN toward the true joint as the conditionals become accurate. This not only formalizes the ad hoc hypothesis favoring accurate conditionals but also demonstrates that dependency network learning can piggyback on the tractability of conditional estimation.

The FC divergence is shown to be a Bregman divergence induced by a sum of conditional entropies, which situates it within a broader geometric framework shared by KL-based projections and optimization in exponential families.

Structure and Parameter Learning: Decomposability and Consistency

The information-geometric results allow the authors to reformulate both parameter and structure learning in DNs as local, node-wise optimization problems. Each node seeks to minimize conditional entropy of its variable given a candidate information source (potentially regularized by an MDL penalty), subject to the overall FC-limit constraint. This yields the cost function:

$$cost = \sum_i c_i [KL(p^D\|E(\theta_i)) + R(k_i,N) ]$$

where $p^D$ is the empirical distribution. Parameter optimization at each node is closed-form (set conditionals to their empirical distributions), and structure learning may be efficiently performed with local splitting and merging heuristics akin to those in decision trees.

The key theoretical result is that, under minimal regularity conditions and with a sufficiently expressive structure search, the stationary distribution of the learned DN converges almost surely to the true joint distribution as the sample size grows. This consistency theorem closes the theoretical gap left by earlier works which lacked non-asymptotic generalization guarantees or could not rigorously handle the inconsistency of the global distribution.

Conditional Pseudo-Gibbs Sampling for Inference

For inference in DNs (e.g., answering conditional queries), the authors analyze conditional pseudo-Gibbs sampling. Clamping certain variables and running the sampler on the remainder results in (asymptotically) correct conditionals if the learned full conditionals are accurate for all values in question. The paper additionally quantifies when inference for rare events may be unreliable, since the bound on FC divergence is weighted by the empirical frequency of clamped configurations.

A notable property is the absence of a universal joint over all variables: the stationary distribution constructed for each conditional query is query-specific, implying that the approach is not strictly coherent across all possible inference tasks. This feature must be considered in applications demanding joint or marginal probabilities over many clamped sets.

Comparison to Prior Work and Implications

The proposed information-geometric tools generalize and refine prior treatments of DNs and pseudo-likelihood, connecting to the literature on m-projections in exponential families [Amari 1995] and providing theoretical scaffolding for earlier empirical observations. The analysis links DNs to decision tree algorithms in terms of structure learning objectives, clarifying the computational advantages while making explicit the conditions under which these advantages do not undermine statistical consistency.

From a practical standpoint, this work supplies well-justified algorithms for large-scale dependency modeling where other graphical models are intractable, especially in high-dimensional discrete domains. Theoretically, the FC-limit bound and the convergence proof invite extensions to continuous variables, alternative divergence measures, and richer structure search spaces.

Conclusion

This paper offers a coherent information-geometric framework for dependency networks, resolving the non-constructive aspects of their joint modeling by connecting pseudo-Gibbs sampling to iterative m-projections and introducing the full conditional divergence as an actionable criterion for learning and evaluation. The results guarantee that—with appropriate local learning and sufficient data—the stationary distribution of a dependency network model will converge to the true data-generating distribution. These advances should facilitate broader and more robust use of dependency networks in large-scale probabilistic modeling and suggest further lines of inquiry in divergence-based learning under partial or inconsistent constraints.