Probabilistic Dependency Graphs

• Probabilistic Dependency Graphs are graphical models that encode local probabilistic dependencies with attached confidence parameters, allowing overlapping and even conflicting beliefs.
• They extend traditional Bayesian networks and factor graphs by optimizing a joint distribution to minimize inconsistency via measures like relative entropy and information deficiency.
• PDGs enable tractable inference through convex optimization methods under bounded treewidth and inspire algorithmic approaches such as Local Inconsistency Resolution.

Probabilistic Dependency Graphs (PDGs) are graphical formalisms for representing probabilistic information as local dependency statements while allowing those statements to be overlapping, redundant, or mutually inconsistent. In the formulation introduced as a new class of directed graphical models, a PDG attaches conditional probability distributions and confidence parameters to edges or hyperarcs, evaluates candidate joint distributions by how well they fit those local beliefs, and thereby extends the semantics of Bayesian networks while also faithfully representing factor graphs and their exponential families (Richardson et al., 2020). Later work develops inconsistency as the central PDG quantity, showing that many standard objectives in machine learning and statistics arise as special cases, and provides a tractable inference procedure for discrete PDGs of bounded treewidth via convex optimization with exponential cone constraints (Richardson, 2022, Richardson et al., 2023).

1. Formal structure and semantics

In the core PDG formalism, a probabilistic dependency graph may be written as

$$M = (\mathcal{V}, \mathcal{D}, \mathcal{E}, \mathbf{p}, \boldsymbol\alpha, \boldsymbol\beta),$$

where $\mathcal{V}$ is a set of variables, $\mathcal{D}(X)$ is the domain of each variable $X$, $\mathcal{E}$ is a set of labeled directed edges, $\mathbf p$ assigns to each edge $L:X\to Y$ a conditional probability distribution $p_L(Y\mid X)$, $\alpha_L$ is confidence in the structural dependence of $Y$ on $\mathcal{V}$0, and $\mathcal{V}$1 is confidence in the numerical correctness of $\mathcal{V}$2. The formalism also allows edges whose source or target is a set of variables, so that joint factors $\mathcal{V}$3 and unconditional distributions can be represented directly. Multiple edges into the same node are allowed, and these edges may encode incompatible local beliefs (Richardson, 2022).

The original semantic move is to separate exact compatibility from graded incompatibility. A set-based semantics $\mathcal{V}$4 collects those joint distributions $\mathcal{V}$5 that satisfy every edge cpd exactly; this semantics is adequate for consistent models but collapses all inconsistent PDGs to the empty set. The scoring semantics instead evaluates any candidate joint distribution $\mathcal{V}$6 by two functionals. The first is incompatibility,

$\mathcal{V}$7

where $\mathcal{V}$8 is relative entropy. The second is information deficiency,

$\mathcal{V}$9

These combine into

$\mathcal{D}(X)$0

The inconsistency of a PDG is the minimum achievable incompatibility,

$\mathcal{D}(X)$1

and later work also considers the $\mathcal{D}(X)$2-inconsistency $\mathcal{D}(X)$3. A basic structural property is monotonicity: adding edges or increasing confidences cannot decrease inconsistency (Richardson et al., 2020, Richardson, 2022).

2. Inconsistency as a universal objective

A central claim of the modern PDG literature is that many familiar losses arise not as arbitrary design choices but as inconsistency scores of natural PDGs. For a single distribution $\mathcal{D}(X)$4 together with an infinitely confident event $\mathcal{D}(X)$5, the inconsistency is

$\mathcal{D}(X)$6

namely surprisal or negative log-likelihood. For a dataset with empirical distribution $\mathcal{D}(X)$7, the corresponding PDG yields

$\mathcal{D}(X)$8

so average NLL appears up to an additive constant. In supervised learning with empirical joint $\mathcal{D}(X)$9 and predictor $X$0, the inconsistency becomes

$X$1

which is cross-entropy loss up to the data-dependent constant $X$2. For deterministic classifiers $X$3 and input distribution $X$4, a PDG with deterministic edges for $X$5 and $X$6 yields

$X$7

so log accuracy appears as inconsistency. For Gaussian regressors, the inconsistency reduces to

$X$8

that is, mean squared error (Richardson, 2022).

The same construction recovers regularization from priors. With data distribution $X$9, likelihood $\mathcal{E}$0, prior $\mathcal{E}$1, and a point observation $\mathcal{E}$2, the PDG inconsistency is

$\mathcal{E}$3

If $\mathcal{E}$4, the second term yields L2 regularization; if $\mathcal{E}$5, it yields L1 regularization. More generally, choosing a Gibbs prior $\mathcal{E}$6 produces regularizer $\mathcal{E}$7, so regularization strength equals prior confidence.

A large class of divergences also appears as PDG inconsistency. With two distributions $\mathcal{E}$8 and $\mathcal{E}$9, if $\mathbf p$0 is enforced with infinite confidence and $\mathbf p$1 has confidence $\mathbf p$2, then

$\mathbf p$3

so forward KL is a PDG inconsistency. For two finite confidences $\mathbf p$4, the PDG divergence is

$\mathbf p$5

which is a scaled Rényi divergence with $\mathbf p$6. Chernoff divergence appears as the smallest possible PDG inconsistency when total confidence is fixed at $\mathbf p$7 but split arbitrarily between $\mathbf p$8 and $\mathbf p$9.

Variational objectives arise in the same way. For latent-variable models $L:X\to Y$0, observation $L:X\to Y$1, and variational distribution $L:X\to Y$2, the PDG inconsistency is exactly negative ELBO:

$L:X\to Y$3

The standard ELBO bound follows from monotonicity: adding the extra belief $L:X\to Y$4 can only increase inconsistency. For VAEs, the PDG containing prior $L:X\to Y$5, decoder $L:X\to Y$6, encoder $L:X\to Y$7, and observation $L:X\to Y$8 yields

$L:X\to Y$9

and a $p_L(Y\mid X)$0-VAE is obtained by simply giving the prior confidence $p_L(Y\mid X)$1. In the factor-graph setting, if a weighted factor graph $p_L(Y\mid X)$2 is embedded as a PDG $p_L(Y\mid X)$3, then the minimizer of $p_L(Y\mid X)$4 is the factor-graph distribution $p_L(Y\mid X)$5, and the $p_L(Y\mid X)$6-inconsistency is

$p_L(Y\mid X)$7

This identifies PDG free energy with minus the log partition function (Richardson, 2022).

3. Relation to other graphical-model traditions

PDGs were designed to subsume Bayesian networks and factor graphs while relaxing their consistency requirements. A Bayesian network assigns one cpd $p_L(Y\mid X)$8 per node in a DAG and thereby defines a single joint distribution. A PDG may instead have several edges into the same variable, such as $p_L(Y\mid X)$9 and $\alpha_L$0, even when these local beliefs are mutually incompatible. Factor graphs can be embedded into PDGs by turning each factor into an unconditional edge into the corresponding variable subset and setting confidence weights appropriately; conversely, for fixed $\alpha_L$1 satisfying $\alpha_L$2, PDG scoring reduces to weighted-factor-graph free energy, but general PDGs with arbitrary $\alpha_L$3 or $\alpha_L$4 semantics are not captured by factor graphs without distortion (Richardson et al., 2020).

Formalism	Primitive local object	Global semantics
PDG	Edge or hyperarc cpd with $\alpha_L$5	Minimizer of $\alpha_L$6
Bayesian network	Node cpd on a DAG	Product factorization
Dependency network	Node-wise conditional $\alpha_L$7 on a possibly cyclic digraph	Stationary distribution of Gibbs or pseudo-Gibbs sampling
Weighted dependency graph	Edge weights in $\alpha_L$8 controlling cumulant bounds	Asymptotic-normality machinery

Earlier dependency-network work is closely related but semantically distinct. A dependency network is a pair $\alpha_L$9 with a possibly cyclic directed graph $Y$0 and one local conditional $Y$1 per node. It does not define its joint distribution by a product formula; instead, an ordered Gibbs sampler over the local conditionals induces a stationary joint distribution. If the local conditionals are consistent with a positive joint $Y$2, then the stationary distribution is exactly $Y$3. Later work reinterprets pseudo-Gibbs sampling information-geometrically as iterative m-projections onto full-conditional manifolds, and introduces a full-conditional divergence that bounds how far the stationary distribution can lie from a target distribution (Heckerman et al., 2013, Takabatake et al., 2021).

The phrase has also been used in more domain-specific ways. In transportation, a “probabilistic dependency network” is a sparse, temporally ordered Bayesian-type network with GLM local conditionals learned via lasso regression and used for prediction and diagnostics; temporally earlier events serve as candidate parents, guaranteeing a DAG (Edakunni et al., 2015). In asymptotic probability theory, “weighted dependency graphs” encode quantitative dependence through cumulant bounds

$Y$4

where $Y$5 is the maximum spanning-tree weight of the induced weighted graph; this framework supports central-limit theorems for combinatorial and stochastic-process models (Féray, 2016). A separate cyclic directed proposal, the probabilistic relation network, assigns probability directly to structured outcomes $Y$6 that are themselves graphs, so directed cycles are handled by a probability distribution over outcome-graphs rather than by local-recursive factorization (Sirotkin, 2023). This suggests a broader umbrella usage of “probabilistic dependency graph” across several related traditions, even though the Richardson-style PDG has a more specific optimization-based semantics.

4. Inference and computational properties

For discrete PDGs, tractable inference is based on an explicit convex-optimization view of semantics. Writing a PDG as

$Y$7

one defines observational incompatibility

$Y$8

structural incompatibility

$Y$9

and total score

$\mathcal{V}$00

For proper PDGs and $\mathcal{V}$01, $\mathcal{V}$02 is strictly convex and has a unique minimizer; the empirical-limit semantics $\mathcal{V}$03 is also unique for proper PDGs (Richardson et al., 2023).

The key computational result is that minimizing these scores can be formulated as convex optimization with exponential cone constraints. The $\mathcal{V}$04 problem minimizes a sum of KL divergences, and the small-$\mathcal{V}$05 problem adds entropy terms using the alternate expression

$\mathcal{V}$06

The $\mathcal{V}$07 semantics is obtained in two stages: first minimize $\mathcal{V}$08, then among all minimizers choose the one minimizing $\mathcal{V}$09. Interior-point methods for exponential-cone programs then provide polynomial-time algorithms in the size of the conic formulation.

The exponential dependence on the full joint state space is removed, for bounded treewidth, by a clique-tree construction. A Markov property for PDGs states that if a PDG is decomposed into two sub-PDGs over variable sets $\mathcal{V}$10, then every optimal distribution satisfies

$\mathcal{V}$11

Hence every optimizer can be represented by a calibrated tree marginal over a tree decomposition $\mathcal{V}$12, and joint entropy can be rewritten in cluster form via

$\mathcal{V}$13

This yields cluster-based exponential-cone programs whose size is polynomial in the number of clusters and exponential only in treewidth. The main complexity theorem gives runtime

$\mathcal{V}$14

for approximating the unique calibrated tree marginal of a proper PDG with $\mathcal{V}$15 variables, $\mathcal{V}$16 arcs, maximum domain size $\mathcal{V}$17, treewidth $\mathcal{V}$18, and parameter bit-length $\mathcal{V}$19. At the same time, general approximate PDG inference remains $\mathcal{V}$20P-hard, and approximate inconsistency calculation is likewise $\mathcal{V}$21P-hard, reflecting the fact that PDGs inherit the hardness of the graphical models they generalize (Richardson et al., 2023).

5. Algorithms and modern reinterpretations built on PDGs

Recent work treats PDGs not only as representational objects but also as algorithmic substrates. Local Inconsistency Resolution (LIR) defines a generic iterative procedure: choose an attention mask $\mathcal{V}$22 specifying which arcs and structural terms matter, choose a control mask $\mathcal{V}$23 specifying which parameters may change, and then follow the gradient flow of the focused inconsistency $\mathcal{V}$24. By appropriate choices of refocusing, LIR recovers EM, belief propagation, adversarial training, GANs, and GFlowNets. In the GFlowNet case, the derivation leads to a length-normalized trajectory-balance objective,

$\mathcal{V}$25

and experiments on HyperGrid environments show improved convergence speed or stability relative to unnormalized TB and LPV variants (Richardson et al., 18 Apr 2026).

In natural-language processing, the “Probabilistic Transformer” is a conditional random field over latent label variables $\mathcal{V}$26 and head-selection variables $\mathcal{V}$27 for each word and attention channel. The model does not enforce tree constraints; the dependency structure is a general directed graph, possibly with cycles and multiple heads. Mean-field variational inference uses updates of the form

$\mathcal{V}$28

and the resulting computation graph closely matches transformer self-attention, with attention weights interpreted as posterior probabilities of dependency arcs and contextual embeddings as approximate posteriors over latent states (Wu et al., 2023).

In visual reasoning, EVPG reconstructs the execution of a non-differentiable visual program as exact probability inference on a directed probabilistic graph. LOC, VQA, and EVAL modules become nodes with probabilistic or deterministic dependencies, and final answer probabilities are obtained by marginalizing over latent module outputs. For example, a sub-answer distribution is computed by

$\mathcal{V}$29

which replaces hard module calls by differentiable probabilistic inference. This permits end-to-end supervised learning from only final labels on complex visual-reasoning tasks (Wan et al., 16 Dec 2025).

6. Conceptual implications, misconceptions, and open directions

The most compact statement of the PDG viewpoint is the slogan “Choose your model, not your loss function.” In this view, losses, divergences, ELBOs, regularizers, and free energies are all instances of PDG inconsistency induced by different collections of local probabilistic beliefs. The loss value measures epistemic conflict between trusted beliefs and the current joint distribution; regularization strength is prior confidence; and inequalities between divergences can often be derived visually from monotonicity under adding edges or increasing confidences (Richardson, 2022).

Several recurrent misconceptions are corrected by the literature. A PDG is not merely a Bayesian network with extra edges, because its local pieces need not be jointly satisfiable and its semantics are optimization-based rather than purely factorized. It is also not fully subsumed by factor graphs: for a fixed $\mathcal{V}$30 and matched $\mathcal{V}$31 ratios, a PDG may coincide with weighted-factor-graph free energy, but general PDGs with arbitrary confidence structure or $\mathcal{V}$32 semantics cannot be reproduced without changing the meaning of repeated or conflicting information. Inconsistency is therefore not a pathology to be eliminated before modeling; it is a representable and measurable property of the model itself. Cycles are likewise model-dependent: dependency-network and structured-outcome traditions accept cyclic directed graphs directly, while the Richardson-style PDG handles conflict and dependence through global optimization over joint distributions rather than through local-recursive factorization (Richardson et al., 2020, Heckerman et al., 2013, Sirotkin, 2023).

Open problems remain substantial. Tractable inference currently depends on bounded treewidth and discrete state spaces, and the best current guarantees scale exponentially in treewidth. Continuous or hybrid PDGs are not yet part of the tractable theory. For parameter regimes in which structural terms dominate and $\mathcal{V}$33, the objective becomes nonconvex and current methods rely on heuristic extensions such as convex-concave procedures. On the algorithmic side, LIR suggests that adaptive or learned refocusing policies could become a general theory of attention over probabilistic models, but this remains largely undeveloped. A plausible implication is that future PDG research will divide between stronger exact theory for restricted classes and scalable approximate procedures that preserve the explicit semantics of inconsistency while borrowing optimization technology from variational inference, message passing, and modern deep learning (Richardson et al., 2023, Richardson et al., 18 Apr 2026).