Probabilistic Semantic Scene Graphs

• Probabilistic Semantic Scene Graphs are graphical structures that attach explicit probability distributions to nodes and edges, enabling uncertainty quantification and semantic ambiguity handling.
• They support a range of applications including generative modeling, long-tail debiasing, and structured reasoning through methods like Bayesian inference and autoregressive flows.
• PSSGs facilitate efficient semantic compression and enhanced spatial commonsense interpretation, yielding empirical improvements in recall metrics and prediction diversity.

Probabilistic Semantic Scene Graphs (PSSG) generalize conventional scene graph representations by assigning explicit probability distributions to graph entities and relations, supporting uncertainty quantification, generative modeling, debiasing, semantics-driven compression, and structured reasoning. This probabilistic formalism has emerged as a unifying framework across discriminative, generative, and communication settings, principally via the introduction of learned or structured distributions over nodes, predicates, and geometric attributes. The resulting graphs permit posterior inference, stochastic sampling, uncertainty-aware predictions, and principled handling of semantic ambiguity in real-world images.

1. Foundational Formalisms and Definitions

A Probabilistic Semantic Scene Graph is any scene graph structure in which at least some nodes or edges are endowed with probabilistic semantics—either distributions over labels, spatial locations, co-occurrences, or higher-order properties. Representative instantiations include:

• Graph-based joint probability distributions: Each triplet $(h, r, t)$ in the scene graph has associated $p_{\text{rel}}(r|h, t)$ (predicate conditional) and $p_{\text{co}}(t|h)$ (co-occurrence) distributions, constructed via normalized empirical counts over training data, i.e.,

$$p(r | h, t) = \frac{\mathrm{Count}(h, r, t)}{\sum_{r'} \mathrm{Count}(h, r', t)} \qquad p(t | h) = \frac{\mathrm{Count}(h, t)}{\sum_{t'} \mathrm{Count}(h, t')}$$

(Zhu et al., 16 Jul 2025).

• Probabilistic node and edge marginals: In grammar-based models, node existence and edge instantiation are cast as binary random variables in a factor graph, with message-passing giving posterior probabilities for each entity's presence and relationship (Chua et al., 2016).
• Hybrid continuous-discrete flows: Recent scene graph generation methods assign distributions not only to categorical slots (object categories, predicates) but also to geometric attributes (bounding box states), and cast the full scene graph as a time-evolving probability state under ODE/CTMC dynamics (Hu et al., 18 Apr 2026).
• Spatial composition distributions: Given a set of observed and latent (“blind”) objects in a 3D scene, PSSGs can assign, for every semantic class $c_i$, a full $S\times S$ heatmap $P(c_i \ \text{at}\ (u,v))$ representing the predicted spatial occupancy field (Saucedo et al., 5 May 2025).

This general formalism enables both plug-in uncertainty modeling in discriminative pipelines and explicit generative modeling for scene graph synthesis.

2. Discriminative PSSG Modules: Uncertainty and Ambiguity

One principal application of PSSG methods is quantifying the uncertainty and semantic ambiguity intrinsic to visual relationship detection. The Probabilistic Uncertainty Modeling (PUM) paradigm, introduced in the context of scene graph generation, replaces deterministic feature representations with learned Gaussian distributions for union regions: $$f_{ij} \sim \mathcal N(\boldsymbol\mu_{ij}, \boldsymbol\Sigma_{ij})$$ with diagonal covariance. The stochastic embedding $z_{ij}$ is produced by the reparameterization trick: $$z_{ij} = \boldsymbol\mu_{ij} + \boldsymbol\sigma_{ij} \odot \epsilon, \quad \epsilon \sim \mathcal N(0, I)$$ and propagated to a softmax head for relationship classification. Losses combine cross-entropy over both $p_{\text{rel}}(r|h, t)$0 and sampled $p_{\text{rel}}(r|h, t)$1, together with an entropy regularizer to prevent variance collapse: $p_{\text{rel}}(r|h, t)$2 where $p_{\text{rel}}(r|h, t)$3 (Yang et al., 2021).

This approach addresses:

• Synonymy, hyponymy, and multiview ambiguity: Distributional variance covers overlapping or hierarchical labels and supports diverse predictions.
• Long-tail debiasing: The ability to sample from broader regions improves coverage of rare predicates.

Integration with GCN-based backbones (e.g., ResCAGCN) is straightforward, and ablation experiments show performance gains in mean-recall metrics, with saturation around 8–16 stochastic samples per prediction (Yang et al., 2021).

3. Generative, Graphical, and Bayesian PSSG Architectures

A broader class of PSSG approaches target generative or inference-centric tasks. Notable models include:

• SceneGraphGen (autoregressive generative model): The joint probability $p_{\text{rel}}(r|h, t)$4 of a scene graph $p_{\text{rel}}(r|h, t)$5 is factorized autoregressively over object and edge sequences, implemented with hierarchically nested GRUs and MLPs, trained via maximum likelihood over sequences comprising node and edge generation steps. Diversity and semantic fidelity are evaluated via Maximum-Mean-Discrepancy kernels over generated scene graphs; samples from the model are successfully used for story completion and anomaly detection tasks (Garg et al., 2021).
• Bayesian debiasing layer: To correct for empirically skewed relationship marginals in black-box SGG models, a per-triplet Bayesian Network is constructed with a within-triplet prior $p_{\text{rel}}(r|h, t)$6 (possibly augmented via embedding-based neighborhood counts), and “virtual evidence” is injected using the softmax predictions of the base classifier. The resulting posterior yields rebalanced predictions, improving mean-recall (mR) at modest cost in head-class recall (Biswas et al., 2022).
• Grammar/factor-graph models: Stochastic grammars define scene structures as random derivations, and sum–product message passing on the corresponding factor graph produces marginal probabilities for every entity and relationship; these marginals collectively define a PSSG (Chua et al., 2016).
• Semantics-for-communication: PSSGs that encode $p_{\text{rel}}(r|h, t)$7 and $p_{\text{rel}}(r|h, t)$8 distributions over entity pairs enable semantic-aware two-stage and multi-round compression algorithms for semantic image transmission. At the transmitter, highly predictable triplet attributes are omitted, and at the receiver, omitted entities and predicates are reconstructed via MAP inference. This results in a compression ratio $p_{\text{rel}}(r|h, t)$9 (pruned relation and tail-entity rates), and empirical 5×–10× throughput improvements over pixel-based baselines in noisy channels (Zhu et al., 16 Jul 2025).

4. Structured Uncertainty, Commonsense, and Spatial Reasoning

Probabilistic semantic scene graphs also support spatial commonsense interpretation in partially observed or belief-driven settings:

• Belief Scene Graphs (BSG): These extend 3D scene graphs with “blind” nodes (for hidden/unseen objects), treat the object-location composition as a probabilistic spatial field, and employ a GCN-based CECI model to infer class-conditional joint distributions $p_{\text{co}}(t|h)$0 over the occupancy grid. Augmenting the GCN with symbolic ontological constraints, derived via LLM-mediated spatial rules, further improves spatial prediction accuracy in real and simulated environments (Saucedo et al., 5 May 2025).

Evaluation leverages metrics such as Wasserstein Distance and Energy Distance between predicted and ground-truth spatial distributions; improvements from ontology inclusion are consistent but modest.

5. Unified Hybrid-Generative Models and Flow-based PSSG Synthesis

Recent advances model the entire SGG process as a progressive, hybrid discrete–continuous generative flow. The FlowSG framework parameterizes the state $p_{\text{co}}(t|h)$1 of a scene graph as jointly evolving continuous geometry (bounding boxes) and discrete tokens (labels, appearances, predicates), each transformed from noisy priors to data-aligned posteriors via ODE-driven and CTMC-driven denoising steps: $p_{\text{co}}(t|h)$2 with all components predicted using a graph transformer that embeds graph structure, image features, and flow/timestep information (Hu et al., 18 Apr 2026).

Key empirical findings:

• Few-step inference: High-quality SGG with only 10 flow steps.
• Posterior sampling: Graphs sampled from the hybrid flow capture semantic and geometric multi-modality.
• mR/R gains: Yields +2–4 point improvements in both recall and mean-recall metrics over prior SOTA (Hu et al., 18 Apr 2026).

6. Applications and Impact

• Uncertainty quantification: Enables not only confidence measures but also diverse, plausible outputs for ambiguous queries and multi-label relationships (Yang et al., 2021).
• Semantic compression and recovery: PSSGs enable practical culling and MAP-based recovery of semantic content over constrained-channel communication, demonstrating orders-of-magnitude improvements in throughput (Zhu et al., 16 Jul 2025).
• Long-tail debiasing and balanced recall: Probabilistic posteriors or stochastic sampling support