FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning

Abstract: Recent work in 3D scene understanding is moving beyond purely spatial analysis toward functional scene understanding. However, existing methods often consider functional relationships between object pairs in isolation, failing to capture the scene-wide interdependence that humans use to resolve ambiguity. We introduce FunFact, a framework for constructing probabilistic open-vocabulary functional 3D scene graphs from posed RGB-D images. FunFact first builds an object- and part-centric 3D map and uses foundation models to propose semantically plausible functional relations. These candidates are converted into factor graph variables and constrained by both LLM-derived common-sense priors and geometric priors. This formulation enables joint probabilistic inference over all functional edges and their marginals, yielding substantially better calibrated confidence scores. To benchmark this setting, we introduce FunThor, a synthetic dataset based on AI2-THOR with part-level geometry and rule-based functional annotations. Experiments on SceneFun3D, FunGraph3D, and FunThor show that FunFact improves node and relation discovery recall and significantly reduces calibration error for ambiguous relations, highlighting the benefits of holistic probabilistic modeling for functional scene understanding. See our project page at https://funfact-scenegraph.github.io/

• The paper introduces a unified framework that constructs probabilistic functional 3D scene graphs through dual factor-graph reasoning and belief propagation.
• It leverages foundation VLMs and LLMs along with geometric, cardinality, and proximity priors to refine object and part detection in RGB-D sequences.
• Empirical results on datasets like FunThor demonstrate improved mapping recall, functional relation prediction, and confidence calibration over previous methods.

Probabilistic Functional 3D Scene Graph Construction via Factor-Graph Reasoning with FunFact

Introduction

FunFact introduces a unified framework for functional scene understanding that addresses the limitations of prior approaches by constructing probabilistic open-vocabulary functional 3D scene graphs from posed RGB-D sequences. Unlike methods focusing only on static geometric or semantic reasoning, FunFact explicitly models joint functional relationships and their inherent ambiguities via belief propagation in a dual factor graph. The system leverages foundation VLMs for open-vocabulary part/object proposal and LLMs for proposing semantically grounded functional interactions, which are regularized and optimized through geometric and commonsense priors. The framework is accompanied by the introduction of FunThor, a synthetic AI2-THOR-based dataset with comprehensive functional annotations, enabling robust evaluation in mapping, relation recall, and calibration error.

Figure 1: FunFact reconstructs an object- and part-centric 3D scene and models candidate functional relations as binary variables in a dual factor graph, optimizing marginal probabilities via belief propagation.

Pipeline Overview

FunFact's pipeline consists of two principal stages: object/part-centric 3D reconstruction and probabilistic functional scene-graph inference.

Semantic 3D Scene Reconstruction

Given posed RGB-D frames, a large VLM (GPT-4.1 or alternatives) is queried for each image to extract an object list, hierarchy descriptions, and part proposals. The system then uses GroundingDINO for grounding object/part queries, refines proposals by spatial filtering and consistency checks, and performs hierarchical merging across views to generate a 3D part-aware scene map. This approach supports fine-grained interactive element recovery (e.g., stove knobs, coffee machine buttons, etc.), and incorporates hallucination filtering to mitigate noise from VLMs.

Figure 2: The FunFact pipeline: scene reconstruction with VLM-based proposals followed by grounded detection, part refinement, multi-view fusion, and factor-graph based relation inference.

Functional Scene Graph Generation via Dual Factor Graphs

Candidate functional relations (object-object, part-object, part-part) are generated by LLMs using semantic and geometric cues from the constructed 3D map. For each relation template, FunFact constructs a dual factor graph where binary variables encode candidate relations, and constraints are enforced via two principal priors:

• Cardinality Factors: Soft constraints on the plausible number of incident relations (e.g., one-to-one mapping for knob-burner).
• Proximity Priors: Unary factors favor connections with shorter Euclidean distance, adaptive per-relation type.

Joint probabilistic inference via belief propagation yields well-calibrated marginal confidence for each relation, resolved in the context of all candidate edges rather than independently.

Figure 3: A candidate functional scene graph (left), and its dual factor graph (right) encoding edge variables and cardinality/proximity factors.

Empirical Evaluation

Datasets and Benchmarks

FunFact is evaluated on three settings:

• SceneFun3D and FunGraph3D: Real-scene datasets with open-vocabulary functional annotation, used for benchmarking object/part/node recovery and functional relation recall.
• FunThor: Introduced in this work, FunThor is a synthetic dataset built on AI2-THOR, with part-level geometry, 26 functional relation types, comprehensive annotation, and dense frame coverage.

  Figure 4: Example FunThor environments with mapped functional objects; shows top-down and segmentation views with dense functional annotation.

Quantitative Results

Mapping Recall

FunFact achieves substantial improvements in Recall@K over previous baselines such as Open3DSG, ConceptGraph, and OpenFunGraph on both SceneFun3D and FunGraph3D, most notably with respect to interactive elements—demonstrating the value of hierarchical object/part proposal and robust merging. The method preserves high object recall even in cases where fine-grained annotation may diverge from ground truth.

Functional Relation Prediction

In triplet-based evaluation (subject, interaction, object), FunFact yields higher recall and F1 on FunThor and FunGraph3D compared to OpenFunGraph, with notably lower expected calibration error (ECE)—especially for ambiguous relations (e.g., stove knob-burner, switch-light)—evidencing the efficacy of joint factor-graph reasoning. On SceneFun3D, FunFact's open-vocabulary predictions are penalized due to coarse, generic ground-truth labels but still rival or surpass state-of-the-art baselines across most metrics.

Figure 5: Qualitative results: input image with detected functional objects, 3D point clouds with color-coded relation confidence, and the final functional 3D scene graph.

Confidence Calibration

FunFact's calibration, as measured via ECE on the FunThor dataset, indicates a significant reduction in overconfidence for ambiguous edges compared to the baseline, affirming the merit of holistic probabilistic modeling.

Figure 6: Confidence histograms and reliability diagrams on FunThor: FunFact (left) achieves better calibration, especially for ambiguous classes, compared to OpenFunGraph (right).

Factor-Graph Inference Effect

Ablations confirm that removing factor-graph inference substantially degrades both calibration and F1, highlighting that global joint reasoning (beyond independent pairwise classification) is critical for accurate, calibrated functional scene understanding.

Figure 7: Illustration of belief propagation in the dual factor graph, demonstrating calibrated updates of edge confidences after ground-truth verification.

Practical and Theoretical Implications

FunFact provides a tractable framework for integrating VLM/LLM-based commonsense and geometric priors with a probabilistic graphical model, offering well-founded confidence estimates for ambiguous functional relationships. This approach enables more reliable autonomous scene understanding for downstream robotics, AR assistants, or planning tasks, where knowing what is unknown is crucial for safe decision making. The factor-graph construction is inherently modular—new evidence can be incorporated in an online or active learning setting, supporting interactive or embodied robotic systems.

The introduction of FunThor addresses the critical gap in synthetic and real datasets for benchmarking joint entity–relation–confidence modeling, facilitating fairer evaluation under dense annotation.

Limitations and Future Work

The method can be sensitive to VLM/LLM inference errors and over/under-segmentation in 3D instance association. Its current reliance on nontrivial LLM latency (several seconds/frame) may limit deployment in strict real-time applications. Future research may advance toward online/active learning variants (e.g., incremental evidence integration for functional verification during robotic exploration) and explore adaptation to dynamic or multi-agent environments.

Conclusion

FunFact advances functional scene understanding by fusing open-vocabulary scene reconstruction with joint probabilistic functional inference via factor graphs, producing well-calibrated, context-aware edge confidence scores. Its performance across standard and new benchmarks consistently exceeds that of prior pairwise or heuristic methods, particularly in ambiguous settings where functional reasoning requires holistic context. This paradigm can directly inform robust scene understanding for embodied agents, interactive reasoning systems, and downstream AI applications.

Reference:

For full technical details and datasets, see "FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning" (2604.03696).