- The paper introduces a two-stage framework that integrates relation proposals with counterfactual evidence verification to mitigate hallucinated scene graph relations.
- It decomposes each candidate relation into a distribution of evidence types and employs predicate-conditioned attention to tightly couple visual features with linguistic cues.
- Empirical results on VG150, OV-VG, and PSG benchmarks demonstrate notable gains in recall, reliability, and reduction of unsupported relations.
Evidence-Grounded Open-Vocabulary Scene Graph Generation via Counterfactual Verification
Motivation and Background
Scene graph generation (SGG) operationalizes visual understanding as a graph structure where nodes represent objects and edges encode semantic relations. Traditionally, SGG approaches have been confined to closed predicate vocabularies, limiting their ability to capture open-ended, fine-grained, and compositional relations inherent in real-world imagery. Vision-LLMs have lifted these constraints, enabling open-vocabulary SGG and substantially expanding the semantic scope of scene graphs. However, this shift has introduced critical reliability issues: relations predicted by these models are often driven by language priors or object co-occurrence instead of visually grounded evidence, resulting in hallucinated relationships that may not accurately represent the underlying scene.
The inadequacy of purely generative models for relation prediction, especially under open-vocabulary settings, underscores a fundamental need for evidence-grounded verification. Current protocols emphasize maximizing triplet recall, but neglect explicit verification that relations are visual, geometric, or contextually supported. True visual relation recognition requires that each relation in a scene graph can be justified by observable, predicate-specific evidence and that prediction confidence decreases under counterfactual perturbations targeting the relevant evidence.
CAGE-SGG: Methodology
CAGE-SGG formulates open-vocabulary SGG as a two-stage process: (1) relation proposal using vision-LLMs, followed by (2) evidence-grounded verification using counterfactual interventions.
Relation Proposal
The initial stage generates a compact set of candidate relation phrases per object pair. Object-centric visual and spatial features (bounding boxes, masks, geometry, depth, appearance) are encoded, and top-K candidate relations are produced via vision-language similarity between object pair features and textual embeddings of predicate phrases. This proposal remains intentionally permissive, allowing noisy and linguistically plausible but unsupported relations to be included for subsequent verification.
Evidence-Type Decomposition
A core challenge in open-vocabulary SGG is the diversity and ambiguity in unseen predicates, which may reference heterogeneous evidence types. CAGE-SGG overcomes this by decomposing each relation phrase into a soft distribution over evidence bases: support, contact, containment, depth, proximity, part, motion, state, and functional. This allows sharing of evidence patterns across semantically distinct predicates (e.g., standing on, resting on, supported by) and facilitates generalization to previously unannotated or rare relations.
Predicate-Conditioned Evidence Encoding
For each proposed triplet, multimodal evidence tokens are extracted—capture object regions, union features, masks, geometric cues, contextual information, and temporal features for video—and filtered through a predicate-conditioned attention mechanism. Evidence gating further suppresses irrelevant evidence channels, ensuring that the final edge representation is tightly coupled to predicate-specific cues.
Counterfactual Relation Verification
The central innovation is predicate-specific counterfactual verification. For each candidate triplet, two families of counterfactual interventions are constructed: relation-breaking (target and disrupt the necessary evidence—for example, mask supporting regions for "on," disrupt depth ordering for "in front of") and relation-preserving (modify nuisance factors while preserving evidence—such as background replacement or mild cropping). The model is trained to decrease relation confidence under evidence removal and maintain confidence under nuisance changes.
The counterfactual validity score quantitatively reflects whether relation prediction is causally dependent on evidence (evidence necessity and nuisance invariance), forming the foundation for final edge scoring. Contradiction-aware predicate learning further enforces discrimination among fine-grained or semantically opposing predicates (e.g., in front of vs. behind).
Graph-Level Preference Optimization
CAGE-SGG extends verification to the graph level via preference ranking over sets of candidate scene graphs. Graphs are scored for semantic agreement, geometric validity, counterfactual sensitivity, and global consistency. This collective, group-relative optimization ensures global coherence and reduces contradictory or hallucinated edge sets.
Empirical Results
CAGE-SGG demonstrates strong empirical improvements across multiple SGG benchmarks—VG150 (closed-vocabulary), OV-VG (open-vocabulary), and PSG (panoptic region-level SGG):
- Mean recall (mR@K) gains: Outperforms state-of-the-art baselines (e.g., PE-Net, Pix2Graphs, VL-IRM) with notable margins on rare and unseen predicates.
- Open-vocabulary generalization: Harmonic mean of seen and unseen predicate recall improved substantially, indicating successful transfer from seen to unseen relations via evidence-type sharing.
- Counterfactual grounding metrics: CF-Acc and Inv-Stab metrics show significant increases; Hallu-Rate is reduced, quantifying reduction in hallucinated relations unsupported by evidence.
- Panoptic SGG: Gains in precision and mean recall are observed for mask-grounded region-level relations, affirming the method's applicability beyond bounding-box SGG.
Ablations confirm the primacy of counterfactual verification, evidence-type decomposition, and graph-level preference optimization in achieving these improvements. Sensitivity analyses show that moderate candidate selection (K=10) is optimal and that relation-aware intervention mixtures outperform generic masking.
Implications, Limitations, and Future Prospects
Practical
The evidence-grounded paradigm delivers more trustworthy and interpretable scene graphs for downstream tasks—robotic manipulation, embodied navigation, visual question answering, and commonsense reasoning. By tying relation prediction to observable evidence and providing evidence traces per edge, the model supports robust, transparent reasoning and human-AI collaboration. Robustness to language-prior bias and co-occurrence statistics positions CAGE-SGG as a superior alternative for deployment in open-world settings.
Theoretical
CAGE-SGG reframes SGG as a causal inference task, emphasizing verification rather than generation. This shift encourages broader integration of causal reasoning, counterfactual interventions, and evidence regularization in vision-language representation learning. The evidence-type decomposition and predicate-conditioned verification bridge the gap between linguistic richness and visual grounding.
Future Directions
- Extending verification to multi-modal, temporal, and 3D scene graphs will further enhance the framework's universality.
- Integrating explicit causal modeling for group-level graph consistency may yield even stronger guarantees against hallucination and contradiction.
- Automated discovery of evidence bases and relation-specific intervention protocols could improve adaptation to new domains and relation types.
- Application to embodied AI scenarios can facilitate action-planning and dynamic reasoning under task constraints.
Conclusion
CAGE-SGG advances the state-of-the-art in open-vocabulary scene graph generation by moving from relation generation to evidence-grounded verification. Through predicate-conditioned evidence encoding and counterfactual interventions, the framework substantially reduces language-prior hallucination and enhances reliability, especially for rare and unseen relations. Its verification-centric approach, validated by strong empirical results, signals a matured direction for SGG research—toward interpretable, trustworthy, and causally robust scene graphs (2604.22274).
