VCG-32K: Visual Causal Graph Dataset

• VCG-32K is a comprehensive dataset for visual causal discovery, featuring over 32K images with entity-level causal graphs based on counterfactual removal criteria.
• It employs a stringent two-stage annotation pipeline with bounding box refinement and causal labeling to achieve high accuracy and sparsely distributed, directed edges.
• The dataset underpins advanced benchmarks for evaluating causal inference, with applications in robotics, autonomous driving, and physical commonsense reasoning.

Visual Causal Graph Dataset (VCG-32K) is a large-scale dataset for visual causal discovery: the task of inferring cause-and-effect relations among entities in a single image rather than merely detecting objects or describing spatial relations. It contains over 32,000 images annotated with entity-level causal graphs, in which entities are localized by bounding boxes and connected by directed causal edges grounded in a counterfactual removal criterion. In the accompanying formulation, the dataset is positioned as a resource for learning not only what is present in an image but why a visual configuration is maintained, with an emphasis on local physical mechanisms such as support, carry-on, and lift (Zhang et al., 1 Dec 2025).

1. Formal definition and causal semantics

VCG-32K formalizes visual causal discovery as causal graph construction from a single image $I$. The graph is defined as $\mathcal{G} = (\mathcal{V}, \mathcal{E})$, where the entities are $\mathcal{V} = \{v_i\}_{i=1}^N$ and each entity is represented as $v_i = (c, b)$, with $c$ a category label and $b \in \mathbb{R}^4$ a bounding box. The edge set is $\mathcal{E} = \{e_{ij}\}_{i \ne j}$, with directed, predicate-labeled edges of the form $e_{ij} = (v_i, \kappa, v_j)$ and $\kappa \in \mathcal{K}$, where representative mechanism types include “support,” “carry_on,” and “lift” (Zhang et al., 1 Dec 2025).

The central causal criterion is counterfactual and disposition-based. An edge belongs to the graph if and only if

$$e_{ij} \in \mathcal{E} \iff p(v_j \mid do(v_i = 0)) \ne p(v_j),$$

where $\mathcal{G} = (\mathcal{V}, \mathcal{E})$0 indicates entity presence or removal and $\mathcal{G} = (\mathcal{V}, \mathcal{E})$1 denotes intervention, instantiated here as removal of $\mathcal{G} = (\mathcal{V}, \mathcal{E})$2. The accompanying intuition is explicit: if removing $\mathcal{G} = (\mathcal{V}, \mathcal{E})$3 changes the state of $\mathcal{G} = (\mathcal{V}, \mathcal{E})$4, then the directed relation $\mathcal{G} = (\mathcal{V}, \mathcal{E})$5 is present. This makes the dataset structurally different from corpora that encode only co-occurrence, relative position, or descriptive predicates.

The adjacency structure is represented by $\mathcal{G} = (\mathcal{V}, \mathcal{E})$6 with $\mathcal{G} = (\mathcal{V}, \mathcal{E})$7 iff $\mathcal{G} = (\mathcal{V}, \mathcal{E})$8. The task objective is to predict $\mathcal{G} = (\mathcal{V}, \mathcal{E})$9, or equivalently $\mathcal{V} = \{v_i\}_{i=1}^N$0, given image $\mathcal{V} = \{v_i\}_{i=1}^N$1 and detected entities $\mathcal{V} = \{v_i\}_{i=1}^N$2. No explicit acyclicity constraint is imposed on $\mathcal{V} = \{v_i\}_{i=1}^N$3; edges are directed, but cycles are not ruled out by the formulation. In practice, the paper states that most edges reflect local physical support or containment and are sparsely distributed.

2. Corpus composition and annotation pipeline

VCG-32K contains 32,256 images with entity-level causal graphs, drawn from two existing visual corpora: 11,428 images from MS-COCO and 20,828 images from Objects365. Across the full dataset, there are 299,262 entities spanning 2,287 categories and 185,321 directed causal relationships. The reported averages are approximately 9.27 entities per image and 5.75 causal edges per image, with approximate graph density

$\mathcal{V} = \{v_i\}_{i=1}^N$4

indicating sparse graphs (Zhang et al., 1 Dec 2025).

Component	Value
Total images	32,256
MS-COCO images	11,428
Objects365 images	20,828
Total entities	299,262
Categories	2,287
Directed causal relationships	185,321

The dataset targets diverse everyday scenes from COCO and Objects365, but its relation inventory is explicitly centered on physical causality. Social or temporal causality is not the primary target. Because the annotation criterion requires direct contact or force interaction, the resulting edges largely encode local physical mechanisms such as support, carry-on, and inclusion; long-range relationships are described as rare by design.

Annotation follows a two-stage pipeline with 50 trained annotators and 10 senior reviewers. The first stage is bounding box refinement. Annotators review and correct existing boxes, remove boxes with boundary or label errors, exclude overly broad or abstract entities such as “sky” or “air,” and correct incorrect labels such as “man” versus “woman.” Entities without direct physical contact or interaction with any other entity are excluded, and missing entities participating in direct physical contact or force interaction are added. Entities smaller than $\mathcal{V} = \{v_i\}_{i=1}^N$5 pixels are excluded to preserve reliability.

The second stage is causal relationship labeling. For each ordered pair $\mathcal{V} = \{v_i\}_{i=1}^N$6, annotators determine the existence and direction of $\mathcal{V} = \{v_i\}_{i=1}^N$7 if three conditions are satisfied: $\mathcal{V} = \{v_i\}_{i=1}^N$8 is in direct contact with $\mathcal{V} = \{v_i\}_{i=1}^N$9; the presence of $v_i = (c, b)$0 maintains $v_i = (c, b)$1’s current state; and removing $v_i = (c, b)$2 causes $v_i = (c, b)$3 to lose its current state. For each positive pair, a mechanism type $v_i = (c, b)$4 is assigned. The reported average annotation accuracy exceeds 95%, and 10 senior annotators perform random reviews with corrections. No explicit inter-annotator agreement statistic such as Cohen’s $v_i = (c, b)$5 is reported.

3. Data representation and benchmark protocol

Each dataset item includes a dataset identifier (“COCO” or “365”), an image identifier, a list of entities, and a dictionary of mechanism types mapping to directed pairs. In the provided schema, each entity has an entity_id, entity_name, and bbox, while the causal_relationships field groups directed pairs under mechanism labels such as "carry_on" and "support" (Zhang et al., 1 Dec 2025).

Images originate from COCO and Objects365, and the stated preprocessing requirements are minimal: standard image loading suffices, entities are provided as bounding boxes in absolute pixel coordinates, and evaluation expects a structured output schema. Conformant formatting is explicitly rewarded during reinforcement learning through a “Format” term in the reward design. Code, model, and dataset are open-sourced at the project repository, while license terms and exact on-disk size are to be checked on the project page.

The benchmark protocol is structure-only and direction-aware. Entity matching is performed via the Hungarian algorithm with GIoU, using a typical threshold of 0.5. An edge $v_i = (c, b)$6 is counted as correct only if there is a matched pair between predicted and ground-truth entities and the directed edge $v_i = (c, b)$7 exists in the ground truth. Predicate labels and category names are ignored for scoring; correctness is purely structural and directional.

The primary metrics are

$v_i = (c, b)$8

and

$v_i = (c, b)$9

Evaluations report graph-level scores.

The paper also introduces diagnostic quantities to separate detection from reasoning. Reachable recall, $c$0, is defined as the maximum achievable recall given correctly recognized entities. Reasoning-induced loss is

$c$1

which quantifies the gap due to reasoning rather than detection. Detection sensitivity across GIoU thresholds $c$2 is probed with the Recall Stability Index:

$c$3

By contrast, GED and SHD are not reported, and no partial credit is given for predicate labels.

4. Distinction from scene graphs and visual relationship detection

VCG-32K is explicitly differentiated from scene graph generation and visual relationship detection. Scene graph generation typically outputs undirected or weakly directed predicates such as “on,” “next to,” and “behind,” while visual relationship detection focuses on recognizing a triplet $c$4 without counterfactual semantics. VCG-32K instead provides entity-level, directed causal graphs grounded in counterfactual criteria defined through removal and intervention (Zhang et al., 1 Dec 2025).

A common misunderstanding is to treat VCG-32K as a conventional relation-detection benchmark with more directed edges. The paper’s formulation rejects that interpretation. First, the semantics are causal rather than descriptive: an edge is intended to explain why an entity’s state is maintained. Second, each edge carries a mechanism type $c$5, which the paper describes using the notion of “supersense” as a training signal for high-level causal mechanisms, although no closed taxonomy beyond representative examples is enumerated. Third, the evaluation protocol ignores predicate labels and category names at scoring time, prioritizing directionally correct structure over lexical agreement.

The comparison to Visual Genome and related scene graph datasets is also framed around data quality. Prior corpora are described as often suffering from bounding box or label noise and as being agnostic to causality. VCG-32K responds with refined bounding boxes, strict inclusion criteria, exclusion of entities smaller than $c$6 pixels, and a structure-first protocol centered on directed causal discovery. This suggests a deliberate shift from relational description toward counterfactual physical reasoning in static images.

5. Integration into CauSight training

Within the associated CauSight framework, VCG-32K is not only a benchmark but also the training substrate for a multi-stage learning pipeline. The training corpus used in the paper is the MS-COCO subset with 11,078 samples. Tree-of-Causal-Thought (ToCT) synthesis is performed on 6,000 training images, yielding 3,631 retained trajectories after filtering for superiority over vanilla one-step reasoning. Reinforcement learning is then conducted on the remaining 7,447 COCO samples (Zhang et al., 1 Dec 2025).

ToCT organizes reasoning as an iterative action loop:

$c$7

corresponding to region selection, entity recognition, and causality orientation. The state at step $c$8 is

$c$9

A teacher policy generates intermediate results $b \in \mathbb{R}^4$0 up to a step limit $b \in \mathbb{R}^4$1, with $b \in \mathbb{R}^4$2 used in the reported setup.

Trajectory quality is enhanced with MCTS. The selection score is

$b \in \mathbb{R}^4$3

and backpropagation uses

$b \in \mathbb{R}^4$4

Leaf rewards are derived from structural match against the ground-truth causal graph. The reported search configuration runs 20 search iterations and expands up to 10 children per step.

Supervised initialization is performed by SFT on filtered ToCT trajectories with

$b \in \mathbb{R}^4$5

Reinforcement learning then uses GRPO with objective

$b \in \mathbb{R}^4$6

where

$b \in \mathbb{R}^4$7

The causal reward is

$b \in \mathbb{R}^4$8

The reported GRPO configuration uses 5 samples per prompt, 15 epochs, 4 nodes $b \in \mathbb{R}^4$9 8 H200 GPUs, batch size 512, and maximum response length 4096. The paper notes that GRPO eliminates a separate value network by using group-relative advantage.

6. Benchmark performance and diagnostic findings

The reported baselines on VCG-32K include proprietary systems—Gemini 2.5 Pro, OpenAI o3, GPT-4.1, and GPT-5—as well as open-source Qwen2.5-VL models at 7B, 32B, and 72B scales, plus a direct supervised fine-tuning baseline without process supervision. On the average of the COCO and Objects365 benchmarks, GPT-4.1 achieves Recall 10.2, Precision 9.6, and F1 9.2; Qwen2.5-VL-72B achieves Recall 15.4, Precision 23.6, and F1 16.8; the SFT variant based on a 7B model achieves Recall 11.7, Precision 17.2, and F1 12.9; and CauSight achieves Recall 31.2, Precision 45.5, and F1 34.4 (Zhang et al., 1 Dec 2025).

The paper reports an absolute recall gain of +21.0% over GPT-4.1, described as approximately $\mathcal{E} = \{e_{ij}\}_{i \ne j}$0, and a +15.8% absolute recall advantage over Qwen2.5-VL-72B. Against the SFT variant in cross-domain evaluation on Objects365, CauSight shows +16.0% recall and +17.3% F1. These results are paired with in-domain and cross-domain breakdowns: on COCO, CauSight obtains Recall 34.2, Precision 48.7, and F1 37.6, compared with GPT-4.1 Recall 16.5; on Objects365, CauSight obtains Recall 28.1, Precision 42.3, and F1 31.1, compared with GPT-4.1 Recall 3.9.

Ablation experiments attribute much of the performance to the full training recipe. Without ToCT, average performance drops to Recall 24.5, Precision 43.7, and F1 28.6, with reduced generalization. Without RL, average performance falls to Recall 10.1, Precision 6.6, and F1 7.0. The paper also states that integrating SFT on ToCT before RL boosts cross-domain recall by +10.1% over RL alone.

The reasoning-versus-detection analysis further localizes the gain. On COCO recall, the reasoning-induced loss is reported as 8.0% for CauSight, 27.9% for GPT-4.1, 12.9% for Qwen2.5-VL-72B, and 44.8% for Gemini. The interpretation advanced in the paper is that CauSight reduces the gap toward the reachable upper bound while maintaining solid detection stability as measured by RSI.

7. Limitations, biases, and research uses

The dataset’s stated limitations follow directly from its design. Because it is built from COCO and Objects365, it is dominated by object-centric daily scenes. Social and temporal causality are underrepresented, and the requirement of direct physical contact further narrows the operative notion of causality to local physical mechanisms. A plausible implication is that performance on VCG-32K should not be read as evidence of broad causal competence outside that regime (Zhang et al., 1 Dec 2025).

The exclusion of entities smaller than $\mathcal{E} = \{e_{ij}\}_{i \ne j}$1 pixels improves reliability but may omit fine-grained causal structures in cluttered scenes. Evaluation is deliberately strict: structurally correct causal predictions are counted as errors if bounding boxes fail the matching criterion, and no partial credit is awarded for correct predicate labels. The annotation guidelines are detailed, but judgments about causality in marginal-contact cases can remain subjective, and no explicit inter-annotator agreement statistic is reported.

The paper also includes a responsible-use caution. The causal graphs should not be overinterpreted beyond local physical mechanisms, and the dataset should not be deployed as-is for safety-critical decisions without accounting for detection robustness, confounders, and uncertainty. It additionally warns about spurious correlations introduced by object co-occurrence in the underlying source datasets.

For practical usage, the recommended evaluation protocol is to detect entities, match predicted entities to ground truth using Hungarian matching with $\mathcal{E} = \{e_{ij}\}_{i \ne j}$2, and compute direction-aware Precision, Recall, and F1 on edges between matched nodes while ignoring category and predicate labels. Optional analyses include RSI across thresholds and reasoning-induced loss to separate detection from reasoning. The paper identifies prospective applications in robotics manipulation, autonomous driving scene topology, physical commonsense reasoning about stability and stacking, and, as a future extension rather than a present capability, social interaction understanding. It also identifies future directions: extension to videos for temporal causality, incorporation of confidence or strength on edges, and a broader supersense taxonomy.