3D ReasonSeg: 3D Spatial Reasoning Benchmark

• 3D ReasonSeg is a high-fidelity dataset featuring per-point segmentation and explicit reasoning tags for multi-step spatial queries in indoor 3D scenes.
• It unifies sensor-captured ScanNet scans and synthetic SceneVerse reconstructions, covering over 20 object categories with compositional reasoning tasks.
• Evaluation using metrics like generalized IoU highlights significant advancements in 3D reasoning performance and exposes model weaknesses.

The 3D ReasonSeg Dataset, introduced as “3D ReasonSeg” and described in “Enhancing Spatial Reasoning in Multimodal LLMs through Reasoning-based Segmentation,” is a large-scale, high-fidelity benchmarking resource designed to advance the spatial reasoning and language understanding capabilities of multimodal AI agents operating on 3D point clouds. The dataset addresses the persistent gap in rigorous testing and development for vision-LLMs requiring multi-step, compositional reasoning over complex indoor scenes, where relationships such as proximity, containment, and composite spatial queries must be solved via language-aligned per-point segmentation. By combining detailed point-level masks, explicit reasoning tags, and compositional queries grounded in both real and synthetic indoor scenes, 3D ReasonSeg defines a challenging testbed for the next generation of 3D reasoning architectures (Ning et al., 29 Jun 2025).

1. Dataset Scope, Composition, and Statistics

3D ReasonSeg comprises 29,151 high-quality samples, partitioned into 25,185 training and 3,966 validation samples, following rule-based filtering and manual verification. The scenes are drawn from two sources: sensor-captured ScanNet scans and synthetic SceneVerse reconstructions, ensuring diversity across bedrooms, offices, living rooms, kitchens, and other indoor environments. Object coverage includes over 20 semantically distinct indoor categories (such as chairs, tables, beds, cabinets, monitors, and containers), each with per-instance masks for fine spatial delineation.

Each sample consists of:

• A 3D point cloud with original (x, y, z) coordinates.
• Per-point semantic class and instance labels.
• A natural-language question demanding spatial reasoning, and corresponding per-point segmentation mask(s) identifying the answer region.
• Explicit “reasoning tags” denoting which scene objects (by instance ID) constitute intermediate steps in the required multi-step reasoning chain.

Statistically, the dataset averages 5.4 relevant objects per query and 19.6 words per question, substantially exceeding the corresponding values for previous 3D vision-language datasets such as ScanQA (1.5 objects) and ScanRefer (1.8 objects). This composition enforces multi-object, multi-hop reasoning and compositional understanding well beyond simple instance identification (Ning et al., 29 Jun 2025).

Dataset	#Train	#Val	Avg. Words	Avg. Objects
ScanQA	26,563	4,675	8.8	1.5
ScanRefer	36,665	9,508	17.8	1.8
3D ReasonSeg	25,185	3,966	19.6	5.4

2. Annotation Format and Representation

Each sample is annotated according to a strict schema:

• Per-point segmentation masks are provided, with each point assigned to at most one instance mask.
• Instance-level masks are generated using a transformer-based decoder inspired by the Segment-Anything paradigm.
• No 3D bounding box annotations are included; spatial reasoning is supported solely by point-wise masks and metadata.

Every point is annotated with:

• Metric-space (x, y, z) position.
• Semantic category (e.g., “chair”).
• Instance ID.
• Reasoning tags—a list of instance IDs tracing the path of required intermediate objects in the reasoning process.

All spatial cues are encoded within mask-level annotations; box parameterizations such as $(x, y, z, w, h, d, \theta)$ are omitted. This design compels models to reason compositionally rather than relying on shortcut heuristics.

3. Generation and Quality Assurance Workflow

The data generation pipeline blends synthetic and real-world scene sources:

• SceneVerse metadata provides object descriptions (function, color, relative spatial context).
• Annotated object lists and textual metadescriptions are passed to LLaMA 3.1, which outputs a triplet: (Natural-language question; Multi-step reasoning steps with explicit relevant objects; Answer).
• “Relevant object” tokens are mapped back to instance masks via model segmentation heads.

A multi-stage quality-control process includes:

• Rule-based filtering: ambiguous queries (“in the right of”) and outliers (e.g., invalid superlative claims, improbable distances) are excluded.
• Manual spot-checks remove template violations and nonsensical samples.
• During training, relevant-object priors may be randomly omitted or added to mitigate train/test distribution shift.

4. Types of Spatial Reasoning Tasks

The dataset enforces a rich spectrum of spatial reasoning phenomena, including:

• Relative positions (“in front of,” “behind,” “left of,” “right of”).
• Proximity (“nearest,” “farthest,” “closest to”).
• Size comparison (“largest,” “smallest,” “tallest”).
• Containment or inclusion (“object inside a cabinet/container”).
• Composite relations (e.g., “the smallest chair nearest the window”).

Every query is constructed to require multi-step reasoning:

1. Identify all potentially relevant objects (e.g., “all desks,” “all chairs”).
2. Apply additional linguistic and spatial constraints to yield the precise object(s) or region(s) (e.g., the single chair closest to a particular desk).

Sample 1: “What is the object used for sitting and located near the desk with a monitor on it?” demands (i) segmentation of all desks and chairs, followed by (ii) minimal-distance reasoning between individual desk and chair instances. Sample 2: “Which container is inside the largest cabinet?” requires sequential (i) volumetric comparison over all cabinets, then (ii) containment constraint applied to possible containers (Ning et al., 29 Jun 2025).

5. Benchmarks, Evaluation Protocols, and Performance

Evaluation is based primarily on the generalized Intersection-over-Union (gIoU) metric:

$gIoU = \frac{1}{N}\sum_{i=1}^{N} \mathrm{IoU}_i,$

where $\mathrm{IoU}_i$ denotes the intersection-over-union for the $i$-th sample, computed over the per-point predicted and ground-truth masks.

Additional metrics for specific tasks include:

• Acc@25 / Acc@50 (percentage of predicted masks matching ground-truth with IoU ≥ 0.25 / 0.50) for ScanRefer.
• BLEU-4, CIDEr, METEOR, ROUGE-L for ScanQA-style language grounding tasks.
• Loss functions encompass cross-entropy for text, binary cross-entropy, and Dice loss for mask outputs.

Baseline and state-of-the-art performance on the 3D ReasonSeg validation set:

Method	gIoU
Grounded 3D-LLM (SoTA)	25.6
+ 3D ReasonSeg only	29.2
+ Relevant Reasoning Seg	33.1

On representative samples, gIoU can exceed 0.8 for well-localized queries; the dataset’s multi-hop structure exposes marked deficiencies in single-step or non-reasoning systems.

6. Comparative Analysis and Relationship to Other Datasets

3D ReasonSeg occupies a distinct position in the 3D vision-language dataset landscape. In comparison with benchmarks such as ReasonSeg3D (Jiang et al., 2024) and SURPRISE3D (Huang et al., 10 Jul 2025):

• 3D ReasonSeg’s queries systematically require compositional, multi-object reasoning; SURPRISE3D focuses on spatial language without object category bias by omitting object names, emphasizing spatial constraints, while ReasonSeg3D integrates explicit QA triplets with spatial-relational explanations.
• The average number of relevant objects per query (5.4) exceeds other benchmarks, highlighting its value for multi-element reasoning.
• The annotation pipeline and reasoning-tag schema enforce stepwise reasoning with explicit object dependencies.

A plausible implication is that while all datasets advance 3D VL research, 3D ReasonSeg is especially rigorous for developing and benchmarking systems with compositional and multi-hop spatial reasoning abilities.

7. Access, Licensing, and Recommendations

3D ReasonSeg scenes and annotations are available according to the terms of the underlying ScanNet and SceneVerse licenses. Researchers may access the dataset and accompanying codebase via the project’s repository as stated in (Ning et al., 29 Jun 2025). The dataset design encourages future models to exploit explicit reasoning tags and compositional mask generation. Recommendations for extension include expanding the annotation set to include more classes, scene types, and reasoning constructs, as well as integrating temporal or dynamic scene queries.

• “Enhancing Spatial Reasoning in Multimodal LLMs through Reasoning-based Segmentation” (Ning et al., 29 Jun 2025).
• For context: “Multimodal 3D Reasoning Segmentation with Complex Scenes” (Jiang et al., 2024), “SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes” (Huang et al., 10 Jul 2025), and “PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Model” (Kareem et al., 2024).