MesaTask-10K: 3D Tabletop Scene Benchmark

• MesaTask-10K is a large-scale dataset that drives task-driven 3D tabletop scene generation with explicit spatial reasoning and realistic inter-object relations.
• It leverages a multi-stage pipeline integrating LLM-driven prompt synthesis, coarse 3D reconstruction, and human-in-the-loop refinement to ensure physical plausibility.
• The benchmark supports robotic policy learning and spatial reasoning research with robust metrics, including a 99% success rate and near-zero collision rate.

MesaTask-10K is a large-scale dataset and research benchmark introduced to advance task-driven 3D tabletop scene generation with explicit 3D spatial reasoning and realistic inter-object relations. Its primary aim is to bridge the gap between high-level manipulation task instructions and physically plausible tabletop scene layouts, supporting both robust LLM-based scene generation and robot manipulation policy learning. Leveraging over 10,700 manually curated synthetic scenes and a comprehensive 3D asset library, MesaTask-10K establishes a rigorous environment for evaluating and developing spatial reasoning algorithms, relation-aware generation frameworks, and embodied intelligence systems (Hao et al., 26 Sep 2025).

1. Dataset Composition and Construction

MesaTask-10K comprises approximately 10,700 distinct tabletop scenes, each instantiated with programmatically synthesized reference images, geometric reconstruction, and intensive human-in-the-loop refinement. The asset library contains over 12,000 unique 3D models mapping to more than 200 everyday object categories, with scenes spanning six archetypal table settings: office desks, dining tables, kitchen counters, coffee tables, bathroom vanities, and dressing tables. The per-scene object count follows a bell-shaped distribution (mean ≈ 15 objects, range: 3–30), with even distribution across table types to ensure broad coverage of manipulation and arrangement domains.

The construction pipeline integrates several technical stages:

• Reference Image Synthesis: Prompts generated via LLMs encode object lists paired with spatial relations (e.g., "a bowl with a spoon inside, a glass left of the plate"). The FLUX text-to-image model renders plausible photographic images of these descriptions.
• Coarse 3D Reconstruction: DepthAnything v2 provides pixel-level depth estimates, while Grounded-SAM yields semantic segmentation masks. Masked point clouds are processed to obtain per-object bounding boxes, labeled using a multimodal LLM, which facilitates semantic 3D asset retrieval. Asset placement respects isometric scaling to align with extracted bounds.
• Human-in-the-Loop Layout Correction: Twenty expert annotators refine object positions, orientations, and scales in Blender, referencing a 1.7 m virtual robot for calibration. Fine-grained geometric and semantic corrections address collisions, floating objects, and restore specific spatial relations (e.g., containment, stacking, precise adjacency). All layouts are validated in NVIDIA IsaacSim for collision-free physical feasibility.

The table below summarizes core dataset statistics:

Scene Count	Object Classes	Avg. Objects per Scene	Table Types
≈10,700	>200	≈15 (min 3, max 30)	6 (office, dining, etc.)

2. Represented Inter-Object Relations and Annotations

MesaTask-10K explicitly encodes a broad spectrum of spatial relations critical for task-driven manipulation:

• Support: ("above", "below")
• Containment: ("in")
• Adjacency and Ordering: ("left of", "right of", "in front of", "behind")
• Alignment and Spacing: ("centered", "equally spaced")
• Orientation: Eight cardinal bins (e.g., front, right, back-left)

Annotations are constructed at the scene-graph level, where each scene is formalized as a directed graph $G = (V, E)$—with nodes representing instances and edges encoding relational predicates. Adjacency matrices $A_{ij}$ represent edge presence, supporting both discrete and numerical computation of relation types based on centroids $p_i,\,p_j$, sizes $s_i,\,s_j$, and oriented bounding-box geometry.

3. Spatial Reasoning Chain for Task-to-Scene Generation

MesaTask-10K introduces a structured, three-stage spatial reasoning pipeline for mapping task instructions to concrete 3D scene layouts:

1. Object Inference: Given parsed task elements $(E, G, O)$—comprising the environment, sub-goals, and seed objects—an LLM infers the full object set $V$ to appear in the layout.
2. Spatial Interrelation Reasoning: Conditioned on $(E, G, O, V)$, the model predicts pairwise relations $E$ via explicit geometric functions:

$R_{ij} = f(p_i, p_j, s_i, s_j)$

Examples include discretizing relative 2D vectors or imposing minimum overlap constraints for containment.

1. Scene Graph Construction and Layout Generation: The set $(V, E)$ forms a scene-graph $A_{ij}$0, which is then unrolled by a second LLM pass into a detailed 3D layout $A_{ij}$1, with each $A_{ij}$2 (position, size, orientation, type). The pseudo-code is:

$A_{ij}$4 This controlled pipeline facilitates both end-to-end learning and fine-grained analysis of spatial reasoning failure cases.

4. Dataset Usage, Metrics, and Baseline Performance

MesaTask-10K supports both supervised fine-tuning (SFT) and preference-based reinforcement learning (DPO) for 3D spatial reasoning tasks. The benchmark provides several evaluation metrics:

• Success Rate: Fraction of generated layouts parsed as valid (≥1 object) JSON.
• Fréchet Inception Distance (FID): Visual realism of rendered scenes, computed as $A_{ij}$3.
• GPT-Score (Multi-dimensional): LLM ratings on five axes: consistency with task, object size reasonableness, placement/intersections, layout coherence/realism, and visibility.
• Scene Graph Consistency: Proportion of scene-graph predictions matching the ground truth.
• Collision Rate: Percentage of infeasible object pairs (negative signed distance in Drake).

MesaTask achieves the following key metrics:

• Success Rate ≈ 99% (baseline: 91–99%)
• FID ≈ 40.3 (best among compared methods)
• GPT-Score Avg ≈ 8.25/10
• Collision Rate ≈ 0% (post-physics checks) (Hao et al., 26 Sep 2025).

5. Applications in Learning and Robotics

MesaTask-10K enables several downstream research and engineering use cases:

• End-to-End Task-to-Scene Generation: Trains/fine-tunes LLMs (e.g., Qwen3-8B) to synthesize 3D layouts directly from natural-language task descriptions, superseding rule-based generators.
• Robotic Policy Learning: Provides a scalable set of physically plausible, semantically diverse tabletop configurations for RL-based or imitation learning robotic agents. The curated physical validity of MesaTask-10K reduces the sim-to-real gap for task-conditioned manipulation.
• Spatial Reasoning and Scene Graph Modeling: Offers a controlled challenge for new algorithmic paradigms targeting graph-driven layout generation, multi-object relation modeling, and instruction-to-scene generalization.

The representational granularity, combined with both automatic and manual layout validation, positions MesaTask-10K as a foundation for research in grounded reasoning, physically aware learning, and interactive scene modeling.

6. Integration With Scalable Long-Sequence Modeling

The scale and heterogeneity of MesaTask-10K necessitate advanced sequence modeling techniques for efficient learning from long records (potentially up to 10,000 tokens per trajectory or user history). Recent work demonstrates the utility of Stacked Target-to-History Cross Attention (STCA), Request Level Batching (RLB), and length-extrapolative training strategies for this regime (Guan et al., 8 Nov 2025):

• STCA: Reduces attention complexity from quadratic to linear in sequence length, making it computationally feasible to process 10,000-token contexts.
• RLB: Batches multiple tasks/queries for a single context, improving memory usage and throughput without biasing the optimization.
• Length-Extrapolative Training: Exposes models to variable-length truncated sequences (mean ≈2,000, occasional samples at full 10,000) during training, facilitating robust generalization across long contexts with reduced training cost.

Applying these approaches directly to MesaTask-10K preserves full gradient flow across long trajectories, with empirical evidence for monotonic improvement in task metrics as context length scales. Limits of STCA for certain within-history dependencies suggest supplementing with local attention layers or convolutional modules when high-order sequential transitions are critical.

7. Limitations and Future Directions

Known constraints include:

• Manual Annotation Bottleneck: Each scene refinement requires 10–20 minutes of expert effort, impacting scalability for even larger or more diverse domains.
• Physical Simulation Fidelity: While IsaacSim validation eliminates basic collisions, some real-world physical affordances (viscosity, deformables) are not modeled.
• Fixed Taxonomy: The 200+ object categories are comprehensive for tabletop domains but may underrepresent rare or emerging manipulation targets.
• Hierarchical Model Extensions: For ultra-long sequences or layouts, hierarchical batching or memory modules could further improve scalability and context adaptation (Guan et al., 8 Nov 2025).

Future work includes expanding object and relation taxonomies, incorporating richer physics, and benchmarking novel generative or graph-based scene reasoning models operating under the challenging, multi-relational constraints exemplified by MesaTask-10K (Hao et al., 26 Sep 2025).