MesaTask: 3D Tabletop Scene Generation

• MesaTask is a framework that transforms natural language instructions into detailed 3D tabletop scene layouts for robotic training and simulation.
• It employs a three-stage spatial reasoning pipeline—object inference, spatial relation reasoning, and scene graph construction—to ensure task-conforming and realistic scenes.
• The framework leverages supervised fine-tuning and direct preference optimization to reduce errors, enhance realism, and improve data efficiency.

MesaTask is a framework for task-driven 3D tabletop scene generation via LLM-based spatial reasoning. It addresses the translation of high-level, natural-language manipulation instructions into plausible tabletop layouts suitable for robotic training and simulation. Traditional methods for creating such scenes—manual layout design or random sampling—either lack efficiency or fail to yield task-conforming, realistic environments. MesaTask formalizes the challenge as task-oriented tabletop scene generation and introduces algorithmic, architectural, and dataset contributions to bridge the semantic gap between abstract tasks and detailed 3D scene representations (Hao et al., 26 Sep 2025).

1. Formal Problem Definition

The input to MesaTask is a natural language manipulation instruction $T$ (e.g., “Organize the nuts from the jar into bowls and place the tray to the left of the lamp”). The framework aims to generate a 3D tabletop scene $S$ composed of $N$ objects with a layout $L = \{\ell_1, \ldots, \ell_N\}$. Each object layout $\ell_i$ is represented as $[p_i, s_i, \theta_i, t_i]$, where:

• $p_i \in \mathbb{R}^3$: 3D position
• $s_i \in \mathbb{R}^3$: axis-aligned bounding box size
• $\theta_i \in [0, 2\pi)$: yaw around the vertical axis
• $t_i$: textual description (category and attributes)

A pretrained LLM expands $S$0 into structured components:

• $S$1: Environment (table type, free placement zones, etc.)
• $S$2: Ordered subgoals
• $S$3: Seed list of task-relevant objects

The generative model $S$4 then computes the layout: $S$5. Objects matching $S$6 are retrieved from a 3D asset library to instantiate the final scene $S$7.

2. Spatial Reasoning Chain

MesaTask decomposes scene generation into a three-stage spatial reasoning pipeline to systematically ground task semantics into physical layouts:

2.1 Object Inference

From $S$8, the LLM infers a complete object set $S$9 according to:

$N$0

The LLM is prompted to enumerate all objects necessary to fulfill $N$1, including supplemental, scene-typical props.

2.2 Spatial Interrelation Reasoning

Given $N$2, the framework predicts pairwise spatial relations $N$3. Relations include: LeftOf, RightOf, InFrontOf, Behind, Above, Below, and In.

$N$4

For each candidate relation $N$5:

$N$6

The highest-scoring $N$7 is selected.

2.3 Scene Graph Construction and Layout Generation

An attributed scene graph $N$8 is constructed. Each $N$9 is enriched with grid-based coarse position (e.g., "center-front") and an orientation bin among eight yaw sectors. The LLM predicts for each $L = \{\ell_1, \ldots, \ell_N\}$0 a layout $L = \{\ell_1, \ldots, \ell_N\}$1 with precise 3D position, size, and orientation. Final layout inference is formulated as:

$L = \{\ell_1, \ldots, \ell_N\}$2

3. MesaTask LLM Architecture and Learning

MesaTask employs a decoder-only LLM (Qwen-3-8B), trained in two stages:

3.1 Supervised Fine-Tuning (SFT)

"Reasoning-chain" prompts are constructed by extracting $L = \{\ell_1, \ldots, \ell_N\}$3 from each labelled MesaTask-10K scene using GPT-4o, producing natural language traces, JSON scene graphs, and layouts. The LLM is fine-tuned to map $L = \{\ell_1, \ldots, \ell_N\}$4 in one shot.

Prompt example:

$\ell_i$6

3.2 Direct Preference Optimization (DPO)

Despite SFT, outputs may contain object collisions or omissions. For each $L = \{\ell_1, \ldots, \ell_N\}$5, (positive, negative) layout pairs are generated:

• Positive: ground-truth layout $L = \{\ell_1, \ldots, \ell_N\}$6
• Negative: $L = \{\ell_1, \ldots, \ell_N\}$7, generated by perturbing positions/sizes, corrupting relations in $L = \{\ell_1, \ldots, \ell_N\}$8, or omitting mission-critical objects

The DPO objective is maximized as:

$L = \{\ell_1, \ldots, \ell_N\}$9

where $\ell_i$0 is the current policy, $\ell_i$1 the SFT model, $\ell_i$2 the sigmoid, and $\ell_i$3 a temperature hyperparameter.

4. MesaTask-10K Dataset

MesaTask-10K is a large-scale collection of $\ell_i$4 manually refined tabletop scenes, spanning six canonical table types (office, dining, kitchen counter, coffee table, bathroom vanity, dressing table). The asset library comprises over 12,000 high-fidelity 3D models in 200+ categories, each annotated with semantic and physical metadata (category, description, material, isContainer, onTable). The scenes average 15 objects per scene (standard deviation $\ell_i$5), exhibiting complex inter-object relations (stacking, containment, alignment).

Scene layouts were initially synthesized using text-to-image rendering (FLUX), depth estimation, segmentation (Grounded-SAM), asset retrieval, and then refined through manual annotation and collision testing in IsaacSim.

5. Experimental Protocol and Results

5.1 Metrics

• Success Rate: Percentage of outputs parseable as valid JSON layouts
• FID: Fréchet Inception Distance between renders of generated and ground-truth scenes (lower is better)
• GPT-Score (scale 1–10): Assessed by an LLM on
  • Consistency with Task (CwT)
  • Object Size Reasonableness (OSR)
  • Placement Plausibility & Intersections (PPI)
  • Layout Coherence & Realism (LCR)
  • Object Visibility (OV)
• User Study: Human ratings (1–7) for realism, task alignment, spatial coherence

5.2 Baselines

• GPT-4o zero-shot (with/without explicit chain prompts)
• Holodeck-table (adapted room-scale LLM+optimizer)
• I-Design-table (LLM-driven scene-graph + backtracking placement)

5.3 Results

The table below summarizes major results on 500 test tasks:

Model Name	Success %	FID↓	AvgGPT	User Study
GPT-4o w/o reasoning	91.6	84.3	6.95	3.11
Holodeck-table	99.3	91.3	5.42	2.29
I-Design-table	56.5	96.0	6.87	1.73
MesaTask w/o reasoning	100.0	40.8	8.20	5.43
MesaTask w/o DPO	98.4	41.4	8.23	5.75
MesaTask (full)	99.1	40.3	8.25	6.12

MesaTask achieves the lowest FID and highest multi-dimensional GPT-based scores. DPO reduces collision rates and missing object frequency relative to the SFT-only model.

6. Applications, Limitations, and Future Work

MesaTask enables the automated generation of physically plausible, instruction-aligned tabletop scenes for robotic policy learning and interactive design tools, significantly augmenting data efficiency and scene variety. The current system is domain-confined to six tabletop types and bounded by the diversity of its retrieval-based 3D asset library. Physics plausibility is enforced only by post-simulation validation.

Anticipated research directions include integrating 3D generative models for asset synthesis conditioned on predicted layouts, extending the method to arbitrary furniture and room-scale environments, and incorporating end-to-end physics-aware reasoning into the chain. These improvements address key bottlenecks in asset diversity and layout physicality.

---

References:

• MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning (Hao et al., 26 Sep 2025)