ReSem3D: Semantic 3D Constraint Manipulation

• ReSem3D is a robotic manipulation framework that dynamically constructs fine-grained 3D semantic constraints using Vision Foundation Models and Multimodal LLMs.
• It employs a two-stage hierarchy—part-level extraction followed by region-level refinement—to enable closed-loop control in semantically diverse settings.
• Experimental evaluations in household and chemical lab environments demonstrate its superior performance, adaptability, and robust real-time manipulation.

ReSem3D is a robotic manipulation framework for semantically diverse environments that uses the synergy between Vision Foundation Models and Multimodal LLMs to achieve fine-grained visual grounding and to dynamically construct hierarchical 3D spatial constraints for real-time manipulation. It is introduced as a response to three limitations in existing methods: coarse semantic granularity in constraint modeling, lack of real-time closed-loop planning, and compromised robustness in semantically diverse environments. The framework is driven by hierarchical recursive reasoning in MLLMs, interacts with VFMs to construct 3D spatial constraints from natural language instructions and RGB-D observations in two stages, and encodes those constraints as real-time optimization objectives in joint space, with reported zero-shot performance in semantically rich household and sparse chemical lab environments (Su et al., 24 Jul 2025).

1. Problem setting and system definition

The central object in ReSem3D is the semantics-driven 3D spatial constraint, which aligns high-level semantic representations with low-level action spaces. In the formulation reported for each subtask $L_i$, the system receives an RGB-D observation $O_i\in\mathbb R^{H\times W\times 4}$, extracts an initial constraint set $\mathcal C_i^{\rm init}$, refines it into $\mathcal C_i$, and then uses the resulting constraint in a real-time control loop.

The paper characterizes the framework as unified manipulation for semantically diverse environments. The phrase is technically significant because the experiments explicitly span semantically rich household environments and sparse chemical lab environments. This suggests that the intended generalization axis is not limited to object category variation alone, but includes variation in the density and structure of semantic cues available to the grounding pipeline.

A key distinction in ReSem3D is that semantic grounding is not treated as a one-shot localization problem. Instead, the framework constructs a two-stage hierarchy of 3D spatial constraints, first at the part level and then at the region level. The paper therefore positions semantic understanding, geometric localization, and low-level execution as a single coupled pipeline rather than as separate modules connected only by fixed keypoints or open-loop plans.

2. Two-stage hierarchical constraint construction

For each subtask $L_i$, ReSem3D constructs a two-stage hierarchy of 3D spatial constraints $\mathcal C_i^{\rm init}$, consisting of part-level extraction followed by region-level refinement.

In the part-level extraction stage, a Vision Foundation Model, specifically FastSAM, produces an initial mask set

$$M=\{\,m_1,\dots,m_g\},\quad m_j\in\{0,1\}^{H\times W}\,.$$

This set is then filtered by area and containment constraints: $$M_{\rm area}=\{m_j\mid \alpha\,A_{\rm img}\le{\rm Area}(m_j)\le\beta\,A_{\rm img}\},$$

$$M_{\rm contain}=\{m_j\mid N_{\rm sub}(m_j)<3\},\quad N_{\rm sub}(m_j)=|\{\,m_k\mid m_k\subset m_j\}|\,,$$

$$M_{\rm filtered}=M_{\rm area}\cap M_{\rm contain}\,.$$

The filtered masks are then clustered semantically using DBSCAN: $O_i\in\mathbb R^{H\times W\times 4}$0

For each clustered mask $O_i\in\mathbb R^{H\times W\times 4}$1, the centroid is computed as

$O_i\in\mathbb R^{H\times W\times 4}$2

Numeric labels $O_i\in\mathbb R^{H\times W\times 4}$3 are overlaid at these centroids on the RGB image, and the combined visual prompt together with instruction $O_i\in\mathbb R^{H\times W\times 4}$4 is provided to the MLLM. The MLLM selects a subset of centroids, and the corresponding 2D coordinates $O_i\in\mathbb R^{H\times W\times 4}$5 are lifted to 3D via depth $O_i\in\mathbb R^{H\times W\times 4}$6, producing the initial part-level constraint set $O_i\in\mathbb R^{H\times W\times 4}$7.

The functional role of this stage is to reduce raw scene observations into semantically indexed candidate constraints. The paper’s emphasis on dual filtering and clustering indicates that the front end is not merely segmenting all visible content; it is shaping the candidate set before language-conditioned selection. A plausible implication is that this preconditioning is meant to reduce the burden on the MLLM by removing masks that are either too small, too large, or structurally over-nested.

3. Region-level refinement and fine-grained semantic grounding

Once the MLLM has selected a mask $O_i\in\mathbb R^{H\times W\times 4}$8 and centroid $O_i\in\mathbb R^{H\times W\times 4}$9, ReSem3D applies one of two refinement modes: geometric refinement or positional refinement. This explicit bifurcation is central to the framework’s claim of fine-grained semantic grounding.

In geometric refinement, the selected mask is first normalized by cropping it to its bounding box $\mathcal C_i^{\rm init}$0, where

$\mathcal C_i^{\rm init}$1

with $\mathcal C_i^{\rm init}$2. The crop is resized via nearest-neighbor using factors

$\mathcal C_i^{\rm init}$3

and centered on a canvas $\mathcal C_i^{\rm init}$4.

The normalized mask is then split into $\mathcal C_i^{\rm init}$5 grids of size

$\mathcal C_i^{\rm init}$6

For each grid $\mathcal C_i^{\rm init}$7, foreground density is computed as

$\mathcal C_i^{\rm init}$8

If $\mathcal C_i^{\rm init}$9, the grid receives a label $\mathcal C_i$0, with centroid

$\mathcal C_i$1

The original centroid $\mathcal C_i$2 is associated to its grid label $\mathcal C_i$3 through inverse mapping,

$\mathcal C_i$4

The MLLM then reads the original RGB image, the original mask, the target mask with grid labels, and the numeric prompt $\mathcal C_i$5, and returns refined labels $\mathcal C_i$6. These are mapped back to $\mathcal C_i$7, projected back into the original 2D image, and then lifted to 3D via $\mathcal C_i$8.

Positional refinement is used for open-top or symmetric objects. In this mode, edge pixels are extracted: $\mathcal C_i$9 These are lifted to 3D: $L_i$0 Kernel density estimation is then applied to the $L_i$1-heights $L_i$2,

$L_i$3

retaining first the subset satisfying $L_i$4 to form $L_i$5, and then the subset satisfying $L_i$6 to form $L_i$7 and $L_i$8. A symmetric pair is selected by

$L_i$9

and the refined 3D point is

$\mathcal C_i^{\rm init}$0

The paper’s two refinement modes clarify that ReSem3D does not assume a single geometry class for all manipulation targets. Geometric refinement addresses finer localized regions within a selected part, whereas positional refinement addresses open-top or symmetric objects through edge structure and height statistics. A common simplification would be to describe the system as only a mask-selection pipeline; the explicit refinement procedures show that this description would be incomplete.

4. Recursive reasoning, program structure, and closed-loop control

ReSem3D’s language-model program uses a layered “Code-as-Policy” architecture with seven modules: TaskPlanner, ConstraintExtraction, ConstraintRefinement, PreconditionsBuilding, CostFunctionBuilding, SubtaskExecution, and PostconditionsBuilding (Su et al., 24 Jul 2025). The top-level workflow takes instruction $\mathcal C_i^{\rm init}$1 and observation $\mathcal C_i^{\rm init}$2, iteratively obtains a subtask, extracts and refines constraints, builds preconditions and a cost function, executes motion planning, and backtracks to $\mathcal C_i^{\rm init}$3 when the result is FAIL_PRE; otherwise it updates the observation and continues.

The VFM→MLLM loop is defined identically for both ConstraintExtraction and ConstraintRefinement: the VFM segments and filters masks into a visual prompt, the MLLM reads the prompt together with the instruction and returns numeric or spatial labels, and these labels are converted into 3D constraints. This organization is important because it places the MLLM not at the very beginning or very end of the pipeline, but inside an iterative perception-reasoning loop.

For each subtask $\mathcal C_i^{\rm init}$4, the end-effector pose $\mathcal C_i^{\rm init}$5 is governed by the robot model $\mathcal C_i^{\rm init}$6 and the joint-velocity command $\mathcal C_i^{\rm init}$7. At each time $\mathcal C_i^{\rm init}$8, the controller solves

$\mathcal C_i^{\rm init}$9

The per-step cost is

$$M=\{\,m_1,\dots,m_g\},\quad m_j\in\{0,1\}^{H\times W}\,.$$0

where $$M=\{\,m_1,\dots,m_g\},\quad m_j\in\{0,1\}^{H\times W}\,.$$1 and

$$M=\{\,m_1,\dots,m_g\},\quad m_j\in\{0,1\}^{H\times W}\,.$$2

The motion planner is MPPI. At each $$M=\{\,m_1,\dots,m_g\},\quad m_j\in\{0,1\}^{H\times W}\,.$$3, it samples $$M=\{\,m_1,\dots,m_g\},\quad m_j\in\{0,1\}^{H\times W}\,.$$4 velocity sequences $$M=\{\,m_1,\dots,m_g\},\quad m_j\in\{0,1\}^{H\times W}\,.$$5, simulates

$$M=\{\,m_1,\dots,m_g\},\quad m_j\in\{0,1\}^{H\times W}\,.$$6

computes rollout costs

$$M=\{\,m_1,\dots,m_g\},\quad m_j\in\{0,1\}^{H\times W}\,.$$7

and forms weights

$$M=\{\,m_1,\dots,m_g\},\quad m_j\in\{0,1\}^{H\times W}\,.$$8

The first control $$M=\{\,m_1,\dots,m_g\},\quad m_j\in\{0,1\}^{H\times W}\,.$$9 is applied and the procedure iterates at 15 Hz.

Dynamic encoding of semantic constraints is realized through

$$M_{\rm area}=\{m_j\mid \alpha\,A_{\rm img}\le{\rm Area}(m_j)\le\beta\,A_{\rm img}\},$$0

where external evolution or disturbance $$M_{\rm area}=\{m_j\mid \alpha\,A_{\rm img}\le{\rm Area}(m_j)\le\beta\,A_{\rm img}\},$$1 is tracked using TAPNext at approximately 20 Hz. The updated constraint enters the MPPI cost at each iteration, enabling reaction to moving objects or shifted targets in real time. Within the paper’s formulation, closed-loop semantics and closed-loop control are therefore directly coupled: the constraint itself is a time-varying object.

5. Experimental regime and reported performance

The experimental program includes both simulation and real-world evaluation. The real-world platform is a UR5e with a two-finger gripper and an Intel D435i. The simulation environments are in Omnigibson with UR5e and Franka Panda. The evaluation includes 12 real-world tasks, comprising 6 household and 6 chemical tasks, with 10 trials each and success rate measured in both “Static” and “Disturbance” settings. It also includes 10 simulated tasks, comprising 4 household and 6 chemical tasks, again with 10 trials each, as well as visual-grounding sub-experiments at object scales $$M_{\rm area}=\{m_j\mid \alpha\,A_{\rm img}\le{\rm Area}(m_j)\le\beta\,A_{\rm img}\},$$2 (Su et al., 24 Jul 2025).

The baselines are VoxPoser, described as VLM+LLM $$M_{\rm area}=\{m_j\mid \alpha\,A_{\rm img}\le{\rm Area}(m_j)\le\beta\,A_{\rm img}\},$$3 3D value maps; ReKep, described as LLM keypoints + optimization; and CoPa, described as part-level constraint prompting + planning. The baseline selection is notable because each comparison targets a different alternative design choice: value-map construction, keypoint-based optimization, and part-level prompting.

The reported quantitative results are as follows.

Setting	Reported results
Simulation, static, household success	VoxPoser 20%, ReKep 42.5%, ReSem3D 70%
Simulation, static, chemical lab success	VoxPoser 36.6%, ReKep 10%, ReSem3D 65%
Real world, static/disturbance, household	VoxPoser 16.6/13.3%, ReKep 31.6/20.0%, ReSem3D 58.3/43.3%
Real world, static/disturbance, chemical lab	VoxPoser 13.3/8.3%, ReKep 15.0/6.6%, ReSem3D 60.0/46.6%
Visual grounding, real	CoPa 2.5% vs. ReSem3D 60% success

For simulation visual grounding on Vase, Toaster, and Pestle at $$M_{\rm area}=\{m_j\mid \alpha\,A_{\rm img}\le{\rm Area}(m_j)\le\beta\,A_{\rm img}\},$$4 scales, ReSem3D outperforms ReKep by 3–8 trials out of 10. Figures 7–9 are reported to show that part-level points and region-level refinements remain robust under scale changes, and that ReSem3D’s closed-loop trajectories recover from perturbations faster than baselines, with 10–15 Hz versus 5–10 Hz.

The paper interprets these results as evidence that ReSem3D performs diverse manipulation tasks under zero-shot conditions and exhibits strong adaptability and generalization. A plausible implication is that the gains do not arise from a single subsystem alone, because the experimental claims combine improvements in visual grounding, disturbance response, and cross-environment manipulation.

6. Failure modes, limitations, and implications

The paper attributes ReSem3D’s strengths to three factors: two-stage constraint refinement for fine-grained semantic grounding, MPPI in joint space at 15 Hz for reactive closed-loop control, and MLLM-driven TAMP that automatically decomposes tasks, generates preconditions and postconditions, and backtracks when violated (Su et al., 24 Jul 2025). These strengths summarize the intended integration of semantic reasoning, task structure, and control.

The reported failure analysis distributes failures across five causes: VFM segmentation misses relevant parts, 7%; MLLM extraction errors, 16%; MLLM refinement errors, 28%; MPPI slow convergence under large pose error, 11%; and TAPNext tracking breaks under occlusion, 38%. The largest category is therefore not initial segmentation or low-level optimization but tracking failure under occlusion. This suggests that, within the tested pipeline, online constraint updating is especially sensitive to the reliability of the disturbance-tracking channel.

The stated limitations and future work are threefold: incorporating LLM-based self-correction loops for erroneous constraints, employing global optimizers for multi-objective convergence, and fusing multi-view RGB-D for more reliable depth on transparent objects. These limitations are consistent with the identified failure modes. Self-correction targets extraction and refinement errors, global optimization targets convergence issues, and multi-view RGB-D targets depth reliability beyond the single-view setting.

A common point of confusion would be to treat ReSem3D primarily as a perception framework because of its use of VFMs and MLLMs. The paper’s actual contribution is broader: a refinable two-stage constraint hierarchy, a layered “Code-as-Policy” program, and a real-time MPPI controller are all part of the system definition. Conversely, it would also be incomplete to interpret the method only as a controller with semantic inputs, because the reported gains in visual grounding and robustness under scale changes depend explicitly on the part-level and region-level grounding procedures.

Taken together, the framework defines a specific research direction in which natural-language instructions, RGB-D observations, fine-grained grounding, and joint-space control are coupled through a dynamic constraint representation. The paper’s own formulation supports the view that ReSem3D is best understood not as a single algorithmic primitive, but as an integrated manipulation architecture built around refinable semantic 3D constraints.