---
title: 'ReSem3D: Semantic 3D Constraint Manipulation'
url: https://www.emergentmind.com/topics/resem3d
type: topic
---

# ReSem3D: Semantic 3D Constraint Manipulation

ReSem3D is a robotic manipulation framework for semantically diverse environments that uses the synergy between Vision Foundation Models and Multimodal Large Language Models to achieve fine-grained visual grounding and to dynamically construct hierarchical 3D spatial constraints for real-time manipulation. It is introduced as a response to three limitations in existing methods: coarse semantic granularity in constraint modeling, lack of real-time closed-loop planning, and compromised robustness in semantically diverse environments. The framework is driven by hierarchical recursive reasoning in MLLMs, interacts with VFMs to construct 3D spatial constraints from natural language instructions and RGB-D observations in two stages, and encodes those constraints as real-time optimization objectives in joint space, with reported zero-shot performance in semantically rich household and sparse chemical lab environments [2507.18262].

## 1. Problem setting and system definition

The central object in ReSem3D is the semantics-driven 3D spatial constraint, which aligns high-level semantic representations with low-level action spaces. In the formulation reported for each subtask \(L_i\), the system receives an RGB-D observation \(O_i\in\mathbb R^{H\times W\times 4}\), extracts an initial constraint set \(\mathcal C_i^{\rm init}\), refines it into \(\mathcal C_i\), and then uses the resulting constraint in a real-time control loop.

The paper characterizes the framework as unified manipulation for semantically diverse environments. The phrase is technically significant because the experiments explicitly span semantically rich household environments and sparse chemical lab environments. This suggests that the intended generalization axis is not limited to object category variation alone, but includes variation in the density and structure of semantic cues available to the grounding pipeline.

A key distinction in ReSem3D is that semantic grounding is not treated as a one-shot localization problem. Instead, the framework constructs a two-stage hierarchy of 3D spatial constraints, first at the part level and then at the region level. The paper therefore positions semantic understanding, geometric localization, and low-level execution as a single coupled pipeline rather than as separate modules connected only by fixed keypoints or open-loop plans.

## 2. Two-stage hierarchical constraint construction

For each subtask \(L_i\), ReSem3D constructs a two-stage hierarchy of 3D spatial constraints \(\mathcal C_i^{\rm init}\), consisting of part-level extraction followed by region-level refinement.

In the part-level extraction stage, a Vision Foundation Model, specifically FastSAM, produces an initial mask set
\[
M=\{\,m_1,\dots,m_g\},\quad m_j\in\{0,1\}^{H\times W}\,.
\]
This set is then filtered by area and containment constraints:
\[
M_{\rm area}=\{m_j\mid \alpha\,A_{\rm img}\le{\rm Area}(m_j)\le\beta\,A_{\rm img}\},
\]
\[
M_{\rm contain}=\{m_j\mid N_{\rm sub}(m_j)<3\},\quad
N_{\rm sub}(m_j)=|\{\,m_k\mid m_k\subset m_j\}|\,, 
\]
\[
M_{\rm filtered}=M_{\rm area}\cap M_{\rm contain}\,.
\]
The filtered masks are then clustered semantically using DBSCAN:
\[
M_{\rm cluster}=\mathrm{DBSCAN}(M_{\rm filtered})
=\{\tilde m_1,\dots,\tilde m_e\}\,.
\]

For each clustered mask \(\tilde m_j\), the centroid is computed as
\[
\mathbf c_j
=\frac1{|\tilde m_j|}\sum_{(x,y)\in \tilde m_j}\begin{bmatrix}x\\y\end{bmatrix}.
\]
Numeric labels \(z_j\in\{0,\dots,q-1\}\) are overlaid at these centroids on the RGB image, and the combined visual prompt together with instruction \(L_i\) is provided to the MLLM. The MLLM selects a subset of centroids, and the corresponding 2D coordinates \(\{\mathbf c_{j_k}\}\) are lifted to 3D via depth \(I_{\rm depth}(x,y)\), producing the initial part-level constraint set \(\mathcal C_i^{\rm init}\).

The functional role of this stage is to reduce raw scene observations into semantically indexed candidate constraints. The paper’s emphasis on dual filtering and clustering indicates that the front end is not merely segmenting all visible content; it is shaping the candidate set before language-conditioned selection. A plausible implication is that this preconditioning is meant to reduce the burden on the MLLM by removing masks that are either too small, too large, or structurally over-nested.

## 3. Region-level refinement and fine-grained semantic grounding

Once the MLLM has selected a mask \(\tilde m_j\) and centroid \(\mathbf c_j\), ReSem3D applies one of two refinement modes: geometric refinement or positional refinement. This explicit bifurcation is central to the framework’s claim of fine-grained semantic grounding.

In geometric refinement, the selected mask is first normalized by cropping it to its bounding box \(\bigl[t_j{:}b_j,\;l_j{:}r_j\bigr]\), where
\[
\mathcal S_j=\{(x,y)\mid \tilde m_j(x,y)=1\}\,,
\]
with \(t_j=\min y,\;b_j=\max y,\;l_j=\min x,\;r_j=\max x\). The crop is resized via nearest-neighbor using factors
\[
\alpha_x=W_r/(r_j-l_j+1),\;\alpha_y=H_r/(b_j-t_j+1)
\]
and centered on a canvas \(H_t\times W_t\).

The normalized mask is then split into \(G_h\times G_w\) grids of size
\[
\omega_c=W_t/G_w,\qquad h_c=H_t/G_h.
\]
For each grid \(\mathcal G^j_{k,s}\), foreground density is computed as
\[
\rho^j_{k,s}=|\{(x,y)\in\mathcal G^j_{k,s}\mid \tilde m_j^t(x,y)=1\}|/(\omega_c\,h_c)\,.
\]
If \(\rho^j_{k,s}>\tau\), the grid receives a label \(\ell^j_{k,s}\), with centroid
\[
\mathbf c_{k,s}^j
=\frac1{|\mathcal P^j_{k,s}|}\sum_{(x,y)\in\mathcal P^j_{k,s}}\begin{bmatrix}x\\y\end{bmatrix}.
\]
The original centroid \(\mathbf c_j\) is associated to its grid label \(\ell_{k,s}^j\) through inverse mapping,
\[
\bar x_j=(x_j-l_j)/\alpha_x+\Delta_x,\qquad \bar y_j=(y_j-t_j)/\alpha_y+\Delta_y.
\]
The MLLM then reads the original RGB image, the original mask, the target mask with grid labels, and the numeric prompt \(z_j\), and returns refined labels \(\hat\ell_{k,s}^j\). These are mapped back to \(\mathbf c_{k,s}^j\), projected back into the original 2D image, and then lifted to 3D via \(I_{\rm depth}\).

Positional refinement is used for open-top or symmetric objects. In this mode, edge pixels are extracted:
\[
\mathcal M_j=\{(x,y)\mid \tilde m_j^t(x,y)=1,\,\exists (x',y')\in\mathcal N_4(x,y)\colon \tilde m_j^t(x',y')=0\}\,.
\]
These are lifted to 3D:
\[
\mathcal P_j=\{\mathcal F_{3D}(x,y,I_{\rm depth})\mid(x,y)\in\mathcal M_j\}\,.
\]
Kernel density estimation is then applied to the \(z\)-heights \(\{z_i\}\),
\[
\hat f(z)=(1/Nh)\sum_i K((z-z_i)/h),\qquad z^*=\arg\max\hat f,
\]
retaining first the subset satisfying \(|z_i-z^*|\le\delta\) to form \(\mathcal P_j^{\rm KDE}\), and then the subset satisfying \(z_i\ge z_{\max}-\eta\) to form \(\mathcal P_j^{\rm peak}\) and \(\mathcal M_j^{\rm peak}\). A symmetric pair is selected by
\[
(\mathbf u_a^*,\mathbf u_b^*)
=\arg\min_{u_a,u_b\in\mathcal M_j^{\rm peak}}
\bigl\|\tfrac{u_a+u_b}2-\mathbf c_j\bigr\|_2,
\]
and the refined 3D point is
\[
\mathcal C_i^{\rm init}=\tfrac12(\mathbf p_a+\mathbf p_b)\,.
\]

The paper’s two refinement modes clarify that ReSem3D does not assume a single geometry class for all manipulation targets. Geometric refinement addresses finer localized regions within a selected part, whereas positional refinement addresses open-top or symmetric objects through edge structure and height statistics. A common simplification would be to describe the system as only a mask-selection pipeline; the explicit refinement procedures show that this description would be incomplete.

## 4. Recursive reasoning, program structure, and closed-loop control

ReSem3D’s language-model program uses a layered “Code-as-Policy” architecture with seven modules: TaskPlanner, ConstraintExtraction, ConstraintRefinement, PreconditionsBuilding, CostFunctionBuilding, SubtaskExecution, and PostconditionsBuilding [2507.18262]. The top-level workflow takes instruction \(L=\{L_1,\dots,L_n\}\) and observation \(O_0\), iteratively obtains a subtask, extracts and refines constraints, builds preconditions and a cost function, executes motion planning, and backtracks to \(i-1\) when the result is `FAIL_PRE`; otherwise it updates the observation and continues.

The VFM→MLLM loop is defined identically for both ConstraintExtraction and ConstraintRefinement: the VFM segments and filters masks into a visual prompt, the MLLM reads the prompt together with the instruction and returns numeric or spatial labels, and these labels are converted into 3D constraints. This organization is important because it places the MLLM not at the very beginning or very end of the pipeline, but inside an iterative perception-reasoning loop.

For each subtask \(i\), the end-effector pose \(\mathbf T_e(t)\in SE(3)\) is governed by the robot model \(S\) and the joint-velocity command \(\mathbf v(t)\in\mathbb R^n\). At each time \(t\), the controller solves
\[
\arg\min_{\mathbf v(t)}\;\mathcal J\bigl(\mathbf T_e(t),\mathcal C_i\bigr)
\quad
\text{s.t.}
\begin{cases}
\mathcal C_i^{\rm init}=f(L_i,O_i),\\
\mathcal C_i = g\bigl(\mathcal C_i^{\rm init},\mathcal E_i\bigr),\\
\epsilon_{\rm pre}(\mathbf T_e(t),\mathcal C_i)\le\varepsilon_{\rm pre},\\
\epsilon_{\rm post}(\mathbf T_e(t),\mathcal C_i)\le\varepsilon_{\rm post},\\
(\mathbf v(t),\mathbf T_e(t))\in\mathrm{Feasible}(S).
\end{cases}
\]
The per-step cost is
\[
\mathcal J(\mathbf T_e,\mathcal C)
=\lambda_p\,d_p\bigl(P(\mathbf T_e),P(\mathcal C)\bigr)
+\lambda_r\,d_r\bigl(R(\mathbf T_e),R(\mathcal C)\bigr)
+\lambda_c\,\mathcal D_c(\mathbf T_e)\,,
\]
where \(d_p(p_1,p_2)=\|p_1-p_2\|_2\) and
\[
d_r(q_1,q_2)=2\arccos(\max(-1,\min(1,\langle q_1,q_2\rangle))).
\]

The motion planner is MPPI. At each \(t\), it samples \(K\) velocity sequences \(\{\mathbf V_k\}\), simulates
\[
\mathbf x_k(t+1)=S(\mathbf x_k(t),\mathbf v_k(t)),
\]
computes rollout costs
\[
C_k=\sum_{t=0}^{T-1}\mathcal J\bigl(\varphi(\mathbf x_k(t)),\mathcal C_i\bigr),
\]
and forms weights
\[
\omega_k\propto\exp[-(C_k-\min C)/\beta],\qquad
\mathbf U^*=\sum_k\omega_k\mathbf V_k.
\]
The first control \(\mathbf u_0^*\) is applied and the procedure iterates at 15 Hz.

Dynamic encoding of semantic constraints is realized through
\[
\mathcal C_i=g\bigl(\mathcal C_i^{\rm init},\,\mathcal E_i\bigr),
\]
where external evolution or disturbance \(\mathcal E_i\) is tracked using TAPNext at approximately 20 Hz. The updated constraint enters the MPPI cost at each iteration, enabling reaction to moving objects or shifted targets in real time. Within the paper’s formulation, closed-loop semantics and closed-loop control are therefore directly coupled: the constraint itself is a time-varying object.

## 5. Experimental regime and reported performance

The experimental program includes both simulation and real-world evaluation. The real-world platform is a UR5e with a two-finger gripper and an Intel D435i. The simulation environments are in Omnigibson with UR5e and Franka Panda. The evaluation includes 12 real-world tasks, comprising 6 household and 6 chemical tasks, with 10 trials each and success rate measured in both “Static” and “Disturbance” settings. It also includes 10 simulated tasks, comprising 4 household and 6 chemical tasks, again with 10 trials each, as well as visual-grounding sub-experiments at object scales \(\{150\%,70\%,40\%\}\) [2507.18262].

The baselines are VoxPoser, described as VLM+LLM \(\rightarrow\) 3D value maps; ReKep, described as LLM keypoints + optimization; and CoPa, described as part-level constraint prompting + planning. The baseline selection is notable because each comparison targets a different alternative design choice: value-map construction, keypoint-based optimization, and part-level prompting.

The reported quantitative results are as follows.

| Setting | Reported results |
|---|---|
| Simulation, static, household success | VoxPoser 20%, ReKep 42.5%, ReSem3D 70% |
| Simulation, static, chemical lab success | VoxPoser 36.6%, ReKep 10%, ReSem3D 65% |
| Real world, static/disturbance, household | VoxPoser 16.6/13.3%, ReKep 31.6/20.0%, ReSem3D 58.3/43.3% |
| Real world, static/disturbance, chemical lab | VoxPoser 13.3/8.3%, ReKep 15.0/6.6%, ReSem3D 60.0/46.6% |
| Visual grounding, real | CoPa 2.5% vs. ReSem3D 60% success |

For simulation visual grounding on Vase, Toaster, and Pestle at \(150\%/70\%/40\%\) scales, ReSem3D outperforms ReKep by 3–8 trials out of 10. Figures 7–9 are reported to show that part-level points and region-level refinements remain robust under scale changes, and that ReSem3D’s closed-loop trajectories recover from perturbations faster than baselines, with 10–15 Hz versus 5–10 Hz.

The paper interprets these results as evidence that ReSem3D performs diverse manipulation tasks under zero-shot conditions and exhibits strong adaptability and generalization. A plausible implication is that the gains do not arise from a single subsystem alone, because the experimental claims combine improvements in visual grounding, disturbance response, and cross-environment manipulation.

## 6. Failure modes, limitations, and implications

The paper attributes ReSem3D’s strengths to three factors: two-stage constraint refinement for fine-grained semantic grounding, MPPI in joint space at 15 Hz for reactive closed-loop control, and MLLM-driven TAMP that automatically decomposes tasks, generates preconditions and postconditions, and backtracks when violated [2507.18262]. These strengths summarize the intended integration of semantic reasoning, task structure, and control.

The reported failure analysis distributes failures across five causes: VFM segmentation misses relevant parts, 7%; MLLM extraction errors, 16%; MLLM refinement errors, 28%; MPPI slow convergence under large pose error, 11%; and TAPNext tracking breaks under occlusion, 38%. The largest category is therefore not initial segmentation or low-level optimization but tracking failure under occlusion. This suggests that, within the tested pipeline, online constraint updating is especially sensitive to the reliability of the disturbance-tracking channel.

The stated limitations and future work are threefold: incorporating LLM-based self-correction loops for erroneous constraints, employing global optimizers for multi-objective convergence, and fusing multi-view RGB-D for more reliable depth on transparent objects. These limitations are consistent with the identified failure modes. Self-correction targets extraction and refinement errors, global optimization targets convergence issues, and multi-view RGB-D targets depth reliability beyond the single-view setting.

A common point of confusion would be to treat ReSem3D primarily as a perception framework because of its use of VFMs and MLLMs. The paper’s actual contribution is broader: a refinable two-stage constraint hierarchy, a layered “Code-as-Policy” program, and a real-time MPPI controller are all part of the system definition. Conversely, it would also be incomplete to interpret the method only as a controller with semantic inputs, because the reported gains in visual grounding and robustness under scale changes depend explicitly on the part-level and region-level grounding procedures.

Taken together, the framework defines a specific research direction in which natural-language instructions, RGB-D observations, fine-grained grounding, and joint-space control are coupled through a dynamic constraint representation. The paper’s own formulation supports the view that ReSem3D is best understood not as a single algorithmic primitive, but as an integrated manipulation architecture built around refinable semantic 3D constraints.

Source: https://www.emergentmind.com/topics/resem3d