Grasp-MPC: Value-Guided Visual Grasping
- The paper introduces Grasp-MPC, a closed-loop 6-DoF vision-based grasping method that leverages a learned value function with MPC to update grasp execution in real time.
- It integrates initial grasp prediction with recurrent MPC optimization to correct errors from object pose shifts and noisy visual inputs in cluttered scenes.
- Experimental results demonstrate up to 32.6% improvement in simulation and 33.3% higher success in real-world settings compared to open-loop baselines.
Searching arXiv for the Grasp-MPC paper and closely related grasping/MPC work to ground the article in current literature. {"query":"Grasp-MPC Closed-Loop Visual Grasping via Value-Guided Model Predictive Control arXiv", "max_results": 10} {"query":"Grasp-MPC value-guided model predictive control grasping arXiv (Yamada et al., 7 Sep 2025)", "max_results": 5} Grasp-MPC is a closed-loop, 6-DoF vision-based grasping policy designed for robust and reactive grasping of novel objects in cluttered environments. It couples a learned value function, trained on visual observations from a large-scale synthetic dataset of 2 million grasp trajectories that include successful and failed attempts, with model predictive control so that grasp execution can be updated online while also respecting collision avoidance and smoothness objectives. The method is positioned against the failure modes of open-loop grasping—particularly grasp prediction errors and object pose changes during grasping—and is evaluated on FetchBench and real-world settings across diverse environments, where it improves grasp success rates by up to 32.6% in simulation and 33.3% in real-world noisy conditions (Yamada et al., 7 Sep 2025).
1. Problem formulation and conceptual scope
Grasp-MPC addresses grasping of diverse objects in unstructured environments, with particular emphasis on cluttered scenes in which open-loop methods struggle. The motivating observation is that open-loop grasping methods, while effective in controlled settings, fail when the predicted grasp is inaccurate or when the target object moves during execution. Closed-loop alternatives can address such effects, but prior closed-loop methods are described as operating in simplified settings, such as a single object on a table, on a limited set of objects, with no path to generalization (Yamada et al., 7 Sep 2025).
Within this formulation, the central design choice is to use a learned value function as a grasp-execution prior rather than to rely exclusively on a fixed grasp target or a directly extracted policy. The value function estimates expected future cost-to-go from the current robot and local object scene, and the MPC optimizer uses that estimate as one cost term among other model-based costs. This makes Grasp-MPC a hybrid architecture: grasp prediction and motion planning are used to obtain an initial pre-grasp configuration, while the terminal stage of grasp execution is delegated to a reactive, value-guided MPC controller (Yamada et al., 7 Sep 2025).
A common source of ambiguity is the phrase “grasp MPC” as a generic description of model-predictive grasp control. In the present literature, “Grasp-MPC” specifically denotes the method titled “Grasp-MPC: Closed-Loop Visual Grasping via Value-Guided Model Predictive Control,” which is defined by its vision-based value function, CuRobo-based optimization, and closed-loop transition from pre-grasp to final grasp execution (Yamada et al., 7 Sep 2025).
2. Perception, state representation, and execution pipeline
Grasp-MPC relies on visual observations in the form of segmented object point clouds and the robot’s end-effector pose relative to the object. These inputs are used by the value function and, through it, by the MPC controller. The representation is explicitly intended to support generalization to new objects and environments, to handle sensor occlusion and noise, and to react adaptively to changes in the scene, including object movement (Yamada et al., 7 Sep 2025).
In real-world deployments, visual perception is performed via RGB-D sensors and segmentation models such as SAM-Track and Grounding DINO, with the resulting point clouds provided directly to the MPC’s neural value network. The execution process is staged. An initial grasp prediction model and motion planner produce grasp and pre-grasp poses; the robot moves to the pre-grasp pose in open loop; it then switches to Grasp-MPC for closed-loop motion from pre-grasp to final grasp. This division separates global initialization from local reactive correction, and suggests an architecture in which the learned component is concentrated in final approach and error recovery rather than replacing planning outright (Yamada et al., 7 Sep 2025).
| Component | Role | Details |
|---|---|---|
| Grasp prediction model & motion planner | Initial open-loop planning | Used for initial open-loop planning to pre-grasp poses |
| Learned value function | Grasp guidance | Guides the MPC policy toward successful grasps |
| MPC framework | Real-time optimization | CuRobo integrates the value term with safety and smoothness costs |
3. Value function construction, dataset, and supervision
The value function is trained on synthetic grasp trajectories generated in simulation across 8,515 unique Objaverse objects. The dataset contains over 2 million trajectories and 115M states, with up to 256 trajectories per object. The trajectories include both successful (70.2%) and failed (29.8%) attempts; success is defined as reaching a feasible grasp pose from a perturbed pre-grasp, while failure includes infeasible or unreachable poses. Both pre-grasp and grasp poses are perturbed with translation and orientation noise to increase generalization (Yamada et al., 7 Sep 2025).
Architecturally, the value function takes as input the segmented point cloud of the object and the end-effector pose relative to the object centroid. A PointNet++ encoder processes the point cloud, an MLP encodes the end-effector pose, and the resulting features are concatenated and passed to a final MLP to predict the value, interpreted as cost-to-go. A Softplus activation ensures nonnegative outputs. Robustness to observation quality is incorporated during training by using both full and partial point clouds, Gaussian noise injection, and random camera perspectives (Yamada et al., 7 Sep 2025).
The supervision scheme is sparse. Final or near-final states in successful trajectories are labeled with cost $0$, and all other states with cost $1$. Training minimizes Bellman error using
with , and with obtained by exponential moving average of parameters. Training is reported to have been performed on a single RTX 4090 GPU for six days. Due to the data coverage, ensembles for risk aversion did not meaningfully improve performance in ablation (Yamada et al., 7 Sep 2025).
This labeling scheme implies that the network is not trained to regress geometric distance to a grasp pose; instead, it learns a bootstrapped notion of downstream feasibility and progress under the distribution induced by the synthetic dataset. A plausible implication is that the controller inherits robustness from the diversity of failure cases, not only from successful demonstrations.
4. MPC objective, optimization machinery, and closed-loop control
In deployment, the learned value function is injected into an MPC objective. Over a rollout horizon , the cumulative value cost is
and the full optimization problem is
where comprises the default CuRobo costs and $1$0 is set to $1$1 (Yamada et al., 7 Sep 2025).
The CuRobo term includes collision and smoothness objectives. Collision avoidance covers both self-collision and world collision avoidance and is enforced via signed distance fields using NVBlox. Smoothness is implemented through minimum jerk and B-spline smoothing, with acceleration, velocity, and jerk constrained. Control sampling uses a Halton sequence, and a degree-5 B-spline is fitted (Yamada et al., 7 Sep 2025).
Operationally, Grasp-MPC is a receding-horizon controller. At each update, it replans action sequences using the current visual state and the combined model-based and value-based costs. This makes the learned value function a heuristic inside a constraint-aware optimizer rather than a stand-alone policy. The distinction is important: safety-relevant properties such as collision avoidance remain encoded at the control level, which is one reason the method is reported as suitable for real-world deployment (Yamada et al., 7 Sep 2025).
5. Experimental evaluation in simulation and on hardware
Simulation experiments are conducted on FetchBench in Isaac Sim using a UR10 with a Robotiq 2F-140 gripper. The benchmark comprises 5,400 test cases, corresponding to 90 scenes and 60 trials, across on-table, on-shelf, in-basket, and in-drawer tasks. Multiple cameras are available, and the camera with the best object view is used. Success is defined as lifting the object by at least $1$2 cm after grasping. Baselines include OSC as an oracle open-loop baseline with ground-truth grasp pose, a diffusion policy, a transformer policy, and IQL (Yamada et al., 7 Sep 2025).
Under ground-truth grasp poses, Grasp-MPC achieves 74.9% success, compared with 79.0% for OSC, while surpassing the closed-loop baselines, including IQL at 64.4%. Under significant noise in pre-grasp and grasp specification, Grasp-MPC drops by 14% to 60.3%, whereas OSC drops by 40%. With predicted grasps from M2T2, an off-the-shelf grasp predictor trained for a Franka gripper, Grasp-MPC reaches 67.2%, compared with 63.6% for OSC, indicating resilience to grasp prediction errors. When only partial point clouds are available, success decreases by only 1.4%. Across scene types, the method outperforms all non-oracle baselines, with especially marked gains under clutter or shelf scenarios (Yamada et al., 7 Sep 2025).
Real-world experiments use a UR10 with a Robotiq 2F-140 gripper, RGB-D cameras, and NVBlox for signed distance fields. Scenes include an empty table, a cluttered table, and a cluttered shelf, each with 5 objects and evaluated over 3 poses and 2 trials, for 30 trials per scene. Because other learning baselines lack collision avoidance for safe deployment, comparison is restricted to the open-loop CuRobo-GraspAPI baseline. In these experiments, Grasp-MPC achieves up to 33.3% higher success rate than open loop, especially in highly cluttered shelf and table tasks. When the object pose is perturbed after pre-grasp to simulate disturbances, Grasp-MPC maintains 60% success by tracking and adapting to object movement, a behavior not available to open-loop execution (Yamada et al., 7 Sep 2025).
| Setting | Result | Comparator |
|---|---|---|
| Ground-truth grasp poses | 74.9% success | OSC: 79.0% |
| Significant pre-grasp/grasp noise | 60.3% success after a 14% drop | OSC drops by 40% |
| Predicted grasps from M2T2 | 67.2% success | OSC: 63.6% |
6. Relation to adjacent grasping and MPC literature
Grasp-MPC belongs to a broader line of work that combines learned task representations with predictive control, but its precise formulation differs from earlier and parallel approaches. “Neural Motion Fields: Encoding Grasp Trajectories as Implicit Value Functions” encodes object point clouds and relative task trajectories as an implicit value function over $1$3, with a path-length module and collision module queried by sampling-based MPC in STORM (Chen et al., 2022). Grasp-MPC retains the principle of value-guided grasp optimization, but shifts to a Bellman-trained cost-to-go over visual observations, uses CuRobo rather than STORM, and emphasizes cluttered scenes, partial observations, and robustness to noisy grasp prediction (Yamada et al., 7 Sep 2025).
The method is also distinct from tactile-reactive MPC systems. “LeTac-MPC: Learning Model Predictive Control for Tactile-reactive Grasping” uses a GelSight sensor, a neural tactile embedding, and a differentiable MPC layer operating at 25 Hz, and is targeted at dynamic and force-interactive tasks with daily objects (Xu et al., 2024). “Learning-Based Collaborative Control for Bi-Manual Tactile-Reactive Grasping” extends this direction to a multi-agent MPC using two GelSight Mini sensors and centralized collaborative control for objects with different softness and shapes (Giacobbe et al., 24 Sep 2025). These tactile methods and Grasp-MPC share the use of MPC as the execution substrate, but they differ in sensing modality, state representation, and task structure.
A further neighboring direction is robust belief-space grasp planning under multimodal uncertainty. “Variational Neural Belief Parameterizations for Robust Dexterous Grasping under Multimodal Uncertainty” formulates grasp acquisition as a risk-sensitive POMDP, uses a differentiable Gaussian-mixture belief, and optimizes a differentiable CVaR surrogate for dexterous grasp planning (Enwerem et al., 28 Apr 2026). This suggests that the current grasp-MPC landscape spans visual value-guided control, tactile-reactive control, and explicitly risk-sensitive belief-space control, with Grasp-MPC occupying the visual, clutter-aware, closed-loop 6-DoF grasp-execution regime.
7. Limitations, interpretation, and prospective extensions
The reported limitations are explicit. Real-world grasp success rates, although improved, are not 100%. The paper identifies incorporating physics simulation in data labeling or fine-tuning on real-world trajectories as possible ways to improve performance. The method is validated for grasping, and extension to broader manipulation remains to be studied (Yamada et al., 7 Sep 2025).
The paper also argues for value optimization over policy extraction. By combining MPC with a learned value function, Grasp-MPC avoids brittle policy extraction steps that can hinder offline RL approaches such as IQL. At the same time, the method does not treat risk aversion through value ensembles as essential: ensembles were considered, but with the available data coverage they did not meaningfully improve performance; this could be revisited as tasks become more complex (Yamada et al., 7 Sep 2025).
One of the more consequential empirical observations is that the system is trained on “empty scenes” yet handles real-world clutter robustly because of its modular, value-guided design. This suggests that Grasp-MPC’s generalization does not derive from end-to-end scene imitation alone, but from the interaction between a visual cost-to-go estimator and a constraint-aware optimizer. In that sense, Grasp-MPC can be understood as a specific synthesis of data-driven grasp valuation and classical trajectory optimization, with the value function acting as an execution-time objective rather than as a full replacement for model-based control.