Papers
Topics
Authors
Recent
Search
2000 character limit reached

TAMPAgent: Framework for Mobile Manipulation

Updated 6 November 2026
  • TAMPAgent is an agentic planning framework for mobile manipulation in open-world, partially observable environments. It integrates large vision-language models (LVLMs), task-and-motion planning, geometric constraint satisfaction, and perception, grasping, inverse-kinematics, and motion-planning tools.
  • The framework allows the agent to propose semantic hypotheses and partial task plans. The system then uses classical search, perception, and motion-planning tools to verify constraints, ensuring feasibility before execution, especially beneficial in environments with incomplete information, occluded objects, or hidden containment.
  • TAMPAgent achieved 77.2% task success in household kitchen scenarios, outperforming baseline models like CaP and GaP, by effectively integrating LVLM reasoning and classical TAMP tools.

TAMPAgent is an agentic planning framework for mobile manipulation in open-world, partially observable environments. It combines large vision-LLM (LVLM) reasoning over semantic beliefs with task-and-motion planning (TAMP), geometric constraint satisfaction, tool-mediated parameter binding, and closed-loop execution. Rather than requiring a complete world model before planning or allowing an agent to invoke motion tools without certification, TAMPAgent permits the agent to formulate partial task plans and semantic hypotheses, then uses classical search and perception, grasping, inverse-kinematics, and motion-planning tools to ground and certify only the next executable action. The system was evaluated on household mobile-manipulation tasks using Rainbow Robotics RB-Y1 and Unitree G1 platforms and reported higher success than direct tool-calling baselines (Mishra et al., 2 Nov 2026).

1. Problem setting and motivation

TAMPAgent addresses mobile manipulation when the relevant objects, obstacles, receptacles, and spatial relations are initially unknown or only partially observed. Objects may be hidden, occluded, located in another room, behind clutter, or inside containers. The robot must therefore solve both a manipulation problem and an information-gathering problem: it must determine what to observe, where to navigate, which blockers to clear, how to manipulate discovered objects, and when to revise its plan.

Direct LVLM tool-calling systems, including Code as Policies (CaP)-style architectures, can compose perception, grasping, navigation, and manipulation tools, but the paper identifies several limitations. Such systems generally lack an explicit geometric constraint model, provide weak treatment of inter-step dependencies, may behave locally or open loop, and can begin execution before reachability, collision, grasp, or placement constraints have been jointly verified. A LLM may also infer that a grasp, viewpoint, or trajectory exists when robot kinematics or scene geometry make it infeasible.

Classical TAMP provides symbolic preconditions, effects, continuous parameters, inverse-kinematics constraints, collision avoidance, stable-placement constraints, and motion feasibility checks. Its usual formulation, however, assumes a hand-authored planning problem, a fixed object inventory, known predicates and operators, a closed world, and already observed geometric states. These assumptions are unsuitable when the robot does not yet know whether, for example, an apple is on a table, behind a box, or inside a refrigerator.

TAMPAgent therefore treats the LVLM as a belief-space policy. The LVLM proposes semantic hypotheses and partial plans, while a TAMP planner and geometric tools determine which variables must be concretized and which constraints must be satisfied before execution. The resulting architecture is neither unrestricted language-driven tool use nor conventional closed-world TAMP; it is a hybrid in which semantic reasoning and geometric certification are interleaved.

2. Formal TAMP and belief-space representation

In the classical TAMP formulation, the world state sts_t contains variables such as robot configuration, the object held by the robot, object poses, containment relations, and support relations. An action has discrete arguments, continuous parameters, symbolic preconditions, symbolic effects, and geometric constraints. A pick action, for example, may require that the robot hold nothing and may set the holding state to the selected object.

A plan skeleton is a sequence of discrete actions, such as

a⃗=⟨pick[A],place[A,B]⟩,\vec{a}=\left\langle \mathop{\mathrm{pick}}[A],\mathop{\mathrm{place}}[A,B]\right\rangle,

where AA is an object and BB is a support or container. TAMPAgent compiles such a skeleton into a factor graph

Φ(a⃗)=(V,F),\Phi(\vec{a})=(V,F),

where VV contains planning variables such as object poses, robot configurations, grasps, placements, and trajectories, while FF contains constraint factors. The skeleton is feasible when there exists an assignment β\beta satisfying every factor:

∃βsuch that∀f∈F,  f(β)=True.\exists \beta \quad \text{such that} \quad \forall f\in F,\; f(\beta)=\mathrm{True}.

Shared variables encode cross-action dependencies. A grasp selected for picking may also be used by a later placement constraint, so the grasp must support both actions rather than merely enable immediate pickup.

Under partial observability, the robot does not directly observe sts_t. It maintains a belief

a⃗=⟨pick[A],place[A,B]⟩,\vec{a}=\left\langle \mathop{\mathrm{pick}}[A],\mathop{\mathrm{place}}[A,B]\right\rangle,0

where a⃗=⟨pick[A],place[A,B]⟩,\vec{a}=\left\langle \mathop{\mathrm{pick}}[A],\mathop{\mathrm{place}}[A,B]\right\rangle,1 is the history of actions and observations. Conventional belief-space TAMP represents uncertain planning variables with distributions and often requires sampling, inference, integration, and chance constraints. TAMPAgent instead retains high-level semantic uncertainty in language and requires geometric tools to produce execution-critical values.

This design avoids requiring a calibrated probability distribution over every semantic hypothesis. A statement such as “the apple is probably behind the penguin” functions as a language-valued belief that can guide observation and manipulation. It is not claimed to be a calibrated probability distribution. A wrong hypothesis is exposed through tool outcomes, such as an observation returning no usable mask or a motion planner finding no feasible trajectory.

3. Hybrid beliefs and system architecture

TAMPAgent represents planning variables using a hierarchy of semantic and geometric forms:

  • a⃗=⟨pick[A],place[A,B]⟩,\vec{a}=\left\langle \mathop{\mathrm{pick}}[A],\mathop{\mathrm{place}}[A,B]\right\rangle,2: a uniform or uninformative prior;
  • a⃗=⟨pick[A],place[A,B]⟩,\vec{a}=\left\langle \mathop{\mathrm{pick}}[A],\mathop{\mathrm{place}}[A,B]\right\rangle,3: an agent-written semantic hypothesis;
  • a⃗=⟨pick[A],place[A,B]⟩,\vec{a}=\left\langle \mathop{\mathrm{pick}}[A],\mathop{\mathrm{place}}[A,B]\right\rangle,4: a segmentation mask;
  • a⃗=⟨pick[A],place[A,B]⟩,\vec{a}=\left\langle \mathop{\mathrm{pick}}[A],\mathop{\mathrm{place}}[A,B]\right\rangle,5: a metric object estimate;
  • a⃗=⟨pick[A],place[A,B]⟩,\vec{a}=\left\langle \mathop{\mathrm{pick}}[A],\mathop{\mathrm{place}}[A,B]\right\rangle,6: a tool-certified grasp;
  • a⃗=⟨pick[A],place[A,B]⟩,\vec{a}=\left\langle \mathop{\mathrm{pick}}[A],\mathop{\mathrm{place}}[A,B]\right\rangle,7: a robot configuration;
  • a⃗=⟨pick[A],place[A,B]⟩,\vec{a}=\left\langle \mathop{\mathrm{pick}}[A],\mathop{\mathrm{place}}[A,B]\right\rangle,8: a collision-checked trajectory.

An object-location belief may consequently evolve as

a⃗=⟨pick[A],place[A,B]⟩,\vec{a}=\left\langle \mathop{\mathrm{pick}}[A],\mathop{\mathrm{place}}[A,B]\right\rangle,9

The agent may initially hypothesize an object location, use a viewpoint and an observation tool to obtain a segmentation mask, and then refine the result into an RGB-D estimate suitable for grasp generation and motion planning.

The architecture contains an LVLM agent, an extensible perception and planning API, a TAMP service, geometric and control tools, a robot or simulator, and an event and camera recorder. The agent writes or updates a planning problem AA0, inventories objects and categories, records observed facts and semantic location hypotheses, parses the user goal, proposes a partial plan, selects among certified candidates, invokes tools through computation graphs or Python scripts, interprets failures, and replans.

The API provides perception, object and belief authoring, task planning, grasp generation, placement generation, inverse kinematics, base and arm motion planning, execution grants, and effect observation. The described formal domain includes AA1, AA2, and AA3; the full architecture additionally lists exploration, approach, holding-motion, reorientation, fixture manipulation, and confirmation operations.

The agent does not directly move the robot or fabricate feasibility evidence. It operates in a sandbox through an allowlisted gateway. The gateway removes credentials supplied by the agent, restricts request sizes and response formats, prevents direct access to private network resources, and mediates calls to the TAMP API. The agent cannot directly access the robot, cameras, simulator, or planner backend. Execution requires a planner-issued grant, trajectories are signed or otherwise validated, and tool outputs, factor graphs, camera streams, and execution events are recorded.

4. Parameterized actions and certification

Detection

A detection action is represented as

β\beta0

where AA4 is the observed object, AA5 is a language description, AA6 is the prior belief, AA7 is the refined belief, AA8 is a robot configuration, and AA9 is a trajectory. The action requires

BB0

and produces

BB1

Its geometric constraints include

BB2

and

BB3

The observation constraint can require that BB4 be a segmentation mask or an RGB-D estimate. Failure modes include OCCLUDED, insufficient depth, and the absence of a reachable viewpoint satisfying the observation requirement.

Pick

A pick action is represented as

β\beta1

where BB5 is an RGB-D object estimate and BB6 is a grasp. Its preconditions are

BB7

and its effects include

BB8

The principal constraints are

BB9

Φ(a⃗)=(V,F),\Phi(\vec{a})=(V,F),0

and

Φ(a⃗)=(V,F),\Phi(\vec{a})=(V,F),1

The requirement that Φ(a⃗)=(V,F),\Phi(\vec{a})=(V,F),2 be an Φ(a⃗)=(V,F),\Phi(\vec{a})=(V,F),3 estimate forces a detection step before picking. Grasp candidates are generated by GraspGenX, while the LVLM can express preferences such as “top grasp, clear of the table edge” or “a grasp that lets me drop it into the bowl.”

Place

A placement action is represented as

β\beta2

where Φ(a⃗)=(V,F),\Phi(\vec{a})=(V,F),4 is the held object, Φ(a⃗)=(V,F),\Phi(\vec{a})=(V,F),5 is the support or container, Φ(a⃗)=(V,F),\Phi(\vec{a})=(V,F),6 is the current grasp, Φ(a⃗)=(V,F),\Phi(\vec{a})=(V,F),7 is the placement pose, Φ(a⃗)=(V,F),\Phi(\vec{a})=(V,F),8 is the support’s RGB-D estimate, Φ(a⃗)=(V,F),\Phi(\vec{a})=(V,F),9 is the final robot configuration, and VV0 is a trajectory. Preconditions include

VV1

For a support, the effects include

VV2

while a container can instead produce VV3. Placement requires stable placement, kinematic, and motion constraints. Because the grasp variable is shared between picking and placing, the planner can reject a grasp that permits pickup but makes later placement infeasible.

First-action binding

The agent may propose a multi-action plan, but it binds the arguments of the first action only. A typical computation graph selects a viewpoint, plans a base trajectory, obtains a segmentation mask and RGB-D estimate, generates grasps or placements, selects a candidate using a language description, computes inverse kinematics, and plans collision-free motion. Execution is granted only when the required factors for the first action have been certified and all arguments have the required types.

This staged binding allows future actions to remain language-valued when their geometry depends on observations that have not yet occurred. It also prevents the system from executing an unverified long-horizon plan.

5. Closed-loop planning, execution, and recovery

The central operating loop is

VV4

where VV5 is the current planning problem, VV6 is the belief, VV7 is a partial plan, VV8 is its compiled constraint graph, VV9 denotes the computation graph or tool-binding process, FF0 is a certified parameter assignment, and FF1 is the executed action.

After execution, the robot observes the effect and updates its belief. Confirmed effects are added to the belief state. If an effect is unconfirmed, the system uses observation to resolve the discrepancy and replans. The next planning problem is rewritten from the updated history rather than treating the old plan as an open-loop command sequence.

Newly observed objects can be added to FF2 and categorized as movable objects, supports, containers, fixtures, or blockers. The agent can maintain multiple semantic location hypotheses before acting, rather than testing one location and inventing alternatives only after failure. Navigation is represented through motion constraints, allowing the robot to move to viewpoints or manipulation stances and to reobserve an object while holding another object.

Blocker clearing is treated as planning rather than ad hoc reaction. If a direct pick or place is uncertifiable, the planner can introduce actions such as moving a cup, mustard bottle, cracker box, or other object; closing an open drawer; opening a refrigerator; or approaching from another stance. The blocker-clearing action must itself satisfy geometric constraints, and its placement can be selected with later actions in mind.

Failures are attached to unsatisfied factors. An infeasible placement may be associated with a FF3, FF4, or FF5 factor; an occluded object may produce an FF6 failure. The agent can respond by trying another candidate or viewpoint, writing another computation graph, selecting a different task plan, introducing a new semantic hypothesis, or adding an action such as blocker relocation.

The paper distinguishes epistemic and kinematic gaps. An epistemic gap, such as failure to see an apple, is addressed by revising beliefs or moving occluders. A kinematic gap, such as the absence of a reachable placement from the current base pose, is addressed by changing geometry, changing the robot stance, or adding a navigation action.

6. Tools, platforms, and evaluation

TAMPAgent uses SAM 3 for segmentation masks, SAM 3 combined with Fast-FoundationStereo for RGB-D estimates, GraspGenX for grasp candidates, cuRobo for inverse kinematics and arm motion planning, and a placement sampler and cuRobo for stable placements and motion. Base motion planning is used for navigation and viewpoint selection.

The RB-Y1 is a wheeled bimanual mobile manipulator with an XHAND dexterous hand. Its perception and control stack includes stereo RGB-D through Project Aria glasses, Fast-FoundationStereo, a look-ahead camera, a base motion planner, and cuRobo. The Unitree G1 uses an Intel RealSense D435i ego camera, a Luxonis OAK-D Pro look-ahead camera, Dex3 hands, SONIC whole-body control, visual-inertial and LiDAR-inertial odometry, nvblox local occupancy mapping, and LocateAnything for metric 3-D localization.

The evaluation used MuJoCo simulation, household kitchen assets from MolmoSpaces and RoboCasa, RB-Y1 and G1 platforms, and tasks in which the robot initially faced away from relevant objects and received only a short goal prompt without a scene description. Representative tasks included placing an apple on a plate, finding an apple inside a refrigerator, clearing cups and drawers, moving boxes and bottles, handling occluded objects, and transporting objects between rooms.

For RB-Y1, the evaluation used 20 episodes per task and method, with time budgets generally set to 15 minutes plus 5 minutes per blocker and 25 minutes for especially difficult occlusion tasks. G1 experiments used five episodes per method on T1–T6. Success required confirmation of the goal, such as FF7 or containment of an object in a receptacle. Episodes failed when an object fell, the scene was destroyed, the robot fell, or the time budget expired. Metrics included absolute success, partial success, failure causes, and uncached input and output tokens.

Across the RB-Y1 kitchen benchmark, TAMPAgent achieved 77.2% absolute success, compared with 57.2% for CaP and 61.1% for GaP. On non-prehensile tasks T4–T6, T8, and T10, it achieved approximately 75% success, compared with 50% for CaP and 56% for GaP. On reported occlusion tasks, the apple was found in 14/20 TAMPAgent episodes in one task and 13/20 in another, compared with 9/20 and 10/20, respectively, for both baselines.

Across the five principal blocker tasks, episodes involving object drops or scene destruction numbered 20 for CaP, 11 for GaP, and 9 for TAMPAgent. Timeouts numbered 30, 33, and 16, respectively. In three-room tasks, all methods achieved 5/5 on T11 and T12, while T13 produced 4/5 for all methods.

On G1 T1–T6, averaged across the six tasks, TAMPAgent increased absolute success from 6.7% to 53.3% and partial success from 30.3% to 76.6%. In a repeated-attempt experiment on T9, TAMPAgent and GaP solved the task on the third attempt, while CaP solved it on the fourth. TAMPAgent used 42% fewer uncached input tokens than GaP and 31% fewer than CaP, with 9% fewer output tokens than GaP and 12% fewer than CaP.

On RB-Y1 kitchen tasks, normalized by simulated time, TAMPAgent used 19.3k uncached input tokens per minute and 1.75k output tokens per minute. The corresponding values were 20.3k and 1.91k for CaP and 27.4k and 2.73k for GaP. The paper attributes this reduction to the compact structured state FF8, planner-supplied constraint list FF9, and shorter tool-binding scripts.

Real-world demonstrations included simple pick and place, carrying an object while searching for an out-of-view receptacle, and moving an object that hid a target. Approximate uncached token usage per episode increased from 376k input and 39k output for simple pick and place to 438k and 58k for out-of-view receptacle search and 973k and 127k for hidden-object manipulation. On a real G1, the system identified and moved a toy dog blocking a yellow can, but the run ended because the dog’s placement could not be certified.

7. Contributions, limitations, and relation to adjacent agentic planning

TAMPAgent’s principal contribution is the use of an LVLM as an active belief-space policy rather than merely as a one-time translator from language to a closed-world planning problem. Hybrid beliefs allow variables to remain semantic until observations make continuous grounding possible. Planner-mediated tool use ensures that perception, grasping, inverse kinematics, placement, and motion constraints are satisfied before execution. Closed-loop partial-plan execution permits long-horizon behavior without requiring all future geometry to be known in advance, while factor-specific failures enable targeted recovery.

The framework also provides a design pattern for tool-using agents more broadly: maintain a structured state, expose typed tool interfaces, compile proposed actions into explicit constraints, bind only execution-critical parameters, validate the immediate action, and update the state after observing its effect. This pattern is related to tool-driven engineering orchestration such as TACS-RF, which likewise uses structured artifacts, cross-tool feedback, multi-fidelity evaluation, and verification gates, although TAMPAgent addresses embodied mobile manipulation rather than electronic-design automation (Liang et al., 22 Oct 2026).

The assignment and motion-planning interaction highlighted by ITA-CBS is also relevant to extensions involving multiple robots or interchangeable tasks. ITA-CBS maintains a single constraint tree and incrementally recomputes the best target assignment when new collision constraints alter path costs, avoiding multiple assignment-specific search trees (Tang et al., 2023). Its applicability to TAMPAgent is direct only for discrete coordination layers; continuous geometry, temporal constraints, heterogeneous agents, and non-additive objectives require substantial extensions.

Several limitations constrain the interpretation of the results. TAMPAgent depends on the reliability of perception, grasp, placement, inverse-kinematics, and motion-planning tools. Language beliefs are not calibrated probabilities, so the agent may continue searching when acting would probably succeed, causing timeouts. The available action domain is fixed, and the system cannot autonomously invent new primitive operators. Geometric feasibility can be certified, but semantic interpretation of the user’s intention is not guaranteed. Real-world experiments are demonstrations rather than a large statistically controlled hardware evaluation, and repeated perception, planning, and replanning remain computationally and token intensive.

The reported comparison is not a conventional component-level ablation. It primarily compares TAMPAgent with CaP and GaP under matched tools, models, robot stacks, sandboxing, and episode harnesses. The paper does not provide confidence intervals, statistical significance tests, a complete per-task numerical table in text, or a calibrated uncertainty evaluation. It also does not establish that the framework will generalize beyond the available operators, tools, platforms, and household task distributions.

TAMPAgent is therefore best characterized as a coordination layer between open-world semantic reasoning and certified geometric execution. Its central architectural principle is that an LVLM may propose hypotheses, plans, and preferences, but only a planner-mediated tool chain may bind and execute the immediate action. This separation is particularly consequential in scenes involving occlusion, hidden objects, clutter, blocker manipulation, coupled grasp-placement constraints, and long-horizon mobile manipulation.

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to TAMPAgent.