ACE: Agentic Control for Embodied Manipulation via Zero-shot Workflow Reasoning

Abstract: Open-ended tabletop manipulation requires agents to not only understand natural language but also adapt to dynamic environments and execution failures. We present ACE (Agentic Control for Embodied Manipulation), a zero-shot workflow reasoning framework for tabletop pick-and-place from natural language. Rather than relying on direct low-level action mapping, ACE combines agentic workflow reasoning with two robot-facing executable skills: a visual grounding interface and a reusable pick-and-place primitive. To bridge semantic reasoning and physical control, the active sub-goal is grounded into a mask-mediated vision-action interface. This unified mask specifies the target object and destination, is tracked over time, exposed for human verification, and ultimately passed to a task-agnostic downstream policy for execution. Crucially, ACE operates in a closed loop supported by a multi-timescale memory. After an action is executed, the system automatically verifies whether the intended sub-goal succeeded, using the outcome to advance, retry, repair, or replan. This enables online adaptation to user corrections, scene changes, and physical failures. We evaluate ACE on logically complex, long-horizon tasks, including zero-shot multi-step equation formation with number cubes and constraint-based object retrieval. ACE demonstrates task-level zero-shot generalization on novel semantic constraints and randomized tabletop scenes without task-specific retraining. Specifically, while standard end-to-end baselines struggle to complete these logically demanding tasks, ACE achieves a 50% success rate in equation formation and a 70% success rate in constraint retrieval. This contrast demonstrates that explicit workflow reasoning and mask-mediated control offer a robust, practical route toward adaptable robotic manipulation.

• The paper introduces ACE, a framework that decouples high-level workflow reasoning from low-level action, enabling robust zero-shot manipulation.
• It employs modular components including agentic planning, mask-mediated visual grounding, and closed-loop verification for improved task performance.
• Quantitative evaluations demonstrate significant gains with 50-70% task success rates and 90% grounding accuracy compared to end-to-end methods.

Agentic Control for Embodied Manipulation via Zero-shot Workflow Reasoning: A Technical Analysis

Problem Statement and Motivation

Open-ended tabletop manipulation presents substantial challenges for embodied agents, particularly in achieving robust task-level generalization in dynamic environments. Traditional systems, reliant on task-specific policies or direct vision-action/language-action mapping, demonstrate fundamental brittleness when faced with variable semantic instructions or physical execution errors. These systems exhibit marked deficiencies in long-horizon reasoning, compositional task decomposition, and online adaptation. The ACE (Agentic Control for Embodied manipulation) framework addresses these limitations by introducing a decoupled architecture wherein explicit agentic reasoning orchestrates reusable low-level manipulation primitives via a closed-loop, verifiable interface. The core aim is to demonstrate that explicit workflow reasoning, combined with mask-centered visual grounding, enables zero-shot task-level generalization without retraining on task-specific trajectories.

Methodology

Architectural Decoupling and Modularization

ACE formalizes open-ended pick-and-place manipulation as a zero-shot workflow reasoning problem. The architecture comprises:

• Agentic Planner and Replanner: High-level reasoning is managed by a language agent (powered by Qwen3.6-35B-A3B via Qwen-Agent [24, 25]) that decomposes natural language instructions $u$ and scene context $c$ into explicit, revisable semantic sub-goals, producing a hierarchical workflow $W = P(u, c) = (w_1, w_2, ..., w_K)$.
• External Robot-Facing Skills: Only two robot-facing skills are exposed to the agent: MaskInterface (for grounding sub-goals in the visual scene as spatial masks) and PickPlace (for executing manipulation actions according to these masks).
• Mask-mediated Vision-Action Interface: Each semantic sub-goal is grounded into a grayscale mask $M_k$ (encoding pick and place regions) using a combination of SAM3 [26] for segmentation and Cutie [27] for temporal mask tracking. These masks undergo human verification before execution, ensuring safety and interpretability.
• Reusable Pick-and-Place Primitive: The downstream vision-action controller is instantiated via a Diffusion Policy (DP) [29], trained exclusively on generic pick-and-place tasks with mask-conditioned inputs, deliberately omitting any task-level demonstrations.
• Multi-Timescale Memory: Centralized memory supports live execution state, persistent object/role identity, visual references for re-acquisition, and dialogue context tracking. This supports robust recovery, consistency, and stateful adaptation across asynchronous components.

Closed-Loop Workflow and Recovery Mechanisms

After each pick-and-place action, the system verifies execution success by analyzing the spatial overlap of target masks with the new scene. Failures (due to grounding, grasping, or changed user intent) trigger immediate repair, local/global replanning, or retries at the sub-goal level. Human-in-the-loop verification is leveraged prior to execution, mitigating the risk of catastrophic errors and allowing for intuitive correction of mask misgrounding.

Experimental Evaluation

Task Suite

The evaluation targets zero-shot, task-level generalization across two open-ended physically grounded tasks:

• Semantic Formula Assembly: Multi-step spatial sequencing of number and operator blocks to physically instantiate target equations specified in natural language (e.g., "make an equation that equals 7"). This necessitates symbolic reasoning, open-vocabulary grounding, spatial arrangement, and adaptation under evolving task and scene conditions.
• Constraint-Based Retrieval: Logical/numerical analysis of the scene to select and transfer objects satisfying user-specified constraints (e.g., sort cubes by value).

In all evaluations, object initializations and instruction phrasings are randomized per trial to rigorously probe generalization.

Quantitative Results

ACE demonstrates 50% success rate on Semantic Formula Assembly (avg. FGS 17.8/20) and 70% success rate on Constraint Retrieval, achieving 90% grounding accuracy in both domains. Baselines—ACT (vision-action, imitation) and 70.5 (end-to-end vision-language-action learning)—trained on task-specific demonstrations, exhibit 0% end-to-end task completion and low fine-grained scores despite retaining proficiency in motor execution. The result isolates the necessity of explicit cognitive workflow decomposition for successful long-horizon reasoning under low-data settings.

Ablation Analysis

• Mask-only vs. Mask+RGB for VA Policy: The mask-only representation is robust under domain shift, yielding a 90% grasp success rate, whereas the addition of raw RGB input induces overfitting and severe performance degradation (30%).
• Removal of Agentic Reasoning: Without the workflow decomposition engine (i.e., direct DP control), end-to-end task performance collapses to 0%, underscoring the criticality of high-level planning.
• Effect of Human Verification: Disabling user approval reduces the success rates and grounding accuracy notably, highlighting the interface's practical utility in real-world deployments.

Implications and Theoretical Significance

The explicit decoupling of semantic planning from low-level execution, together with verifiable mask-mediated grounding, provides several advantages:

• Interpretability and Diagnosability: The explicit workflow exposes sub-goals, mask groundings, and failure points, facilitating fine-grained debugging and targeted repair compared to opaque end-to-end policies.
• Data Efficiency and Generalization: By repurposing generic primitives, ACE alleviates the need for task-specific retraining, delivering robust zero-shot performance in previously unseen semantic manipulation tasks.
• Human-Compatible Control: The mask interface enhances transparency and supports efficient shared autonomy, essential for safe operation in diverse environments.
• Adaptation and Recovery: Closed-loop verification and multi-timescale memory enable ACE to recover from physical/memory/semantic errors and scene perturbations, broadening applicability in real-world dynamic scenarios.

While the framework is tailored to tabletop pick-and-place, the underlying principles—modular executable skill interfaces, interpretable intermediates, and tight reasoning-control coupling—are extensible to more complex, contact-rich, or multi-modal manipulation tasks. Scaling of agentic planners and memory architectures is an immediate direction for improving cognitive and temporal contextualization across extended task horizons.

Limitations and Prospects

ACE's reliance on human-in-the-loop approval inserts additional latency and operational overhead, which may impede autonomy in high-throughput scenarios. The architecture currently centers on spatially local pick-and-place actions, necessitating further development for rich contact manipulations, deformable objects, and multi-agent interaction. Advancements in planner reasoning abilities, mask-generation fidelity, and calibrated confidence estimation for semi-autonomous operation are promising future avenues. Stronger, possibly hierarchical, memory systems could further enable robust context retention over extended interactions.

Conclusion

The ACE framework systematically demonstrates the advantage of explicit, decoupled workflow reasoning for embodied manipulation in open-ended, zero-shot settings (2607.04162). Through modular skill coordination, mask-mediated visual grounding, and a multi-timescale memory-informed closed loop, ACE achieves measurable improvements over traditional direct mapping policies, offering a principled, interpretable approach to generalizable robotic control. The findings substantiate the theoretical and practical importance of explicit cognitive-semantic planning as a route to robust, adaptable embodied agents.