GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks

Abstract: For robots to work reliably in commercial and industrial applications, can recent advances in agentic coding systems combine interpretable robot programming with the open-world adaptability of model-free policies? We focus on "Variational Automation" (VA), a class of tasks that have larger variations in object geometry and pose than fixed automation. Model-free policies often struggle to close the reliability gap for VA tasks, which must be executed persistently and reliably in commercial and industrial applications. Motivated by prior work on Task and Motion Planning (TAMP) and the Robot Operating System (ROS), we introduce Graph-as-Policy (GaP), a multi-agent coding harness that generates directed computation graphs with perception, planning, and control nodes from a Modular Open Robot Skill Library (MORSL). GaP then generates an internal simulation environment to rehearse task instances with different graphs in parallel to iteratively refine the graph structure and parameters to improve success rates and throughput. Evaluation with 8 new open VA task benchmarks, 4 in-simulation and 4 in real-world, suggests that GaP can achieve success rates that significantly outperform baselines. Details, code, and data can be found online: https://graph-robots.github.io/gap

• The paper introduces GaP, demonstrating a multi-agent, graph-structured approach to synthesize robust robot policies for variational automation tasks.
• It employs modular skill libraries and simulation-driven optimization to achieve over 0.93 success rates under significant pose and geometry variations.
• The framework integrates seamlessly with legacy systems and enhances policy adaptability through iterative graph refinements and targeted failure analysis.

GaP: Multi-Agent Graph-Structured Policy Synthesis for Variational Automation

Motivation and Variational Automation Task Formalism

The paper introduces GaP, a multi-agent harness that leverages agentic coding with LLMs/VLMs to synthesize modular robot policies for variational automation (VA) tasks (2607.05369). VA tasks, formally defined by the tuple $$\mathcal{T} = \langle \mathcal{L}, \mathcal{E}, \mathcal{R}, \mathcal{O}, \mathcal{X}, \mathcal{B}, \mathcal{J} \rangle$$, require robots to operate persistently within a bounded workspace, over repeated instances with significant variation in object geometry and pose. Distinct from fixed automation (FA) and generalist robotics (GR), VA tasks demand both reliability and open-world adaptivity as seen in industrial use cases (e.g., logistics, manufacturing, data center maintenance). Traditional model-free policies fail to deliver the reliability needed for VA deployment, motivating agentic coding approaches that unify modular skill invocation and simulation-driven optimization.

GaP Architecture: Graph-as-Policy and Multi-Agent Harness Design

GaP operationalizes robot policies as directed computation graphs, each node encapsulating an atomic perception, planning, or control skill drawn from the Modular Open Robot Skill Library (MORSL), comprising both model-based (e.g., ROS-exposed procedures) and model-free (e.g., diffusion-based grasp planners) functionalities. Task specification begins with an Orchestration Agent that semantically partitions objectives and delegates graph construction to specialist Skill Agents. Each agent is responsible for a bounded segment, curating subgraphs to simultaneously optimize modularization, composability, and constraint adherence. Aggregated execution graphs are evaluated via simulation rollouts, where self-learning proceeds by parallel rehearsal across variationally sampled task instances, followed by LLM-driven failure analysis and iterative graph refinement.

Figure 1: Automatic graph generation for the "Fulfill Grocery Orders" VA benchmark, demonstrating segmentation into perception, grasp, and transport subgraphs.

Modular Open Robot Skill Library and Perception/Planning Integration

MORSL exposes 51 initial skills spanning VLM-based segmentation (SAM2, Grounding DINO), 2D/3D vision processing, grasp synthesis (GraspGen, Contact GraspNet), and motion planning (cuRobo, cuRobov2). Each skill is specified by input/output signatures, semantic parameters, and preconditions, facilitating agentic orchestration and robust cross-node connectivity. The explicit graph structure enables integration with legacy robot systems such as ROS via interface nodes.

Figure 2: Comparison between hand-authored ROS graphs and GaP-generated graphs integrating skills for cable insertion; GaP supports modular subgraph reuse for extended long-horizon tasks.

Self-Learning: Simulation-Driven Graph Optimization

GaP employs internal simulation (Isaac Sim) for self-learning and iterative graph optimization. The harness orchestrates parallel rollouts across sampled task instances, capturing object/robot state transitions and registering failures (e.g., grasp instability, placement misalignment). These are analyzed to derive geometric or semantic root causes, prompting targeted node/edge/topology edits to the current computation graph. Optimization terminates upon performance plateau or diagnostic failure.

Figure 3: GaP-generated graph for popcorn-making; modular subgraphs govern knob turn, pan grasp/place, and stove operations, supporting simulation-driven improvement.

Experimental Evaluation: Robustness and Generalization in VA Benchmarks

GaP is evaluated across 8 VA benchmarks (4 simulation, 4 real-world), covering fulfillment, packing, cooking, cable insertion, and industrial crate washing. On positional robustness, GaP significantly outperforms VLA baselines ($\pi_{0.5}$, MolmoAct2) and TAMP-based TipTop. GaP achieves $>$0.93 success rates under large pose/geometry variation, where end-to-end policies drop as low as 0.20. The modular pipeline enables high transferability from simulation to real, maintaining perfect success on physical grocery order fulfillment (25/25) and $>$0.90 on real popcorn-making.

Figure 4: Real-world seven-port cable insertion setup used in GaP benchmarking, evaluating positional/orientation robustness and goal sequencing flexibly.

In physical cable insertion, GaP achieves $0.93$ success rate (121/130) across extensive position and orientation perturbations, integrating with ROS nodes for high-fidelity control. For cooperative crate washing in simulation, GaP matches hand-engineered expert graphs with $0.95$ reliability and similar throughput metrics, demonstrating nearly equivalent efficiency and long-form robustness.

Figure 5: GaP-synthesized bimanual crate washing policy graph, coordinating dual-arm manipulation for industrial VA tasks.

Ablation and Failure Analysis

Ablation studies reveal that replacing graph-structured multi-agent generation with single-agent or monolithic script code collapses success rates due to context window overload and semantic connectivity errors. Graph validation and modular agent separation are indispensable for persistent reliability and composability. Failure modes are localized via checkpoint feedback in simulation, guiding efficient repair and subgraph swap.

Practical/Theoretical Implications and Future Directions

GaP provides a formal framework for agentic robotics in VA deployment, blending interpretable modularization with open-world model-free adaptivity. This approach enables persistent, reliable automation across high-variation task spaces, bridging GOFE and end-to-end learning. The agentic harness and simulation-driven graph optimization address context window limitations and constraint violation risks inherent in LLM coding.

Practically, GaP's modular graph-centric design supports scalable integration with legacy robot software and real-time adaptability, amplifying throughput and reliability in industrial settings. Theoretically, it advances policy representation as hierarchical graphical models, suitable for hierarchical reinforcement learning, task/motion planning, and compositional affordance discovery. Future developments will need to expand skill catalog coverage (especially for deformable/dynamic tasks), accelerate execution throughput, and increase autonomy in unsupervised skill acquisition.

Conclusion

GaP establishes a structured, multi-agent paradigm for persistent robot policy synthesis in variational automation. Through explicit computation graph construction, modular skill invocation, and rehearsal-driven optimization, it significantly advances robust open-world automation while preserving interpretability and composability. The results highlight that graph-structured agentic coding is essential to scaling reliable robot learning for complex VA deployments.