GenSwarm: Scalable Multi-Robot Code-Policy Generation and Deployment via Language Models (2503.23875v1)

Abstract: The development of control policies for multi-robot systems traditionally follows a complex and labor-intensive process, often lacking the flexibility to adapt to dynamic tasks. This has motivated research on methods to automatically create control policies. However, these methods require iterative processes of manually crafting and refining objective functions, thereby prolonging the development cycle. This work introduces \textit{GenSwarm}, an end-to-end system that leverages LLMs to automatically generate and deploy control policies for multi-robot tasks based on simple user instructions in natural language. As a multi-language-agent system, GenSwarm achieves zero-shot learning, enabling rapid adaptation to altered or unseen tasks. The white-box nature of the code policies ensures strong reproducibility and interpretability. With its scalable software and hardware architectures, GenSwarm supports efficient policy deployment on both simulated and real-world multi-robot systems, realizing an instruction-to-execution end-to-end functionality that could prove valuable for robotics specialists and non-specialists alike.The code of the proposed GenSwarm system is available online: https://github.com/WindyLab/GenSwarm.

GenSwarm is presented as an end-to-end system for automatically generating and deploying control policies for multi-robot systems directly from natural language user instructions (Ji et al., 31 Mar 2025 ). This approach addresses the limitations of traditional manual development (complex, labor-intensive, inflexible) and previous automatic methods (requiring manual objective function crafting).

The core idea is to leverage LLMs to translate natural language task descriptions into executable Python code policies for individual robots in a swarm. Unlike approaches that use LLMs for online decision-making, GenSwarm generates code, which is described as "white-box," offering advantages in reproducibility, interpretability, and efficiency for execution on resource-constrained robots. The system is designed to be zero-shot, adapting to new or altered tasks without requiring explicit training examples, and features scalable software and hardware architectures for efficient deployment.

GenSwarm Pipeline

The GenSwarm system operates through a three-module pipeline:

1. Task Analysis:
   • Takes natural language instructions (e.g., "The robots need to surround the target prey...").
   • An LLM agent extracts task constraints (e.g., "CollisionAvoidance"). These constraints form a pool that subsequent steps must adhere to.
   • Another LLM agent generates a skill library, defining necessary skills as Python function names and descriptions (e.g., "Allocate_initial_angles" for global goal assignment, "update velocity" for local control). These skills are classified as global (executed on a control station) or local (executed on individual robots).
2. Code Generation:
   • An LLM agent constructs a skill graph representing hierarchical dependencies between skills and their associated constraints.
   • The skill graph guides the generation of the main body code for each skill function, starting with low-level skills to promote reuse.
   • LLM agents review the generated code for constraint alignment and logical errors, making necessary modifications.
   • Static code checks are performed to ensure executability.
3. Code Deployment and Improvement:
   • Automatically deploys the generated code and necessary runtime environments to simulated or real-world robots.
   • Uses multi-modal feedback for policy refinement:
     • A Vision-LLM (VLM) agent analyzes video clips of simulation execution to provide feedback on task completion.
     • A human interface allows users to provide natural language feedback (e.g., "Forming a circle with a radius of 1 meter seems a bit large; perhaps 0.8 meters would be better."). GenSwarm can adjust parameters based on this feedback.
   • Policies are refined based on this feedback and then deployed to real-world robots if validated.

Software and Hardware Platform for Scalable Deployment

A key aspect of GenSwarm is its scalable deployment system. Manual setup on numerous robots is inefficient. GenSwarm uses a combination of Ansible and Docker to automate environment setup and code deployment.

• Software: A control station uses Ansible to wirelessly connect to robots via SSH. Ansible playbooks automate tasks like installing and configuring Docker. Pre-built Docker images containing the ROS environment for robot operation and a Python environment with required dependencies are pulled. The generated code is then transmitted to robots and executed within these containers. This process significantly reduces deployment time, taking minutes for environment setup and seconds for code deployment across multiple robots, nearly independent of the number of robots. The software architecture is modular, with seven core modules (Core, Skill, Action, Environment, Constraint, File, Feedback) designed with interfaces and patterns (like Composite Pattern) for scalability and flexibility.
• Hardware: A custom multi-robot platform, an upgrade of a previous design, supports the system. Each robot has onboard computational, control, and communication resources. Features like "one-click all start," "one-click all sleep," and wireless data retrieval simplify experiments. Perception is emulated using an indoor motion capture system and an MQTT coordination server, distributing local information to each robot.

Demonstration and Performance

The paper demonstrates GenSwarm on a "predator-prey encircling" task, showing the step-by-step workflow from instruction to deployment and refinement. The time taken for code generation, deployment, and human-feedback-based improvement are reported as approximately six, two, and two minutes, respectively, using GPT-4o.

GenSwarm was evaluated on ten diverse multi-robot tasks: aggregation, flocking, shaping, encircling, crossing, coverage, exploration, pursuing, bridging, and clustering. Evaluation metrics specific to each task were used (e.g., spatial variance for flocking, Procrustes Distance for shaping, average distance to prey for pursuing).

Across 1,000 simulation trials using the o1-mini LLM (100 trials per task), GenSwarm achieved an average success rate of 81%. For comparison, using GPT-4o on six representative tasks (100 trials per task and method), GenSwarm achieved a 74% average success rate, outperforming Code-as-Policy (40%) and MetaGPT (31%). Including VLM feedback improved the success rate from 71% (without VLM) to 74% (with VLM). The quality of user instructions significantly impacted success rates, with more comprehensive instructions generally leading to better performance, although success is not guaranteed even with detailed prompts due to LLM limitations.

Implementation Considerations and Limitations

• Computational Requirements: While generated code policies are efficient, the generation process relies on powerful LLMs, requiring significant computational resources on the control station.
• Deployment: The use of Ansible and Docker is crucial for practical, scalable deployment on robot fleets. Setting up and maintaining these environments is a necessary prerequisite.
• Perception: The current implementation uses an emulated perception system via a central server. Integrating onboard sensing is a necessary future step for real-world autonomy in diverse environments.
• Task Complexity: While demonstrated on various tasks, scaling to more complex, real-world multi-robot challenges may require advancements in LLM capabilities or integration with other methods like multi-agent reinforcement learning.
• Code Correctness: LLM hallucinations mean generated code is not always perfect, necessitating review (automated and human) and testing in simulation before real-world deployment.

GenSwarm represents a significant step towards automating the development lifecycle for multi-robot systems, enabling rapid prototyping and deployment based on natural language commands. Its practical implementation relies on a robust software/hardware platform designed for scalability and incorporates feedback loops for policy refinement.