--- title: LEO-RobotAgent Framework url: https://www.emergentmind.com/topics/leo-robotagent type: topic --- # LEO-RobotAgent Framework LEO-RobotAgent is a general-purpose language-driven robotic agent framework that enables large language models (LLMs) to perform automated operation, task planning, and high-level reasoning across a diverse range of robot types and environments, with a focus on modularity, generalization, efficiency, and robust human-robot interaction [2512.10605]. ## 1. Formal Structure and Operational Cycle LEO-RobotAgent is realized as a self-cycling agent, defined as the tuple $A = \langle M, T, H, \pi \rangle$: - $M$: The LLM, prompted to produce structured JSON outputs and equipped to handle both planning and reasoning. - $T = \{\tau_1, \ldots, \tau_K\}$: The set of registered tools, where each tool $\tau_i$ is a quadruple $\langle \text{name}, \text{fn}_i, \text{desc}_i, \text{status}_i \rangle$. - $H_t$: The dynamic history at iteration $t$, containing the user’s task description, all agent messages, tool observations, and human feedback. - $\pi$: The implicit agent policy, realized through the LLM and driven by the current history $H_t$ and tool set $T$. At each step, the system operates as follows: 1. LLM processes system prompt $P_\text{sys}$ and $H_t$, outputting an object $o_t = \{ \text{“Message”}: m_t, \text{“Action”}: a_t, \text{“Action\_Input”}: p_t \}$. 2. The Executor dispatches $\tau_{a_t}.\text{fn}(p_t) \rightarrow \text{obs}_t$. 3. The history buffer is updated: $H_{t+1} = H_t \cup \{o_t, \text{obs}_t\}$. 4. The loop halts if the agent explicitly declares either “Task Completed” or “Cannot Proceed”. The agent’s planning objective is to generate action-parameter sequences $\{a_1, p_1; \ldots; a_N, p_N\}$ that maximize expected cumulative reward: $$ J = \mathbb{E}_{\pi}\left[\sum_{t=1}^N r(s_t, a_t)\right] $$ subject to system dynamics $f(x_t, u_t) = 0$ and state/action constraints $g(x_t, u_t) \leq 0$, where $r(\cdot)$ is a sparse success reward, $u_t$ are low-level tool parameters, and $s_t$ denotes the current (implicitly represented) world state [2512.10605]. ## 2. Modular Toolset and Invocation Mechanics LEO-RobotAgent’s extensible tool system decomposes perception, manipulation, planning, and communication actions into callable modules. Each tool is defined as: $$ \tau = \left\lbrace \text{name},\ \text{fn: Callable},\ \text{desc: InputSchema} \rightarrow \text{OutputSchema},\ \text{status: Boolean} \right\rbrace $$ Registration employs the following API: ```python def register_tool(name, fn, description, active=True): Tools[name] = { "fn": fn, "desc": description, "active": active } ``` Tool invocation is determined by the LLM through structured JSON: ```json { "Action": "", "Action_Input": { ... parameters ... } } ``` The Executor inspects tool status: if the selected action is available and active, the tool function is executed; otherwise, a standardized error observation is returned. This design allows straightforward extension and dynamic adaptation of the agent’s capabilities across hardware and software stacks [2512.10605]. ## 3. Bidirectional Human–Robot Interaction The framework integrates a real-time human–robot interaction loop supporting: - **History-mediated intent grounding**: User utterances $U_t$ are appended directly to $H_t$, with the LLM webbed as a “partner,” explicitly instructed to monitor, interpret, and immediately react to human input. - **Interruptibility and clarification**: The policy $\pi$ is modified on-the-fly by user corrections, formally expressed as: $$ \pi_{\text{eff}}(a_t \mid H_t) \propto \exp(\text{LLM}(\operatorname{softmax}(H_t \cup \delta U))) $$ where $\delta U$ is the embedded latest human correction. If ambiguity is detected, the agent may issue an “ask\_clarification” meta-action, explicitly routing control to the human for further direction. - **Interleaved pipeline**: Human input can preempt LLM-initiated actions, and the LLM will re-plan in context. This architectural strategy systematically lowers barriers for non-expert users in embodied contexts and enables robust mixed-initiative collaboration [2512.10605]. ## 4. Task Planning and Execution Pipeline The end-to-end cycle progresses as: 1. **Task Input**: Free-form, natural language instruction $d$ from the user. 2. **LLM Reasoning**: Using the locked-down system prompt $P_\text{sys}$, the LLM generates both explanatory “Message” fields and prescriptive “Action”/“Action_Input” pairs as JSON. 3. **Execution**: Action forwarded to the relevant tool, returning an observation. 4. **History Update**: Augmentation of $H$ with all new reasoning, actions, parameters, and observations. 5. **Iteration**: Looping continues until “Task Completed” or “Cannot Proceed”. For motion tools, path generation solves the optimization: $$ \min_{u_1 \ldots u_M} \sum_{i=1}^{M} c(u_i) \quad \text{s.t.} \quad x_{i+1} = f(x_i, u_i), \ h(x_i) \in \text{Safe},\ x_0 = \text{current},\ x_M = \text{goal} $$ with $c(u) = \|\Delta \text{pos}\|_2 + \lambda \|\Delta \text{yaw}\|_1$ typically weighting travel and effort [2512.10605]. ## 5. Experimental Evaluation and Comparative Performance LEO-RobotAgent is validated across UAVs (simulation and real flights), wheeled mobile robots equipped with articulated arms, and complex maps (café-style, urban). The evaluation suite includes delivery, search, and handover tasks, with metrics: - Success Rate (%) - Average Time to Completion (s) - Token Usage (number of LLM tokens) - Task-specific Score (out of 10) - Perfect Rate (% fully completed tasks) Results for prompt engineering configurations are summarized below: | Method | Success Rate | Token Usage | Time (s) | Time/item (s) | |----------------|-------------|-------------|----------|--------------| | Zero-shot | 20% | 32,656 | 175.2 | 183.8 | | One-shot | 50% | 32,048 | 156.7 | 123.4 | | CoT | 60% | 37,791 | 155.0 | 126.9 | | One-shot+CoT | 70% | 44,985 | 180.2 | 172.8 | Agent architecture comparisons for key tasks: | Task / Agent | DAS | CGE | DLLMs | TLLMs | LEO-Agent | |--------------------------|-------|-------|-------|-------|-----------| | Delivery (Score) | 9.16 | 9.34 | 7.94 | 8.38 | 9.16 | | Searching (Score) | – | 5.38 | 3.13 | 5.88 | 7.88 | | Handover (Score) | – | – | 4.93 | 4.87 | 7.87 | | Perfect Rate (Handover) | – | – | 13.3% | 13.3% | 46.7% | Ablation reveals chain-of-thought (CoT) and one-shot prompting drive a 50% relative increase in success rate. The single-LLM LEO-Agent structure is empirically more robust than decoupled (DLLMs/TLLMs) designs [2512.10605]. ## 6. Generalization, Robustness, and Efficiency The framework demonstrates substantial sim-to-real transfer: UAV search-and-drop achieves 90% success in simulation and 70% in the real world, with principal failures attributed to low-level tool or control errors, not to LLM planning. Performance plateaus at LLM parameter scales $P>20$B, with inference latency $L\approx \alpha P^{0.6}$ in ROS deployments, indicating diminishing returns for further scaling [2512.10605]. ## 7. Software Stack, Codebase, and Reproducibility The implementation leverages ROS Noetic for agent-to-tool orchestration, with a modular structure: - `agent_node.py`: wraps LLM, maintains history, dispatches tool calls. - `tools/`: each Python file implements a ROS node for perception, control, or summarization. - `web_ui/`: front-end in React JS for registering tools, monitoring status, inputting tasks, and real-time video overlay. Launch and reproduction: 1. Clone `github.com/LegendLeoChen/LEO-RobotAgent` 2. Install dependencies: `pip install -r requirements.txt; sudo apt install ros-noetic-rosbridge-server` 3. Launch system: `roslaunch leo_agent agent_system.launch` 4. Register tools and interact via web UI on localhost:8080 [2512.10605]. This architecture supports direct cross-platform application on UAVs, manipulators, and mobile platforms while maintaining extensibility, interpretability, and integration with modern large language models. Source: https://www.emergentmind.com/topics/leo-robotagent