Task-LLM: LLM-Based Human–Robot Collaboration

• Task-LLM is a framework that uses a GPT-2 based language model with hierarchical planning to convert high-level instructions into detailed robotic motion commands.
• It integrates YOLO-based real-time object detection to fuse linguistic plans with dynamic environmental data for precise manipulation.
• The system employs teleoperation and Dynamic Movement Primitives (DMP) to incorporate human corrections, ensuring smooth and adaptable trajectory adjustments.

A LLM-based framework for human–robot collaboration in manipulation tasks refers to a systematic approach that leverages LLMs to interpret high-level human instructions, translate them into executable robot motion commands, and supplement the system with vision-based perception and teleoperation-guided corrections. Such frameworks are designed to empower robots to autonomously and flexibly perform complex object manipulation tasks in real-world, dynamic environments, while retaining practical recoverability and adaptability through human-in-the-loop interventions.

1. Logical Inference and Hierarchical Task Planning

A core component of the framework is the use of a LLM, specifically a GPT-2–based model trained on a targeted text corpus to facilitate logical inference. The LLM interprets natural language commands, such as “clean the top of the cabinet,” and infers a sequence of low-level motion functions necessary for robotic execution.

The translation from language to action is formalized through a code template that maps user instructions to predefined executable motion functions. The system adopts a hierarchical planning algorithm to manage both long-horizon and short-horizon tasks:

• Level 1: Tasks involving more than 10 motion functions are decomposed by the LLM into multiple short-horizon subtasks.
• Level 2: Each short-horizon task is further mapped into a sequence of executable motion functions.

This structured decomposition reduces system complexity and mitigates error accumulation over extended actions, thus uniquely enabling flexible task planning that scales to varied task complexities.

2. Integration with YOLO-Based Environmental Perception

The framework incorporates a YOLO-based perception module to achieve robust, real-time object detection and environmental situational awareness. After YOLO identifies relevant objects in the robot’s workspace, their positions are registered and continuously updated.

• Context-Aware Decision-Making: The LLM’s inference pipeline fuses linguistic instruction with live environmental input. For example, if asked to “catch the bowl,” YOLO delivers the precise position, ensuring that generated motion functions align with the actual location and state of the environment.
• Adaptability: This integration is critical in dynamic or uncertain settings, such as domestic environments where object positions are not guaranteed to be fixed. Continuous updates from the perception system ensure that robotic actions are always based on the most current state, thereby maximizing the feasibility and reliability of plans inferred by the LLM.

3. Teleoperation and Dynamic Movement Primitive (DMP)-Based Corrections

Despite the strengths of LLM-driven automation, inaccuracies or illogical plans may arise, especially in edge cases where the model’s reasoning does not seamlessly translate to feasible robotic execution (e.g., improper grasping strategies for irregular objects).

• Human-in-the-loop Teleoperation: In such scenarios, a human operator intervenes via an intuitive interface—often using a VR device—to demonstrate the correct action or trajectory.
• DMP for Trajectory Modeling: Demonstrations are captured and modeled using Dynamic Movement Primitives (DMP), a framework based on differential equations:

$$\tau \dot{y} = \alpha(\beta(g - y) - \dot{y}) + f(x)$$ where \( y \) is position, \( \dot{y} \) velocity, \( g \) the goal, and \( f(x) \) a nonlinear term capturing complex motion patterns. The DMP model allows for smooth, adaptive reproduction of demonstrated movements, ensuring that robot trajectories are both dynamically robust and responsive to environmental variation. - **Generalizability and Corrective Feedback:** This dual application—autonomous inference supplemented by human-guided corrections—confers practical reliability and adaptability, allowing for learning from human interventions and continuous improvement over time. ## 4. Technical Implementation Details Implementation integrates three principal modules: - **LLM-based Function Prediction Module:** Utilizes GPT-2 to produce sequences of motion function calls, mapped via code templates to the robot’s control API. - **YOLO-based Vision Module:** Provides ongoing real-time perception inputs for object and scene state estimation. - **Teleoperation+DMP Module:** Allows human demonstrations to be incorporated as correctives, which are encoded and replayed via DMP trajectory representations. The DMP’s formulaic core, as used in action correction, can also be written with velocity-based notation:$$</p> <p>\tau \dot{y} = \alpha_v(\beta_v(g - y) - v) + f(s)</p> <p>$$

Here, state evolution towards the goal is modulated by feedback from environmental events and human interventions. The overall computational pipeline is modular, with clear interfaces for integrating perception, inference, and teleoperation, thus supporting extensibility in complex robotic systems.

5. Applications in Collaborative and Service Robotics

The described framework is directly applicable in domestic robotics for tasks such as cleaning, object retrieval, and tidying—contexts that require robust manipulation, adaptability to environmental changes, and the ability to interpret high-level user commands.

Broader applications include:

• Collaborative Industrial Robotics: Where LLMs can augment production line robots, making them more responsive to high-level operator instructions.
• Healthcare and Assistive Robotics: For supporting individuals with disabilities or in eldercare environments, where the language-to-action pipeline and the safety net of teleoperation are both critical.
• Service Robotics: Any scenario in which non-expert users must interact with robots via natural language, requiring the system to handle ambiguous commands and dynamically changing contexts.

6. Implications and Future Directions

The integration of LLM inference, vision-based perception, and DMP-driven teleoperation corrections in this framework paves the way for more robust and practical human–robot collaboration systems. Key directions for ongoing research and development include:

• Improving Hierarchical Planning: Reducing error propagation and making long-horizon plans even more reliable by refining intermediate task decomposition strategies.
• Enhancing Sensor Fusion: Deepening the integration between language-driven inference and real-time sensory data to handle complex, dynamic, and cluttered environments.
• Scalable Learning from Teleoperation: Leveraging DMP and related techniques for more extensive and automated learning from human demonstrations, making corrective interventions increasingly rare over time.

In summary, this LLM-based human–robot collaboration framework establishes a comprehensive pipeline from language understanding through perception and motion control, augmented by rigorous mathematical modeling (via DMP) and practical human-in-the-loop correction. Such a design underpins a new generation of intelligent robots capable of robustly executing manipulation tasks in real-world, human-centric environments (Liu et al., 2023).