Human-in-the-loop Multi-Robot Collaboration

• HMCF is a collaborative framework combining human intuition with robotic scalability to safely and adaptively coordinate multiple robots.
• It employs methodologies such as LLM-based task allocation, dynamic human oversight, and structured supervisory interfaces to optimize mission performance.
• Empirical evaluations highlight improved task success rates, reduced operator workload, and robust coordination across varied communication scenarios.

A Human-in-the-loop Multi-Robot Collaboration Framework (HMCF) is a class of architectures and methodologies wherein humans interact with and supervise teams of autonomous robots to achieve collaborative tasks. In these frameworks, the human operator may provide intermittent input, instantiate dynamic guidance, or serve as an arbiter for task allocation, safety, and exception handling. The core design principle is to leverage the synergy between human intuition, adaptability, and global situational awareness and robotic scalability, speed, and precision. HMCF has been instantiated across diverse domains, including multi-agent imitation learning, LLM-mediated robot orchestration, symbolic planning with dynamic trust estimation, flexible shared control, vision-based teamwork in physical tasks, and robust mission execution under communication constraints.

1. Architectural Paradigms in HMCF

Human-in-the-loop multi-robot frameworks adopt a range of system architectures, each tailored to particular collaboration modalities and task domains.

• Dynamic Human Guidance of Agent Teams: In single-human multi-agent imitation learning, such as (Ji et al., 2024), a solitary human operator is empowered to selectively override moment-to-moment policy outputs for an arbitrary robot (seeker) in a team, with other robots proceeding autonomously. The interface allows agent selection, waypoint override, and seamless transfer of control back to the autonomous policy.
• LLM-mediated Task Allocation and Execution: HMCFs leveraging LLMs (Li et al., 1 May 2025, Sun et al., 30 Nov 2025) are organized into hierarchical or distributed agent layers. Typically, a human supervisor specifies a mission using natural language; an assistant LLM decomposes the mission, allocates subtasks to robots based on explicit capability profiles, oversees execution, and invokes the human operator only upon uncertainty or error.
• Server–Client and Communication Abstractions: Architectures such as CoHRT (Sarker et al., 2024) utilize a server–client paradigm, where the server maintains global state and coordinates distributed teams of humans and robots via locking protocols and real-time, vision-based tracking.
• Latency-Guaranteed and Sparse-Network Frameworks: In settings with restricted communication, frameworks like iHERO (Tian et al., 2024) design data flow topologies (e.g., ring graphs) for intermittent, pairwise robot data exchange, periodic return-to-operator events, and ensure that human requests (map update, prioritized region, relocation) are honored within a bounded latency.

The following table summarizes representative architectural features across selected HMCF instances:

Framework	Control Mode	Coordination Backbone	Human Roles
(Ji et al., 2024)	Dynamic switching	Imitation + ToM policy	Direct agent control, finetune
(Li et al., 1 May 2025)	LLM-mediated	Assistant LLM + JSON comm	High-level commands, safety
(Sarker et al., 2024)	Server–Client	Vision + TCP, GUI	Collaborative task share
(Tian et al., 2024)	Sparse network	Pairwise comm, ring graph	Directive, region selection
(Zhu et al., 17 Feb 2025)	Shared control	Vector fields, intention	Trajectory, priority input
(Wang et al., 2018)	Symbolic, trust	Distributed TS, DBN trust	Real-time mode switching

2. Algorithmic and Modeling Principles

HMCF designs rely on a range of modeling constructs for state representation, policy learning, communication, trust/adaptability, and safety.

• State and Action Specification: States are typically augmented stacks of sensor data (e.g., last $N$ RGB frames, agent/teammate masks) (Ji et al., 2024), structured as discrete symbolic cells (Wang et al., 2018), or multi-modal sensory fusions (Sun et al., 30 Nov 2025). Actions may be continuous waypoints, symbolic commands, or low-level actuator primitives.
• Policy Learning: Learning often proceeds via a two-stage regime: (i) pretraining on demonstrations or heuristics (e.g., $\mathcal{L}_{IL}$, $\mathcal{L}_{IL\text{-}Long}$), and (ii) fine-tuning on human interventions, with frozen encoders and focused adaptation of policy heads (Ji et al., 2024).
• Theory-of-Mind (ToM) and Policy Embedding: Frameworks explicitly embed predictors for teammates’ actions (“Policy Embedding Team”), enabling each agent to anticipate and coordinate with others by learning ToM-style representations (Ji et al., 2024).
• LLM-based Task Decomposition and Verification: LLM agents encode robot capabilities as explicit vectors, generate plans, estimate confidence, and invoke human oversight on low-confidence or infeasible assignments. Verification mechanisms include both LLM-based logic checks and localized safety validators (Li et al., 1 May 2025).
• Probabilistic Trust and Switching Models: Quantitative human–robot trust is modeled as a dynamic Bayesian network, updated by observing robot/human performance, detected faults, and explicit human feedback. Trust-valued switching logic governs real-time transitions between autonomous and manual modes to balance safety and efficiency (Wang et al., 2018).
• Vector Field and Intention Propagation: Shared control paradigms utilize guiding vector fields, overlaid with human intention inputs (via BCI, gestures, or gaze) and propagate intention consensus across the robot network via gradient-based updates, ensuring stable and efficient group trajectory adaptation (Zhu et al., 17 Feb 2025).

3. Human–Robot Interaction and Supervisory Interfaces

A fundamental characteristic of HMCFs is the diversity and precision of human–robot interfaces:

• Dynamic Agent Selection and Intervention: Interfaces permit runtime selection and override for arbitrary robots, with unconstrained intervention criteria, thus minimizing operator load and system non-stationarity (Ji et al., 2024).
• Structured High-Level Input: Human missions are specified in natural language; systems synthesize knowledge bases, capability maps, and assignable tasks through retrieval-augmented generation (RAG) and LLM planning (Li et al., 1 May 2025, Sun et al., 30 Nov 2025).
• Direct Shared Control: Input modalities span brain–computer interfaces (BCI), myoelectric (EMG) bands, and eye-tracking devices, enabling translational mapping from human physiological or gesture input to trajectory guidance in real time (Zhu et al., 17 Feb 2025).
• Supervisory Arbitration and Safety: Interfaces mediate exception handling (e.g., collision detection, sensor failure) and safety arbitration, initiating mode switching or invoking the human on demand (Wang et al., 2018, Li et al., 1 May 2025).
• Persistence and Real-Time Feedback: GUIs visualize task progress, agent selection, and locking protocols, while recording multimodal data streams (e.g., gaze, pose, control commands) for offline learning and analysis (Sarker et al., 2024).

4. Task Domains and Experimental Validation

HMCFs have been validated across diverse tasks and environments:

• Collaborative Hide-and-Seek: Demonstrated with seekers using partial SLAM-based maps and hiders operating egocentrically, HMCF achieved catch-rate improvements up to 58 pp in simulation, and 25 pp in real-robot deployments under less than 1 hour of human guidance (Ji et al., 2024).
• Generalized Zero-Shot Task Solving: Through LLM-based reasoning over heterogeneous teams in BEHAVIOR-1K and real labs, HMCF yielded a 4.76% absolute increase in task success rates over the strongest baselines, and handled near full generalization with minimal interventions (Li et al., 1 May 2025).
• Collaborative Manipulation and Puzzle Solving: CoHRT enabled synchronous, fairness-allocated, and legible teamwork (one Franka Panda, two humans) with real-time, vision-based object detection, hierarchical allocation, and quantitative workload balancing (Sarker et al., 2024).
• Exploration Under Scarce Communication: iHERO enforced data-latency bounds and robust operator interactivity during multi-robot mapping missions with intermittent, local pairing for map fusion, outperforming prior methods for area coverage, efficiency, and latency assurance (Tian et al., 2024).
• Firefighting and Motion Coordination: HI-GVF-based HMCF propagated dynamically drawn human trajectory intentions to a robot fleet, yielding faster intention convergence and reduced workload as measured by NASA-TLX compared to leader-follower baselines (Zhu et al., 17 Feb 2025).
• Symbolic Motion Planning with Trust Feedback: Trust-based frameworks demonstrated livelock-free goal accomplishment and lower path lengths, with dynamic GUI trust visualization and selective waypoint intervention, in non-trivial multi-obstacle simulation runs (Wang et al., 2018).

5. Evaluation Metrics and Quantitative Analysis

Frameworks report a range of quantitative and qualitative metrics:

• Task Success Rate (SR): Fraction of fully completed missions or caught targets (Ji et al., 2024, Li et al., 1 May 2025, Sun et al., 30 Nov 2025).
• Completion Rate (CR), Redundancy Rate (RR), Execution Time (TIME): Disaggregated subtask and resource utilization measures (Sun et al., 30 Nov 2025).
• Idle Time, Concurrent Activity Fraction, Functional Delay: Metrics operationalized for measuring inter-agent fluency and efficiency (e.g., CoHRT) (Sarker et al., 2024).
• Workload and Trust Assessments: NASA-TLX for subjective/physiological workload, SUS/ROSAS for usability/safety, trust and fairness questionnaires (Sarker et al., 2024, Zhu et al., 17 Feb 2025).
• System Robustness and Ablation: Removing any module (e.g., Perceiver or Validator) in multi-agent LLM systems saw success rate drops of 10–32 pp, highlighting modular indispensability (Sun et al., 30 Nov 2025).
• Coverage, Update Latency, Exploration Efficiency: For mapping missions, maximal area coverage, mean/max data-latency, and efficiency metrics (Tian et al., 2024).
• Comparative Gains: All cited HMCFs reported statistically significant improvements over controlled baselines (e.g., IL-Long+PE-T surpassing heuristic by 136% relative in SR (Ji et al., 2024), LLM HMCF outperforming HMAS-2 by 4.76% (Li et al., 1 May 2025)).

6. Limitations, Scalability, and Future Directions

• Scalability Bottlenecks: LLM context and cloud latency may limit systems managing very large robot teams; ongoing research aims at lightweight inference and mesh comms (Li et al., 1 May 2025, Sun et al., 30 Nov 2025).
• Communication Constraints: Scarce, intermittent exchange requires topological communication planning to guarantee latency and coverage (Tian et al., 2024).
• Trust Modeling and Human Workload: Operator overload and optimal trust management remain open problems requiring large-scale user studies, with a plausible implication being the necessity for adaptive HMI and shared autonomy regimes (Wang et al., 2018, Zhu et al., 17 Feb 2025).
• Transition from Monolithic to Multi-Agent Foundation Models: Evidence suggests that multi-agent orchestration of vision/language/action models yields more robust, scalable autonomy than continuing to scale single-model architectures (Sun et al., 30 Nov 2025).
• Heterogeneity and Safety: HMCFs integrating diverse robots (mobile, manipulator, UAV) must encode explicit capability maps, enforce safety and collision-checking both at plan and execution levels (Li et al., 1 May 2025).

7. Comparative Synthesis and Cross-Framework Insights

HMCFs represent a unifying methodology for endowing robot teams with adaptive, efficient, and safe collaborative intelligence, by structuring interaction between human guidance and autonomous control. Key advances include dynamic role assignment via ToM and policy embedding (Ji et al., 2024), high-level semantic reasoning and task allocation via LLMs (Li et al., 1 May 2025, Sun et al., 30 Nov 2025), scalable multi-modal perception and fairness-aware learning (Sarker et al., 2024), and formal guarantees for system responsiveness and goal accomplishment under constrained communication (Tian et al., 2024). The convergence of deep imitation learning, foundation models, probabilistic trust, and shared vector field control enables frameworks to balance generalization, explainability, safety, and performance in heterogeneous, real-world settings.

Cross-experiment results repeatedly demonstrate the necessity of rigorous human-in-the-loop mechanisms, not only for safety and recovery from uncertainty or execution failures, but also for rapid policy learning, robust generalization, and alignment with human priorities and social constraints. Future work entails scaling such systems to broader scenarios, incorporating richer forms of user feedback and delegation, and advancing principled approaches to communication, trust, and cooperative autonomy across multi-agent, multi-human collectives.