MH-COPILOT: Multimodal Human-Steered Copilot

• MH-COPILOT is a systems paradigm that integrates multimodal perception, structured tool use, human guidance, and closed-loop verification to execute complex, safety-critical tasks.
• Its architecture decomposes roles into specialized agents—such as Planner, Viewer, Programmer, and Mentor—that collaboratively plan, execute, and verify steps.
• MH-COPILOT enables diverse applications from clinical informatics to aviation and mobile operations, mitigating errors through explicit human oversight and structured verification.

Searching arXiv for the cited papers to ground the response. MH-COPILOT denotes a copilot-oriented systems paradigm that integrates multimodal perception, structured tool use, human guidance, and verification loops to assist complex tasks in environments that are only partially API-accessible or operationally safety-critical. In the supplied literature, closely related systems span autonomous PC operation, electronic health record navigation, scientific hypothesis discovery, mobile task execution, runtime monitoring for autonomous flight, multimodal aviation conflict detection, and code-generation assistants. Across these settings, the recurrent design objective is not generic chat interaction, but closed-loop assistance that can perceive state, decompose tasks, invoke specialized tools or agents, verify outcomes against observable evidence, and remain constrained by domain-specific governance requirements (Song et al., 2024, Zakka et al., 2024, An et al., 28 May 2026, Dutle et al., 2020, Fan et al., 2 Sep 2025, Garib et al., 24 Nov 2025).

1. Conceptual scope and motivating problem classes

MH-COPILOT is best understood as a family of copilot designs for environments where single-agent, single-modality operation is brittle. In the PC-application setting, text-only operators miss GUI layout, visual cues, spatial relationships, and dynamic content, which leads to planning errors, brittle scripts, and hallucinations about nonexistent APIs or commands. MMAC-Copilot addresses this by using multi-modal agents—vision, text, and video—within a team collaboration chain so that visually dependent steps are grounded in the actual on-screen state rather than inferred from text alone (Song et al., 2024).

A closely related motivation appears in clinical informatics. Almanac Copilot is introduced in response to clinician burden from EMRs, including poor usability, excessive data entry, interruptive alerts, and non-intuitive workflows. Its stated role is Level 1 autonomy: offloading routine EHR tasks such as information retrieval and order preparation while keeping clinicians in the loop for review and approval (Zakka et al., 2024). In this formulation, a copilot is not an unconstrained autonomous actor, but a constrained assistant embedded in an auditable workflow.

In scientific discovery, MOOSE-Copilot frames the central failure of autonomous LLM systems differently: exploratory ideation and fine-grained refinement are often treated as isolated tasks, and the generative process is left largely unguided by human experts. The system’s response is a formalized human–AI interaction protocol with three explicit signals—initial blueprints, inter-stage routing, and regenerative feedback—so that the scientist becomes a navigator of the search process rather than merely a consumer of outputs (An et al., 28 May 2026).

In mobile and aviation contexts, the problem expands further from tool selection to long-horizon execution under multimodal uncertainty. AppCopilot identifies four core problems for practical mobile agents—generalization, accuracy, long-horizon capability, and efficiency—and responds with a full-stack, closed-loop multimodal assistant. AIRHILT, by contrast, focuses on synchronized radio, vision, and ADS-B streams for conflict detection, emphasizing time-to-first-warning and human-in-the-loop advisory delivery (Fan et al., 2 Sep 2025, Garib et al., 24 Nov 2025). This suggests that MH-COPILOT is not tied to one application vertical; rather, it denotes a systems pattern for coupling perception, planning, control, and oversight in domains where observable state, latency, and correctness all matter.

2. Core architectural pattern: specialized agents, staged reasoning, and closed-loop execution

A canonical MH-COPILOT architecture, as implied by the supplied systems, is multi-role rather than monolithic. MMAC-Copilot provides the clearest decomposition. Its Planner acts as strategic manager and orchestrator; the Librarian supplies information retrieval and external knowledge; the Programmer executes Bash/Python and performs error analysis and code refinement; the Viewer grounds UI elements in screenshots and turns them into clicks, text entry, and hotkeys; the Video Analyst extracts temporal context from video; and the Mentor performs post-execution oversight through screenshot-based verification and structured feedback to the Planner (Song et al., 2024).

This role specialization is paired with a planning/execution loop. MMAC-Copilot formalizes the sequence as follows: the Planner drafts an initial plan $P_0$ from user request $R$; each subtask $s$ is assigned to a best-fit agent $A$; the agent executes and updates system state $S_i$; the Mentor inspects $S_i$ and emits feedback $F_i$; if adjustments are needed, the Planner revises the plan and re-enters assignment/execution; the process terminates when the state satisfies the request or failure criteria are met. The same source also gives the VIBench state-transition view,

$$S_0 \rightarrow^{(a_1)} S_1 \rightarrow^{(a_2)} S_2 \rightarrow \cdots \rightarrow^{(a_n)} S_f,$$

with the objective of verifying that $S_f$ meets $R$ (Song et al., 2024).

Almanac Copilot uses a simpler but structurally analogous loop: perception $R$0 planning $R$1 tool use $R$2 verification. The LLM parses natural-language queries, selects an appropriate subset of tools from a predefined catalog, decides parameter values from EHR context, executes tool-wrapped API calls, and verifies that the selected functions, parameters, and composed script are appropriate and executable. The reported setup explicitly limits the number of functions to nine pre-defined modules to constrain behavior and improve reliability (Zakka et al., 2024). This is architecturally significant because it treats controllability as a first-class systems concern rather than a post hoc safeguard.

MOOSE-Copilot generalizes the architecture to a human-steered state machine. Exploratory ideation is modeled as iterative inspiration-conditioned hypothesis generation,

$R$3

while fine-grained refinement is handled by hierarchical optimization from coarse-grained corrections to finer methodological details. The full process is factorized by human signals $R$4, yielding a routed transition between conceptual exploration and executable detail (An et al., 28 May 2026). A plausible implication is that MH-COPILOT, in research settings, should expose stage transitions and correction hooks explicitly rather than bury them inside a single opaque prompt.

AppCopilot extends this pattern into a mobile multi-agent stack with a Perception agent, Planner agent, Executor agents, and Speech agent. Its planner constructs a task graph $R$5, assigns endpoints across apps or devices, and schedules execution subject to predecessor constraints. Multi-agent collaboration then uses majority voting on action types and parameter aggregation, followed by region calibration against parsed UI widgets (Fan et al., 2 Sep 2025). Relative to MMAC-Copilot, the emphasis shifts from desktop task allocation to long-horizon orchestration and resource-aware execution, but the underlying pattern remains a closed loop of perceive, decompose, execute, verify, and adapt.

3. Interfaces, modalities, and tool grounding

A defining property of MH-COPILOT systems is that they do not rely on language alone. In MMAC-Copilot, the Viewer uses GPT-4V plus SeeClick to parse screenshots, read labels, detect buttons, inputs, menus, dialog boxes, and contextual elements, and convert grounded targets into atomic GUI actions. The Video Analyst, using Gemini 1.5 Pro Vision, extracts events and temporal context from videos, which is especially relevant for dynamic or 3D environments. The Mentor then validates state transitions via screenshots, confirming whether actions had the intended effect (Song et al., 2024).

AppCopilot supplies a more explicit action ontology for this kind of system. Its execution layer supports compact JSON actions such as POINT, Swipe, TYPE, PRESS, STATUS, duration, and terminate, and it relies on OCR plus object recognition through OmniParser to output structured widgets with bounding boxes, semantic labels, and interactivity. For precise click targeting, it defines a region-calibration rule: if a predicted point lies outside all candidate boxes, it is re-targeted to the center of the nearest box, with distance computed against box boundaries. It also uses perceptual hash–based avoidance of repeated ineffective corrections (Fan et al., 2 Sep 2025). This suggests a general design principle for MH-COPILOT: interface actions should be represented in a constrained machine-readable schema, then corrected against parsed interface structure before execution.

In clinical systems, the tool layer is correspondingly standardized rather than purely visual. Almanac Copilot is built around FHIR-based EHR function wrappers, a browser to medical sources such as PubMed, clinical calculators stored as markdown in a vector database and evaluated through a Python REPL, and a Qdrant vector similarity database combining sparse and dense retrieval. Structured EHR data are accessed through FHIR resource endpoints such as MedicationRequest and Encounter, while write operations are only prepared for clinician approval (Zakka et al., 2024). In this setting, GUI grounding is secondary to schema-constrained API grounding.

AIRHILT provides a third modality pattern: synchronized multimodal streams with standardized JSON-based interfaces. Its ASR input/output, vision detections, ADS-B tracks, and advisory outputs are all represented as explicit JSON payloads with timestamps, confidence values, slot parses, and evidence fields. The testbed’s modularity allows ASR, visual detection, decision-making, and TTS models to be swapped by editing scenario JSON to point to alternate endpoints (Garib et al., 24 Nov 2025). The systems significance is that interoperability is achieved through typed interfaces and monotonic time alignment, not merely by colocating multiple models.

Runtime assurance systems based on Copilot introduce yet another interface regime. In the Monitoring ICAROUS project, requirements authored in FRETish are translated into temporal logic and then into executable Copilot monitors, which compile to C99 with no dynamic memory allocation and fixed-size arrays. The monitor logic is stream-based; for example, a detect-and-avoid separation property can be encoded as $s$6 and bound to ICAROUS topics through a cFS wrapper (Dutle et al., 2020). Here the relevant “interface” is not a GUI or API endpoint but a typed telemetry stream that can be monitored with predictable time and memory behavior.

4. Human guidance, verification, and hallucination control

A recurrent theme in the literature is that copilots become materially more reliable when they expose explicit verification and human guidance mechanisms. MMAC-Copilot’s hallucination mitigation relies on four concrete mechanisms: coarse-to-fine planning, structured JSON communication with role-specific fields, screenshot-based verification by the Mentor, and the Programmer’s evaluate-and-refine cycle. The details emphasize that “observation” is produced only by Mentor and Viewer and consumed only by the Planner, thereby reducing ambiguous message passing and planning drift (Song et al., 2024).

The Discord case study from MMAC-Copilot makes this concrete. The Planner initially assigns application launch to the Programmer, the Mentor observes from a screenshot that the intended contact is already visible, the Planner revises the task, the Programmer fails because the interface is proprietary and not API-accessible, and the Planner reassigns the task to the Viewer, who performs grounded UI clicks and text entry. The Mentor then verifies from the screenshot that “Hi” appears in the chat, and execution terminates (Song et al., 2024). This sequence illustrates a central MH-COPILOT principle: visual or operational evidence should be allowed to override text-only assumptions about capabilities.

In MOOSE-Copilot, hallucination control takes the form of formalized human intervention. The three explicit HAII signals are not auxiliary annotations but structural factors in the generation process: $R$6 constrains the initial region of the exploratory space, $R$7 controls bidirectional transitions between exploration and exploitation, and $R$8 appends feedback to the context for regeneration. Feedback is integrated as

$R$9

and re-generation is triggered within the active stage (An et al., 28 May 2026). In this framework, human guidance is a routing and correction operator over search, not merely a final approval step.

Clinical systems express the same idea through Level 1 autonomy. Almanac Copilot prepares orders, notifications, and conditional actions, but clinicians review and approve all write operations. The tool schemas explicitly define purpose, parameters, types, and return values, and the limited function catalog constrains the action space (Zakka et al., 2024). This governance model is structurally similar to Mentor verification in MMAC-Copilot, but operationalized through pre-submission approval rather than screenshot comparison.

AIRHILT similarly embeds the human operator in the loop through severity-gated advisories, confidence-gated clarifications, and addressed transmissions. ASR slot confidence below $s$0 triggers clarification; vision confidence below $s$1 requires corroboration or persistence; only advisories above a configured severity threshold are spoken aloud (Garib et al., 24 Nov 2025). In safety-critical contexts, this indicates that MH-COPILOT should communicate uncertainty and request clarification rather than silently continue under low-confidence conditions.

5. Empirical performance and evaluation paradigms

The strongest direct quantitative evidence for the MH-COPILOT design pattern comes from MMAC-Copilot, Almanac Copilot, MOOSE-Copilot, AppCopilot, and AIRHILT. On GAIA, MMAC-Copilot was evaluated on 466 QA tasks with exact-match evaluation for string, number, and float outputs. Its reported scores were 45.16 on Level-1, 20.75 on Level-2, and 6.12 on Level-3, for an average of 25.91, outperforming FRIDAY’s 24.25 by 6.8% on average. On VIBench, which targets non-API-interactable applications in 3D gaming, recreation, and office domains, MMAC-Copilot achieved an average success rate of 70.32%, with 63.16% in 3D gaming, 69.23% in recreation, and 78.57% in office, compared with UFO at 14.65% and FRIDAY at 35.07% (Song et al., 2024).

Almanac Copilot’s benchmark is EHR-QA, a synthetic dataset of 300 clinician-derived queries over information retrieval, summarization, and data entry. The scoring rubric allocates one point each for Functions Called, Parameter Choice, and Script Validity, with sequential scoring. Almanac Copilot achieved a task completion rate of 74% $s$2, a mean score of 2.45/3, and a 95% confidence interval of 2.34–2.56. Baselines reported in the same study include Claude 3 Opus with mean 2.28 and GPT-4 with mean 2.39, while BioMistral scored 0.30 (Zakka et al., 2024).

MOOSE-Copilot’s evaluation is framed as recall of ground-truth fine-grained elements on TOMATO-Chem2. The reported autonomous baselines are MC at 11.44% recall and MC2 at 10.33%. Injecting structured human guidance improves these values: MC plus initial blueprint reaches 15.37%; oracle routing into MC2 reaches 18.26%; standard feedback x3 reaches 22.35% with 123.1 steps; and strong feedback x4 reaches 26.96% with 90.1 steps, compared with the MC2 baseline at 478.6 steps (An et al., 28 May 2026). The quantitative pattern is that structured expert intervention both increases recall and reduces search steps.

AppCopilot reports results across grounding and step-level mobile-action benchmarks. On CAGUI grounding accuracy over 1,500 samples, it achieves an average of 71.3%, with Fun2Point at 79.1%, Text2Point at 76.5%, and Bbox2Text at 58.2%. For step-level action prediction, it reports 94.4% TM / 90.2% EM on AndroidControl-Low, 77.7% TM / 69.2% EM on AndroidControl-High, 90.9% TM / 75.0% EM on GUI-Odyssey, 85.7% TM / 76.4% EM on AITZ, and 96.9% TM / 91.3% EM on CAGUI. Its RFT via GRPO also improves EM relative to SFT on Odyssey, AITZ, and AndroidControl-Low (Fan et al., 2 Sep 2025).

AIRHILT’s preliminary results focus on multimodal timing rather than benchmark accuracy alone. In representative runway-overlap scenarios, the assistant reports an average time-to-first-warning of approximately 7.66 s, average ASR latency of approximately 5.88 s, average vision latency of approximately 0.415 s, and TTS latency of approximately 0.9 s, with first vision detection at approximately 125 m range (Garib et al., 24 Nov 2025). This evaluation style is especially relevant to MH-COPILOT in operational domains because it measures end-to-end detection and advisory timing rather than only offline classification quality.

6. Safety, governance, limitations, and the role of “copilot” in practice

The term “copilot” in the supplied literature does not uniformly imply full autonomy. In clinical deployment, Almanac Copilot is explicitly Level 1: the agent drafts orders and actions, but clinicians review, edit, and approve them within the EHR. The paper emphasizes local execution behind institutional firewalls, avoidance of external API calls with PHI, and compatibility with FHIR; it further notes that SMART on FHIR/OAuth2, role-based access controls, audit logging, and approval workflows are required for deployment, even though those features were not experimentally evaluated (Zakka et al., 2024).

In embedded autonomy, the Monitoring ICAROUS project supplies a complementary notion of safety: not task assistance, but runtime verification of requirements. FRET is used to author requirements in structured natural language; those requirements are formalized into MTL/PTLTL; Ogma translates the past-time logic into Copilot monitors; and Copilot generates predictable C99 code integrated into cFS. The central claim is not superior task completion, but the direct integration of requirements-based monitors into the autonomy stack so that violations can be detected during execution and drive mitigation (Dutle et al., 2020). This suggests that MH-COPILOT, in safety-critical deployments, may need a second channel of assurance beyond the planning agent itself.

The empirical literature on GitHub Copilot adds an important cautionary dimension. Practitioner studies report that the significant benefit is useful code generation, but the main limitation encountered is difficulty of integration, and the overall experience is described as a “double-edged sword.” The 2023 study over 303 Stack Overflow posts and 927 GitHub Discussions reports Visual Studio Code as the dominant IDE at 48.0%, Node.js as the most common technology at more than 45%, useful code generation as the leading benefit, difficulty of integration at 28.1%, and “integrate with more IDEs” as the most common expected feature at 28.8% (Zhang et al., 2023). The earlier 2023 study over 169 Stack Overflow posts and 655 GitHub discussions reports similar patterns, including difficulty of integration at 28.0% and privacy concerns at 7.5% (Zhang et al., 2023).

Performance studies also show that copilot assistance can improve correctness while degrading efficiency if outputs are accepted without expert review. In a within-subjects study of 32 systems programmers solving C++ tasks, Copilot-associated solutions were statistically significantly slower on average, particularly for I/O-heavy code: for Problem A, no-Copilot solutions were approximately 26–29% faster on average, with Wilcoxon $s$3 and mean-of-means $s$4; for Problem B, no-Copilot solutions were approximately 15% faster on average, with Wilcoxon $s$5, although the mean-of-means analysis was not significant. At the same time, Copilot helped participants produce correct solutions within the time budget, especially on Problem A (Erhabor et al., 2023). A separate Python-centered study similarly concludes that humans had a higher correct ratio overall than Copilot’s suggestions, though Copilot’s buggy outputs were often easier to repair (Dakhel et al., 2022). For MH-COPILOT, this indicates that a copilot can accelerate baseline task completion while still requiring expert scrutiny for optimization, constraint satisfaction, and domain correctness.

The principal limitations across the broader copilot literature are consistent. MMAC-Copilot reports remaining failures from complex UI understanding, multi-step operations in intricate interfaces, limited spatial reasoning in 3D contexts, and longer inference times on temporally sensitive tasks (Song et al., 2024). MOOSE-Copilot notes noisy LLM self-evaluation, bias from the inspiration corpus, and the absence of closed-loop empirical experiment execution (An et al., 28 May 2026). AppCopilot highlights Chinese-heavy dataset coverage, difficulties with dynamic UIs and rare workflows, and current limits of fully on-device deployment on mid-tier phones (Fan et al., 2 Sep 2025). AIRHILT identifies sim-to-real gap, limited human-factors validation, and unoptimized latency as present constraints (Garib et al., 24 Nov 2025).

Taken together, these systems imply that MH-COPILOT is most rigorously conceived as a verified, multimodal, human-steered copilot architecture rather than an unconstrained autonomous agent. Its mature forms combine specialized roles, structured interfaces, explicit verification, constrained tool schemas, and domain-specific governance. Where these elements are absent, the literature consistently reports hallucination, integration friction, degraded optimization quality, or brittle behavior. Where they are present, the reported gains are in grounded execution, improved task completion, reduced search steps, or measurable end-to-end operational performance (Song et al., 2024, Zakka et al., 2024, An et al., 28 May 2026, Fan et al., 2 Sep 2025, Garib et al., 24 Nov 2025, Dutle et al., 2020).