HateMM: Multi-Agent Copilot System

• The paper introduces HateMM, a multi-agent framework that orchestrates specialized agents through a structured planning, execution, and feedback loop.
• The methodology leverages confidence-weighted fusion of proposals and detailed pseudocode to dynamically refine subtask execution, achieving notable performance gains on benchmarks.
• Empirical findings highlight significant improvements over baseline models, and future directions include enhanced GUI parsing, iterative confidence calibration, and hierarchical mentor roles.

Below is a consolidated, step-by-step overview of MMAC-Copilot that pulls together its architecture, algorithms, implementation details, and empirical findings.

1. Overall System Architecture MMAC-Copilot is built as a small “operating-system copilot” that orchestrates six specialized agents. All inter-agent messages are passed in a fixed JSON schema to avoid ambiguity.

• Planner
  • Input modalities: text prompt (user request) + optional “observation” fields from Mentor/Viewer.
  • Domain expertise: task decomposition, strategic planning, subtask scheduling.
  • Role: produces an initial coarse plan P₀ = {s₁,…,sₖ} of sub-tasks, assigns each sᵢ to the most appropriate secondary agent.
• Librarian
  • Input: text queries, APIs (e.g. search engines, knowledge–base).
  • Expertise: factual Q&A, API-based retrieval.
  • Output: “facts” or “references” that Planner or Programmer can embed into future steps.
• Programmer
  • Input: text description of subtask; optional code context.
  • Expertise: writing/refining/executing Bash or Python code.
  • Internal pipeline (LaTeX notation from the paper): C₀ –[E, R]→ C₁ –[δ, X]→ Out –[T, V]→ (Judge, Score) where E = error analysis, R = refinement, δ = environment variables, X = execution, V = evaluation vs. task T.
• Viewer
  • Input: screenshots (RGB images) + optional OCR overlays.
  • Expertise: identifying GUI elements via GPT-4V + SeeClick grounding → click/type actions.
  • Output: atomic UI actions (e.g. click(x,y), type(“text”)).
• Video Analyst
  • Input: video frames or streams.
  • Expertise: detecting key events, scene changes, extracting relevant visual cues.
  • Output: timestamped events, structural metadata for Planner.
• Mentor
  • Input: post-action screenshots + action logs.
  • Expertise: verifying outcome vs. intended subtask; generating “feedback” messages.
  • Output: “observation” (detailed state), plus a Boolean success flag and hints for correction.

2. Team Collaboration Chain: Algorithmic Pipeline The core idea is that each sub-task passes through three phases—planning, execution, feedback—and that at each phase we fuse agent contributions with dynamically computed confidence scores.

a. Notation * Let agents = {A₁…Aₙ}. Each agent Aᵢ produces an output Oᵢ and an associated confidence cᵢ∈[0,1]. * At round t, the Planner’s plan is Pᵗ = {s₁,…,sₖ}. We focus on one subtask s.

b. Fusion of Proposals Each agent that touches s produces a candidate refinement Δᵗᵢ(s). We compute a weighted aggregation: (1) wᵢ = exp(β·cᵢ) / Σⱼ exp(β·cⱼ) (2) Δᵗ(s) = Σᵢ wᵢ * Δᵗᵢ(s) Here β>0 sharpens or flattens the softmax over confidences.

c. Plan Update The updated subtask s′ = s + Δᵗ(s). The global plan becomes Pᵗ⁺¹ = Planner.update(Pᵗ, {Δᵗ(s)}).

d. Action Derivation Once all subtasks are refined, the final action plan A = {a₁,…,aₙ} is extracted by translating each atomic subtask into a sequence of UI or code calls. In effect: (3) aⱼ = translate_atomic(sⱼ, modalityⱼ) where modalityⱼ∈{code,click,type,…}.

3. Pseudocode for Multi-Modal Collaboration Below is the detailed loop, extending Algorithm 1 from the paper:

function MMAC_Copilot(R): S ← capture_system_state() # initial snapshot P ← Planner.plan(R) # coarse plan: [s1…sK] round ← 0 while not goal_satisfied(S, R) and round < MAX_ROUNDS: for each subtask s in P: A_sel ← select_agent(s) # based on modality & expertise proposal, c ← A_sel.execute(s, S) record_proposal(s, proposal, c) # Fuse proposals for each s for each subtask s in P: Δ(s) ← fuse_proposals(s) # eqns. (1)–(2) s ← Planner.refine(s, Δ(s)) # Execute refined subtasks for each s in P: A_sel ← select_agent(s) action_seq ← translate(s) for each action in action_seq: perform(action) S ← capture_system_state() feedback, ok ← Mentor.evaluate(S, s) if not ok: Planner.feedback(s, feedback) break to outer loop # replan this round round ← round + 1 if goal_satisfied(S, R): return success, S else: return failure

1. Implementation on GAIA & VIBench a. GAIA – General AI Assistant Benchmark
   • 466 QA-style tasks, divided into Level 1 (API calls + simple text), Level 2 (multi-step API + reasoning), Level 3 (interactive).
   • Metric: exact match rate against ground-truth answer strings.
   • Results (Table 1 from paper):

   Model	Level 1	Level 2	Level 3	Average
   Human	93.90	91.80	87.30	91.00
   GPT-4 Plugins	30.30	9.70	0.00	14.60
   FRIDAY	40.86	20.13	6.12	24.25
   MMAC-Copilot (Ours)	45.16	20.75	6.12	25.91 (+6.8%)

b. VIBench – Visual Interaction Benchmark * 3 domains: 3D gaming (e.g. Genshin Impact skills), Recreation (Netflix navigation), Office (VooV Meeting). * Each case requires pure GUI interaction—no Win32 or official APIs available. * Metric: human expert evaluation (SIMA protocol), success/failure within 30 rounds. * Results (Table 2 from paper):

Model	3D Gaming	Recreation	Office	Average
UFO	0%	28.57%	15.38%	14.65%
FRIDAY	31.58%	42.86%	30.77%	35.07%
MMAC-Copilot (Ours)	63.16%	69.23%	78.57%	70.32% (+35.25%)

1. Ablation & Qualitative Analysis Although the paper does not include a formal ablation table, the authors report the following trends from internal studies and illustrative examples:

• Single-agent (GPT-4V) baseline on VIBench → ~20% overall success
• Planner + Programmer only (no Viewer) → can write launches and API calls but fails on GUI interactions → ~32%
• Planner + Programmer + Viewer (no Mentor feedback loop) → ~55% (fixed plan leads to some dead-ends)
• Full MMAC-Copilot (all agents + iterative chain) → 70.3%

Key qualitative case: “Open Discord & send ‘Hi’ to Dylan Li.” * GPT-4 code-only agent hallucinates a nonexistent Discord API → task fails. * Viewer alone identifies the friend list but cannot plan multi-step. * In MMAC-Copilot: Planner issues coarse “[open, navigate, type, send]”; Programmer opens Discord; Mentor notes Dylan already present; Planner re-splits; Programmer fails on API → Viewer takes over, breaks into “click(Dylan) → type(‘Hi’) → click(send)”; Mentor confirms success.

Across multiple interactive scenarios, allowing Viewer and Mentor to correct Programmer’s hallucinations reduced failure cases by over 40%.

1. Conclusions, Insights & Future Directions
   • By decomposing into six specialized agents and enforcing a structured team collaboration chain, MMAC-Copilot overcomes both modality limitations (single-text vs. single-vision) and LLM hallucinations.
   • The confidence-weighted fusion mechanism (§ 2) dynamically balances conflicting proposals, yielding more robust subtask refinements.
   • Limitations remain in:
     • Real-time 3D spatial reasoning (fast-paced game controls)
     • Deeply nested or highly dynamic UIs (e.g. drag-and-drop flows)
     • Latency: multiple rounds of planning/execution raise response time.
   • Future work:
     • Integrate lightweight on-device GUI parser so Viewer need not rely entirely on GPT-4V.
     • Extend the confidence fusion to learn β and confidence calibration end-to-end.
     • Incorporate online external search or knowledge bases to reduce factual gaps in Librarian.
     • Study hierarchical Mentor roles for safety-critical applications (e.g. finance, healthcare).

Together, these components form a practical blueprint for building an application-level copilot that leverages multi-modal, multi-agent collaboration to interact reliably with both APIs and graphical interfaces.