--- title: 'HateMM: Multi-Agent Copilot System' url: https://www.emergentmind.com/topics/hatemm type: topic --- # HateMM: Multi-Agent Copilot System Below is a consolidated, step-by-step overview of MMAC-Copilot that pulls together its architecture, algorithms, implementation details, and empirical findings. 1. Overall System Architecture MMAC-Copilot is built as a small “operating-system copilot” that orchestrates six specialized agents. All inter-agent messages are passed in a fixed JSON schema to avoid ambiguity. • Planner – Input modalities: text prompt (user request) + optional “observation” fields from Mentor/Viewer. – Domain expertise: task decomposition, strategic planning, subtask scheduling. – Role: produces an initial coarse plan P₀ = {s₁,…,sₖ} of sub-tasks, assigns each sᵢ to the most appropriate secondary agent. • Librarian – Input: text queries, APIs (e.g. search engines, knowledge–base). – Expertise: factual Q&A, API-based retrieval. – Output: “facts” or “references” that Planner or Programmer can embed into future steps. • Programmer – Input: text description of subtask; optional code context. – Expertise: writing/refining/executing Bash or Python code. – Internal pipeline (LaTeX notation from the paper): C₀ –[E, R]→ C₁ –[δ, X]→ Out –[T, V]→ (Judge, Score) where E = error analysis, R = refinement, δ = environment variables, X = execution, V = evaluation vs. task T. • Viewer – Input: screenshots (RGB images) + optional OCR overlays. – Expertise: identifying GUI elements via GPT-4V + SeeClick grounding → click/type actions. – Output: atomic UI actions (e.g. click(x,y), type(“text”)). • Video Analyst – Input: video frames or streams. – Expertise: detecting key events, scene changes, extracting relevant visual cues. – Output: timestamped events, structural metadata for Planner. • Mentor – Input: post-action screenshots + action logs. – Expertise: verifying outcome vs. intended subtask; generating “feedback” messages. – Output: “observation” (detailed state), plus a Boolean success flag and hints for correction. 2. Team Collaboration Chain: Algorithmic Pipeline The core idea is that each sub-task passes through three phases—planning, execution, feedback—and that at each phase we fuse agent contributions with dynamically computed confidence scores. a. Notation – Let agents = {A₁…Aₙ}. Each agent Aᵢ produces an output Oᵢ and an associated confidence cᵢ∈[0,1]. – At round t, the Planner’s plan is Pᵗ = {s₁,…,sₖ}. We focus on one subtask s. b. Fusion of Proposals Each agent that touches s produces a candidate refinement Δᵗᵢ(s). We compute a weighted aggregation: (1) wᵢ = exp(β·cᵢ) / Σⱼ exp(β·cⱼ) (2) Δᵗ(s) = Σᵢ wᵢ · Δᵗᵢ(s) Here β>0 sharpens or flattens the softmax over confidences. c. Plan Update The updated subtask s′ = s + Δᵗ(s). The global plan becomes Pᵗ⁺¹ = Planner.update(Pᵗ, {Δᵗ(s)}). d. Action Derivation Once all subtasks are refined, the final action plan A = {a₁,…,aₙ} is extracted by translating each atomic subtask into a sequence of UI or code calls. In effect: (3) aⱼ = translate_atomic(sⱼ, modalityⱼ) where modalityⱼ∈{code,click,type,…}. 3. Pseudocode for Multi-Modal Collaboration Below is the detailed loop, extending Algorithm 1 from the paper: ``` function MMAC_Copilot(R): S ← capture_system_state() # initial snapshot P ← Planner.plan(R) # coarse plan: [s1…sK] round ← 0 while not goal_satisfied(S, R) and round < MAX_ROUNDS: for each subtask s in P: A_sel ← select_agent(s) # based on modality & expertise proposal, c ← A_sel.execute(s, S) record_proposal(s, proposal, c) # Fuse proposals for each s for each subtask s in P: Δ(s) ← fuse_proposals(s) # eqns. (1)–(2) s ← Planner.refine(s, Δ(s)) # Execute refined subtasks for each s in P: A_sel ← select_agent(s) action_seq ← translate(s) for each action in action_seq: perform(action) S ← capture_system_state() feedback, ok ← Mentor.evaluate(S, s) if not ok: Planner.feedback(s, feedback) break to outer loop # replan this round round ← round + 1 if goal_satisfied(S, R): return success, S else: return failure ``` 4. Implementation on GAIA & VIBench a. GAIA – General AI Assistant Benchmark – 466 QA-style tasks, divided into Level 1 (API calls + simple text), Level 2 (multi-step API + reasoning), Level 3 (interactive). – Metric: exact match rate against ground-truth answer strings. – Results (Table 1 from paper): Model | Level 1 | Level 2 | Level 3 | Average -------------------|---------|---------|---------|--------- Human | 93.90 | 91.80 | 87.30 | 91.00 GPT-4 Plugins | 30.30 | 9.70 | 0.00 | 14.60 FRIDAY | 40.86 | 20.13 | 6.12 | 24.25 MMAC-Copilot (Ours)| 45.16 | 20.75 | 6.12 | 25.91 (+6.8%) b. VIBench – Visual Interaction Benchmark – 3 domains: 3D gaming (e.g. Genshin Impact skills), Recreation (Netflix navigation), Office (VooV Meeting). – Each case requires pure GUI interaction—no Win32 or official APIs available. – Metric: human expert evaluation (SIMA protocol), success/failure within 30 rounds. – Results (Table 2 from paper): Model | 3D Gaming | Recreation | Office | Average -------------------|-----------|------------|---------|-------- UFO | 0% | 28.57% | 15.38% | 14.65% FRIDAY | 31.58% | 42.86% | 30.77% | 35.07% MMAC-Copilot (Ours)| 63.16% | 69.23% | 78.57% | 70.32% (+35.25%) 5. Ablation & Qualitative Analysis Although the paper does not include a formal ablation table, the authors report the following trends from internal studies and illustrative examples: • Single-agent (GPT-4V) baseline on VIBench → ~20% overall success • Planner + Programmer only (no Viewer) → can write launches and API calls but fails on GUI interactions → ~32% • Planner + Programmer + Viewer (no Mentor feedback loop) → ~55% (fixed plan leads to some dead-ends) • Full MMAC-Copilot (all agents + iterative chain) → 70.3% Key qualitative case: “Open Discord & send ‘Hi’ to Dylan Li.” – GPT-4 code-only agent hallucinates a nonexistent Discord API → task fails. – Viewer alone identifies the friend list but cannot plan multi-step. – In MMAC-Copilot: Planner issues coarse “[open, navigate, type, send]”; Programmer opens Discord; Mentor notes Dylan already present; Planner re-splits; Programmer fails on API → Viewer takes over, breaks into “click(Dylan) → type(‘Hi’) → click(send)”; Mentor confirms success. Across multiple interactive scenarios, allowing Viewer and Mentor to correct Programmer’s hallucinations reduced failure cases by over 40%. 6. Conclusions, Insights & Future Directions • By decomposing into six specialized agents and enforcing a structured team collaboration chain, MMAC-Copilot overcomes both modality limitations (single-text vs. single-vision) and LLM hallucinations. • The confidence-weighted fusion mechanism (§ 2) dynamically balances conflicting proposals, yielding more robust subtask refinements. • Limitations remain in: – Real-time 3D spatial reasoning (fast-paced game controls) – Deeply nested or highly dynamic UIs (e.g. drag-and-drop flows) – Latency: multiple rounds of planning/execution raise response time. • Future work: – Integrate lightweight on-device GUI parser so Viewer need not rely entirely on GPT-4V. – Extend the confidence fusion to learn β and confidence calibration end-to-end. – Incorporate online external search or knowledge bases to reduce factual gaps in Librarian. – Study hierarchical Mentor roles for safety-critical applications (e.g. finance, healthcare). Together, these components form a practical blueprint for building an application-level copilot that leverages multi-modal, multi-agent collaboration to interact reliably with both APIs and graphical interfaces. Source: https://www.emergentmind.com/topics/hatemm