OS-Copilot Framework: Unified OS Agents

• OS-Copilot frameworks are unified, extensible architectures that enable language model-driven agents to perceive, reason, and act across diverse operating system interfaces.
• They employ a modular design with components like Planner, Configurator, Actor, and Critic, leveraging DAG planning, retrieval-augmented generation, and multimodal perception.
• They are deployed on desktop, mobile, and edge environments with continuous self-improvement through iterative tool learning and empirical benchmarking.

An OS-Copilot framework is a unified, extensible architecture for constructing generalist, LLM-driven agents capable of perceiving, reasoning, and acting across heterogeneous applications and interfaces within an operating system. OS-Copilot systems integrate planning, tool invocation, multimodal perception, and self-improvement, targeting both desktop and mobile/edge environments. By orchestrating multi-stage planning, contextual retrieval, interface-agnostic execution, and continuous skill acquisition, OS-Copilot frameworks address the challenges of generalization, robustness, and usability in digital autonomy (Wu et al., 2024, Wang et al., 2024, Fan et al., 2 Sep 2025, Song et al., 2024).

1. Architectural Foundations

OS-Copilot frameworks adopt a modular, agent-centered design separating high-level planning from tool-level execution, with shared declarative, procedural, and working memory modules (Wu et al., 2024). Central to the architecture is a Planner, responsible for goal decomposition and dependency analysis, typically yielding a directed acyclic graph (DAG) of subtasks. The Configurator maintains system, user, and tool state, supporting memory-augmented retrieval of prior experience and tool definitions. The Actor executes atomic actions using a variety of actuators—ranging from code terminals to API endpoints and GUI controls. Critique is incorporated via a Critic module, often LLM-based, iteratively assessing completion and proposing refinements.

The invocation stack typically incorporates:

• LLM Executor (LLM acting as planner/critic/actor)
• Runtimes: Python interpreter (scriptable control), Bash shell (system ops), HTTP/API calls, GUI interaction (mouse, keyboard)
• Memory Index: Dense retrieval over symbolic and experience embeddings

A high-level control loop alternates between plan generation, subtask assignment, execution, and result or critique handling, looping until the user goal is achieved or resources are exhausted (Wu et al., 2024, Song et al., 2024).

2. Retrieval-Augmented and Multimodal Perception

Contextual perception is achieved through hybrid retrieval-augmented generation (RAG) modules and multimodal vision-language grounding. Systems such as CAMP-inspired OS-Copilots index and retrieve symbolic code context via Dynamic Code Symbol Indexing (DCSI), rank candidates using similarity scoring (e.g., cosine or bilinear transform), and chunk relevant snippets into prompt components under token constraints. Formally, document relevance for retrieval is computed as:

$$p_{\eta}(z \mid x, c) = \frac{\exp\left(\psi(z)^\top H \phi(x, c)\right)}{\sum_{z'} \exp\left(\psi(z')^\top H \phi(x, c)\right)}$$

where $z$ is a candidate document, $x$ the current context, $c$ the prompt context, $\phi$ and $\psi$ are embedding functions, and $H$ a parameter matrix (Wang et al., 2024).

Multimodal OS-Copilot variants incorporate vision encoders (ViT-style) and text transformers, merging image (e.g., screenshot) and language features to emit JSON-structured actions. Integrations with SeeClick facilitate vision-to-action mapping for GUI interaction, and video agents analyze time sequences for complex environments. All agent communications are typically mediated via structured JSON schemas (Song et al., 2024, Fan et al., 2 Sep 2025).

3. Planning, Reasoning, and Control Mechanisms

OS-Copilots employ LLM-driven, plan-and-solve paradigms—often refined into formal DAG planning, enabling parallel subagent execution when task dependencies allow (Wu et al., 2024, Fan et al., 2 Sep 2025). Chain-of-thought (CoT) prompting is leveraged for stepwise reasoning, with models emitting an interleaved series of "Thought" and "Action" tokens before converting action plans into executable JSON payloads.

Subtasks are dynamically assigned to agents specialized by domain (e.g., code, API, GUI, vision), with a collaborative "team chain" protocol ensuring planner oversight, domain-specific execution, and mentor/critic feedback aggregation. Notably, Mentor agents hold veto authority to trigger replanning upon detecting errors or subgoal failures (Song et al., 2024).

Tool invocation is abstraction-agnostic: actors may call Bash or Python, trigger REST APIs, or dispatch low-level mouse/keyboard events as appropriate. Experience logs and memory buffers track prior successful trajectories, enabling replay or partial plan reuse for efficiency (Fan et al., 2 Sep 2025).

4. Integration and System Deployment

To operate in OS-constrained or sandboxed environments (e.g., macOS/iOS IDEs), OS-Copilot frameworks segregate heavy logic—file I/O, embedding computation, HTTP clients—into unsandboxed or privileged helpers (e.g., XPC services on macOS), while lightweight UI plugins maintain clipboard, cursor, and context monitoring, mediated by IPC or platform accessibility APIs (Wang et al., 2024).

In mobile and edge deployments (AppCopilot variant), core models are compressed through structured pruning, quantization, and distillation to sub-1 GB footprints, engineered for heterogeneous hardware runtime (mobile GPU/NPU, ONNX/TensorRT). Offloading policies dynamically switch execution between local and cloud inference based on device load and latency targets (Fan et al., 2 Sep 2025).

SDKs provide rich integration points across languages (Java/Kotlin/Swift), facilitating orchestration of voice, cross-app, and cross-device intents and function calls.

5. Self-Improvement and Skill Acquisition

A distinctive facet is self-directed tool learning: given a new application domain, OS-Copilot agents autonomously propose, attempt, and refine a curriculum of tasks, iteratively generating and scoring new atomic tools (Python classes/API wrappers) based on correctness and generality. Tool retention is governed by thresholds on critic ratings (e.g., $s_i \geq 8$ out of 10) (Wu et al., 2024). No gradient updates are performed; instead, iterative LLM synthesis and validation drive procedural memory expansion.

Experience reuse and semantic retrieval of prior episodes further enhance generalization, as agents can rapidly "replay" actions for familiar tasks identified via language similarity (Fan et al., 2 Sep 2025). This process enables long-horizon task mastery and robust adaptation to novel applications.

6. Empirical Evaluation and Benchmarking

OS-Copilot frameworks are empirically benchmarked on generalist agent testbeds such as GAIA (466 tasks across file, web, code, and multimedia domains) and the Visual Interaction Benchmark (VIBench) for GUI-heavy applications, including non-API-interactable targets (e.g., 3D games, office suites) (Wu et al., 2024, Song et al., 2024). Performance metrics include:

• Success rate at multiple task complexity levels (Level 1: 40.86%, Level 2: 20.13%, Level 3: 6.12% for FRIDAY; MMAC-Copilot +6.8% vs baseline)
• GUI location accuracy (Fun2Point: 79.1% with AppCopilot)
• Long-horizon workflow success (85% for >50 step workflows, surpassing prior systems)
• Latency and resource efficiency (on-device inference <200 ms/256 tokens, <1 W power consumption)

Subjective user studies in IDE settings report significant reductions in completion time and user preference for in-IDE assistants (Wang et al., 2024).

Model	L1	L2	L3	GAIA Avg	VIBench Avg	GUI Loc (Fun2Pt)	Latency
FRIDAY	40.86	20.13	6.12	24.25	35.07	—	118 s
MMAC-Copilot	45.16	20.75	6.12	25.91	70.32	—	—
AppCopilot	—	—	—	—	—	79.1%	200 ms

7. Limitations and Future Research Directions

OS-Copilot systems face multiple open challenges:

• Prompting brittleness and sensitivity to LLM backend changes (Wu et al., 2024)
• Ineffective control of closed-source GUI applications without visual grounding; ongoing work integrates models such as SeeClick for vision-action mapping
• Evaluation labor and lack of standardized auditing for subtask success, safety, and interpretability
• Real-time inference constraints, especially in video-rich or 3D task environments, due to current agent round latency
• The need for expanded datasets for broader generalization (task/apps/languages/UI types) and more reliable API–GUI fallback strategies (Fan et al., 2 Sep 2025)

Future directions include hybrid planning with 3D scene understanding, federated and self-supervised learning for broad adaptation, deeper long-term memory for multi-agent orchestration, aggressive model compression (e.g., 4-bit quantization, distillation), and formal safety/alignment research (Fan et al., 2 Sep 2025, Wu et al., 2024, Song et al., 2024).