FRIDAY Agent: Autonomous OS Automation

• FRIDAY is a self-improving embodied agent that automates OS tasks via a multi-stage pipeline combining Planner, Configurator, and Actor modules.
• It decomposes high-level queries into DAG-structured subtasks and applies embedding-based retrieval for dynamic tool generation and reuse.
• Empirical evaluations show FRIDAY achieves up to 40.86% success on benchmark tasks, outperforming conventional digital agents.

FRIDAY is a self-improving, embodied agent for the automation of general computer tasks, designed atop the OS-Copilot framework to provide broad, generalist competence across operating system environments. Unlike conventional digital agents—typically limited to narrow domains—FRIDAY interfaces with a wide array of OS elements, including web resources, code terminals, files, multimedia content, and third-party applications. It employs a multi-stage architecture and a self-directed learning mechanism to accumulate, evaluate, and reuse skills across tasks, enabling generalization and continuous improvement (Wu et al., 2024).

1. System Architecture

FRIDAY is structured as a pipeline of interleaved modules organized around three principal stages: Planner, Configurator, and Actor, together with a universal OS interface layer and a skill memory module.

The logical data flow is as follows:

$$J(\pi) = \mathbb{E}_{\tau \sim \mathcal{T}}\left[\mathbf{1}\{\text{success}(\tau)\}\right],$$1

Subtasks in the directed acyclic graph (DAG) produced by the Planner may independently instantiate their own Configurator and Actor, facilitating parallelization when subgraphs are independent.

The core components are:

• Planner: Decomposes high-level user requests into a DAG of atomic subtasks, utilizing a prompt-driven graph planner. Each node is specified as JSON with fields: name, type (Code, API, QA), description, and dependency list.
• Configurator: Aggregates relevant data for subtask execution.
  • Working Memory: Current subtask, tool documentation, system state (current working directory, file tree), and user profile.
  • Declarative Memory: Factual history—OS version, preferences, and execution trajectories.
  • Procedural Memory: Repository of atomic tools (both seeded and self-generated).
• Actor:
  • Executor: Issues executable actions (Bash commands or Python calls in the form <invoke>ToolClass()(args)</invoke>), routed through one of four OS interface runtimes.
  • Critic: Applies an LLM to before/after state snapshots, returning $$(\mathrm{judge} \in \{\mathrm{true},\mathrm{false}\}, s\in[1,10])$$ as a judgment-success pair, with error analysis and suggestions.
  • Refiner: If $\mathrm{judge} = \mathrm{false}$, attempts up to three rounds of correction using Critic feedback.

The OS interface layer encompasses four distinct runtimes:

• Python Interpreter (e.g., via python-pptx, openpyxl for files such as Excel, PowerPoint, CSV, JSON)
• Bash Shell (system-level I/O, package management, calls)
• API Calls (HTTP endpoints, web or ML backends)
• Mouse/Keyboard Automation (GUI interaction)

All tools within procedural memory conform to a uniform interface—either instantiable Python classes inheriting BaseAction or callable remote APIs.

2. Learning and Self-Improvement Algorithms

FRIDAY eschews end-to-end gradient-based training, instead implementing a self-directed learning loop that iteratively generates, attempts, analyzes, and accrues new tools and skills.

The effective task coverage is informally formalized as:

$$J(\pi) = \mathbb{E}_{\tau \sim \mathcal{T}}\left[\mathbf{1}\{\text{success}(\tau)\}\right],$$

with success(τ) assigned by the Critic. For tool generation, the expected tool quality objective is

$$Q(\theta) = \mathbb{E}_{\text{tool}}\left[s_{\text{tool}}\right],\, s_{\text{tool}} \in [1,10],$$

and only tools with $s_{\text{tool}} \ge 8$ are retained.

The self-improvement loop executes the following steps:

1. For a subtask $\tau$, attempt retrieval in ToolRepo $\mathcal{R}$ using similarity-based search (§3.1).
2. If absent, synthesize a new tool via ToolGenerator and add to $\mathcal{R}$.
3. For up to three execution attempts:
   • Issue action via Executor;
   • Critic evaluates result, returning $(\mathrm{judge}, s)$;
   • On $\mathrm{judge} = \mathrm{true}$ and $\mathrm{judge} = \mathrm{false}$0, retain tool; else, apply Refiner-and-repeat.

Results are measured with:

• Success indicator $\mathrm{judge} = \mathrm{false}$1,
• Tool-quality score $\mathrm{judge} = \mathrm{false}$2,
• Pass@1 for multi-step tasks (e.g., SheetCopilot use-case).

3. Skill Accumulation and Generalization

Tool retrieval is embedding-based: each tool $\mathrm{judge} = \mathrm{false}$3 is indexed by $\mathrm{judge} = \mathrm{false}$4, and for a subtask description $\mathrm{judge} = \mathrm{false}$5 with embedding $\mathrm{judge} = \mathrm{false}$6, similarity is measured by

$\mathrm{judge} = \mathrm{false}$7

retaining $\mathrm{judge} = \mathrm{false}$8 if similarity exceeds the threshold $\mathrm{judge} = \mathrm{false}$9.

After successful execution on new subtasks, FRIDAY appends the new tool (code, metadata) to its repository, making it available for reuse in subsequent tasks. This mechanism has enabled transfer between disparate task domains (e.g., learned Excel tools reused in PowerPoint subtasks) (Wu et al., 2024).

4. Empirical Evaluation

FRIDAY’s performance has been measured primarily on the GAIA benchmark (466 test tasks), summarized below:

System	Level 1	Level 2	Level 3
GPT-4 Plugins	30.30%	9.70%	0.00%
AutoGPT-4	15.05%	0.63%	0.00%
FRIDAY (w/o learning)	36.56%	17.61%	6.12%
FRIDAY (full)	40.86%	20.13%	6.12%

A relative gain of approximately 35% on Level 1 is observed compared to GPT-4 Plugins.

In the SheetCopilot-20 Excel suite (Pass@1):

Agent	Pass@1
SheetCopilot (GPT-4)	55%
FRIDAY (w/o learning)	0%
FRIDAY (self-learn)	60%

By self-instructing on synthetic Excel tasks, FRIDAY autonomously generated openpyxl-based tools and exceeded the performance of dedicated spreadsheet agents.

For PowerPoint tasks, prior to self-learning, FRIDAY fails at tasks such as font size or image positioning (Critic returns $$J(\pi) = \mathbb{E}_{\tau \sim \mathcal{T}}\left[\mathbf{1}\{\text{success}(\tau)\}\right],$$0). After self-learning, it generates specialized python-pptx tools and meets specification targets.

5. Limitations and Open Challenges

Current system limitations are:

• Web interaction is limited to reading; browser automation (clicks, form fill) is not implemented.
• Multimodal capabilities depend on remote APIs; local deployments are slow.
• The system is sensitive to LLM backend: prompt drift induces significant performance variance, and no end-to-end fine-tuning has been conducted.
• For closed-source GUIs, visual grounding (screenshot-to-action) is future work.

A plausible implication is that while FRIDAY demonstrates robust self-improvement and cross-domain skill transfer, its deployment in highly multimodal, visually rich, or security-sensitive OS contexts remains constrained by these limitations (Wu et al., 2024).

6. Context and Significance

FRIDAY demonstrates a distinct approach to general computer task automation: prioritizing skill accumulation, self-critique, and autonomous tool genesis over monolithic model-based learning. By decomposing tasks, embedding-based skill retrieval, and rigorous Critic/Refiner analysis, FRIDAY achieves state-of-the-art results on generalist computer agent benchmarks (GAIA) and provides a blueprint for future research in OS-level autonomous agents capable of rapid adaptation and self-improvement without end-to-end retraining.

The OS-Copilot/FRIDAY architecture, and its empirical strategies for continuous skill harvesting and transfer, potentially mark a substantive advance towards more flexible and intelligent personal computer agents (Wu et al., 2024).