DeskCraft: Desktop Benchmark for AI Workflows

• DeskCraft is a desktop GUI benchmark that assesses AI agents on multi-step, long-horizon creative and engineering workflows using realistic software tasks.
• The framework formalizes evaluation through a phase-conditioned task model, a three-level L1/L2/L3 difficulty taxonomy, and execution-based verification over final system state.
• It incorporates a human–agent collaboration protocol that supports mid-turn clarifications and post-turn feedback to emulate dynamic desktop interactions.

Searching arXiv for the latest DeskCraft-related papers and closely related work to ground the article. I’ll check arXiv for DeskCraft and related papers on desktop agents and craft workflow documentation. DeskCraft is a desktop GUI benchmark for evaluating AI agents on long-horizon professional workflows and proactive human–agent collaboration in live desktop environments. It targets creative and engineering software rather than short atomic computer-use tasks, and formalizes both execution difficulty and interaction dynamics through a phase-conditioned task model, a multilevel L1/L2/L3 taxonomy, and a protocol covering mid-turn and post-turn exchanges. In its published form, DeskCraft contains 538 tasks spanning 11 applications across 5 domains, with execution-based verifiers over final desktop state (Wang et al., 2 Jun 2026).

1. Definition and research scope

DeskCraft was introduced to address a specific gap in desktop-agent evaluation: existing GUI benchmarks are described as valuable, but mainly focused on single, short, atomic tasks with all instructions provided upfront and minimal or no explicit modeling of human–agent interaction during execution. DeskCraft instead targets realistic desktop work in which agents must carry out multi-step pipelines in professional tools, coordinate with users whose goals evolve over time, and deliver verifiable artifacts such as rendered videos, edited documents, code projects, or 3D scenes (Wang et al., 2 Jun 2026).

The benchmark is explicitly situated around long-horizon creative and engineering workflows. Representative settings include video editing in Kdenlive, 3D creation in Blender, audio production in Audacity, image and vector design in GIMP and Inkscape, office work in LibreOffice, and compound developer workflows using VS Code, Chrome, and OS-level tools. This focus distinguishes DeskCraft from benchmarks centered on simpler screen grounding or short instruction following.

A recurrent misconception is to treat DeskCraft as a desktop automation system or a foundation model in its own right. In the published benchmark, DeskCraft is neither: it is an execution-based evaluation framework in a live Ubuntu VM, designed to measure planning, state management, error recovery, and mixed-initiative collaboration under realistic GUI control (Wang et al., 2 Jun 2026).

2. Formal task model and difficulty taxonomy

DeskCraft formulates GUI agent behavior as a phase-conditioned control problem,

$\tau = (s_0,\; u_0,\; \Phi,\; \mathcal{E},\; R),$

where $s_0$ is the initial desktop state, $u_0$ the initial user instruction, $\mathcal{E}$ the desktop environment, $\Phi = (\phi_1,\dots,\phi_K)$ an optional sequence of interaction phases, and $R$ a binary evaluation function over the final state. Each phase is

$\phi_k = (u_k, g_k),$

with $u_k$ a follow-up user message and $g_k(\cdot)\in\{0,1\}$ a trigger deciding when the message is injected. At each step, the agent receives a screenshot and the current active instruction, and selects

$$a_t \in \mathcal{A} \cup \{\text{DONE},\; \text{ASK},\; \text{FAIL}\}.$$

Standard tasks have $s_0$0, while interactive tasks have $s_0$1; success is determined by $s_0$2 on the final desktop state (Wang et al., 2 Jun 2026).

The difficulty structure is a central part of the benchmark. DeskCraft organizes standard tasks into three levels. L1 (Atomic) covers simple atomic operations or very short sequences with negligible composition. L2 (Compositional) consists of related L1 operations, typically 2–4 dependent operations that must be coordinated in the right order. L3 (Long horizon workflows) corresponds to realistic multi-subtask workflows distilled from real delivery pipelines, often over 50 execution steps. The abstract states that long horizon tasks require over 50 execution steps (Wang et al., 2 Jun 2026).

Difficulty is defined by execution capability rather than surface instruction length, but the benchmark reports empirical correlations: median instruction length rises from 186 characters for L1 to 501 for L3; evaluator calls increase from 1.46 to 2.00; rule atoms increase from 3.9 to 7.7. This design makes horizon a property of operational structure, not merely prompt verbosity.

3. Human–agent collaboration protocol

DeskCraft’s interaction protocol formalizes human-in-the-loop collaboration as a sequence of phases triggered during execution. It defines three triggers, described as a closed-loop minimal set spanning mid-turn and post-turn interaction (Wang et al., 2 Jun 2026).

Trigger	When it fires	Collaboration mode
agent_done	When the agent emits DONE	Post-turn feedback or correction
agent_ask	When the agent emits ASK	Agent-initiated clarification under uncertainty
step_count	After a predetermined number of steps	User-initiated interruption

Mid-turn interaction therefore includes both agent-initiated clarification and user-initiated interruption. Post-turn interaction occurs after the agent signals completion and the user provides feedback or follow-up instructions. Scenario families used in analysis include ambiguity / clarification, interruption, progressive refinement, requirement change, correction/feedback, and multi-step workflow (Wang et al., 2 Jun 2026).

To keep evaluation deterministic while remaining interactive, DeskCraft uses an MLLM-based user simulator, Kimi-K2.5, with a structured prompt conditioned on scenario, persona, completed phases, current phase goal, optional next phase goal, the agent’s last reply, and screenshot. The simulator outputs JSON containing "action", "message", "phase_complete", and "reason". When agent_ask is active, agents are given a call_user tool and are instructed to ask when instructions are ambiguous, incomplete, or updated, and not to hallucinate user answers (Wang et al., 2 Jun 2026).

This protocol is significant because it operationalizes mixed-initiative desktop work rather than treating the user as a fixed initial prompt. The benchmark thereby measures whether an agent can recognize uncertainty, solicit clarification, survive interruptions, and continue from the current state instead of restarting.

4. Software coverage, assets, and verifier design

DeskCraft runs in a live Ubuntu VM and covers 11 applications across 5 domains. Tasks involve realistic artifacts, paired gold outputs where appropriate, and domain-aware verifiers inspecting application-native outputs and project files (Wang et al., 2 Jun 2026).

Domain	Applications	Typical artifact types
Office / productivity	LibreOffice Writer, Calc, Impress	.docx, .pptx, .xlsx
Web & development	Chrome, VS Code	.html, .js, local servers
Image / vector, video / audio, 3D, multi-app	GIMP, Inkscape, Kdenlive, Audacity, Blender, OS workflows	.svg, .mp4, .wav, .blend

The benchmark contains 538 tasks, including 386 standard tasks and 152 interactive tasks. It uses 279 unique asset files in 19 formats, and interactive tasks contain 403 phase-level messages. Task construction draws on 224 reference sources and practitioner knowledge, with a capability matrix covering 120+ operation categories. L3 workflows are built by distilling real professional pipelines into self-contained tasks with named inputs and verifiable deliverables (Wang et al., 2 Jun 2026).

Verifier design is unusually domain-specific. SVG evaluators parse XML structure and attributes; Office evaluators parse .docx, .pptx, and .xlsx; audio evaluators analyze waveform properties; video evaluators inspect metadata and project XML; Blender tasks are verified in headless mode through the bpy scene graph; browser and OS workflows inspect configuration files, shell outputs, file existence, and content. Evaluator functions are initially generated by an LLM and then dual-reviewed by a human and an LLM (Wang et al., 2 Jun 2026).

This architecture makes DeskCraft a state-verification benchmark rather than a trajectory-matching benchmark. The agent is judged on the delivered artifact and machine state, which is more aligned with professional desktop work than reward heuristics based only on intermediate clicks.

5. Empirical performance and observed failure modes

DeskCraft evaluates 18 agents, including proprietary frontier models, open-source generalist VLMs, and open-source GUI-specialist CUA foundations. On the Standard split, the best overall result is Kimi-K2.6 with a success rate of 33.8%, while GPT-5.4 reaches 31.6%. On the Interactive split, GPT-5.4 is best with 27.6%, while Kimi-K2.6 reaches 25.7% (Wang et al., 2 Jun 2026).

Performance degrades sharply on long-horizon workflows. For EvoCUA-32B, success falls from 19.9% on L1 to 10.7% on L2 and 1.0% on L3. For Kimi-K2.6, it drops from 41.7% to 41.0% to 21.6%. For GPT-5.4, it drops from 39.0% to 40.7% to 9.5%. Run length increases with difficulty: Kimi-K2.6 averages 30.8, 48.8, and 77.7 steps across L1–L3, while GPT-5.4 averages 25.0, 44.3, and 71.2. The benchmark notes that many failed trajectories are almost as long as successful ones, indicating prolonged drifting rather than early termination (Wang et al., 2 Jun 2026).

A step-budget study on Kimi-K2.6 shows that standard success rises from 17.0% at 25 steps to 34.3% at 100, 34.9% at 150, and 35.7% at 181, with no further gains observed beyond approximately 200 steps. This indicates that longer budgets recover a small but meaningful tail of long-horizon tasks, but do not solve the core planning problem. Reliability analysis is similarly sobering: for Kimi-K2.6 on a subset, standard pass@1 is 28.7% and pass@6 is 45.6%, while interactive pass@1 is 27.0% and pass@6 is 38.8%; pass$s_0$3 decreases with $s_0$4, reflecting low run-to-run consistency (Wang et al., 2 Jun 2026).

The collaboration analysis identifies a specific weakness: lack of proactive clarification. Ask-style tasks, which require agent-initiated clarification, yield the lowest success rates for both GPT-5.4 and Kimi-K2.6. The reported failure mode is that agents rarely emit ASK when critical details are missing and instead proceed with guesses. Harder still are interruption and requirement-change scenarios, which require replanning under partial completion. By contrast, correction/feedback scenarios are handled relatively better because they localize edits rather than forcing global plan revision.

6. Broader interpretations and adjacent “DeskCraft” concepts

In current arXiv usage, DeskCraft most specifically denotes the desktop-agent benchmark just described. However, related papers use the name or “DeskCraft-style” framing in broader conceptual ways. A desk- and office-centric 3D scene generator is described as a natural specialization of RoomCraft’s controllable indoor scene pipeline, reusing multimodal scene extraction, relationship graphs, heuristic-based depth-first search, unified constraints, and Conflict-Aware Positioning Strategy for office layouts (Zhou et al., 27 Jun 2025). PLAICraft presents a time-aligned vision–speech–action data platform and is described as a reference architecture for a general desktop or PC embodied agent that learns from synchronized screen, audio, keyboard, and mouse traces (He et al., 19 May 2025). Dreaming in Code is used to motivate a DeskCraft open-ended crafting world in which foundation models generate executable task code for curriculum learning (Mitsides et al., 9 Feb 2026). “(De)composing Craft” contributes an elementary grammar—Granularity Shifts, Reflective Loops, Note-to-Self, External Links, Segments, Branches, and Revision Loops—that has been proposed as a conceptual basis for a DeskCraft system documenting improvisational craft workflows (Batra et al., 12 Jun 2025).

This suggests that “DeskCraft” has begun to function as a broader label for systems concerned with structured, controllable, and interaction-rich craft or desktop activity. A plausible implication is that the benchmark (Wang et al., 2 Jun 2026) occupies only one point in a wider design space: evaluation of desktop agents, desk-centric 3D synthesis, embodied data collection for computer use, open-ended task curricula, and grammar-based documentation of skilled practice. The established research artifact, however, remains the benchmark: a desktop-specific testbed for long-horizon execution and human-in-the-loop coordination.

According to the benchmark comparison table, DeskCraft is the first desktop benchmark that jointly supports long-horizon workflows as a central axis, a human-in-the-loop interaction protocol covering mid-turn and post-turn exchanges, and a structured L1/L2/L3 difficulty taxonomy (Wang et al., 2 Jun 2026). Within desktop-agent research, that combination makes it a reference point for studying whether agents can move beyond button pressing toward sustained, mixed-initiative delivery of complex digital artifacts.