Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies

Abstract: Workspace learning requires AI agents to identify, reason over, exploit, and update explicit and implicit dependencies among heterogeneous files in a worker's workspace, enabling them to complete both routine and advanced tasks effectively. Despite its importance, existing relevant benchmarks largely evaluate agents on pre-specified or synthesized files with limited real-world dependencies, leaving workspace-level evaluation underexplored. To this end, we introduce Workspace-Bench, a benchmark for evaluating AI agents on Workspace Learning invOlving Large-Scale File Dependencies. We construct realistic workspaces with 5 worker profiles, 74 file types, 20,476 files (up to 20GB) and curate 388 tasks, each with its own file dependency graph, evaluated across 7,399 total rubrics that require cross-file retrieval, contextual reasoning, and adaptive decision-making. We further provide Workspace-Bench-Lite, a 100-task subset that preserves the benchmark distribution while reducing evaluation costs by about 70%. We evaluate 4 popular agent harnesses and 7 foundation models. Experimental results show that current agents remain far from reliable workspace learning, where the best reaches only 68.7%, substantially below the human result of 80.7%, and the average performance across agents is only 47.4%.

• The paper introduces a benchmark with 20,476 interconnected files across 74 formats and 388 dependency-annotated workspace tasks spanning five professional roles.
• AI agents achieve a mean Rubric Pass Rate of 47.4%, compared with 80.7% for human-assisted performance, with results declining from 57.6% on Easy tasks to 40.5% on Hard tasks.
• Lineage tracing and heterogeneous file understanding remain the central bottlenecks, while high dependency-graph recognition does not reliably produce correct task outputs.

Workspace-Bench 1.0 (2605.03596) is a benchmark for evaluating AI agents on "Workspace Learning": the ability to identify, reason over, and exploit explicit and implicit dependencies among heterogeneous files in a realistic digital workspace. The authors argue that existing agent benchmarks either bypass file systems entirely (prompt-driven and tool/API-driven benchmarks), provide pre-packaged task-specific files that reduce to document QA, or simulate workspaces too simplistically—typically with a single directory style, fewer than ten file formats, and no evaluation of version lineage. Workspace-Bench addresses this gap by pairing large-scale, persona-specific workspaces with tasks whose annotated file dependency graphs are explicitly evaluated.

The benchmark comprises five professional workspaces—Operations Manager, Logistics Manager, AI Product Manager, Backend Developer, and Researcher—containing 20,476 interconnected files (up to 20 GB) across 74 file types, 3,299 directories, and a maximum nesting depth of 8. Workspaces are built through a top-down pipeline: LLM-generated role-conditioned directory hierarchies with injected structural noise, followed by hybrid content population combining an agentic crawler over public resources (arXiv papers, GitHub repositories, reports) with grounded generation of related artifacts such as emails and meeting notes. Domain experts verify plausibility.

Tasks derive from 154 authentic scenarios collected via an internal questionnaire at ByteDance's Lark platform, expanded into 388 tasks by 25 human annotators. Each task carries a natural-language instruction (intentionally under-specified), a reference output, a file dependency graph specifying the minimal set of required files, difficulty labels across six capability dimensions, and an average of 19.1 rubrics (7,399 total), split into result-oriented (54.8%), foundation (25.0%), and process-oriented (20.2%) categories. Tasks average 5.1 dependency edges over 4.7 files; 29.4% have high edge density ($\ge$6 edges). Difficulty stratification yields 14% Easy, 53% Medium, and 33% Hard. The paper also releases Workspace-Bench-Lite, a 100-task distribution-preserving subset reducing evaluation cost by roughly 70%.

A notable limitation in construction: workspaces are simulated rather than captured from real user data, so realism rests on expert verification rather than provenance from actual production file systems.

Evaluation uses isolated sandbox environments with dual-level parallelism (workspace-level and task-level sandbox pooling), multi-strategy output-file extraction (instruction-constrained path reporting, replica-based centralized retrieval, metadata fuzzy matching), and parallel BFS workspace rollback against baseline snapshots. Scoring follows an Agent-as-a-Judge paradigm using Seed-2.0-Lite as judge, producing binary rubric scores plus dependency-graph recognition rates computed as node- and edge-level F1 against ground-truth graphs extracted from execution trajectories. Metrics include Rubric Pass Rate, Task Completion Rate at thresholds (TCR@$p$), dependency F1, token consumption, and interaction turns.

Because judging is itself performed by an LLM, evaluation fidelity depends on judge quality; the paper does not report human–judge agreement statistics on rubric scoring, which is an open validity question for the reported numbers.

Across 28 configurations (4 harnesses × 7 backbone models) on Workspace-Bench-Lite, the mean Rubric Pass Rate is only 47.4%, versus a human-in-the-loop baseline of 80.7%. The best configuration, OpenClaw + Claude-Opus-4.7, reaches 68.7%, still well below human performance. Key findings:

Finding	Evidence
Consistent degradation with difficulty	Mean pass rate falls 57.6% (Easy) → 49.2% (Medium) → 40.5% (Hard); some configurations drop below 30% on Hard
Lineage and heterogeneity are bottlenecks	Heterogeneous File Understanding and Lineage Tracing rank lowest across nearly all agents; Edge F1 is universally low relative to Node F1
Harness choice matters unevenly	Harnesses add little for strong models but act as boosters for weaker ones; GLM-5.1 shows unusual stability under DeepAgent
Cost does not buy accuracy	DeepAgent + MiniMax-M2.7 consumes up to 58.1 turns and 0.61M tokens per task while averaging only ~45%; efficient configurations (ClaudeCode/Hermes + Opus-4.7) exceed 65% with fewer than 20 turns
Human-agent collaboration dominates	Expert baselines exceed all autonomous agents at every difficulty tier, without degrading on Hard tasks

Error analysis shows Missing Content and Reasoning Errors dominate failed rubrics, while Format and Process Errors are marginal—indicating foundational workflow execution is largely solved, but deep information recall and cross-file aggregation are not. A comparison against Anthropic's closed-source Claude Cowork + Opus-4.7 on 20 hard tasks shows Cowork achieving the highest average rubric score and most resilient TCR@$p$ curve, which the authors attribute to likely co-optimization between the harness and model.

An important caveat on interpretation: several agents achieve high dependency-graph F1 yet low rubric accuracy, indicating that correctly identifying relevant files does not translate into correct task-supporting and result-providing file utilization—the gap lies in how dependencies are exploited, not merely discovered.

The paper formalizes a capability ladder: L0 (data-insensitive guidance), L1 (user-specified file execution), L2 (file-to-file dependency reasoning, marked by an "orchestration singularity" where harness contribution surpasses the base model), L3 (task-to-file dependency discovery, the "capability singularity"), and L4 (workspace-native self-evolution). Current agents fail monotonically as they approach L3, and the mismatch between required cross-file association and current isolated-file processing paradigms is termed the "Data Association Gap." This taxonomy is a conceptual contribution rather than an empirically validated model; its stage boundaries are asserted, not measured independently of benchmark performance.

Several limitations bear directly on the results. First, the headline experiments run on Workspace-Bench-Lite rather than the full 388-task set, so full-benchmark numbers remain unreported. Second, the Agent-as-a-Judge pipeline introduces potential judge bias, particularly for open-ended deliverables, and no inter-rater reliability analysis is provided. Third, the simulated workspaces, while expert-verified, may not capture the true distribution of enterprise file systems; the claim of "realism" is qualitative. Fourth, the five-stage taxonomy and constructs like the Data Association Gap lack independent operational measurement. Open questions include whether harness-side memory or retrieval architectures can close the Edge-F1 deficit, and whether the observed harness-model synergy effects (e.g., Cowork + Opus-4.7) generalize beyond co-developed pairs.

Workspace-Bench 1.0 provides a large-scale, dependency-annotated testbed that exposes a substantial deficit in current agentic systems: even frontier configurations achieve roughly 69% rubric pass rate against an 80.7% human baseline, with inter-file relationship reasoning—especially lineage tracing and heterogeneous file understanding—as the dominant bottleneck. The benchmark's fine-grained rubrics, dependency-graph metrics, and efficiency analyses offer a concrete diagnostic foundation for developing dependency-aware agent architectures.