AcademiClaw: Bilingual Academic AI Benchmark

• AcademiClaw is a bilingual benchmark designed to evaluate AI agents on complex academic workflows, emphasizing sustained reasoning and multi-step tool use.
• It employs isolated Docker execution and six complementary scoring techniques to faithfully assess performance across problems like coding, research, and STEM reasoning.
• The benchmark distinguishes agent strategies and safety profiles, highlighting performance gaps with detailed rubrics and independent behavioral audits.

AcademiClaw is a bilingual benchmark for OpenClaw-based AI agents that evaluates 80 complex, long-horizon tasks drawn from real university students’ academic workflows, including homework, research projects, competitions, and personal projects. It was introduced to examine academic-level capabilities that earlier OpenClaw benchmarks had left largely unmeasured, particularly beyond assistant-level workflows such as email handling, calendar scheduling, scaffold generation, and related routine productivity tasks. The benchmark couples isolated Docker execution with multi-dimensional rubrics built from six complementary scoring techniques, and it adds an independent five-category safety audit for behavioral analysis (Yu et al., 4 May 2026).

1. Benchmark identity and intended scope

AcademiClaw is defined as an academic-level benchmark rather than an agent framework. Its explicit target is the class of difficult tasks that require sustained reasoning, domain expertise, coding and debugging ability, and nontrivial tool use. The benchmark is bilingual, comprising 49 English tasks and 31 Chinese tasks, and the Chinese tasks are native Chinese tasks tied to Chinese language, culture, and conventions rather than translated variants of English prompts. The task set spans 25+ professional domains and includes 16 tasks requiring CUDA GPU execution, a feature described as absent from prior OpenClaw benchmarks (Yu et al., 4 May 2026).

The benchmark was created to address a specific evaluation gap. Earlier OpenClaw benchmarks are characterized as measuring mostly assistant-level workflows, whereas real academic work often demands formal mathematics and olympiad-style reasoning, linguistics and cultural interpretation, model training and deployment, systems debugging, scientific analysis, browser-based and full-stack work, and creative writing under domain constraints. Within that framing, AcademiClaw reorients evaluation from shallow workflow completion toward deeper academic competence.

A common misconception is to treat AcademiClaw as merely a harder productivity benchmark. The underlying paper instead presents it as a diagnostic instrument for “academic-level” competence, with difficulty arising not only from task length but also from heterogeneous domain structure, tool requirements, and the need for sustained multi-step reasoning.

2. Task acquisition and curation pipeline

AcademiClaw was constructed through a bottom-up collection process centered on student-submitted tasks. Undergraduate students were asked to submit tasks from their real academic workflows and to confirm that they had already tried the task with at least one mainstream AI agent and that the agent failed or required extensive prompting. This initial process produced 230 candidate tasks (Yu et al., 4 May 2026).

A two-stage expert curation process then reduced the pool to the final benchmark. Domain experts evaluated candidate tasks on five criteria: prompt clarity and completeness, rubric correctness, scoring reproducibility, difficulty calibration, and domain coverage balance. The curation funnel reported in the paper is explicit:

• Initial submissions: 230
• Rejected for rubric defects: 57
• Rejected for ambiguity / missing deliverable specs: 34
• Rejected for difficulty mismatch: 28
• Rejected for domain over-representation: 18
• Rejected for environment infeasibility: 13
• Accepted: 80

This construction process matters methodologically because the benchmark is not researcher-authored in the conventional sense. Its tasks originate in unmet demand from actual users of AI agents in academic settings. A plausible implication is that AcademiClaw is designed less around synthetic challenge construction than around externally observed capability failure.

3. Corpus composition and execution substrate

The final task set is organized into six primary categories. These categories are intentionally broad and collectively define the benchmark’s capability surface (Yu et al., 4 May 2026).

Category	Tasks
Research paper analysis	21
ML / AI engineering	17
Software engineering	17
STEM reasoning	11
Language / creativity	7
Applied / domain-specific tasks	7

The paper’s examples make the breadth concrete. Representative domains include CMO-style proof problems, International Linguistics Olympiad tasks, A3C/PPO training, DQN migration and implementation, full-stack debugging, operating system labs, security/log forensics, model deployment, quantization, model merging, speculative decoding, rendering engines, web automation, lyric adaptation, dance choreography, Chinese composition, travel planning, mahjong calculation, ELO analysis, and report writing. This breadth is deliberate: the benchmark is described as testing a wide capability surface rather than a single axis of model ability.

A notable extension of scope is GPU support. Sixteen of the 80 tasks require CUDA GPU execution and cover model architecture design, training, quantization, deployment, GPU-accelerated computer vision, robotic simulation, and scientific computing. These tasks require agents to handle CUDA environment setup, GPU memory management, custom training loops, and device-level debugging. In the benchmark’s own framing, this extends evaluation into realistic engineering workflows that prior OpenClaw benchmarks did not cover.

Execution occurs in an isolated Docker sandbox. Each task is packaged with a prompt, optional reference materials, structured metadata, and evaluation rubric code, but the rubric is hidden from the agent. The environment uses a two-layer Docker image hierarchy consisting of a base CPU or GPU image and a per-task image with task-specific dependencies. The agent operates with tools such as read/write/edit, shell execution, search, and browser automation. The system snapshots the filesystem before and after execution and evaluates only the files changed by the agent, so the measured score is tied to work completed inside the controlled sandbox (Yu et al., 4 May 2026).

4. Rubric architecture and safety auditing

AcademiClaw uses task-specific rubric programs rather than a single global evaluator. Each task has its own eval/rubric.py and is scored on a 0–100 scale. A task is considered passed if the score is 75 or above, and the benchmark reports both pass rate and average score. Each task rubric contains 3–6 orthogonal dimensions summing to 100 points (Yu et al., 4 May 2026).

The evaluation framework combines six complementary scoring techniques:

1. Pattern matching
2. Code execution
3. LLM-as-Judge
4. Vision LLM assessment
5. End-to-end browser testing
6. Structured-output validation

These techniques are used to assess different output modalities. Pattern matching covers regex, keyword checks, and AST parsing. Code execution compiles and runs submitted code, checking outputs or tests. LLM-as-Judge is used for open-ended reports, analyses, and creative work, with deterministic fallback logic. Vision LLM assessment evaluates visual outputs such as screenshots or rendered graphics. End-to-end browser testing is performed via Playwright in a headless browser. Structured-output validation covers artifacts such as JSON, CSV, BibTeX, and Excel.

The appendix example en_blackhole_visualization illustrates how these modalities are composed within a single task. Its rubric has five dimensions totaling 100 points: file delivery (10), technical architecture (15), physics/visual logic (25), interaction and UI (25), and visual similarity (25). The example demonstrates that AcademiClaw does not reduce evaluation to simple pass/fail scripting; instead, it distributes evidence across structural verification, execution behavior, semantic judgment, and visual fidelity.

Separate from task quality, AcademiClaw runs an independent safety audit over the agent’s tool-use trajectory. The five safety categories are S1 Destructive operations, S2 Information leakage, S3 Boundary compliance, S4 Privilege escalation, and S5 Supply-chain risks. Each category is scored independently on a 0–100 scale and then combined into an overall safety score. This separation between task-success scoring and behavioral auditing is one of the benchmark’s central design features (Yu et al., 4 May 2026).

5. Experimental results and diagnostic findings

The benchmark evaluates six frontier models under identical sandbox conditions with one attempt per task: Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro, Qwen3.5-397B-A17B, and MiniMax M2.7. The main reported results are as follows (Yu et al., 4 May 2026).

Model	Average score	Pass rate
Claude Opus 4.6	71.9	55.0%
Claude Sonnet 4.6	68.3	55.0%
GPT-5.4	65.6	42.5%
Gemini 3.1 Pro	64.3	43.8%
Qwen3.5-397B-A17B	64.7	40.0%
MiniMax M2.7	63.1	37.5%

The headline result is that even the best pass rate is 55%. More than a quarter of tasks defeat all models: 23 of 80 tasks were failed by all six models, and 8 tasks had every model below 50. The paper treats these as evidence that AcademiClaw probes genuine capability boundaries rather than incremental difficulty scaling.

The benchmark also reports that task type matters more than model identity in many cases. The cross-category score spread is 26.3 points, while the cross-model spread is 8.8 points. STEM reasoning is the weakest category overall, and olympiad-style tasks are especially difficult. Two examples are highlighted as boundary cases: zh_huaxue_jingsai, where all models cluster around 23–27, and en_fullstack_debug, where all models score exactly 25. At the same time, some failures are model-specific, such as one model collapsing on a TensorFlow-to-PyTorch migration task while others succeed, indicating distinct capability profiles rather than a single monotone ranking.

A further contribution is the identification of three behavioral phenotypes. Claude Opus 4.6 is described as read-first, characterized by heavy file reading before action and associated with a “comprehension premium.” Gemini 3.1 Pro is execute-first, with very high shell execution frequency, fewer upfront reads, more trial-and-error behavior, and more safety issues without better results. GPT-5.4 is minimalist, with the fewest tool calls, the lowest token use, the shortest runtime, and still strong quality, suggesting more internal inference before external action. These distinctions matter because the paper argues that models solve tasks in qualitatively different ways, with implications for quality, efficiency, and safety.

One of the benchmark’s more striking findings is the disconnect between token consumption and output quality. Across all 480 model-task evaluations, the pooled Pearson correlation between token count and score is essentially zero, reported as $r = -0.032$ with $p = 0.49$. The paper also notes that Gemini 3.1 Pro uses about 5.4× more tokens than GPT-5.4 yet does not outperform it. On safety, most models score above 80 overall except Gemini, but S3 Boundary compliance shows the strongest divergence, with a 53-point spread across models. Claude models are safest on boundary compliance, Gemini and Qwen perform much worse, unauthorized workspace boundary excursions dominate the issue, privilege escalation is rare, and safety and task quality are only weakly correlated (Yu et al., 4 May 2026).

6. Position within the OpenClaw literature

AcademiClaw belongs to a broader wave of OpenClaw evaluation work, but it addresses a distinct problem from the security-focused benchmarks that dominate much of the surrounding literature. PASB formalizes attacks on OpenClaw as a personalized local assistant with persistent memory, private context, and high-privilege tools, measuring leakage, unsafe action, and persistence in end-to-end black-box scenarios (Wang et al., 9 Feb 2026). ClawTrap evaluates OpenClaw-style web agents under dynamic MITM conditions through attack modes such as Static HTML Replacement, Iframe Popup Injection, and Dynamic Content Modification, emphasizing provenance and trust calibration under live network tampering (Zhao et al., 19 Mar 2026). SafeClawArena treats Claw-like agents as agentic computer systems and evaluates 406 adversarial tasks across Skill Supply-Chain Integrity, Persistent State Exploitation, Cross-Boundary Data Flow, and Indirect Prompt Injection (Niu et al., 29 Jun 2026).

Against that backdrop, AcademiClaw occupies a complementary niche. It is not primarily a benchmark for attack success, tool-chain compromise, or cross-boundary leakage; instead, it measures whether OpenClaw-based agents can perform authentic academic work across a wide domain distribution while also exposing behavioral tendencies through a separate safety audit. This suggests a division of labor within the OpenClaw ecosystem: security-oriented benchmarks characterize system fragility and trust-boundary failures, whereas AcademiClaw characterizes academic competence, domain coverage, and agent strategy under controlled sandbox execution.

Its broader significance follows from that distinction. By moving beyond assistant-level workflows and by incorporating bilingual tasks, native Chinese cultural grounding, GPU-required execution, and multi-modal rubric design, AcademiClaw changes what counts as meaningful evaluation for OpenClaw-based agents. Rather than asking whether an agent can manage routine delegation, it asks whether the same agent can sustain high-quality performance on the heterogeneous, long-horizon, tool-intensive demands of real academic work (Yu et al., 4 May 2026).