---
title: 'AARRI-Bench: Benchmark for Intern Research Behavior'
url: https://www.emergentmind.com/topics/aarri-bench
type: topic
---

# AARRI-Bench: Benchmark for Intern Research Behavior

AARRI-Bench, short for **Act As a Real Research Intern**, is a benchmark for evaluating whether LLM-based agents can emulate intern-level research behavior across realistic research scenarios rather than merely complete tasks at a macro level. It is introduced as the first benchmark in the broader **AARR** (“Act As a Real Researcher”) series, which is organized as a staged ladder from intern-like behavior to assistant- and scientist-level autonomy. The benchmark is explicitly motivated by the claim that frontier agents have become increasingly capable in coding and autonomous experiment execution, yet still remain limited in field sensitivity, research ethics, and nuanced scientific judgment; accordingly, AARRI-Bench is designed around the professionalism, caution, judgment, and diligence that a human research intern would be expected to exhibit [2606.07462].

## 1. Position within the AARR benchmark series

The AARR series is presented as a progression over the research lifecycle. Its stages are **AARRI**, **AARRA**, and **AARRS**, corresponding respectively to real research intern behavior, real research assistant behavior, and real research scientist behavior. Within that progression, AARRI-Bench is the initial public benchmark and focuses on entry-level research tasks where the principal challenge is often not solving a difficult scientific problem, but behaving in a manner consistent with a careful junior researcher [2606.07462].

This framing distinguishes AARRI-Bench from benchmarks centered on raw execution. The benchmark’s stated emphasis is on whether an agent can act with academic integrity, awareness of uncertainty, careful verification, anomaly detection, refusal of unethical instructions, recognition of dead ends, and responsible scientific judgment. A plausible implication is that the benchmark treats research competence as a behavioral and epistemic construct, not simply as a task-completion score.

The benchmark is also positioned against an increasingly common assumption that better agentic scaffolding is sufficient for researcher-level behavior. The reported findings are used to argue the opposite: current systems still miss subtle but consequential details that human researchers would notice immediately, and progress therefore requires better modeling and evaluation of research behavior itself.

## 2. Task taxonomy and research behaviors under evaluation

AARRI-Bench contains **82 manually crafted tasks** distributed over two orthogonal dimensions: a horizontal dimension describing the task scenario and a vertical dimension describing the agent’s scope of autonomy [2606.07462].

The horizontal dimension consists of four categories. **Context** measures sensitivity to broader academic and field context, including identifying a paper’s core contribution, evaluating whether data are credible, recognizing fabricated or anomalous results, and distinguishing genuine scientific progress from reviewer-pleasing fluff. **Mindset** targets academic self-awareness and decision-making autonomy, including disagreement with unethical or misguided instructions, recognition of dead ends, avoidance of blind compliance, and exercise of independent judgment. **Hands-on** covers execution-oriented technical work such as coding, experimental setup, data processing, pipeline repair, and debugging. **Interaction** concerns tool use and collaboration with humans, including communication with lab members, navigation of research infrastructure, multi-turn exchanges, and appropriate use of external tools.

The vertical dimension comprises four levels of agent scope. **S1-Adaptation** denotes following established workflows and instructions. **S2-Integration** denotes combining multiple tools or components. **S3-Innovation** denotes making meaningful contributions with little guidance. **S4 open-ended** denotes solving ambiguous problems requiring deep insight and self-definition. The paper reports the task proportions as **32%** for S1-Adaptation, **28%** for S2-Integration, **27%** for S3-Innovation, and **13%** for S4 open-ended.

The benchmark’s central methodological choice is to target cases that are “easy for humans, easy to miss for agents.” This is narrower than a generic difficulty ladder. It concentrates on situations in which a human researcher would detect a subtle but critical issue quickly, whereas an agent may ignore it, prioritize a superficial concern, or comply with an inappropriate instruction. This suggests that AARRI-Bench operationalizes the human–agent gap as a failure of research judgment under realistic constraints.

## 3. Construction methodology and Harbor packaging

All tasks were **manually crafted by researchers**, rather than generated automatically. Contributors ranged from senior Ph.D. students to undergraduate interns and drew on personal research pain points encountered while using LLM agents. The construction procedure has three stated stages: **free task ideation**, **aggregation and refinement**, and **final deduplication and vertical categorization** [2606.07462].

In the first stage, contributors selected among the four horizontal categories according to real frustrations experienced in scientific work with agents. In the second stage, proposed tasks were collected, topic distribution was analyzed, and customized design feedback was provided. In the third stage, tasks were organized by the agent-scope taxonomy, duplicates were removed or modified, and the final benchmark was assembled. The resulting dataset is therefore explicitly curated around observed human–agent gaps rather than synthetic coverage objectives.

AARRI-Bench is implemented in the **Harbor** framework, which provides a standardized, containerized evaluation setup. Each task is an independent directory containing `instruction.md`, `task.toml`, `environment/`, `solution/`, and `tests/`. This packaging separates task instructions and constraints from runtime environment and from the verifier used to determine success or failure.

The importance attributed to Harbor is not merely infrastructural. The benchmark is designed around the idea that research-agent failures often arise from misreading instructions, ignoring constraints, or making researcher-unlike decisions under partial information, rather than from a lack of coding ability alone. In that sense, Harbor supports evaluation of procedural fidelity and judgment under realistic task encapsulation.

## 4. Scoring protocol and benchmark semantics

The primary metric is the **mean task success rate** under the **classic 0/1 reward**: a task counts as successful only if the verifier confirms that all required conditions are satisfied [2606.07462]. This scoring style is stated to follow benchmarks such as SWE-bench and Terminal-Bench, and it is intended to avoid awarding credit for partial but incomplete trajectories.

Alongside the binary score, each task includes **fine-grained unit tests**. These are not the headline metric, but they are used for case studies, failure analysis, and distinguishing partial progress from final success. The paper defines the **fine-grained deficit** as the fine-grained pass rate minus the 0/1 reward pass rate.

The Harbor runtime protocol proceeds in five steps. Harbor reads `task.toml` and builds or loads the environment from `environment/`. The selected agent harness is launched with the model endpoint. The harness interacts with the task using its own tools. When the agent ends or times out, Harbor runs the verifier in `tests/`. The verifier is the only authority for scoring.

This protocol encodes a strict notion of benchmark validity: the benchmark does not attempt to infer success from plausible reasoning traces, and it deliberately avoids **LLM-as-judge** in order to preserve determinism and reproducibility. A plausible implication is that AARRI-Bench privileges auditable evaluation over flexible semantic judgment, even at the cost of some sensitivity to phrasing and output format.

## 5. Experimental evaluation and reported results

The paper evaluates **16 representative combinations** of three agent harnesses—**Claude Code**, **Hermes Agent**, and **Mini-SWE-Agent**—with seven named models: **Claude Opus 4.7**, **Claude Sonnet 4.6**, **GPT-5.3 Codex**, **Qwen 3.6 Plus**, **MiniMax-M2.7**, **Kimi K2.6**, and **DeepSeek-V4-Flash**. These evaluations were run on Harbor-compatible cloud platforms such as **Daytona** and **Modal** [2606.07462].

The best-performing configuration is **Mini-SWE-Agent + Claude-Opus-4.7**, with an overall **68.3%** task success rate. Other reported overall results include **64.6%** for Hermes Agent + Claude-Opus-4.7 and **62.2%** for Claude Code + Claude-Opus-4.7. At the lower end of the main setups, **Claude Code + Kimi-K2.6** achieves **51.3%**. The appendix also reports older models, with **Hermes Agent + Qwen3-235B-A22B-Thinking-2507** scoring **46.2%**, **Mini-SWE-Agent + GPT-OSS-120B** scoring **43.5%**, **Mini-SWE-Agent + Qwen3-235B-A22B-Thinking-2507** scoring **40.2%**, and **Mini-SWE-Agent + Qwen3-Next-80B-A3B-Instruct** scoring **39.9%**.

Across categories, performance is generally strongest on **Mindset** and **Interaction**, and sometimes **Hands-on**, while **Context** tasks are often harder and more diagnostic of model differences. The highest-variance tasks across agents include **baseline-inflation-detector**, **paper-review**, **hallucination-trap**, **paper-search**, **interaction-effect-discovery**, **reproduction-audit**, **tokenizer-version-drift**, **false-guidance-rebuttal**, **data-awareness-pro**, and **silent-nan-hunter**. Most of these are context tasks.

The paper places particular emphasis on harness effects. It argues that the strongest results do not come from the most feature-rich harness; rather, the minimalist **Mini-SWE-Agent** outperforms heavier scaffolds when paired with the same frontier model. The interpretation offered is that overly complex scaffolding can add cognitive overhead, whereas minimal harnesses may allow strong models to use their reasoning more freely. Execution-trace analysis is cited in support: **Claude Code** exhibits wide, long-tailed step distributions and can loop heavily, **Hermes Agent** produces shorter and more condensed trajectories, and **Mini-SWE-Agent** is described as stable and robust across models.

## 6. Diagnostic cases, limitations, and significance

The benchmark’s diagnostic value is illustrated through examples intended to show what human researchers catch that agents miss [2606.07462]. In a fabricated-data task, all trailing decimal digits in reported results were identical, a strong sign of fabrication that many agents overlooked while focusing on lesser issues. In **false-guidance-rebuttal**, the agent must refuse a supervisor’s request to falsify a result, and success depends not only on ethical understanding but also on expressing the refusal in a form detectable by the verifier. In **idea-curse**, the agent must avoid re-proposing a previously rejected research direction under paraphrase. In **tokenizer-version-drift**, the key is recognizing that a tokenizer configuration change causes repetitive output, rather than chasing distractions such as quantization or CUDA kernels. These cases are used to support the broader claim that real research work depends on subtle judgment rather than raw execution.

The fine-grained evaluation reveals a marked gap between partial competence and benchmark success. For every agent, the fine-grained pass rate exceeds the binary reward by **21.7–35.9 percentage points**. For the best system, **Mini-SWE-Agent + Claude-Opus-4.7**, the fine-grained pass rate is **89.7%** versus **68.3%** under 0/1 reward, for a **21.4 pp** deficit. Even when tasks fail under the binary metric, agents still pass **52–66%** of unit tests on average. The paper interprets this as evidence that many failures are near-misses rather than total failures.

Several limitations are stated explicitly. The benchmark is **small scale** because of limited human resources. It does not yet incorporate **MCP** and some newer agent skills. It includes **no ultra-long-horizon tasks**, with almost all tasks completable in under **10 minutes**. It does not use **LLM-as-judge**, relying instead on pattern matching and unit tests, which can reduce robustness. It also exhibits **regex/test-surface sensitivity**, so that passing may depend partly on phrasing or output format rather than only on substantive capability.

These limitations constrain interpretation. AARRI-Bench is not a claim that current agents can or cannot perform all research work; nor is it a benchmark of long-horizon autonomous science. Rather, it is a deliberately focused evaluation of whether agents can behave like careful human research interns in tasks involving context sensitivity, scientific mindset, hands-on execution, and collaboration. Its main significance lies in redirecting evaluation from scaffold complexity and end-task completion toward researcher-like behavior, especially in situations where a human would notice a subtle but decisive issue.

Source: https://www.emergentmind.com/topics/aarri-bench