AARRI-Bench: Benchmark for Intern Research Behavior

• AARRI-Bench is a benchmark designed to evaluate LLM agents' ability to emulate realistic research intern behavior in nuanced scientific scenarios.
• It categorizes tasks into context, mindset, hands-on, and interaction, using manually crafted tasks to expose subtle human–agent judgment gaps.
• Results reveal performance deficits that highlight the importance of minimal harnesses and careful decision-making in research-oriented evaluations.

AARRI-Bench, short for Act As a Real Research Intern, is a benchmark for evaluating whether LLM-based agents can emulate intern-level research behavior across realistic research scenarios rather than merely complete tasks at a macro level. It is introduced as the first benchmark in the broader AARR (“Act As a Real Researcher”) series, which is organized as a staged ladder from intern-like behavior to assistant- and scientist-level autonomy. The benchmark is explicitly motivated by the claim that frontier agents have become increasingly capable in coding and autonomous experiment execution, yet still remain limited in field sensitivity, research ethics, and nuanced scientific judgment; accordingly, AARRI-Bench is designed around the professionalism, caution, judgment, and diligence that a human research intern would be expected to exhibit (Wang et al., 5 Jun 2026).

1. Position within the AARR benchmark series

The AARR series is presented as a progression over the research lifecycle. Its stages are AARRI, AARRA, and AARRS, corresponding respectively to real research intern behavior, real research assistant behavior, and real research scientist behavior. Within that progression, AARRI-Bench is the initial public benchmark and focuses on entry-level research tasks where the principal challenge is often not solving a difficult scientific problem, but behaving in a manner consistent with a careful junior researcher (Wang et al., 5 Jun 2026).

This framing distinguishes AARRI-Bench from benchmarks centered on raw execution. The benchmark’s stated emphasis is on whether an agent can act with academic integrity, awareness of uncertainty, careful verification, anomaly detection, refusal of unethical instructions, recognition of dead ends, and responsible scientific judgment. A plausible implication is that the benchmark treats research competence as a behavioral and epistemic construct, not simply as a task-completion score.

The benchmark is also positioned against an increasingly common assumption that better agentic scaffolding is sufficient for researcher-level behavior. The reported findings are used to argue the opposite: current systems still miss subtle but consequential details that human researchers would notice immediately, and progress therefore requires better modeling and evaluation of research behavior itself.

2. Task taxonomy and research behaviors under evaluation

AARRI-Bench contains 82 manually crafted tasks distributed over two orthogonal dimensions: a horizontal dimension describing the task scenario and a vertical dimension describing the agent’s scope of autonomy (Wang et al., 5 Jun 2026).

The horizontal dimension consists of four categories. Context measures sensitivity to broader academic and field context, including identifying a paper’s core contribution, evaluating whether data are credible, recognizing fabricated or anomalous results, and distinguishing genuine scientific progress from reviewer-pleasing fluff. Mindset targets academic self-awareness and decision-making autonomy, including disagreement with unethical or misguided instructions, recognition of dead ends, avoidance of blind compliance, and exercise of independent judgment. Hands-on covers execution-oriented technical work such as coding, experimental setup, data processing, pipeline repair, and debugging. Interaction concerns tool use and collaboration with humans, including communication with lab members, navigation of research infrastructure, multi-turn exchanges, and appropriate use of external tools.

The vertical dimension comprises four levels of agent scope. S1-Adaptation denotes following established workflows and instructions. S2-Integration denotes combining multiple tools or components. S3-Innovation denotes making meaningful contributions with little guidance. S4 open-ended denotes solving ambiguous problems requiring deep insight and self-definition. The paper reports the task proportions as 32% for S1-Adaptation, 28% for S2-Integration, 27% for S3-Innovation, and 13% for S4 open-ended.

The benchmark’s central methodological choice is to target cases that are “easy for humans, easy to miss for agents.” This is narrower than a generic difficulty ladder. It concentrates on situations in which a human researcher would detect a subtle but critical issue quickly, whereas an agent may ignore it, prioritize a superficial concern, or comply with an inappropriate instruction. This suggests that AARRI-Bench operationalizes the human–agent gap as a failure of research judgment under realistic constraints.

3. Construction methodology and Harbor packaging

All tasks were manually crafted by researchers, rather than generated automatically. Contributors ranged from senior Ph.D. students to undergraduate interns and drew on personal research pain points encountered while using LLM agents. The construction procedure has three stated stages: free task ideation, aggregation and refinement, and final deduplication and vertical categorization (Wang et al., 5 Jun 2026).

In the first stage, contributors selected among the four horizontal categories according to real frustrations experienced in scientific work with agents. In the second stage, proposed tasks were collected, topic distribution was analyzed, and customized design feedback was provided. In the third stage, tasks were organized by the agent-scope taxonomy, duplicates were removed or modified, and the final benchmark was assembled. The resulting dataset is therefore explicitly curated around observed human–agent gaps rather than synthetic coverage objectives.

AARRI-Bench is implemented in the Harbor framework, which provides a standardized, containerized evaluation setup. Each task is an independent directory containing instruction.md, task.toml, environment/, solution/, and tests/. This packaging separates task instructions and constraints from runtime environment and from the verifier used to determine success or failure.

The importance attributed to Harbor is not merely infrastructural. The benchmark is designed around the idea that research-agent failures often arise from misreading instructions, ignoring constraints, or making researcher-unlike decisions under partial information, rather than from a lack of coding ability alone. In that sense, Harbor supports evaluation of procedural fidelity and judgment under realistic task encapsulation.

4. Scoring protocol and benchmark semantics

The primary metric is the mean task success rate under the classic 0/1 reward: a task counts as successful only if the verifier confirms that all required conditions are satisfied (Wang et al., 5 Jun 2026). This scoring style is stated to follow benchmarks such as SWE-bench and Terminal-Bench, and it is intended to avoid awarding credit for partial but incomplete trajectories.

Alongside the binary score, each task includes fine-grained unit tests. These are not the headline metric, but they are used for case studies, failure analysis, and distinguishing partial progress from final success. The paper defines the fine-grained deficit as the fine-grained pass rate minus the 0/1 reward pass rate.

The Harbor runtime protocol proceeds in five steps. Harbor reads task.toml and builds or loads the environment from environment/. The selected agent harness is launched with the model endpoint. The harness interacts with the task using its own tools. When the agent ends or times out, Harbor runs the verifier in tests/. The verifier is the only authority for scoring.

This protocol encodes a strict notion of benchmark validity: the benchmark does not attempt to infer success from plausible reasoning traces, and it deliberately avoids LLM-as-judge in order to preserve determinism and reproducibility. A plausible implication is that AARRI-Bench privileges auditable evaluation over flexible semantic judgment, even at the cost of some sensitivity to phrasing and output format.

5. Experimental evaluation and reported results

The paper evaluates 16 representative combinations of three agent harnesses—Claude Code, Hermes Agent, and Mini-SWE-Agent—with seven named models: Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.3 Codex, Qwen 3.6 Plus, MiniMax-M2.7, Kimi K2.6, and DeepSeek-V4-Flash. These evaluations were run on Harbor-compatible cloud platforms such as Daytona and Modal (Wang et al., 5 Jun 2026).

The best-performing configuration is Mini-SWE-Agent + Claude-Opus-4.7, with an overall 68.3% task success rate. Other reported overall results include 64.6% for Hermes Agent + Claude-Opus-4.7 and 62.2% for Claude Code + Claude-Opus-4.7. At the lower end of the main setups, Claude Code + Kimi-K2.6 achieves 51.3%. The appendix also reports older models, with Hermes Agent + Qwen3-235B-A22B-Thinking-2507 scoring 46.2%, Mini-SWE-Agent + GPT-OSS-120B scoring 43.5%, Mini-SWE-Agent + Qwen3-235B-A22B-Thinking-2507 scoring 40.2%, and Mini-SWE-Agent + Qwen3-Next-80B-A3B-Instruct scoring 39.9%.

Across categories, performance is generally strongest on Mindset and Interaction, and sometimes Hands-on, while Context tasks are often harder and more diagnostic of model differences. The highest-variance tasks across agents include baseline-inflation-detector, paper-review, hallucination-trap, paper-search, interaction-effect-discovery, reproduction-audit, tokenizer-version-drift, false-guidance-rebuttal, data-awareness-pro, and silent-nan-hunter. Most of these are context tasks.

The paper places particular emphasis on harness effects. It argues that the strongest results do not come from the most feature-rich harness; rather, the minimalist Mini-SWE-Agent outperforms heavier scaffolds when paired with the same frontier model. The interpretation offered is that overly complex scaffolding can add cognitive overhead, whereas minimal harnesses may allow strong models to use their reasoning more freely. Execution-trace analysis is cited in support: Claude Code exhibits wide, long-tailed step distributions and can loop heavily, Hermes Agent produces shorter and more condensed trajectories, and Mini-SWE-Agent is described as stable and robust across models.

6. Diagnostic cases, limitations, and significance

The benchmark’s diagnostic value is illustrated through examples intended to show what human researchers catch that agents miss (Wang et al., 5 Jun 2026). In a fabricated-data task, all trailing decimal digits in reported results were identical, a strong sign of fabrication that many agents overlooked while focusing on lesser issues. In false-guidance-rebuttal, the agent must refuse a supervisor’s request to falsify a result, and success depends not only on ethical understanding but also on expressing the refusal in a form detectable by the verifier. In idea-curse, the agent must avoid re-proposing a previously rejected research direction under paraphrase. In tokenizer-version-drift, the key is recognizing that a tokenizer configuration change causes repetitive output, rather than chasing distractions such as quantization or CUDA kernels. These cases are used to support the broader claim that real research work depends on subtle judgment rather than raw execution.

The fine-grained evaluation reveals a marked gap between partial competence and benchmark success. For every agent, the fine-grained pass rate exceeds the binary reward by 21.7–35.9 percentage points. For the best system, Mini-SWE-Agent + Claude-Opus-4.7, the fine-grained pass rate is 89.7% versus 68.3% under 0/1 reward, for a 21.4 pp deficit. Even when tasks fail under the binary metric, agents still pass 52–66% of unit tests on average. The paper interprets this as evidence that many failures are near-misses rather than total failures.

Several limitations are stated explicitly. The benchmark is small scale because of limited human resources. It does not yet incorporate MCP and some newer agent skills. It includes no ultra-long-horizon tasks, with almost all tasks completable in under 10 minutes. It does not use LLM-as-judge, relying instead on pattern matching and unit tests, which can reduce robustness. It also exhibits regex/test-surface sensitivity, so that passing may depend partly on phrasing or output format rather than only on substantive capability.

These limitations constrain interpretation. AARRI-Bench is not a claim that current agents can or cannot perform all research work; nor is it a benchmark of long-horizon autonomous science. Rather, it is a deliberately focused evaluation of whether agents can behave like careful human research interns in tasks involving context sensitivity, scientific mindset, hands-on execution, and collaboration. Its main significance lies in redirecting evaluation from scaffold complexity and end-task completion toward researcher-like behavior, especially in situations where a human would notice a subtle but decisive issue.