Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle

Abstract: As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution. Despite their evolution from research assistants into autonomous research agents, these systems still exhibit significant limitations in field sensitivity, research ethics, and nuanced scientific judgment. Consequently, frontier agents remain unable to fully replace human researchers. To bridge this gap, we conceptualize the AARR (Act As a Real Researcher) benchmark series. Unlike existing benchmarks that primarily assess macro-level execution capabilities, AARR focuses on whether agents can emulate the professionalism, thoroughness, and nuanced reasoning that characterize human researchers in granular research scenarios. In this work, we propose AARRI-Bench (Act As a Real Research Intern), the first benchmark in this series. We conduct extensive experiments across frontier models and agentic systems, revealing that even the best-performing configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves only 68.3\% success rate, frequently overlooking subtle yet critical details that are obvious to real human researchers. Our results indicate that developing researcher-like AI requires further exploration of research behavior, rather than merely complex scaffolding. Our data is released at https://github.com/AARR-bench/AARRI-bench.

• The paper introduces AARRI-Bench, a benchmark that evaluates agentic LLMs on research-oriented tasks using a dual-axis taxonomy covering context, mindset, technical execution, and interaction.
• The paper’s experiments show that minimalist harness designs can outperform more complex setups, yielding a 3–6% task success improvement and increased execution efficiency.
• The paper highlights a significant researcher–agent gap, stressing the need for systems that incorporate nuanced epistemic reasoning, ethical rigor, and efficient context tracking.

AARRI-Bench: Evaluating Researcher-Quality Behaviors in Agentic LLM Systems

Motivation and Benchmark Design

The proliferation of agentic LLM systems has enabled significant advances across research workflows, e.g., autonomous code generation, iterative experiment optimization, and tool-mediated scientific analysis. While prior benchmarks in this line—such as SWE-bench, Terminal-Bench, and various research-specific suites—have expanded the evaluation surface for autonomous agents, they predominantly focus on end-task success and technical execution. This metric-centric paradigm neglects crucial researcher-oriented attributes such as integrity, uncertainty awareness, rigorous verification, and the nuanced judgment expected from human researchers.

AARRI-Bench ("Act As a Real Research Intern") introduces a new evaluation perspective by explicitly targeting these qualities. Rather than accentuating tasks that challenge LLMs but are straightforward for human researchers, AARRI-Bench systematically dissects the "professionalism gap" between agents and early-stage human research collaborators.

Figure 1: Overview of the AARRI-Bench pipeline underlying the benchmark's human-in-the-loop design, dual-axis taxonomy, and unified evaluation environment.

Task Taxonomy and Construction

AARRI-Bench covers 82 tasks, meticulously curated and manually engineered by a heterogeneous pool of academic researchers to mimic real-world intern-level research scenarios. Task coverage follows a two-dimensional taxonomy:

• Horizontal Axis (Task Scenario): Divided into Context (field sensitivity, background reasoning), Mindset (academic self-awareness, independence, research integrity), Hands-on (technical execution, pipeline management), and Interaction (human-in-the-loop coordination, tool use).
• Vertical Axis (Agent Scope): Ranges from S1 (Adaptation) through S4 (open-ended Innovation), reflecting increased autonomy and intellectual creativity.

  Figure 2: Distribution of tasks across agent scope and task scenario, highlighting the proportional representation of both axes in the task set.

Each task adheres to Harbor's containerized environment protocol for reproducibility and standardization. Fine-grained evaluation scripts test for both holistic task success and incremental unit-test coverage, allowing both coarse and differential analysis of agent behavior.

Experimental Protocol

Sixteen model–harness combinations are systematically evaluated, including leading closed-source models (e.g., Claude-Opus-4.7, GPT-5.3 Codex) and high-parameter open-source models (MiniMax-M2.7, DeepSeek-V4-Flash, Kimi-K2.6), each deployed under multiple agent harnesses: Claude Code, Hermes Agent, and mini-SWE-Agent.

The experimental setup uses formal 0/1 task-level reward as well as highly granular unit-level pass rates, providing a fine-grained diagnostic of failure modes invisible to pure success metrics. Evaluations are performed exclusively using cloud-based, containerized environments (Daytona, Modal) to minimize run-to-run variance.

Numerical Results and Performance Analysis

The best-performing configuration (Mini-SWE-Agent + Claude-Opus-4.7) achieves a 68.3% overall task success, notably outpacing more elaborate harnesses using the same underlying model by 3–6%. This finding directly contradicts the assumption that more complex agentic scaffolding or elaborate toolchains yield better researcher-mimetic behavior. Instead, minimalistic harnesses, by virtue of exposing low-level primitives with less structural constraint, enable frontier LLMs to harness their intrinsic reasoning more flexibly.

Performance distribution across models strongly outweighs the influence of agent harness selection; the highest-performing closed-source models consistently attain 60-64% pass rates, while even high-parameter open-source models saturate around 54-59%. The delta between test-case pass rate and 0/1 reward rate (21–36% across configurations) exposes a substantial "diagnostic deficit," where partial but meaningful task progress is omitted by coarse-grained metrics.

Figure 3: Relative success rate improvements across various model–harness combinations, foregrounding the scaling advantage of advanced models under minimalist harness design.

Execution efficiency analyses, as measured in trajectory length and behavioral variance, confirm that the most structured harnesses (e.g., Claude Code) are prone to runaway execution and redundant paths for weaker models, while minimalist and specialized harnesses enforce tighter, more efficient trajectories.

Figure 4: Trajectory step statistics for combinations, elucidating the susceptibility of various harnesses to inefficient execution under different model strengths.

Case Studies and Qualitative Failure Analysis

Detailed case study analysis surfaces critical and persistent human-agent gaps in research settings. For example, on a task requiring review of fabricated data (manuscript with copy-pasted decimal results—a classic scientific misconduct cue), nearly all agentic configurations failed to flag the anomaly, with the exception of Claude Code + Claude Opus 4.7.

Figure 5: Comparative performance on a research review task, showing that only Claude Code + Claude Opus 4.7 detects fabricated data while Hermes Agent with the same model does not.

Long-horizon memory tasks ("idea-curse") further reveal that the observed agent failures are modulated more by harness design (which governs context encoding and retrieval strategy) than by model reasoning capability per se. Similarly, tasks probing research integrity (e.g., refusing an instruction to falsify experimental data) demonstrate that subtle differences in refusal phrasing can mean the difference between passing and failing a rule-based test, indicating a gap between actual research ethics and mechanical grading.

Additional diagnostic failures arise from output-style mismatches and partial credit scenarios (models frequently solve sub-components, but either violate a single constraint or misalign their output structure with grader expectations).

Practical and Theoretical Implications

Empirically, current frontier agentic LLM systems are not yet reliable substitutes for human research interns, even under well-bounded tasks. Aggregate improvements hinge primarily on scaling intrinsic model capability; agent harnesses exert real, but secondary, influence, and harness overengineering may actively suppress LLM strengths. The benchmark establishes that fine-grained, researcher-quality-oriented tasks are essential for diagnosing and tracking the researcher–agent gap, and for driving model and evaluation advances toward more robust, trustworthy scientific AI.

From a practical standpoint, the findings imply that future agent design should prefer architectural minimalism, expose low-level controls, and prioritize context tracking and adaptive memory over fixed pipelines. Theoretically, filling the "professionalism gap" will require systems to encode not only tool proficiency, but nuanced epistemic reasoning, critical self-assessment, and robust ethical behavior—qualities not trivially compositional from current agentic benchmarks.

Speculations and Future Directions

The AARRI-Bench release is the inaugural step in a proposed series, with planned escalation in both challenge breadth (AARRA: real research assistant; AARRS: autonomous scientist scenarios) and openness (community-driven task curation). Large-scale adoption will likely drive further development of agent infrastructure with integrated long-horizon memory, adaptive output styling, and multi-agent system integration.

Anticipated directions include the use of LLM-as-a-judge protocols for robust open-ended evaluation, ultra-long horizon and multi-hour research workflows, and mechanisms for modeling the grounded, meta-cognitive behaviors required for agentic systems to surpass "intern-level" thresholds.

Conclusion

AARRI-Bench systematically characterizes the divergence between agentic LLM system performance and that of real human research interns, demonstrating that current models—even at very large scale—lack researcher-quality diligence, context sensitivity, and epistemic integrity. The diagnostic framework and task design specification introduced here will serve as a reference methodology for the development and evaluation of next-generation research agents.

(2606.07462)