Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Abstract: Evaluating agents on the growing number of agentic benchmarks is challenging because they often require complex environments and agent integrations. We introduce Harbor Adapters, a unified evaluation infrastructure for agentic benchmarks. Our work makes three contributions. First, we develop benchmark adapters that port more than 80 benchmarks to evaluate arbitrary agents, and validate them through rigorous code review and parity experiments. Second, we conduct a large-scale evaluation of 8 models spanning capability tiers across 54 benchmarks; every model is run with Terminus-2 and with one of 3 native harnesses. This enables a broader analysis of agent capabilities and failure modes than was previously possible. Third, we introduce Harbor-Index, a curated set of 82 difficult, diverse, and high-quality tasks spanning 29 benchmarks, refined from the adapted suite through difficulty filtering, AI and human audit, and an audit-and-fix loop. Harbor-Index preserves the challenge and breadth of large-scale agentic evaluations while being affordable to run; no evaluated model-harness configuration exceeds 30% pass rate, and the strongest (GPT-5.5 with Codex) reaches 28.0%. We release the adapters, evaluation results, in-depth analysis, and Harbor-Index as open-source artifacts to support more reliable and comprehensive evaluation of language-model agents.
First 10 authors:
Summary
- The paper proposes Harbor Adapters, a framework standardizing heterogeneous agentic benchmarks, reducing integration burdens from $\mathcal{O}(mn)$ to $\mathcal{O}(m+n)$.
- The methodology includes over 1,000 GitHub review comments and supports 82 tasks across 29 benchmarks in Harbor-Index, showcasing varying difficulty, diversity, and auditability.
- Principally, the paper finds that benchmark difficulty is the main contributor to score variance, suggesting that models, environments, and verifiers are crucial for incorrect scores.
Evaluation fragmentation and the Harbor abstraction
“Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation” (2609.04298) addresses a central methodological problem in agent evaluation: benchmark proliferation has outpaced the infrastructure required to execute heterogeneous tasks under comparable conditions. Agentic benchmarks differ not only in prompts and labels, but also in sandbox environments, tool interfaces, runtime dependencies, termination rules, verifier implementations, and trajectory-level scoring. Consequently, supporting benchmarks and agents through independent integrations imposes an engineering burden and makes cross-paper comparisons vulnerable to implementation variance.
The paper proposes Harbor Adapters, an integration layer built on the Harbor execution framework. Each benchmark is represented through a shared schema consisting of an instruction, an environment, tests, and a solution. Benchmark-specific adapters translate original datasets and evaluators into this representation, while agent integrations are implemented once against Harbor. The resulting architecture reduces the nominal integration burden to .
Figure 1: Harbor standardizes heterogeneous benchmarks through a shared task schema and separates benchmark adapters from agent integrations.
The abstraction is intentionally broader than conventional agent benchmarks. The authors adapt both natively agentic suites, such as SWE-bench and Terminal-Bench, and originally non-agentic evaluations, such as expert question answering and code-generation benchmarks, by exposing tools, files, executable environments, or iterative interaction. This distinction is formalized through three interaction modes: agentic, non-agentic, and modified-agentic. The last category captures evaluations whose original single-pass protocol is extended into an executable agent setting.
The infrastructure supports sandboxed CPU and GPU execution, LLM-based verifiers, custom metric aggregation, and multiple backends, including Daytona, Modal, and E2B. The manuscript reports more than 80 supported benchmarks and 22 supported agents in the infrastructure, while the large-scale study evaluates 54 benchmarks. The broader adapter catalog contains additional integrations that were not included in the compute-intensive experiment.
Adapter construction, parity, and benchmark validity
A major strength of the infrastructure contribution is that adaptation is treated as a semantic-preservation problem rather than a file-format conversion. Each adapter must reconstruct the execution environment, integrate the original scoring logic, validate an oracle solution, and conduct parity experiments against the upstream benchmark. The review process combines automated checks, junior or trained human review, and senior approval. The project reports more than 10,000 GitHub review comments across adapters.
Parity experiments match the model, agent, prompt, decoding configuration, tool set, and execution environment across the original and Harbor implementations. Repeated trials are used to account for stochasticity, and parity sets are documented when full evaluation is too expensive. This procedure is necessary because the paper identifies numerous sources of non-equivalence: platform-specific dependencies, undocumented prompt assumptions, timing-sensitive metrics, malformed oracle solutions, and evaluator bugs.
The adapter audit exposes defects that would otherwise be indistinguishable from model failures. Examples include GAIA2 tasks whose simulation events never fire, a SWE-Bench Pro verifier that checks unspecified metadata, CRUST-Bench scoring tied to a fixed handwritten Rust interface, and audio tasks whose payload is discarded before the agent receives it. Other examples include 13 GSO oracle failures caused by timing variance, invalid or nonterminating SWE-Bench Pro gold patches, and DA-Code requiring 11 evaluator corrections before oracle outputs scored correctly.
These findings support a strong claim: a low benchmark score is not necessarily evidence of a difficult task or a weak agent. It can instead reflect a broken environment, an instruction–verifier mismatch, an invalid oracle, or an exploitable evaluator. Harbor’s explicit oracle validation and parity workflow therefore function as measurement controls, not merely software-engineering conveniences.
Large-scale evaluation design
The principal experiment evaluates eight models from OpenAI, Anthropic, and Google across 54 benchmarks. Each model is tested with Terminus-2 and one vendor-native harness: Codex for GPT models, Claude Code for Claude models, and Gemini CLI for Gemini models. This produces 16 model–harness configurations using four distinct harness implementations. Each benchmark–model–harness configuration is repeated three times.
The evaluation includes approximately 6,627 selected tasks and roughly 0.3 million trajectories, consuming 226 billion input and output tokens and more than $300,000 in compute. The experimental scale is important because it permits model, harness, benchmark, difficulty, cost, and failure-mode analyses within a common execution protocol. At the same time, the task set is not uniformly complete: several large benchmarks are evaluated on sampled subsets, sometimes at rates below 1%, such as MMMLU, SWE-smith, StrongReject, and CyberGym. This makes the results appropriate for comparative analysis under the Harbor setup, but not automatically equivalent to full-benchmark leaderboard scores.
Benchmark progress, redundancy, and effective dimensionality
The study finds substantial variation in benchmark headroom. Mathematics and several function-level coding evaluations are close to saturation, while repository-level feature development, software optimization, scientific research, and broad agentic workflows remain substantially more difficult. The paper reports that 13 of the 54 evaluated benchmarks have current best scores above 90%, whereas seven remain especially hard. Data and analytics has the lowest mean best score at 51%, followed by Agents, Tools and Systems and Scientific Research at 67%.
Figure 2: Reported progress and remaining headroom across the evaluated benchmark suite, grouped by domain.
The central quantitative result is that the benchmark matrix is much lower-dimensional than its nominal size suggests. With the harness fixed to Terminus-2, the first principal component explains 73.7% of variance across the eight models, and the first two explain 81.9%. When native harnesses are added, the first component explains 67.1% and the second 10.1%; together they explain 77.2%. The second component separates native-harness configurations from Terminus-2 configurations, indicating that harness effects introduce an additional but comparatively narrow direction.
A mixed-effects model reaches the same conclusion. Benchmark identity accounts for most score variance, with an intraclass correlation coefficient of 0.75. Six of eight model coefficients are statistically significant, compared with two of four harness coefficients. The model-effect range is 0.451 score units, versus 0.087 for harnesses, a 5.2-fold difference. Thus, base-model capability is substantially more influential than harness identity in this evaluation, although the harness is not irrelevant.
The authors also assess benchmark-level predictability. A mean-score baseline has a median absolute error of 0.117 when predicting held-out benchmark scores. A blended predictor combining cross-benchmark regression and rank-2 SVD reduces this to 0.063, a 46% reduction. Some benchmarks are highly redundant: HumanEvalFix has a median absolute prediction error of 0.004, KUMO 0.017, and BFCL 0.029. Others supply more distinctive signal: ResearchCodeBench reaches 0.221, FinanceAgent 0.202, and LAB-Bench 0.168.
Greedy selection identifies 12 benchmarks before every remaining benchmark correlates with at least one selected benchmark at . The selected set spans software engineering, mathematics, scientific research, tool use, professional domains, and safety. Importantly, uniqueness is not equivalent to difficulty: CodePDE is difficult and informative, while StrongReject is comparatively easy but contributes a distinct safety-related signal. The implication is that benchmark portfolios should optimize coverage of capability axes rather than simply select the hardest available tasks.
Redundancy also appears within benchmarks. Across 52 benchmarks, three representative tasks recover the full system ranking with mean Spearman correlation ; five tasks increase this to 0.941. However, this compression is benchmark-dependent. WideSearch reaches approximately with three tasks, whereas CyberGym reaches only , reflecting greater internal task diversity. The result supports selective evaluation, but it does not justify indiscriminate microbenchmarking: task-level score matrices retain more structure than benchmark aggregates, and a small task set can miss capability dimensions that matter for particular benchmarks.
Figure 3: The singular-value spectrum shows that model capability and harness configuration explain most variance in the benchmark score matrix.
Harness effects and model–harness interactions
Although model identity dominates globally, the harness can materially alter performance on particular models and tasks. Native harnesses generally provide richer interaction patterns, including iterative execution and self-correction, whereas Terminus-2 uses a more linear Plan–Execute–Complete structure. The analysis therefore rejects both extremes: harnesses are neither the primary determinant of performance nor a negligible implementation detail.
The direction of harness effects varies by model family. Claude Code improves over Terminus-2 by an average of 0.239 score units and wins on 65% of benchmark comparisons. Codex has an average difference of 0.007 but wins on 70% of comparisons, indicating that its average effect is small despite frequent benchmark-level wins. Gemini CLI has an average difference of -0.092 and wins on 53% of comparisons. For the strongest GPT model, Codex produces a particularly large advantage over Terminus-2, while Terminus-2 substantially outperforms Codex for GPT-5-nano. These interactions caution against treating a harness as a fixed additive effect independent of the underlying model.
The trajectory analysis gives a behavioral interpretation. Native harnesses support open-ended iteration and correction, while Terminus-2 is more vulnerable when a task requires repeated verification. Gemini frequently uses external search, which can induce “information drift” by replacing a correct initial hypothesis with conflicting retrieved material. GPT-5.4 tends to either solve tasks concisely or terminate early on difficult instances, especially under Terminus-2. Claude models exhibit more stable turn-count and execution patterns across difficulty levels.
Difficulty, token efficiency, and cost
The paper defines empirical task difficulty as one minus the average pass rate across evaluated model–harness configurations. This is explicitly a configuration-dependent proxy rather than an intrinsic task property; task brittleness, verifier errors, and harness affordances can distort it.
The performance advantage of frontier models is bell-shaped across difficulty. Gains are small on very easy tasks because weaker models already succeed, and small on the hardest tasks because all models approach a shared performance floor. The largest gains occur in the medium-to-hard range, approximately 0.3–0.7 difficulty. This result has a direct routing implication: the additional cost of frontier models is most defensible on tasks where the performance margin is substantial rather than at either distributional tail.
Figure 4: Frontier models achieve higher pass rates and use fewer tokens, but their higher token prices preserve a substantial cost premium.
A notable and somewhat counterintuitive result is that frontier models use fewer tokens than weaker models at every empirical difficulty level. In the easiest bucket, frontier models consume approximately 42% of weaker-model tokens; in the upper half of the difficulty distribution, the ratio rises to 61–86%. Manual trajectory inspection attributes the excess usage of weaker models to instruction misunderstanding, repeated tool calls, verbose reasoning, and redundant verification. Frontier models more often execute the intended procedure directly.
However, token efficiency does not offset pricing differences. The weaker models cost approximately 2–3 times less per trial, and the absolute cost gap peaks at about $0.37 per trial in the 0.7–0.8 difficulty bucket. The paper therefore distinguishes token efficiency from economic efficiency: shorter trajectories by frontier models do not imply lower evaluation cost.
Failure modes and the role of task quality
Failure analysis covers 6,028 trajectories from 45 benchmarks, three frontier models, and two harnesses. Two domain-experienced annotators label 200 trajectories across 12 failure modes, obtaining pooled inter-rater $\kappa = 0.66\kappa = 0.51$.
Figure 5: Human- and judge-annotated failure modes show that factual errors, algorithmic bugs, and hidden-test regressions dominate frontier-agent failures.
Wrong factual answers, algorithmic bugs, and hidden-test regressions dominate across models and harnesses. These failures indicate persistent deficits in task comprehension, domain knowledge, algorithmic reasoning, and robust implementation. Operational failures such as syntax errors and missing deliverables occur less prominently but remain non-negligible.
The analysis separates capability failures from infrastructure and task-design failures. Harbor-Index auditing finds that approximately one third of the hardest candidate tasks across more than 30 benchmarks are rejected as broken rather than genuinely difficult. In the GAIA2 examples, the verifier waits for simulation events that the adapter never triggers, making every trial time out independently of agent behavior. In SWE-Bench Pro, a verifier checks metadata and test files absent from the task specification. In CRUST-Bench, solutions are rejected because the verifier assumes a fixed Rust interface even when alternative safe implementations satisfy the stated objective.
This distinction matters for interpreting benchmark ceilings. A verifier with false positives or false negatives can impose an upper bound on measured accuracy, and repeated sampling cannot correct such a defect. The paper’s auditing results therefore support a strong methodological claim: benchmark maintenance and verifier validation are prerequisites for interpreting agent scores as capability measurements.
Harbor-Index as a curated evaluation subset
Harbor-Index 1.0 distills the adapted suite into 82 tasks spanning 29 benchmarks and multiple domains. The curation pipeline begins with 1,311 difficult candidates selected from the larger task pool. A difficulty filter requires three leading models, each under native and Terminus-2 harnesses with three repeats, to succeed on at most 33% of trials. AI auditing reduces the set to 307, human review reduces it to more than 110, and a senior panel selects 100. A final audit-and-fix process yields 82 tasks after removing or repairing structurally defective instances.
Figure 6: Harbor-Index distributes 82 audited tasks across 29 benchmarks and several agentic domains.
The curation criterion is not difficulty alone. A task must be difficult, well-specified, diverse, and resistant to verifier loopholes. Reviewers explicitly inspect trajectory-grounded failures, false positives, false negatives, instruction–verifier alignment, and whether difficulty arises from substantive reasoning, domain expertise, long-horizon interaction, or multi-step execution.
The resulting suite remains challenging under the evaluated configurations. No model–harness configuration exceeds a 30% pass rate, and GPT-5.5 with Codex reaches the maximum at 28.0%. Claude Opus 4.8 with Claude Code reaches 20.7%, while Gemini 3.1 Pro with Gemini CLI reaches 13.4%. Several open-weight models obtain lower pass rates but occupy favorable cost–performance positions under Terminus-2.
| Model–harness configuration | Pass rate | Approximate full-run cost |
|---|---|---|
| GPT-5.5 + Codex | 28.0% | $178 |
| Claude Opus 4.8 + Claude Code | 20.7% | $269 |
| Gemini 3.1 Pro + Gemini CLI | 13.4% | $74 |
| GLM 5.2 + Terminus-2 | 9.8% | $52 |
| Kimi K2.6 + Terminus-2 | 8.5% | $33 |
| MiniMax M3 + Terminus-2 | 6.1% | $18 |
| MiMo V2.5 Pro + Terminus-2 | 2.4% | $4 |
The evaluation also reveals an interaction between model capability and harness affordances. The six lower-scoring models time out on 36.7% of rollouts, compared with 18.7% for the three highest-scoring models. Across matched tasks, native harnesses reduce the timeout rate from 42% to 26% relative to Terminus-2. Terminus-2’s bash-only action space, which lacks native image and web tools, explains part of this difference. Thus, Harbor-Index’s cost–performance frontier is informative, but it is not a pure model frontier: tool availability and harness design materially affect both pass rate and cost.
Figure 7: Harbor-Index exposes a cost–performance frontier in which native harnesses favor the strongest models while Terminus-2 offers lower-cost operation for several open-weight models.
Limitations and open questions
The paper’s empirical conclusions are bounded by the evaluated configuration space. The large-scale experiment covers eight models from three providers, four harness implementations, and 54 benchmarks, not the full adapter catalog or the broader population of agent architectures. Provider aliases and preview endpoints may not correspond to immutable model snapshots, and some models are evaluated with default reasoning settings rather than the stronger settings used in public leaderboards.
Many benchmark comparisons use sampled subsets, and only 16 of the 54 benchmarks have at least one external evaluation configuration directly overlapping with the Harbor model–harness matrix. Historical progress curves are therefore observational rather than strict apples-to-apples comparisons. Harbor-Index also uses LLM-as-a-judge scoring for some free-form tasks, inheriting judge reliability and calibration concerns. Its conversion of continuous metrics into pass/fail outcomes simplifies aggregation but discards information about partial performance.
The redundancy analysis is similarly conditional on the 16 evaluated system configurations. Prediction from a low-rank benchmark matrix may degrade for future models that occupy new capability regions, and the reported task-level compression results vary substantially by benchmark. The paper leaves open whether a compact subset selected from current systems will preserve rankings for substantially different architectures, modalities, tool policies, or training distributions.
Finally, the live maintenance model introduces a methodological tradeoff. Updating Harbor-Index can reduce saturation and repair defects, but changing tasks over time complicates longitudinal comparability. The unresolved question is how to maintain a benchmark that remains difficult and valid without turning it into a moving target whose successive versions cannot be directly compared.
Conclusion
Harbor Adapters provide a practical abstraction for executing heterogeneous agentic benchmarks under a common task, sandbox, and scoring interface. The associated evaluation shows that benchmark difficulty dominates score variance, model identity generally matters more than harness identity, benchmark collections contain substantial redundancy, frontier models are more token-efficient but remain more expensive, and a significant fraction of apparent task difficulty can originate in defective environments or verifiers.
Harbor-Index operationalizes these findings as an 82-task, 29-benchmark evaluation set designed around difficulty, diversity, and auditability. Its low absolute pass rates preserve headroom, while its compact size makes repeated evaluation feasible. The paper’s principal contribution is therefore not a single leaderboard, but an integrated evaluation methodology in which infrastructure, parity validation, task auditing, trajectory analysis, and cost accounting are treated as necessary components of agent measurement (2609.04298).
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
1. What is this paper about?
This paper introduces Harbor Adapters, a system for testing AI agents on many different tasks in a more organized and reliable way. It also introduces Harbor-Index, a smaller collection of especially challenging and carefully checked tasks.
An AI agent is a computer program powered by a LLM that can do more than answer questions. It might:
- write and test computer code,
- search the web,
- use tools,
- operate a computer,
- solve long, complicated problems,
- or complete tasks in a simulated environment.
The researchers wanted to make it easier to compare these agents fairly. They also wanted to discover whether agents fail because they are not smart enough, because their tools are poorly designed, or because the tasks themselves contain mistakes.
2. What questions did the researchers ask?
The paper focuses on four main questions:
- Which tests are actually useful? Do we need hundreds of different benchmarks, or do many of them measure the same abilities?
- What matters more: the AI model or the agent system around it? A model is the LLM doing the thinking. A harness is the software that gives the model tools, instructions, and a way to interact with its environment.
- Are more powerful models worth their higher cost? Do stronger models solve tasks more efficiently, and when is it worth paying extra for them?
- Why do agents fail? Are they making reasoning mistakes, using tools incorrectly, misunderstanding instructions, or being judged by broken tests?
3. How was the research done?
Building a common system
Different benchmarks usually have different formats. One might ask an agent to fix a software bug, while another might ask it to search the internet or analyze financial information. Running all of them can be difficult because each benchmark has its own rules and software.
The researchers created Harbor Adapters to solve this problem. An adapter acts like a translator: it converts each benchmark into the same basic format, including:
- an instruction,
- an environment where the agent works,
- tests for checking the answer,
- and a possible solution.
This is similar to using one universal plug adapter when traveling. Instead of making every device work with every kind of outlet separately, the adapter lets many devices use the same system.
Without this approach, connecting many benchmarks to many agents could require a separate connection for every pair. With Harbor, each benchmark only needs one adapter, and each agent only needs one Harbor connection. This makes large-scale testing much easier.
The researchers adapted more than 80 benchmarks and checked their work through:
- code reviews,
- experiments comparing the adapted versions with the original versions,
- and several rounds of human and automated quality checks.
Testing models and agent systems
The main experiment tested:
- 8 LLMs from Google, OpenAI, and Anthropic,
- 54 benchmarks,
- 4 different harness systems,
- and three repeated trials for each setting.
Repeating trials is important because AI agents can behave differently each time, even when given the same task. It is like asking a student to take the same type of test several times to get a more reliable result.
The researchers collected hundreds of thousands of agent attempts. This required a large amount of computer time and cost more than $300,000.
Studying failures
The researchers also examined thousands of failed attempts. Human experts first labeled some failures, and then an AI judge helped analyze many more. They looked for problems such as:
- wrong facts,
- bugs in computer programs,
- missing files or answers,
- syntax mistakes,
- poor tool use,
- and failures caused by unclear or broken tasks.
Creating Harbor-Index
The researchers wanted a smaller test set that would be cheaper to run but still difficult and useful. They began with many tasks and gradually removed tasks that were:
- too easy,
- repetitive,
- unclear,
- poorly graded,
- or broken in some way.
Human reviewers and AI systems checked the remaining tasks. The final Harbor-Index contains 82 difficult tasks from 29 benchmarks.
4. What did the researchers find?
The model usually matters more than the harness
The LLM itself had a larger effect on performance than the harness surrounding it. In other words, giving an agent better tools can help, but a much stronger “brain” usually helps more.
However, the harness still matters. Some harnesses let agents repeatedly check and improve their work, while others make them follow a more rigid sequence. Agents often did better when they could freely revise their answers.
Many benchmarks measure similar abilities
The researchers found that many benchmarks produce similar rankings of models. If one model performs well on one benchmark, it often performs well on many others.
This means that testing hundreds of benchmarks may not always provide hundreds of different pieces of information. A smaller set of carefully chosen benchmarks could sometimes show nearly the same differences between models.
However, the researchers warn that benchmarks should not be chosen only because they are difficult. Some easier tests are valuable because they measure special abilities, such as financial reasoning or scientific work, that other tests do not measure.
Stronger models often use fewer tokens
A token is a small piece of text processed by a LLM. Using more tokens usually means using more computing power and money.
The stronger models generally used fewer tokens than weaker models, especially on easier tasks. Weaker models often:
- misunderstood instructions,
- repeated tool calls,
- gave unnecessarily long explanations,
- or checked the same thing many times.
Even so, stronger models were still more expensive overall because their individual tokens cost more.
The biggest advantage of stronger models appeared on tasks of medium difficulty. On very easy tasks, almost every model could succeed. On extremely hard tasks, even the strongest models often failed. Medium-to-hard tasks were where paying for a stronger model helped most.
Many failures came from the tasks themselves
One of the most important findings was that some tasks were not genuinely difficult—they were simply badly designed.
For example:
- a task might tell the agent to create something, but the grader might look for something different;
- a simulation might fail to start correctly;
- a test might require exact file names or messages that were never mentioned in the instructions;
- or a correct solution might be rejected because it did not copy the example solution closely enough.
The researchers found that about one third of the hardest candidate tasks examined by human reviewers were rejected because they were broken or unfair.
This matters because a low score does not always mean that an AI is incapable. Sometimes the test itself is the problem.
Agents still struggle with important reasoning
Among genuine failures, the most common problems included:
- incorrect factual answers,
- algorithmic mistakes,
- hidden-test failures in software,
- and misunderstanding what the task required.
Different models also had different habits:
- Some models tried to solve tasks quickly but gave up more often on very difficult problems.
- Some models followed a consistent procedure.
- Some relied heavily on web searches and sometimes replaced a correct idea with confusing or incorrect information from online sources.
Harbor-Index remains difficult
The strongest evaluated system scored only 28% on Harbor-Index. No model-and-harness combination scored above 30%.
This shows that the new test set is still challenging and leaves plenty of room for future improvement. At the same time, it is much smaller and cheaper to run than the full collection of benchmarks.
5. Why is this research important?
This paper could improve how AI systems are tested.
Without a common testing system, researchers may compare models using different tools, task versions, or grading rules. That can make results confusing or unfair. Harbor Adapters offer a shared framework that could make comparisons more consistent.
Harbor-Index is also useful because it tries to avoid a major problem in AI testing: confusing “the AI failed” with “the test was broken.” Carefully checking tasks can make test results more trustworthy.
The findings suggest several lessons:
- Researchers should test models across different kinds of abilities, not just use many similar benchmarks.
- Stronger models are not always worth their cost, especially on very easy or impossibly hard tasks.
- Agent tools and software design matter, even though the underlying model is usually more important.
- Benchmarks should be regularly checked and repaired as AI systems improve.
- A small, diverse, high-quality test set may be more useful than a huge collection of repetitive tests.
Overall, the paper presents Harbor as a way to make AI-agent evaluation more organized, affordable, fair, and reliable. It does not prove that one testing method is perfect, but it provides tools and evidence that could help researchers better understand what AI agents can—and cannot—do.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
- Incomplete benchmark coverage: Although more than 80 adapters are reported, the large-scale analysis includes only 54 benchmarks, leaving the generality of the findings across the broader agentic-benchmark ecosystem unresolved.
- Limited model and harness diversity: The evaluation covers eight models in the large-scale study and nine in Harbor-Index, with only four distinct harness implementations. It does not establish whether the observed model–harness relationships hold for other model families, open-weight checkpoints, agent architectures, scaffolds, or custom tool-use systems.
- Confounding between model, harness, and provider: Native harnesses are tied to particular model families, while most OpenRouter models in Harbor-Index use Claude Code. This prevents a fully crossed comparison in which every model is evaluated with every harness under identical serving and tool-access conditions.
- Insufficient evidence for universal model dominance: The conclusion that base-model capability matters more than harness design is based on a relatively small, selected set of models and harnesses. More diverse harnesses and controlled interventions are needed to determine when scaffold design can outweigh model differences.
- Unclear causal mechanisms behind harness effects: The study attributes performance differences partly to features such as iterative correction, action-space design, and native image or web tools, but does not isolate their individual causal contributions through component-level ablations.
- Limited replication across model versions: The experiments use specific model releases and harness versions. It remains unknown whether the reported rankings, failure patterns, and cost–performance tradeoffs persist after model updates, changed system prompts, revised tool implementations, or altered inference policies.
- Small number of trials for stochastic evaluation: Most configuration–benchmark settings are repeated only three times. This may be insufficient to estimate variance reliably for highly stochastic, long-horizon tasks or to distinguish small performance differences between models and harnesses.
- Uncertainty in benchmark-level rankings: The paper reports correlations and mixed-model effects, but the statistical uncertainty of many individual benchmark rankings and pairwise model comparisons is not fully characterized, especially after accounting for multiple comparisons and task-level dependence.
- Difficulty is configuration-dependent: Empirical task difficulty is computed from the evaluated models and harnesses, so it may change as stronger systems are introduced. The resulting difficulty buckets do not necessarily measure intrinsic task difficulty and may be distorted by verifier errors, tool limitations, or harness-specific behavior.
- No validated method for online difficulty estimation: The proposed cost-aware routing intuition depends on identifying task difficulty before or during execution, but the paper does not provide a practical, validated predictor that can estimate difficulty cheaply at request time without already solving the task.
- Incomplete cost accounting: Reported costs rely on official or OpenRouter pricing and may omit factors such as long-context charges, cache-write costs, infrastructure overhead, human review, environment provisioning, retries, and tool-service fees. The claimed cost–performance frontier may therefore not generalize to other deployment settings.
- Cost comparisons are not fully normalized: Models may differ in context limits, caching behavior, reasoning-token accounting, rate limits, and serving infrastructure. These differences complicate direct comparisons of per-run cost and token efficiency.
- Token efficiency is not necessarily causal: The finding that stronger models use fewer tokens does not determine whether shorter trajectories cause better performance, result from better early decisions, or reflect differences in hidden reasoning and output-token accounting.
- Limited analysis of adaptive inference policies: The study evaluates fixed model–harness configurations rather than dynamic strategies such as model routing, selective escalation, early stopping, verifier-guided retries, or allocating additional computation only to uncertain tasks.
- Low-rank benchmark structure may be sample-specific: PCA and correlation analyses are conducted on a small matrix of model–harness configurations. The reported low dimensionality may change with a larger and more diverse population of agents, models, or task distributions.
- Redundancy estimates may be unstable: Claims that three representative tasks recover benchmark rankings are based on the evaluated systems and selected sampling procedures. They do not establish that the same small subsets preserve rankings for future, stronger, specialized, or adversarial agents.
- Correlation does not establish construct equivalence: High correlations between benchmark scores show similar ranking behavior but do not demonstrate that benchmarks measure the same underlying capability or that replacing one with another preserves important rare abilities.
- Specialized capabilities remain underrepresented: The analysis identifies benchmarks such as FinanceAgent and LabBench as relatively unique, but the study does not systematically determine which capabilities are missing from the suite or how well the benchmark collection covers real-world agent requirements.
- Limited multimodal and embodied evaluation: The benchmark suite contains little evidence about vision, audio, GUI interaction, robotics, physical environments, or multimodal tool use. Conclusions may therefore apply primarily to text-and-terminal agents.
- External-service and environment variability is unresolved: The paper does not fully quantify how changes in network access, package versions, APIs, data freshness, sandbox providers, hardware, or environment resets affect adapter parity and benchmark reproducibility.
- Adapter parity is not exhaustively demonstrated: Code review and parity experiments provide evidence of faithful adaptation, but the paper does not specify coverage, statistical power, or failure rates for parity tests across all adapters and task types. Semantic equivalence may remain uncertain for complex interactive benchmarks.
- Potential adaptation-induced changes in task semantics: Converting non-agentic benchmarks into tool-using environments may alter the construct being measured. The validity of treating such adapted tasks as comparable to their original, non-agentic versions remains unestablished.
- Verifier validity is still an open problem: The audit identifies instruction–verifier mismatches, false positives, false negatives, and exploitable loopholes, but there is no systematic estimate of verifier accuracy for the full 54-benchmark suite or for all Harbor-Index tasks.
- Human audit reliability is limited and uneven: Human reviewers show moderate agreement in the trajectory analysis, and the AI judge has lower agreement with the human gold labels. The reliability of the broader multi-stage audit process, including task acceptance and repair decisions, is not fully quantified.
- AI-assisted auditing may reproduce model biases: Harbor-Index relies substantially on Gemini-based auditing and judging. It remains unclear whether this introduces systematic preferences for particular solution styles, models, tools, or task formulations.
- Harbor-Index selection is potentially circular: Tasks are selected using performance from leading models and then evaluated on related model families and harnesses. This may favor tasks that are difficult for the selection systems rather than tasks with stable, model-independent validity.
- Harbor-Index difficulty may not persist: The claim that no evaluated configuration exceeds a 30% pass rate is time-sensitive. Stronger models, specialized prompting, test-time compute, or benchmark-specific optimization may quickly saturate the fixed 82-task set.
- Task repair may change benchmark comparability: Repairing or modifying tasks can improve validity but may also alter their original difficulty, intended construct, or relationship to the source benchmark. The paper does not report systematic pre/post repair statistics sufficient to assess this tradeoff.
- Limited evidence for real-world validity: It remains unknown whether performance on Harbor-Index predicts success in production agent deployments, expert workflows, user satisfaction, safety-critical settings, or long-term autonomous operation.
- Pass/fail scoring discards useful information: Continuous outcomes, partial progress, quality of intermediate artifacts, efficiency, and degrees of correctness are reduced to binary outcomes for several tasks. This may obscure meaningful differences between near-miss solutions and make optimization sensitive to arbitrary thresholds.
- SOTA-threshold metrics may become obsolete or biased: Converting AlgoTune and GSO outcomes into pass/fail using current state-of-the-art thresholds rewards surpassing a moving frontier but may penalize strong incremental improvements and become unstable as reference systems change.
- LLM-as-a-judge validity remains uncertain: Free-form tasks depend on model-based judging, but judge calibration, robustness to persuasive incorrect answers, sensitivity to verbosity, and consistency across domains are not comprehensively evaluated.
- Failure-mode taxonomy is incomplete: The analysis covers 12 failure modes and collapses several low-agreement categories into “Others,” leaving potentially important causes—such as tool unreliability, prompt injection, strategic deception, memory failures, and distribution shift—underexplored.
- Failure annotations focus mainly on frontier-model failures: The detailed trajectory audit emphasizes failed hard-task runs from frontier models. Failure mechanisms for weaker models, successful trajectories, and cases involving strategic adaptation are less thoroughly characterized.
- Failure-mode prevalence may depend on the annotation sample: The 200 human-annotated trajectories and subsequent judge-labeled sample may not represent the full task, model, and harness distribution. Sampling procedures and class imbalance could affect reported prevalence.
- No systematic analysis of recovery and self-correction: The study notes that iterative refinement matters, but it does not quantify which types of errors agents can detect and repair, how many retries are required, or whether additional verification disproportionately benefits particular models.
- Security and gaming risks remain unresolved: The paper acknowledges that future agents may exploit benchmark artifacts, but it does not provide adversarial evaluations of reward hacking, verifier manipulation, data leakage, prompt injection, sandbox escape, or benchmark-specific overfitting.
- Contamination and memorization are not fully assessed: The study does not establish whether evaluated models were exposed to benchmark tasks, solutions, repositories, adapter code, or Harbor-Index artifacts during training or post-training.
- Reproducibility may be threatened by live dependencies: Online services, vendor APIs, changing model endpoints, package updates, and evolving benchmark data may make exact replication difficult despite releasing adapters and trajectories.
- No principled update protocol for the live Harbor-Index is specified: The paper proposes maintaining the index as models improve but does not define rules for task retirement, replacement, versioning, contamination monitoring, score comparability, or preventing benchmark churn.
- The relationship between benchmark breadth and practical evaluation value is unresolved: The study shows that many benchmarks are statistically redundant, but it does not determine how to balance ranking efficiency against coverage of rare, safety-critical, or operationally important capabilities.
- Generalization beyond pass-rate leaderboards is unclear: The conclusions focus primarily on aggregate pass rates, rankings, token use, and costs. They do not establish how the infrastructure supports richer objectives such as reliability, calibration, latency, interpretability, safety, maintainability, or human–agent collaboration.
- No evaluation of long-term autonomous behavior: Tasks are generally bounded by benchmark-specific execution limits. The paper does not examine degradation, compounding errors, memory management, goal drift, or resource allocation over substantially longer autonomous horizons.
- Human-level and expert baselines are largely absent: Without systematic human or expert performance and cost baselines, it is difficult to interpret whether low model pass rates reflect genuinely challenging tasks, poor task design, or performance far below practical competence.
- The integration-cost claim is not empirically validated: The shift from to describes the adapter architecture, but the paper does not quantify actual engineering time, maintenance burden, debugging effort, or total cost before and after adopting Harbor Adapters.
Practical Applications
Immediate Applications
- Standardized agent-evaluation pipelines for AI developers and research teams (software/AI infrastructure)
- Potential tools: continuous-evaluation services, model-release dashboards, regression-testing systems, and internal agent scorecards.
- Actionable workflow: add a new agent integration once, select a set of compatible adapters, execute evaluations in sandboxes, and compare results using shared metrics.
- Dependencies: adapter correctness, access to benchmark environments, compatible sandbox backends, and sufficient compute or API budget.
- Pre-release testing and regression monitoring for coding agents (software engineering)
- This can reveal failures involving algorithmic bugs, hidden-test regressions, syntax errors, incomplete deliverables, and incorrect task interpretation.
- Potential products: pull-request agent certification, model-upgrade regression gates, automated coding-agent acceptance tests, and enterprise agent reliability reports.
- Dependencies: benchmark tasks must remain representative of the organization’s codebase; passing benchmark tasks does not establish production safety or correctness.
- Cost-aware model and harness selection (cloud AI operations and enterprise software)
- Frontier models may be reserved for medium-to-hard tasks where their performance advantage is most valuable, while less expensive models can handle simpler workloads.
- Lower-cost open-weight configurations may be appropriate when a moderate pass rate is acceptable.
- Potential tools: model routers, workload-specific routing policies, budget-aware orchestration, and Pareto-frontier dashboards.
- Dependencies: task difficulty must be estimated reliably; pricing, caching, context length, latency, and tool access can differ substantially from the paper’s setup.
- Benchmark-quality auditing and verifier validation (academia, software, and evaluation governance)
- Actionable workflow: run representative trajectories, compare human judgments with automated grading, inspect failures, repair task definitions, and repeat the evaluation.
- Potential tools: verifier test suites, trajectory-review interfaces, audit checklists, and automated inconsistency detectors.
- Dependencies: domain expertise, independent reviewers, reliable test environments, and careful versioning of repaired tasks.
- Compact internal evaluation suites for companies and laboratories (industry and academia)
- Actionable workflow: use correlation, PCA, or greedy selection to identify tasks covering different capability axes; retain specialized domains such as finance, laboratory science, and research assistance even when they are not the hardest.
- Potential products: 20–100 task smoke tests for every model release, domain-specific “mini-indexes,” and rapid evaluation suites for daily development.
- Dependencies: redundancy estimates may change as models improve; small subsets require uncertainty estimates and periodic revalidation.
- Failure-mode diagnostics for deployed agents (healthcare administration, finance, research, and enterprise automation)
- For example, an agent handling financial research could be tested for information drift after web search; a healthcare workflow agent could be tested for unsupported factual claims and incomplete actions.
- Potential tools: failure-taxonomy dashboards, automatic trajectory triage, human escalation queues, and post-deployment incident reports.
- Dependencies: failure labels must be validated for the relevant domain; LLM-based judges may introduce classification errors and should not be the sole safety mechanism.
- More reproducible academic comparisons of agents (academic research)
- Potential outputs: reproducible experiment manifests, trajectory repositories, benchmark parity reports, and standardized leaderboards.
- Dependencies: access to the same model versions and APIs, stable benchmark environments, and disclosure of proprietary harness details.
- Human-in-the-loop escalation policies (daily life and enterprise productivity)
- automatically accept only outputs passing deterministic tests;
- route near-misses to human review;
- terminate repeated tool loops or timeouts;
- require additional verification for high-impact actions.
- Potential applications: coding assistants, spreadsheet agents, research assistants, customer-service automation, and personal productivity tools.
- Dependencies: reliable pass/fail tests, clear escalation thresholds, and human reviewers with adequate domain knowledge.
Long-Term Applications
- A general-purpose evaluation marketplace and compatibility layer for agents (AI infrastructure and software ecosystems)
- Potential products: hosted evaluation APIs, benchmark registries, enterprise certification services, and plug-in marketplaces.
- Benefits: lower integration costs, faster replication of results, and broader cross-domain comparisons.
- Dependencies: stable interface specifications, maintenance of environments and dependencies, security isolation, licensing agreements, and governance against benchmark contamination.
- Adaptive evaluation that selects tasks in real time (AI research and model deployment)
- Potential workflow: estimate a model’s capability profile, select tasks with the highest expected information gain, and stop when ranking confidence reaches a target level.
- Applications: rapid model procurement, personalized agent certification, and continuous monitoring after model updates.
- Dependencies: robust uncertainty estimates, prevention of overfitting to the adaptive test, calibration on future stronger models, and preservation of rare capability axes.
- Dynamic model routing based on predicted task difficulty (cloud computing and enterprise automation)
- Potential products: difficulty predictors, token-budget controllers, cost-aware multi-agent systems, and confidence-triggered escalation.
- Applications: customer support, software maintenance, document analysis, research assistance, and data workflows.
- Dependencies: cheap and accurate difficulty prediction; reliable confidence estimates; changing model prices; latency requirements; and safeguards against routing difficult safety-critical tasks to weak models.
- Continuous, self-renewing benchmarks for frontier agents (academia, policy, and AI safety)
- Potential mechanisms: versioned releases, hidden test sets, rotating tasks, independent audits, contamination checks, and public trajectory metadata.
- Applications: frontier-model evaluations, procurement standards, safety cases, and regulatory reporting.
- Dependencies: sustained curation, funding for human review, secure task pools, transparent governance, and careful comparability across benchmark versions.
- Agent certification standards for regulated or high-impact sectors (healthcare, finance, education, public administration, and cybersecurity)
- Examples: a healthcare agent could be tested on evidence retrieval and structured workflow completion; a financial agent on domain-specific analysis and tool execution; an education agent on grounded feedback and safe interaction.
- Potential outputs: procurement checklists, minimum pass-rate thresholds, standardized incident taxonomies, and audit trails.
- Dependencies: sector-specific task design, privacy-preserving environments, legally valid evaluation criteria, and thresholds determined by risk rather than aggregate benchmark scores alone.
- Training and fine-tuning agents using failure trajectories (AI research and developer tooling)
- Potential methods: supervised trajectory refinement, preference optimization, verifier-aware training, tool-use curriculum learning, and self-correction policies.
- Applications: coding agents, scientific assistants, web researchers, and autonomous workflow systems.
- Dependencies: high-quality labels, protection against learning benchmark-specific shortcuts, careful treatment of private or copyrighted trajectories, and evidence that improvements transfer beyond the evaluation suite.
- Standardized simulation infrastructure for robotics and embodied systems (robotics and autonomous systems)
- Potential tools: interchangeable simulator adapters, robot-policy regression testing, hardware-in-the-loop evaluation, and cross-platform task suites.
- Dependencies: realistic simulators, sensor and actuator interfaces, safety validation, sim-to-real transfer, and evaluation metrics that capture physical robustness rather than only task completion.
- Policy and public-sector monitoring of agent capability claims (public policy and standards)
- Potential applications: evidence requirements for AI procurement, reporting standards for model releases, sector-specific benchmark registries, and monitoring of capability progress.
- Dependencies: neutrality of benchmark maintainers, protection against gaming, transparent methodology, model-access agreements, and recognition that benchmark scores are configuration-dependent rather than universal measures of intelligence.
- Personalized daily-life assistants with bounded autonomy (consumer software and accessibility)
- Potential workflow: evaluate a task on a low-cost model, verify the result with deterministic checks or a second agent, and request user confirmation for irreversible actions.
- Dependencies: privacy-preserving local or sandboxed execution, reliable permission systems, robust handling of ambiguous instructions, and strong safeguards for financial, medical, and legal activities.
Glossary
- Agentic benchmark: A benchmark designed to evaluate AI systems that act autonomously through tools, environments, and multi-step interactions. “The benchmark landscape has grown in response: SWE-bench~\citep{jimenez2023swebench}, Terminal-Bench~\citep{terminalbench}, FinanceAgent~\citep{valsai2024financeagent}, and many other agentic, tool-using, long-horizon suites”
- Agent scaffold: The software structure or control framework that organizes an agent’s reasoning, tool use, and execution. “Most variation is explained by a single shared capability factor; adding agent scaffolds introduces one additional direction”
- API: An application programming interface that allows software components or services to communicate programmatically. “GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro are served by first-party APIs”
- Adapter: A software component that converts one benchmark’s data and execution requirements into a standardized interface. “each benchmark needs one adapter exposing its tasks, environments, and scoring logic through Harbor”
- Annotation protocol: A prescribed procedure for labeling or categorizing data, often used to ensure consistent human judgments. “Full taxonomy, annotation protocol, and per-rubric reliability are in Appendix~\ref{app:qual-method}”
- Benchmark saturation: A condition in which models achieve such high scores on a benchmark that it no longer meaningfully differentiates capabilities. “Of the \NumIncludedBenchmarks{} benchmarks we evaluate, \NumSaturatedBenchmarks{} are largely saturated”
- Capability factor: A latent dimension representing a general ability that explains performance across multiple tasks or benchmarks. “Most variation is explained by a single shared capability factor”
- Capability gap: A persistent difference between the abilities required by a task and the abilities an evaluated model possesses. “indicating a persistent ``capability gap'' in task comprehension, domain knowledge, and the reasoning needed to produce substantially correct solutions”
- Closed-weight model: A model whose learned parameters are not publicly released. “four closed-weight models, whose weights are not publicly released”
- Configuration-dependent proxy: A measured quantity that approximates a property but depends on the particular experimental setup used to obtain it. “this is a configuration-dependent proxy for intrinsic task difficulty”
- Contamination: The presence of benchmark data or related information in a model’s training data, potentially inflating evaluation results. “contamination erodes held-out validity”
- Cross-family harness: An agent execution framework designed to operate across models from different model providers or families. “the cross-family Terminus-2”
- False negative: An incorrect failure judgment in which a valid solution is classified as unsuccessful. “false-positive/false-negative analysis”
- False positive: An incorrect success judgment in which an invalid solution is classified as successful. “false-positive/false-negative analysis”
- Frontier model: A highly capable model near the current leading edge of performance. “Frontier models are more capable but more expensive.”
- Generalized linear mixed model (GLMM): A statistical model combining fixed effects and random effects, typically for non-normally distributed outcomes. “we control for it with a linear mixed model using benchmark as a random intercept”
- Harness: Software that manages an agent’s interaction with a model, tools, environment, and task. “Each model runs under exactly two harnesses”
- Hidden-test regression: A failure in which a change passes visible tests but breaks functionality checked by undisclosed tests. “wrong factual answers, algorithmic bugs, and hidden-test regressions dominate across all frontier models”
- Holdout hygiene: Practices that preserve the independence and integrity of evaluation data not used during model development. “interface, scaffold, cost budget, and holdout hygiene all move reported numbers”
- Instruction–verification alignment: The degree to which a task’s stated requirements correspond to what its evaluator actually checks. “a quality rubric based on instruction--verification alignment”
- Inter-rater reliability: The degree to which independent evaluators agree when labeling or scoring the same items. “two domain-experienced human annotators independently label 200 trajectories across 12 failure modes”
- Intraclass correlation coefficient (ICC): A statistic measuring the proportion of outcome variance attributable to differences between groups or subjects. “Benchmark difficulty accounts for most score variance (ICC $=\GLMMIcc{}$)”
- LLM-as-a-judge: The use of a LLM to evaluate another model’s output against a rubric or criterion. “the adapter infrastructure supports GPU interaction, LLM-as-a-judge verifiers”
- Long-horizon task: A task requiring an agent to perform many dependent actions over an extended sequence. “software engineering, web research, finance, and terminal-based problem solving”
- Low-rank: Having substantially fewer underlying dimensions than the apparent number of measured variables. “the space remains low-rank”
- Marginal benefit: The additional performance or value gained from an incremental change, such as using a stronger model. “The absolute performance gain of frontier models is bell-shaped: is marginal on the simplest and hardest tasks”
- Meta-dataset: A curated dataset assembled from tasks or data originating in multiple component datasets or benchmarks. “we introduce Harbor-Index, a curated meta-dataset of 82 tasks”
- Mixed model: A statistical model containing both population-level effects and group-specific effects. “we control for it with a linear mixed model using benchmark as a random intercept”
- Open-weight model: A model whose learned parameters are publicly available, even if its training data or full development process is not. “five open-weight models”
- Pareto frontier: The set of options for which no alternative is simultaneously better on all evaluated objectives, such as cost and performance. “Colored points and the dashed line mark the cost--performance Pareto frontier”
- Parity experiment: An experiment that compares two implementations under matched conditions to determine whether they produce equivalent results. “we run multi-trial parity experiments matching agent, model, and execution configuration”
- Principal Component Analysis (PCA): A dimensionality-reduction method that transforms correlated variables into a smaller set of orthogonal components capturing variance. “we apply Principal Component Analysis (PCA) to the $\AppNumSystems{} \times \NumIncludedBenchmarks{}$ score matrix”
- Random intercept: A group-specific baseline term in a statistical model that allows each group to have its own average outcome. “using benchmark as a random intercept”
- Reward hack: An unintended strategy that exploits weaknesses in an evaluation or reward mechanism rather than accomplishing the intended task. “currently unexploitable reward hacks and unreliable evaluation”
- Sandbox environment: An isolated computational setting in which an agent can execute actions without unrestricted access to the host system. “Harbor, a Python library~\citep{Harbor_Framework} originally developed for Terminal-Bench that defines tasks and runs agents in sandbox environments”
- SOTA threshold: A threshold based on the current state-of-the-art result that a solution must exceed to count as successful. “we ``SOTA-threshold'' outcomes into pass/fail”
- Spearman rank correlation: A nonparametric statistic measuring the association between the rankings of two variables. “Greedy selection on absolute Spearman rank correlations agrees”
- Stochasticity: Variability in results caused by randomness in models, sampling, or execution. “reporting mean sample standard error of the mean (SEM) to account for stochasticity”
- Task brittleness: Sensitivity of a task or evaluator to minor changes, implementation details, or edge cases. “as it may be distorted by task brittleness (false positives/negatives)”
- Trajectory: The ordered sequence of actions, observations, tool calls, and outputs produced during an agent’s attempt. “Each rubric shows six bars grouped by model”
- Verifier loophole: An unintended weakness in an evaluator that allows an agent to pass without satisfying the task’s intended requirements. “free of exploitable verifier loopholes”
- Verifier overreach: An evaluator’s enforcement of requirements that were not specified by the task. “A SWE-bench Pro \cite{deng2025swebenchproaiagents} task shows verifier overreach”
- Zero-shot: An evaluation setting in which the model receives no task-specific examples before attempting the task. “Free-form answer tasks (e.g., HLE, GAIA, GPQA Diamond) use LLM-as-a-judge evaluation”