Harbor Adapters: Evaluation and Integration for AI Models

Updated 8 September 2026

Harbor Adapters are integration layers designed to standardize the evaluation of agents by translating various benchmark representations, execution environments, agent interfaces, and verification procedures into a universal format.
These adapters reduce the engineering overhead from $\mathcal{O}(mn)$ to $\mathcal{O}(m+n)$ by creating common adapatations.

Harbor Adapters are integration layers that translate heterogeneous benchmark representations, execution environments, agent interfaces, and verification procedures into a common infrastructure for large-scale evaluation of language-model agents. In the formal sense introduced by “Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation,” an adapter converts a benchmark into a Harbor task consisting of an instruction, an environment, tests or a verifier, and an optional solution, allowing the resulting task to run with any Harbor-compatible agent (Shi et al., 3 Sep 2026). The term also has broader architectural analogues in maritime networking, logistics, robotics, MLOps, and parameter-efficient machine learning, where adapters mediate between domain-specific assets or models and shared computational infrastructure.

1. Terminology and architectural scope

In agent evaluation, Harbor Adapters address the fragmentation created when benchmarks and agents are integrated independently. With $m$ benchmarks and $n$ agents, direct pairwise integration has an engineering burden of $\mathcal{O}(mn)$ . Harbor introduces a common integration layer intended to reduce this to $\mathcal{O}(m+n)$ : each benchmark is adapted once to Harbor, and each agent is integrated once with Harbor (Shi et al., 3 Sep 2026).

The central abstraction is a Harbor task with four principal elements:

Instruction: the task specification presented to the agent.
Environment: the sandbox, files, services, tools, datasets, and resource limits in which the agent operates.
Tests or verifier: the mechanism that evaluates the agent’s result.
Solution: an oracle or reference implementation demonstrating task feasibility.

An adapter translates a benchmark’s native representation into this schema while attempting to preserve semantic fidelity, agent neutrality, environment reproducibility, comparable scoring, auditability, and quality control. The adapter is therefore not itself an agent, model, benchmark, or verifier; it is a boundary component connecting a benchmark to a standardized execution and evaluation framework.

The term has several related uses in the supplied research:

Maritime and harbor networking: an adapter is a hardware or software component translating between harbor assets, wireless technologies, IP networks, and cloud services. In HarborNet, the NetRider onboard unit is the principal asset/network adapter, supported by roadside units, routing, connection management, delay-tolerant transfer, and cloud-control layers (Ameixieira et al., 2013).
Maritime information integration: RFID, GPS, wireless sensor networks, gateways, service centers, databases, GIS, and web services act as field-level or middleware adapters connecting containers, vehicles, vessels, customs, terminal systems, and logistics platforms (Cimino et al., 2015).
Maritime software architecture: Ports and Adapters, or Hexagonal Architecture, separates domain logic from external systems such as sensors, databases, message brokers, model runtimes, and APIs (Ferreira et al., 9 Dec 2025).
Parameter-efficient machine learning: adapters are small trainable modules attached to a frozen pretrained model, allowing task-, domain-, language-, or retrieval-specific adaptation (Beck et al., 2021).
Robotic and dynamical systems: an adapter can denote a physical interface or configurable model layer, such as an adaptive latching mechanism for robotic boats (Mateos, 2020) or an automatically identified vessel maneuvering model (Miyauchi et al., 2023).

These senses share a general principle: a stable core is separated from variable external interfaces, task-specific behavior, or deployment conditions.

2. Harbor’s unified evaluation architecture

A Harbor evaluation can be represented as:

$\text{dataset} \rightarrow \text{adapter} \rightarrow \text{Harbor task} \rightarrow \text{sandbox} \rightarrow \text{agent harness} \rightarrow \text{model API} \rightarrow \text{trajectory} \rightarrow \text{verifier} \rightarrow \text{reward and metrics}.$

The benchmark adapter reads upstream data and constructs Harbor task records. Each task supplies its instruction, files, environment image or build specification, tests, and, where available, a reference solution. Harbor launches an isolated sandbox, usually Docker-compatible, and an agent harness interacts with that environment through terminal commands, files, tools, APIs, browsing, or task-specific services.

The harness communicates with a model interface, usually a first-party model API or OpenRouter. The infrastructure records the resulting trajectory, including tool calls, token counts, timing, exceptions, and configuration. The verifier runs in a controlled environment and returns a scalar reward together with benchmark-specific metrics where applicable. The system assumes that heterogeneous evaluation outcomes can ultimately be represented by a scalar reward, while retaining continuous metrics such as F1, $R^2$ , speedup, NMSE, NMAE, and benchmark-specific scores.

Environment isolation

For CPU-only tasks, Daytona is the default backend, with reported limits of approximately 4 CPUs, 8 GB of memory, and 10 GB of storage. Local Docker is used when tasks exceed those limits or are incompatible with Daytona; GPU-dependent tasks use Modal. Prebuilt images may be published to a registry such as GHCR to reduce repeated image pulls and registry-rate-limit failures (Shi et al., 3 Sep 2026).

Harbor-Index explicitly isolates the verifier from the agent environment where possible. This is intended to reduce reward hacking by preventing agents from:

reading gold answers or hidden tests;
modifying verifier dependencies;
tampering with the evaluator;
exploiting shared volumes;
accessing public answer files.

The execution record includes setup, execution, and verifier timestamps, task checksums, exceptions, configurations, agent metadata, rewards, and model token counts.

Dataset and trial registry

The experiment database contains eight principal tables:

dataset;
task;
dataset_task;
job;
trial;
agent;
model;
trial_model.

A job stores the Harbor version or commit, configuration, trial count, timestamps, aggregate statistics, and verification flags. A trial stores its UUID, task checksum, agent and model identity, reward, exceptions, timing, and configuration. The trial_model table records input, output, and cached token counts.

The large-scale rollout release contains a manifest with 178,647 rows and a trajectories configuration containing 793,698 trials, approximately 340 GB, organized in benchmark-specific shards. The published archive can contain up to five recent trials per cell, whereas primary aggregates generally use three trials (Shi et al., 3 Sep 2026).

3. Adapter construction and benchmark integration

Harbor Adapters generally perform four operations:

parse upstream benchmark records;
construct a Harbor task directory;
reconstruct the environment and verifier;
expose the resulting tasks through Harbor’s dataset-generation and execution APIs.

A conceptual task structure contains:

$n$ 9

Adapters may additionally copy attachments into the workspace, install upstream evaluation code, wrap native evaluators in shell scripts or pytest, expose task-specific services through HTTP or MCP, translate function-call, browser, FHIR, SQL, or simulator protocols, normalize output formats, and specify resources and timeouts.

The paper distinguishes three benchmark interaction modes:

Agentic: originally designed for iterative agent interaction with environments, tools, or multi-step execution, as in SWE-bench, GAIA, CyberGym, and FinanceAgent.
Non-agentic: originally designed for direct, single-pass model evaluation, as in GPQA, MMLU/MMMLU, AIME, and StrongReject.
Modified-agentic: originally non-agentic but materially changed when agents receive terminals, files, execution, or iterative testing, as in AlgoTune, BigCodeBench, DS-1000, and related coding benchmarks.

Three parity-integration scenarios are used:

Harbor-supported agent: an existing Harbor-compatible agent is aligned with the original benchmark’s versions, tools, prompts, arguments, and environment.
Vanilla LLM: a direct-prompt benchmark is wrapped with a Harbor-compatible CLI agent, with the original and Harbor protocols treated symmetrically.
Custom agent: a benchmark-specific ReAct, research, multi-agent, or other scaffold is ported when feasible; otherwise, both a custom-agent version and a generic CLI-agent version may be released, with divergence documented.

The adapter catalog contains 93 integrations spanning software engineering, mathematics and reasoning, knowledge and long-context evaluation, scientific research, agents and tools, data and analytics, professional domains, safety and security, and multimodal tasks. Terminal-Bench, CompileBench, and SkillsBench already use Harbor’s native task format and therefore require no adapter. The main large-scale study evaluates 54 benchmarks and 6,627 selected tasks rather than the complete catalog (Shi et al., 3 Sep 2026).

4. Validation, parity, and quality control

Adapter validation is intended to establish that a port preserves the original benchmark’s semantics and execution behavior. Each adapter passes through automated bot review, review by one to three trained human reviewers, and final approval by a team lead. The review team consisted of five trained reviewers, and more than 10,000 GitHub review comments were reported across the adapters.

Review checks include:

schema conformance;
documentation completeness;
code style;
oracle execution;
environment reproducibility;
parity configuration and results;
custom-agent ports;
parity-set selection.

Oracle validation

The desired standard is a 100% pass rate for a provided oracle solution. Deviations must be explained per task rather than silently removed. Reported exceptions include:

SWE-bench Verified: four infrastructure or data failures, approximately 99.2% oracle pass;
SWE-Bench Pro: 15 invalid gold patches and nine timeout cases, approximately 97%;
SWT-Bench: nine parsing failures, approximately 97.7%;
GSO: 89 of 102 oracle passes, or 87.3%, because of timing variance;
ScienceAgentBench: 93.1% CPU-only oracle pass rate, with remaining tasks requiring GPUs;
Omni-Math: at most 99.5%, because 18 tasks lack benchmark-provided answers.

Parity experiments

Parity compares the original and Harbor versions under matched agent, model, prompt, tool, environment, and execution settings. Each side is normally run three times. The reported statistics use the sample mean and standard error:

$\bar{x} \pm s_{\bar{x}}, \qquad s_{\bar{x}}=\frac{s}{\sqrt{k}},$

where $s$ is the sample standard deviation and $k$ is the number of trials.

An adapter passes parity when original and Harbor means agree within a margin consistent with their standard errors. When full evaluation is too expensive, parity uses a documented representative subset selected through random stratified sampling, official subsets such as Lite or Verified splits, or curated subsets for expensive or unstable tasks.

Examples include parity comparisons for SWE-Bench Multilingual, SWE-Bench Pro, SWE-bench Verified, FeatureBench, BigCodeBench, LiveCodeBench, HumanEvalFix, CRUST-Bench, GPQA Diamond, SimpleQA, ScienceAgentBench, GAIA, BFCL, FinanceAgent, and StrongReject (Shi et al., 3 Sep 2026).

Parity can be affected by external-service nondeterminism, GPU availability, timing-sensitive metrics, missing upstream inference scripts, broken oracle solutions, and custom-agent differences. It therefore documents equivalence and divergence rather than guaranteeing exact identity between benchmark implementations.

5. Agents, harnesses, and evaluation effects

The large-scale study uses four harness implementations:

Every model runs with Terminus-2. GPT models additionally run with Codex, Claude models with Claude Code, and Gemini models with Gemini CLI. The eight models in the primary study are Gemini 3.1 Pro Preview, Gemini 3 Flash Preview, Claude Opus 4.6, Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4, GPT-5-mini, and GPT-5-nano.

Terminus-2 uses a constrained, linear Plan–Execute–Complete interaction pattern and is primarily bash- and terminal-oriented. Native harnesses generally permit more open-ended iteration, self-correction, and model-specific tool use. Terminus-2 does not natively provide the same image or web tools as some native harnesses.

The study reports that native harnesses allow more iterative refinement, whereas Terminus-2 is more vulnerable on tasks requiring repeated verification and correction. GPT-5.4 is often concise and either succeeds quickly or gives up under Terminus-2; Claude models exhibit relatively stable turn counts and execution patterns; Gemini models rely heavily on web search, sometimes producing “information drift.”

A linear mixed model is used to separate model and harness effects:

$s_{ij} = \beta_0 + \boldsymbol{\beta}_{\mathrm{model}}x_{\mathrm{model},i} + \boldsymbol{\beta}_{\mathrm{harness}}x_{\mathrm{harness},i} + u_j+\varepsilon_{ij},$

with

$n$ 0

The intraclass correlation is 0.75, with estimated variance components $n$ 1 and $n$ 2. The model-effect range is 0.451, compared with a harness-effect range of 0.087, making the model range approximately 5.2 times larger. The authors conclude that harness design matters, but base-model capability is the dominant determinant of performance (Shi et al., 3 Sep 2026).

Relative to Terminus-2, Claude Code produces a mean difference of $n$ 3 and wins on 65% of benchmarks; Codex produces $n$ 4 and wins on 70%; Gemini CLI produces $n$ 5 and wins on 53%. These results show that a Harbor Adapter standardizes execution, but does not make agent–harness interactions interchangeable.

6. Harbor-Index and the role of curated evaluation

Harbor-Index is a curated subset of the adapted suite containing 82 difficult, diverse, and quality-controlled tasks spanning 29 benchmarks. It is distinct from both the 93-integration catalog and the 54-benchmark large-scale study.

Construction proceeds through four stages.

Difficulty filtering

Each candidate task is run with Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro using native and Terminus-2 harnesses. With three trials per model–harness pair, each task receives 18 trials. A task is retained when it has no more than six successes:

$n$ 6

This reduces the initial pool to 1,311 tasks.

AI audit

A Gemini 3 Flash auditor examines task files, environments, tests, reference solutions, verifier output, trial summaries, and selected trajectories. The audit requires test–instruction alignment and essential difficulty. It reduces the candidate pool from 1,311 to 307 tasks.

An inter-judge check on 74 tasks reports three-way agreement of 89.2%, Cohen’s $n$ 7, and binary accept/non-accept agreement of 93.2%.

Human review

Fourteen domain-experienced reviewers re-audit the 307 survivors. Each candidate receives at least one senior review or two junior reviews, leaving more than 110 tasks. A three-member senior panel selects 100 based on difficulty, diversity, task quality, insight into agent behavior, and coverage of capability axes and domains.

Audit-and-fix loop

Each of the 100 tasks receives at least two senior reviews. Reviewers inspect false positives, false negatives, instruction–verifier mismatches, broken environments, exploitable graders, and tasks that become easy after repair. Fixable tasks are repaired, filtering models are rerun, and the audit is repeated. The final result is Harbor-Index 1.0 with 82 tasks.

Approximately one third of the hardest candidates across more than 30 benchmarks were found to be broken rather than genuinely difficult. Reported defects include missing simulation events in GAIA2, undisclosed verifier requirements in SWE-Bench Pro, fixed-interface assumptions in CRUST-Bench, incorrect judge answers in some Humanity’s Last Exam tasks, unit inconsistencies in AA-LCR, discarded raw audio in MMAU, and undisclosed import paths or symbols in FeatureBench (Shi et al., 3 Sep 2026).

Harbor-Index evaluations show that no model–harness configuration exceeds a 30% pass rate. GPT-5.5 with Codex is strongest at 28.0%, followed by Claude Opus 4.8 with Claude Code at 20.7% and Gemini 3.1 Pro with Gemini CLI at 13.4%. Several open-weight models occupy cost–performance Pareto positions under Terminus-2 because of substantially lower cost.

The principal failure categories are timeouts with no answer, near-miss solutions, and fundamentally wrong answers. Lower-scoring models time out on approximately 36.7% of rollouts, compared with 18.7% for the three strongest models. Matched task-by-task, native harnesses reduce timeout rates from 42% to 26%.

7. Broader architectural relationships and limitations

Harbor Adapters belong to a larger family of adapter architectures in which heterogeneous components are translated into stable interfaces.

In maritime networking, the NetRider onboard unit combines an SBC, IEEE 802.11p/WAVE, conventional Wi-Fi, 3G, GPS, and antennas, while the connection manager hides interface changes from applications (Ameixieira et al., 2013). In maritime logistics, RFID, GPS, WSN, ZigBee, Wi-Fi, cellular, TETRA, satellite, gateways, databases, GIS, and web services connect physical logistics events to customs and terminal systems (Cimino et al., 2015). In MLOps, Ports and Adapters separate domain logic from sensors, databases, brokers, model runtimes, and APIs, allowing multiple microservices to reuse common integrations (Ferreira et al., 9 Dec 2025).

In machine learning, adapters typically preserve a frozen base model while attaching small task-specific modules. AdapterHub Playground operationalizes this through reusable adapter artifacts, continual adapter training, metadata-driven selection, and no-code execution (Beck et al., 2021). Cross-language retrieval studies show that adapter modularity does not automatically ensure representation compatibility: naïvely replacing an English language adapter with a target-language adapter can degrade dense retrieval, particularly for DPR (Yang et al., 2022). Sparse retrieval experiments show that adapters can update approximately 2% of parameters while matching or exceeding full fine-tuning for SPLADE, although transfer between retrieval stages is not automatic (Pal et al., 2023).

Recent work extends adapter infrastructure through post-hoc extraction and compression. PHLoRA extracts LoRA-compatible adapters from the difference between a base checkpoint and a full-rank fine-tuned checkpoint using truncated SVD, without training data or gradients (Vasani et al., 13 Sep 2025). HydraOpt compresses collections of low-rank adapters by sharing components while retaining task-specific factors, reporting approximately 48% storage reduction with a 0.2–1.8% performance drop in its evaluated settings (Ceritli et al., 23 Jul 2025). These methods are relevant to Harbor-style registries but do not themselves define Harbor’s benchmark-adapter protocol.

The formal Harbor Adapter system nevertheless has distinct limitations:

Coverage: the main study evaluates only 54 of the available integrations.
Cost: the large-scale evaluation consumed approximately 226 billion tokens and more than $300,000 in compute.
Model drift: preview endpoints, aliases, APIs, and external services may change.
Harness asymmetry: model families are paired with different native harnesses.
Judge uncertainty: LLM-as-a-judge evaluations introduce bias and variance; reported failure-mode agreement against human gold is $n$ 8.
Parity limitations: external nondeterminism, GPU constraints, timing variance, upstream defects, and custom-agent differences can prevent exact equivalence.
Verifier validity: a failed task may be broken, underspecified, incorrectly graded, or impossible rather than genuinely difficult.
Benchmark saturation: among 54 evaluated benchmarks, 13 are largely saturated and seven are classified as hard.

Harbor-Index therefore treats benchmark evaluation as an auditable engineering process rather than a simple aggregation of model scores. The adapter is responsible not only for translating task formats, but also for preserving execution semantics, documenting deviations, validating oracle behavior, isolating the verifier, recording provenance, and exposing the resulting evaluation to systematic analysis.