The Evaluation Context Protocol (ECP): A Portable Contract for AI Agent Evaluation

Abstract: The evolution of artificial intelligence has necessitated a fundamental shift from evaluating isolated LLMs to assessing autonomous agentic architectures. This paper explores the critical methodologies for evaluating AI agents and the essential role of advanced observability infrastructure. We analyze the architectural components of agents and identify the severe limitations of current evaluation paradigms, including benchmark exploitation, the "confidently wrong" phenomenon, and the discrepancy between theoretical capability and operational reliability. To begin addressing the fragmentation in current evaluation infrastructure, this paper proposes the Evaluation Context Protocol (ECP), an early-stage, vendor-neutral framework intended to act as a portable evaluation contract layer for agentic systems. In its current form ECP defines a small JSON-RPC interface over which an agent exposes its user-visible output, the tool calls it made, and evaluator-safe audit context, and against which programmatic checks can be run uniformly across frameworks and continuous integration systems. We describe an open-source reference implementation that includes adapters for LangChain, LlamaIndex, CrewAI, and PydanticAI, and we situate the design against failure modes documented in the recent literature. ECP is presented as work in progress rather than a finished standard: the evaluation surface, method set, and grader families are all expected to change as the protocol is exercised against more systems, and the empirical validation required to justify adoption is outlined as future work.

• The paper proposes ECP, a vendor-neutral JSON-RPC contract that exposes agent outputs, tool calls, and evaluator-safe audit evidence for trajectory-based grading.
• ECP addresses outcome-only evaluation failures, including the pass@3 versus pass^3 reliability gap, benchmark shortcuts, tool misuse, and biased LLM judges through declared checks and conformance tooling.
• The paper presents a runnable reference implementation and framework adapters, but reports no controlled validation; planned studies will test detection rates, portability, and evaluation overhead.

The paper proposes the Evaluation Context Protocol (ECP), an early-stage, vendor-neutral contract layer intended to standardize how autonomous AI agents expose their outputs, tool invocations, and audit evidence to evaluators. The contribution is a design and a runnable reference implementation rather than a validated standard: the authors state plainly that the protocol surface, method set, and grader families are provisional, and that no controlled empirical validation has yet been performed. This essay summarizes the motivation, the failure modes in current evaluation practice that motivate the design, the architecture of ECP itself, the empirical findings the paper surveys, and the limitations and roadmap the authors articulate.

From model scoring to trajectory-based agent evaluation

The paper begins by distinguishing agent evaluation from classical LLM evaluation. A chatbot's hallucination is confined to text; an agent's hallucinated tool invocation can corrupt databases, trigger unauthorized transactions, or compromise security. Because agents integrate an LLM policy module with planners, tool routers, memory subsystems (episodic, semantic, procedural), and critic/verifier modules (2608.19263), meaningful evaluation must capture the full trajectory: plans, every tool invocation with parameters and responses, intermediate reasoning, and environmental side effects. Without this observability, an evaluator cannot distinguish genuine capability from lucky or illegitimate paths — e.g., an agent that reaches a correct answer via hallucinated intermediate data or unauthorized tool access.

Failure modes in current evaluation paradigms

The paper catalogues several structural weaknesses in existing methods:

• Static benchmarks (MMLU, HumanEval) test capability under fixed conditions and fail to capture behavioral reliability; interactive benchmarks such as AgentBench, GAIA, SWE-bench, WebArena, and CORE-bench shift toward multi-turn, environment-grounded assessment across dimensions including reasoning quality, tool selection accuracy, conversation quality, and trajectory efficiency.
• Benchmark exploitation and contamination: agents pass tasks by shortcuts such as retrieving the actual patch from public Git history, and iterative tuning against evaluation sets destroys generalization unless developmental and withheld test sets are strictly separated.
• The $pass@k$ vs. $pass^k$ reliability gap: the industry-standard $pass@k$ measures success at least once in $k$ attempts, while production demands $pass^k$ — success on all attempts. The paper highlights an illustrative 63-point gap: an agent with 97% under $pass@3$ may achieve only ~34% under $pass^3$. This is one of the paper's strongest quantitative claims about how outcome-only reporting overestimates operational reliability.
• LLM-as-a-judge biases: position bias (~70% of uncalibrated comparative judgments), verbosity bias (>90% of judgments absent length-penalizing rubrics), and self-preference bias (10–25% score inflation). The proposed governance thresholds — calibration on ≥100 human-labeled examples, Cohen's $\kappa \ge 0.6$ against experts, binary rubric-based grading, ~30-day recalibration cadence — are concrete but sourced from practitioner literature rather than peer-reviewed studies.
• "Confidently wrong" behavior and context rot: as context windows grow through multi-step loops, recall degrades, agents lose track of prior steps, and fabricate confident answers that outcome-only checks miss.

Infrastructure fragmentation as the core problem

Beyond methodology, the paper identifies deep fragmentation: evaluation scripts are coupled to specific orchestrators (LangChain, LlamaIndex, CrewAI, AutoGen) and proprietary observability platforms, degrading reproducibility and creating vendor lock-in. The Model Context Protocol (MCP) solved standardized tool access on the execution plane, but no analogous contract exists for the evaluation plane. The paper's survey of observability platforms (Braintrust, LangSmith, Maxim, Galileo, Langfuse, Arize Phoenix/AX, Patronus AI) documents their strengths and lock-in trade-offs, positioning ECP as complementary infrastructure that feeds these platforms rather than replaces them.

The Evaluation Context Protocol

ECP is specified as a minimal JSON-RPC 2.0 contract with three methods: agent/initialize (capability handshake), agent/step (advance one turn and return the result envelope), and agent/reset (clear transient state). Two transports are defined — stdio (default) and Streamable HTTP. The result envelope exposes three graded fields and one optional field:

1. public_output: the user-visible answer, graded by traditional outcome checks (text_match, calibrated llm_judge).
2. tool_calls: the invocation trace with tool names and arguments, graded by tool_usage (name plus argument subset match), targeting hallucinated schemas, unauthorized access, and shortcut exploitation.
3. evaluation_context: structured, evaluator-safe audit evidence justifying actions. Crucially, ECP does not mandate raw chain-of-thought disclosure; private_thought survives only as a deprecated compatibility alias. This disclosure boundary is what makes closed-model provider participation plausible.
4. logs: optional, currently without a dedicated grader, included to illustrate how the surface grows incrementally.

Scenario verdicts are logical ANDs over all declared checks, so an agent reaching the correct answer through an invalid path records as a failure. Graders are declared in a manifest.yaml validated against published JSON Schemas; the CLI suite (ecp run/init/validate/doctor/conformance) executes identically on developer machines and CI runners, breaking builds on regression. Incremental adoption paths include a pytest plugin fixture and a trend command for cross-run pass-rate regression signals — though the latter aggregates pass rates rather than estimating $pass^k$.

Framework neutrality is demonstrated by thin adapters for LangChain (callback handler), LlamaIndex (workflow bridging), CrewAI (crew kickoff mapping), and PydanticAI (structured message history extraction), plus plain Python and HTTP examples. Two honest concessions emerge here: because frameworks expose reasoning differently, evaluation_context is often a concatenation of captured text rather than structured evidence, and ECP lacks any native delegation representation — multi-agent handoffs are expressed as ordinary tool calls by "an agent that happens to call other agents."

Surveyed empirical evidence

The paper situates its argument against recent findings rather than presenting its own experimental results. MAESTRO-style studies show MAS executions can be structurally stable yet temporally highly variable, with system architecture dominating performance over backbone model choice. CloudOps evaluations using MOYA found high task-completion rates masking security-policy deviations visible only in trajectories. A study of agent-authored pull requests reports that human interventions occur less frequently than for human-authored PRs (52.17% vs. 83.59%) but require substantially higher review effort, with 58.02% of effort spent on guidance-level intervention — evidence that human work is shifting toward supervision and trajectory quality control. A healthcare scoping review (43 relevant studies from 1,070 records) found evaluations overwhelmingly simulated, focused on process metrics, and rarely addressing clinical safety endpoints.

Limitations, open questions, and roadmap

The authors are explicit that the principal limitation is the absence of controlled evaluation of ECP itself. Three planned studies define the empirical agenda: a fault injection study comparing detection rates of full-surface versus outcome-only grading; a portability study measuring verdict agreement across frameworks under identical manifests; and an overhead study quantifying wall-clock and monetary cost. Until these exist, ECP should be read as a proposal with a working artifact, not an empirically validated standard.

Additional open problems acknowledged in the paper include: the evaluation surface is not theoretically derived and may be reshaped (e.g., decomposing the overloaded free-text evaluation_context into typed fields; adding cost, latency, observation results, subagent identity); capability negotiation is informational rather than normative; delegation semantics, formalized human-in-the-loop suspension points, provider-neutral judge adapters, machine-checkable calibration records, signed reports with content-addressed provenance, sealed manifests for contamination control, OTEL semantic-convention mapping, and TypeScript SDK parity all remain unfinished. Structural cost concerns persist — LLM-judge-heavy evaluation consumes 10–15% of typical inference budgets, telemetry storage exhausts entry-tier limits within days, and golden-dataset construction bottlenecks on scarce domain experts. Non-goals are clearly drawn: ECP is not an observability backend, benchmark corpus, orchestration framework, or chain-of-thought disclosure mechanism.

Conclusion

This paper contributes a diagnosis — that agent evaluation is fragmented, outcome-biased, and statistically over-optimistic — and a concrete, runnable response in the form of a small JSON-RPC evaluation contract with framework adapters, conformance tooling, and CI integration. Its most defensible claims are architectural and negative: that trajectory-aware grading closes documented detection gaps, and that a portable contract separating evaluation definition from orchestration platform would improve reproducibility. Whether ECP's particular decomposition of agent behavior is correct, whether independent implementations can interoperate, and whether its overhead is acceptable remain open questions the authors themselves frame as the necessary next steps.