Agentified Agent Assessment (AAA) Framework

Updated 4 July 2026

Agentified Agent Assessment (AAA) is a standardized evaluation paradigm that unifies agent benchmarking by using judge agents to encapsulate benchmark logic and subject agents to expose a common interface.
It reduces integration complexity from O(N×M) to O(N+M) by separating benchmark-specific code from agent-specific code, ensuring modular and fair evaluations.
Multiple operational modes (Local, Remote, Hosted, Proxy, CI) enable flexible, secure, and reproducible assessments across diverse evaluation scenarios and agent designs.

Searching arXiv for papers on Agentified Agent Assessment and related agent-as-a-judge evaluation frameworks. Agentified Agent Assessment (AAA) is an evaluation paradigm in which assessment is carried out by judge agents and evaluated systems appear as subject agents, with both sides interacting through standardized protocols rather than bespoke benchmark-specific harnesses. In the formulation introduced with AgentBeats, benchmarks are realized as judge agents, subjects are realized as subject agents, and the shared interface is $I_{\mathrm{AAA}}=\{\mathrm{A2A},\mathrm{MCP}\}$ , where A2A handles task management and high-level messaging and MCP handles tool and environment access (Liu et al., 11 Jun 2026). Across adjacent literature, the same general shift is also described as Agent-as-a-Judge: evaluation moves from single-pass, purely parametric scoring toward planning, tool-augmented verification, multi-agent collaboration, persistent memory, and stateful interaction with executable environments (You et al., 8 Jan 2026).

1. Formal model and interface unification

Traditional agent benchmarking requires a separate integration for each pair of benchmark $B_i$ and agent $A_j$ , yielding $O(N \times M)$ bespoke integrations for $N$ benchmarks and $M$ agents. AAA replaces this with a single unified interface shared by all participants. In the AgentBeats formalization, benchmarks are $J=\{J_1,\ldots,J_N\}$ , subjects are $S=\{S_1,\ldots,S_M\}$ , and the AAA interface is $I_{\mathrm{AAA}}=\{\mathrm{A2A},\mathrm{MCP}\}$ . The integration cost is then written as

$C_{\mathrm{traditional}} = N \cdot M,\qquad C_{\mathrm{AAA}} = N + M.$

Evaluation is modeled as

$B_i$ 0

where $B_i$ 1 is the space of reported metrics such as accuracy, solve-rate, and cost (Liu et al., 11 Jun 2026).

This formulation makes a specific architectural claim: benchmark logic resides inside judge agents, while subject agents only need to expose the standardized interface. The agent side contains no benchmark-specific code, and the benchmark side contains no agent-specific code. In the broader Agent-as-a-Judge literature, this separation corresponds to the replacement of single-pass scoring $B_i$ 2 by an agentic judge represented as a tuple $B_i$ 3, where the evaluator has a state space, an action space, a transition model, a reward or cost function, and a persistent memory store (You et al., 8 Jan 2026).

A central consequence is that AAA defines a common assessment substrate for heterogeneous agents. In AgentBeats, this is presented as the root remedy for fragmented evaluation, test-production mismatch, and unfair comparison across diverse agent designs (Liu et al., 11 Jun 2026). A plausible implication is that the main unit of interoperability becomes the protocol boundary rather than the benchmark harness.

2. Protocol stack and execution semantics

AAA builds on two protocols with distinct roles. A2A provides task management and orchestration; MCP provides tool and environment access. Under the A2A layer, a delegator sends a KickOffRequest to a judge agent, the judge sends TaskInstruction(task_id, payload) to a subject, the subject returns TaskResult(task_id, outputs), and the judge returns EvaluationResult({metrics_j}) to the delegator (Liu et al., 11 Jun 2026).

The single-agent session described for AgentBeats is:

Delegator $B_i$ 4 Judge: KickOffRequest(subject=S, config=C)
Judge sets up MCP server, data, and related resources
Judge $B_i$ 5 Subject: TaskInstruction(task_id, env_info)
Subject invokes MCP tools and returns TaskResult(task_id, output)
Judge scores output into metrics
Judge $B_i$ 6 Delegator: EvaluationResult(metrics) (Liu et al., 11 Jun 2026)

At the MCP layer, the subject agent receives a URL to an MCP endpoint and a tool manifest. Each tool $B_i$ 7 has a signature

$B_i$ 8

and MCP messages are self-described JSON-RPC calls of the form $N$ 2 The judge can dynamically register or revoke tools through an MCP gateway when openness or private-data constraints require it (Liu et al., 11 Jun 2026).

In related work, the same protocolized structure appears with domain-specific emphases. In the logical reasoning assessment framework, the evaluator is an assessor agent that issues tasks, enforces execution budgets, parses outputs into labels or structured failures, and emits aggregate metrics while the agent under test exposes a standardized A2A interface (Ni et al., 3 Mar 2026). In AgentCanary, the equivalent emphasis is trajectory-grounded scoring over real executable interaction traces $B_i$ 9, where events include user queries, tool calls, tool results, and agent replies (Li et al., 9 Jun 2026). These formulations share the same core move: assessment is not an external wrapper around an LLM response but an agent-mediated process over explicit trajectories.

3. Operational realization in AgentBeats

AgentBeats instantiates AAA through five operation modes designed to trade off openness, privacy, and reproducibility. All five modes share the same three-stage lifecycle—Agent Construction, Registration, and Assessment Execution—but differ in who provides agent binaries or live endpoints and in where orchestration occurs (Liu et al., 11 Jun 2026).

Mode	Mechanics	Use case
Local	Judge and subject agents run as local A2A services; a script sends `KickOffRequest` and prints results	On-premise development or private evaluation
Remote	Developers deploy private live A2A services; a central platform orchestrates and displays results	Publicly orchestrated evaluation of closed-source agents
Hosted	Developers upload blueprints such as a Git repo or Docker image; the platform instantiates containers and health-checks A2A endpoints	Open-source agents evaluated at scale with minimal deployment effort
Proxy	A local A2A service is exposed through a reverse-proxy tunnel so remote judge agents can interact	Rapid iteration without full deployment
CI	Agents and an evaluation script are checked into a repo; each push triggers evaluation in public CI	Auditable, reproducible assessments integrated into development workflows

Formally, AgentBeats distinguishes between a blueprint = {GitURL, DockerImage} and an instance = {A2A URL, credentials}, and each mode imposes a mapping from mode to {blueprint, instance, delegator location, result display} (Liu et al., 11 Jun 2026).

This five-mode design is significant because the paper does not treat openness as the only acceptable operating condition. Local Mode supports private on-premise evaluation; Remote Mode keeps agent code private while centralizing orchestration; Hosted Mode standardizes scalable public execution; Proxy Mode supports real-time local debugging; and CI Mode prioritizes auditable reproducibility (Liu et al., 11 Jun 2026). This directly addresses a common assumption that standardized evaluation necessarily requires full code disclosure.

4. Empirical evidence from community-scale and controlled studies

AgentBeats validates AAA through two studies: a five-month open competition and a controlled coding-agent case study. The open competition, “AgentX-AgentBeats,” drew 298 judge agents across 12 categories and 467 subject agents from independent developers. The categories included coding, web, healthcare, multi-agent games, finance, and related areas. The collected metrics included number of submissions per category, lines of code and prompt sizes in judge versus subject repositories, and the submission timeline over five months (Liu et al., 11 Jun 2026).

The reported quantitative findings were specific. Judge agents averaged approximately $A_j$ 0 KLOC in Python, with 78% containing natural-language prompts averaging 241 LOC. Subject agents averaged approximately $A_j$ 1 KLOC, with 87% containing prompts averaging 184 LOC. The categories with highest uptake were coding, finance, and multi-agent games. The first three months focused on judge-agent creation, and the next two months focused on subject-agent registrations (Liu et al., 11 Jun 2026). The associated interpretation in the paper is that dozens of existing benchmarks, including Tau $A_j$ 2, MedAgentBench, and OSWorld, could be agentified with manageable engineering effort, and that adoption in Python, TypeScript, and Rust confirmed practicality. The paper further states that semantic internalization—benchmarks specified via prompts—accounted for approximately 5% of code size in judge agents (Liu et al., 11 Jun 2026).

The controlled coding case study evaluated four model-harness pairings: Claude Opus 4.7 $A_j$ 3 Claude Code, GPT-5.4 $A_j$ 4 Codex CLI, Gemini 3.1 Pro $A_j$ 5 OpenCode, and Qwen 3.5 $A_j$ 6 mini-SWE-Agent. These were tested on three agentified benchmarks: DevEval with 1,222 function-level code-generation instances, SWE-Bench Pro with 731 long-horizon repository tasks, and Terminal-Bench 2.0 with 89 CLI-mediated workflow instances (Liu et al., 11 Jun 2026).

The solve-rate and average cost per instance were reported as follows:

DevEval: Opus 4.7 achieved 91.9% at $A_j$ 7; Gemini 3.1 Pro achieved 88.8% at $A_j$ 8; Qwen 3.5 achieved 71.4%.
SWE-Bench Pro: Opus 4.7 achieved 69.1% at $A_j$ 9; Gemini 3.1 Pro achieved 54.4% at $O(N \times M)$ 0; Qwen 3.5 achieved 45.8%.
Terminal-Bench 2.0: Opus 4.7 achieved 68.5% at $O(N \times M)$ 1; Gemini 3.1 Pro achieved 65.2% at $O(N \times M)$ 2; Qwen 3.5 achieved 36.0% (Liu et al., 11 Jun 2026).

The study reported three main findings. First, no single system dominated all three tasks. Second, GPT-5.4 led DevEval, while Opus 4.7 led SWE-Bench Pro and Terminal-Bench 2.0. Third, per-instance cost varied by at most $O(N \times M)$ 3 across all systems, which the paper describes as well within an order of magnitude (Liu et al., 11 Jun 2026). The comparison with public records is methodologically important: under consistent evaluation, the paper concludes that much of the reported vendor advantage stems from proprietary harness co-adaptation rather than the underlying model. The harness-swap experiment quantified a 5.3 percentage-point native-harness advantage on average, and in one Terminal-Bench case the non-native pairing solved more quickly under a fixed time budget, suggesting a speed-versus-robustness trade-off (Liu et al., 11 Jun 2026).

These results underpin the three empirical claims emphasized by AgentBeats: coverage, practicality, and fidelity. Coverage refers to the agentification of 12 benchmark categories; practicality refers to average judge-agent code size, modest prompt share, and adoption by 298 independent teams plus 467 subject agents; fidelity refers to outcome alignment with public records when comparable and to the recovery of previously missing head-to-head comparisons under identical conditions (Liu et al., 11 Jun 2026).

5. Relationship to adjacent evaluation frameworks

AAA sits within a broader family of agentic evaluation frameworks, but the neighboring proposals emphasize different objects of assessment. The Agent-as-a-Judge survey organizes the field along four orthogonal axes—planning, tool-augmented verification, multi-agent collaboration, and persistent memory—and places existing systems into three developmental stages: Procedural, Reactive, and Self-Evolving (You et al., 8 Jan 2026). In that literature, AAA names the general transition from monolithic LLM judging toward agentic evaluation processes.

HAAF extends the idea from interoperability toward representativeness. It defines a scenario manifold

$O(N \times M)$ 4

where the axes denote task type, tool interface, interaction depth, environmental constraints, social context, and risk severity. An agent’s deployment-level trustworthiness is represented as

$O(N \times M)$ 5

and the evaluation problem becomes weighted sampling over a representative scenario set rather than optimization on disconnected benchmark instances (Qi et al., 16 Mar 2026). HAAF decomposes assessment into static cognitive and policy analysis, interactive sandbox simulation, social-ethical alignment assessment, and a distribution-aware representative sampling engine, all embedded within an iterative red-team/blue-team Trustworthy Optimization Factory (Qi et al., 16 Mar 2026).

AgentCanary specializes AAA to agent security. It models risk as an orthogonal Entry $O(N \times M)$ 6 Impact taxonomy,

$O(N \times M)$ 7

with five entries—Direct Prompt Injection, Indirect Prompt Injection, Skill/Tool Poisoning, Memory Contamination, and Intrinsic Failures—and seven impacts ranging from Local Environment Damage to Unauthorized Financial Transactions (Li et al., 9 Jun 2026). Its evaluation consumes full trajectories rather than reply text or a single tool call and scores them along Outcome Safety Score, Security Awareness Score, and Task Utility Score (Li et al., 9 Jun 2026).

The logical reasoning framework shows a narrower but highly auditable form of AAA. There, the assessor agent enforces budgets, deterministic parsing, retry logic, and a structured failure taxonomy containing TIMEOUT, RUNTIMEERROR, and PARSEERROR; on the cleaned FOLIO validation set, the auto-formalization agent achieved 86.70% accuracy versus 73.89% for a chain-of-thought baseline (Ni et al., 3 Mar 2026). The international joint testing exercise supplies a further methodological layer by emphasizing multilingual and cross-cultural validity, translated task suites, synthetic tool implementations for safe testing, and composite metrics covering performance, leakage risk, fraud vulnerability, cybersecurity risk, linguistic fidelity, comprehensibility, hallucination absence, and logical consistency (Seah et al., 22 Jan 2026).

Taken together, these frameworks indicate that AAA is not a single benchmark format. It is a family of assessment architectures in which evaluation is agentified, protocolized, and increasingly trajectory-based. AgentBeats foregrounds interoperability and reproducibility; HAAF foregrounds representative trustworthiness; AgentCanary foregrounds executable security; the logical reasoning framework foregrounds auditable failure handling; and the international exercise foregrounds methodological harmonization across domains and languages (Liu et al., 11 Jun 2026, Qi et al., 16 Mar 2026, Li et al., 9 Jun 2026, Ni et al., 3 Mar 2026, Seah et al., 22 Jan 2026).

6. Methodological implications, limitations, and open questions

The strongest explicit theoretical claim made for AAA is the reduction of interface cost from $O(N \times M)$ 8 to $O(N \times M)$ 9. AgentBeats does not claim a broader formal guarantee beyond this integration-cost reduction; instead, it relies on empirical evidence from a community-scale field study and a controlled case study to argue for coverage, practicality, and fidelity (Liu et al., 11 Jun 2026). This boundary matters because it distinguishes protocol standardization from guarantees about representativeness, robustness, or safety.

A central methodological question is whether broad benchmark coverage is sufficient. HAAF argues that the deeper limitation of current evaluation practice is not merely insufficient coverage of dimensions but the absence of a principled notion of representativeness over a socio-technical scenario distribution (Qi et al., 16 Mar 2026). This suggests that an AAA system can be interoperable and reproducible while still undersampling rare but high-consequence tail risks.

Several other limitations recur across the literature. The Agent-as-a-Judge survey identifies computational cost and latency, safety and privacy, verifiability of self-modification, and residual bias propagation in multi-agent setups as open challenges (You et al., 8 Jan 2026). The international joint testing exercise reports methodological pitfalls including small sample size, dataset heterogeneity, subjectivity and annotation bias, tool-fidelity gaps, translation errors, and Judge-LLM drift, with discrepancy rates up to 40% with humans on nuanced failure modes (Seah et al., 22 Jan 2026). AgentCanary shows that measured security depends critically on runtime: cross-framework variation reached 15 points on the same model, agents often “silently obey” indirect prompt injection with high OSS but low SAS, and adaptive attacks such as Dynamic Attack Evolution or script-camouflaged skill poisoning can substantially worsen outcomes (Li et al., 9 Jun 2026). These findings indicate that agentified assessment does not eliminate evaluator dependence; it relocates it into protocols, runtimes, scenario design, and judging logic.

An additional misconception is that AAA necessarily collapses evaluation into a single scalar. In practice, the frameworks surveyed here are markedly multi-dimensional. AgentBeats reports solve-rate and cost under fixed harness conditions (Liu et al., 11 Jun 2026). AgentCanary decomposes outcomes into safety, awareness, and utility (Li et al., 9 Jun 2026). HAAF defines a trustworthiness vector $N$ 0 and an estimated deployment profile $N$ 1 rather than a univariate score (Qi et al., 16 Mar 2026). The international exercise likewise distinguishes pass rate, leakage risk, fraud risk, cybersecurity risk, and cultural-linguistic metrics (Seah et al., 22 Jan 2026).

In sum, AAA denotes a shift from ad hoc benchmark-agent integration toward a standardized, agent-mediated assessment interface. In its AgentBeats realization, this shift is operationalized through A2A and MCP, five deployment modes, and large-scale empirical validation across heterogeneous benchmark categories (Liu et al., 11 Jun 2026). In the surrounding literature, the same paradigm expands toward trustworthiness assessment over scenario manifolds, executable security testing, structured failure auditing, and multilingual safety methodology (Qi et al., 16 Mar 2026, Li et al., 9 Jun 2026, Ni et al., 3 Mar 2026, Seah et al., 22 Jan 2026). The shared premise is that as agents become tool-using, stateful, and socially embedded, evaluation must itself become tool-using, stateful, and protocolized.