2000 character limit reached
Human-on-the-Bridge: Scalable Evaluation for AI Agents
Published 15 Jun 2026 in cs.MA | (2606.16871v1)
Abstract: AI agents must be evaluated as behavioral systems, not as isolated response generators. They reason across turns, call tools, preserve context, follow policies, and act under uncertainty. Existing methods provide useful but fragmented signals: benchmarks measure fixed capabilities, Human-in-the-Loop review preserves expert judgment but does not scale easily, LLM-as-judge methods depend on evaluator design, red teaming is often episodic, and trace auditing requires explicit evidence rules. This paper introduces Human-on-the-Bridge (HOB), a scalable evaluation paradigm for agentic AI. HOB places human expertise upstream, where experts curate reusable evaluation intelligence before testing begins, including domain context, Red-Team Traps, Juror Personas, scoring guidelines, audit rules, and fallback policies. ProofAgent Harness then executes this curated intelligence repeatedly through multi-turn adversarial evaluations, trace capture, multi-juror scoring, and evidence-linked reporting. We evaluate HOB through symmetric and cost-efficient asymmetric settings across frontier LLM-based agents and Harness LLM tiers. The study covers 23,500 agent turns and produces evidence-linked findings across finance, healthcare, and code generation. The results show that HOB can amplify evaluation quality without requiring equally large evaluator models, allowing smaller Harness LLMs to challenge agents built on frontier LLM backbones. The evaluation surfaces failures often missed by static benchmarks and single-evaluator scoring, including phantom tool-call claims, missing mandatory tool calls, policy drift, manipulation paths, and safe but non-resolving refusals. These findings support HOB as a paradigm for scaling human-curated evaluation intelligence, where expert judgment is encoded upfront and reused across repeated agent evaluations rather than applied manually inside every run.
Summary
- The paper introduces the Human-on-the-Bridge paradigm that shifts human expertise to upstream, reusable evaluation intelligence, enabling systematic detection of agent failures.
- It employs adversarial multi-turn trials integrating Red-Team Traps, juror personas, and audit rules to measure both subjective and objective performance.
- Empirical results reveal robust failure detection, cost-efficient scaling, and traceable evaluations across diverse agent configurations.
Human-on-the-Bridge: Scalable, Evidence-Linked Evaluation for Agentic AI
Motivation and the Limits of Conventional Agent Evaluation
Agentic AI systems diverge fundamentally from static LLM evaluation settings. Instead of isolated single-turn text generation, agents operate as multi-step, memoryful, tool-integrated decision systems. Their operational complexity introduces novel failure modes—such as phantom tool usage, procedural drift, and silent compliance omission—that are invisible to classical benchmark-based or answer-only evaluations. The fragmentation of current evaluation methodologies, including static benchmarks, HITL review, LLM-as-judge schemes, episodic red teaming, and trace-based auditing, underscores an orchestration deficit in scalable, domain-grounded agent evaluation.
Figure 1: Overview of contemporary methodologies in AI agent evaluation; benchmarks and model-centric protocols do not address system-level behavioral, procedural, and adversarial risks.
The Human-on-the-Bridge Paradigm
Human-on-the-Bridge (HOB) reframes the locus of human expertise in evaluation pipelines: expert curation moves upstream to author reusable evaluation intelligence comprising domain context, Red-Team Traps, Juror Personas, precise scoring guidelines, programmable audit rules, and fallback policies. Rather than invoking manual human review in each test cycle, HOB renders human insight as an infrastructural layer, reusable across agents, configurations, domains, and model versions. The ProofAgent Harness system operationalizes this paradigm, executing multi-turn adversarial evaluations, capturing agent trace evidence, applying multi-juror behavioral scoring, and generating evidence-linked, auditable reports.
Figure 2: The HOB architecture—experts curate all evaluation intelligence components upstream; the harness executes adversarial trials, applies juror scoring, and links scores to behavioral evidence.
This model contrasts sharply with Human-in-the-Loop and Human-on-the-Loop traditions by positioning the expert as policy author rather than as continuous supervisor or reviewer. The scalable and reproducible nature of this evaluation intelligence enables agents to be systematically challenged across regulatory, operational, and adversarial axes.
Operationalizing Scalable Evidence-Linked Evaluation
The core elements of HOB are instantiated as follows:
- Red-Team Traps: Expert-devised adversarial scenarios and edge cases designed to probe policy, workflow, and tool-use vulnerabilities, extensible across emerging risks, domains, and compliance requirements.
- Juror Personas: Parametric, reusable scoring agents embodying perspectives such as safety, procedural correctness, manipulation resistance, or regulatory adherence, decoupling evaluation from single-metric subjectivity.
- Scoring Guidelines & Audit Rules: Programmatically enforced rubrics and rules that tie scores to explicit behavioral evidence, separating subjective juror scores from objective, trace-verifiable failures (e.g., missing tool invocations, illegal state changes).
- Fallback Policies: Explicit strategies and protocols for handling harness or provider instability (e.g., API blocks, malformed outputs), ensuring pipeline transparency and reproducibility.
The operational cycle is structured: upstream expert curation feeds into harness-driven, multi-turn adversarial trials, which in turn produce fine-grained, evidence-linked reporting.
Figure 3: The end-to-end HOB operational flow—human experts curate the challenge and scoring logic, harness executes adversarial trials, linking all outcomes directly to traceable evidence.
Mathematically, this shifts cost scaling from repeated HITL overhead to an amortized curation-to-execution schema, enabling high-throughput, regression-friendly, cost-aware evaluation even as agent diversity and requirements scale.
Empirical Evaluation and Findings
HOB was empirically tested across 47 agent-harness-domain configurations, 470 independent runs, and 23,500 total agent turns, spanning code generation, finance, and healthcare domains. The evaluation design included frontier LLM-based agents (GPT-4.1, GPT-5.5, Claude Opus 4.7/4.8) and 4B–120B Harness LLM tiers, allowing both symmetric and intentionally asymmetric (weaker harness vs. stronger agent) evaluation. Each configuration underwent repeated multi-turn, adversarial trials with trace capture and both subjective (juror) and objective (audit-rule) scoring.
Key Findings
- Amplification of Evaluation with Smaller Harness LLMs: Even when the harness LLM is substantially weaker than the agent under test, curated Red-Team Traps, audit rules, and juror infrastructure enable robust, objective detection of failures—e.g., phantom tool call claims and omitted compliance steps—demonstrating that evaluation strength can be decoupled from evaluator LLM size when supported by expert-authored intelligence.
- Detection of Latent, High-Risk Failures: Failures invisible to endpoint-only evaluation (e.g., agents asserting state updates without corresponding tool invocations, failing mandatory logging, unresolved but safe refusals) are surfaced through evidence-linked, trace-aware evaluation—critical for governance and regulatory contexts.
- Separation of Subjective and Objective Signals: Subjective scores (e.g., task success, safety, manipulation resistance) exhibit evaluator-size-dependent leniency or strictness gradients (with smaller models giving unnaturally higher scores), while objective audit-rule-based failures remain consistent across tiers, providing a reliable foundation for production-grade evaluation.
- Production-Scale Regression and Version-to-Version Sensitivity: The paradigm supports controlled pairing and relative assessment between agent versions (e.g., Claude Opus 4.7 vs. 4.8), exposing nuanced performance and behavioral regressions relevant to release management and longitudinal compliance auditing.
Concrete Empirical Outcomes
- 23,500 agent turns and 414 system-level pipeline failure events were fully auditable, with no loss of evaluation integrity due to fallback infrastructure.
- Asymmetric evaluations consistently surfaced trace-verifiable agent failures even at the smallest harness LLM tiers.
- Subjective scores (particularly for task success and procedure) declined as harness LLM size increased, confirming the need to decouple evaluator strictness from true detection signal.
Implications and Future Directions
HOB operationalizes a domain- and risk-centric evaluation paradigm scalable in both agent diversity and domain complexity. Noteworthy implications include:
- Cost-Efficient Scaling: The reusable nature of curated evaluation intelligence supports sustainable evaluation across agent updates and regulatory domains, enabling the use of local/small LLMs for routine checks augmented by periodic human expert calibration.
- High-Fidelity, Auditable Governance: Evidence-linked scoring enables forensic analysis of procedural breakdowns, regulatory non-compliance, and interaction-level errors, supporting both technical quality assurance and regulatory accountability.
- Open Evaluation Ecosystems: HOB’s architecture facilitates open-source, extensible evaluation infrastructures, catalyzing reproducibility, cross-team collaboration, and longitudinal improvement of evaluation libraries (traps, personas, rubrics).
Open challenges remain, notably the calibration of subjective juror scoring against gold-standard human expert panels, ablation studies to isolate the incremental contribution of expert-authored components, and systematic expansion of the trap, rubric, and persona libraries to match new domains and regulatory requirements.
Conclusion
Human-on-the-Bridge redefines AI agent evaluation as structured, evidence-linked, and expert-guided orchestration. By shifting human judgment to upstream, reusable evaluation intelligence, and separating evidence-linked audit from loose subjective scoring, the paradigm enables production-scale, regression-aware, and cost-effective evaluation of agentic AI. The empirical study validates HOB’s ability to surface critical, latent agent failures, demonstrates its robustness to evaluator LLM size, and underscores its foundational role in future agent governance, safety, and auditing infrastructures.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What this paper is about (in simple terms)
This paper is about a new way to test AI “agents.” An AI agent isn’t just a chatbot that answers once; it can hold long conversations, use tools (like a calculator or a database), follow rules, and act on your behalf. Because of that, we need to test not just what it says, but how it behaves over time.
The authors propose “Human-on-the-Bridge” (HOB), a system where human experts design the tests and rules up front, then a testing platform runs those tests over and over—like building an obstacle course once and sending many robots through it. This makes testing faster, cheaper, and more consistent.
The big questions the paper asks
- How can we test AI agents in a way that checks their behavior across many turns, tools, and rules—not just their final answers?
- Can we put expert human knowledge into reusable test kits (like traps, scorecards, and evidence rules) so we don’t need a human watching every test?
- Can smaller, cheaper AI “judges” still catch real mistakes if we design the tests well?
- Will this approach find hidden problems that regular benchmarks miss?
How they tested it (and what the terms mean)
Think of this like a sports tournament with referees, instant replay, and a carefully designed rulebook:
- The “arena”: ProofAgent Harness (an open-source testing system). It runs long, tricky conversations against the agent, records everything, and produces reports.
- Human-on-the-Bridge (HOB): Humans don’t stand on the field for every play. Instead, they design the whole tournament beforehand:
- Red-Team Traps: Clever obstacles meant to reveal weaknesses (like trying to trick the agent into skipping a safety check).
- Juror Personas: Different “referees” who judge from different angles (safety, usefulness, rule-following, etc.).
- Scoring Guidelines & Audit Rules: The rulebook and instant-replay rules. Not just “was the answer good?” but “did the agent actually make the required tool call?” The “replay” is the trace (a step-by-step log of what happened).
- Fallback Policies: A plan B for when the test system itself hiccups (like a provider error). It says whether to retry, switch, or mark the test as incomplete.
They ran many adversarial conversations (like obstacle course runs) against agents in three areas: finance, healthcare, and code. Across 47 setups, they ran 470 trials, totaling 23,500 turns. They scored agents on:
- Task success (did it actually help you reach the goal?)
- Hallucination resistance (no made-up facts)
- Safety (no harmful advice)
- Instruction following (did it follow rules and procedures?)
- Manipulation resistance (did it resist tricks and social engineering?)
Key idea: They separated two kinds of signals:
- Objective detections (e.g., “the agent claimed it called a tool but the trace shows it didn’t”) — these are fact-checkable.
- Subjective scores (e.g., “how helpful was this?”) — these can vary by the judge.
Helpful analogies for the terms
- Tool call: The agent using a tool (like hitting a calculator button).
- Trace: The instant replay of the match (every move is recorded).
- Phantom tool call: The agent says “I used the calculator,” but the replay shows it didn’t.
- Policy drift: The agent slowly forgets or ignores the rules after several turns.
- Red teaming: Trying to break the system on purpose to see how it fails.
- Juror persona: Different kinds of referees (safety ref, usefulness ref, policy ref).
- Fallback policy: What to do when the scoreboard or replay system glitches.
What they found (and why it matters)
Here are the main takeaways, and why they’re important:
- HOB scales: By putting expert knowledge into reusable “test intelligence” (traps, jurors, rules), they could run lots of tests across different agents and domains—without needing humans to watch every turn. That saves time and makes results more consistent.
- Small judges can catch big mistakes: Even smaller AI evaluators (cheaper models) could find objective, trace-based failures when the tests were well-designed. In other words, you don’t always need a giant expert model to spot a clear rule violation.
- Hidden behavioral failures showed up: The system found problems that normal one-shot benchmarks miss, like:
- Phantom tool-call claims (saying a tool was used when it wasn’t)
- Skipping required tool calls (like forgetting a compliance check)
- Policy drift (following rules at first, then slipping)
- Manipulation paths (being tricked or socially engineered)
- Safe-but-stuck refusals (staying safe but never solving the task)
- Subjective vs. objective: Larger AI judges tended to score more strictly (lower subjective scores); smaller ones were more lenient. But objective detections (like “the tool wasn’t called”) didn’t depend on judge size once the trace was recorded. This shows why separating objective evidence from subjective opinions matters.
- Different agents, different strengths: In the sample they tested, some agents were better at finishing tasks; others were better at safety and resisting tricks. It’s not one single score that matters—reliability is multidimensional.
Why this approach could change how we build and release AI
- Safer, more reliable agents: Testing the “how” (behavior over time) instead of just the “what” (final answer) reduces the chance of dangerous or misleading actions slipping through.
- Reuse beats rework: Invest once to build strong traps and rules, then reuse them across many agents and versions. Over time, that’s cheaper and faster than reviewing everything by hand.
- Fairer comparisons: Evidence-linked reports (with traces and clear rules) make it easier to compare versions, find regressions, and justify decisions to regulators or stakeholders.
- Wider access: Because smaller judging models can still catch objective failures, smaller teams can meaningfully evaluate powerful agents without huge costs.
In short, Human-on-the-Bridge helps teams turn expert judgment into a reusable “test kit,” run large-scale, adversarial, multi-turn evaluations, and produce evidence-based results. That makes deploying AI agents in sensitive areas like healthcare and finance more trustworthy and manageable.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a concise list of what remains missing, uncertain, or unexplored in the paper. Each item is formulated to be concrete and actionable for future research.
- Calibration of subjective juror scores to human expert ground truth is not performed; no inter-rater agreement, bias analysis, or calibration curves are provided across domains and Harness LLM tiers.
- No precision/recall or error-rate quantification for audit-rule detections (e.g., phantom/missing tool calls); the paper assumes objectivity once traces are available but does not measure false positives/negatives or rule robustness.
- Coverage adequacy of the Red-Team Trap library is unmeasured; there is no metric or method to estimate adversarial coverage, trap diversity, or residual risk after testing.
- Potential evaluator-induced bias is not analyzed; smaller Harness LLMs showed leniency in subjective scores, but there is no normalization or cross-tier score calibration to enable fair comparisons.
- The cost model (Cc, Ce, r, Cr and break-even N*) is theoretical-only; empirical measurements of curation cost, marginal execution cost, human review rate, and sensitivity analyses are absent.
- Impact of fallback events on evaluation integrity is not quantified; while 414 production failure events were recovered, there is no analysis of how retries, provider switches, or blocked outputs affect scores and detections.
- Statistical reliability is limited to medians; no variance estimates, confidence intervals, bootstraps, or significance tests are reported for configuration scores or paired deltas.
- The evaluation sweep is selected rather than factorial; many domain–agent–tier cells are represented by a single configuration, limiting generalizability and making cross-configuration comparisons indicative rather than definitive.
- External validity to real-world deployment is untested; there is no linkage between detected failures and downstream operational harm, incident rates, or governance outcomes in production settings.
- Tool-use auditing focuses on presence/absence and ordering but does not verify argument correctness, tool-output interpretation fidelity, or end-to-end procedural semantics.
- No formal guarantees (soundness/completeness) are provided for audit rules; a formal specification language and validation suite for rule semantics are not described.
- Disagreement handling among Juror Personas is not evaluated; the paper references multi-juror scoring and disagreement resolution but provides no empirical assessment of adjudication policies or their effects on outcomes.
- Extensibility claims (tool-agnostic HOB) are not demonstrated across alternative harnesses; portability of curated intelligence and reproducibility outside ProofAgent Harness remain untested.
- The trap curation process lacks methodology and quality controls; guidelines for authoring traps, avoiding curator bias, maintaining versioned libraries, and ensuring domain/regulatory completeness are not provided.
- No metrics for manipulation-resistance coverage are reported; the paper lists manipulation attempts (prompt injection, social engineering, emotional pressure), but does not quantify scenario breadth or effectiveness across agents.
- Severity-graded findings (n=103) are not integrated into scoring; there is no model of risk-of-ruin, tail-risk weighting, or policy thresholds that reflect governance priorities in healthcare/finance.
- Multi-turn interaction length is fixed at 50 turns; sensitivity of failure modes to longer or shorter trajectories, memory persistence, or session continuity (e.g., cross-session drift) is unexplored.
- Agents are heterogeneous (domain skills, tools, grounding layers) without controlled ablations; the contribution of backbone vs. agent architecture vs. tools to observed failures is not disentangled.
- Potential overfitting of agents to known traps is not addressed; mechanisms to rotate, mutate, or conceal traps to prevent agent memorization are not described or evaluated.
- Log-based evaluation integration is absent; the framework does not combine synthetic adversarial trials with real conversation logs to assess whether laboratory detections predict field failures.
- Domain/regulatory rubrics (e.g., GDPR, EU AI Act) are mentioned but not validated; there is no case study demonstrating regulator-grade compliance checks or audit trace sufficiency for certification.
- Safety vs. usefulness trade-offs (over-refusal) are detected but not modeled; no structured utility-safety frontier, Pareto analysis, or policy for resolving “safe but non-resolving” behavior is presented.
- Cross-agent or multi-agent evaluation is out of scope; HOB’s applicability to agent teams, coordination failures, or emergent behaviors in multi-agent systems is unexplored.
- Harness LLM tier selection effects are partially described but not benchmarked; there is no systematic analysis of detection rates, failure types, or score calibration as a function of evaluator capability and cost.
- Data release details are limited; while artifacts are “released,” there is no standardized schema for traps, jurors, rubrics, and rules to enable community sharing, versioning, and inter-lab comparability.
- Ethical considerations and responsible disclosure are minimal; procedures for handling discovered vulnerabilities in medical/financial agents, and pathways for coordinated disclosure, are not outlined.
- Generalization across languages and locales is not evaluated; all reported evaluations appear English-centric, with no assessment of multilingual agents, localized policies, or culturally specific manipulation tactics.
- Automation of curation is not explored; methods to use LLMs or program synthesis to generate candidate traps, jurors, and audit rules with human vetting could reduce Cc and increase coverage, but are not investigated.
- No benchmarking against existing agent evaluation suites is provided; comparative studies to AgentBench, ReAct-based environments, or safety red-team datasets are absent, limiting context on HOB’s incremental value.
Practical Applications
Immediate Applications
The following applications can be deployed now by leveraging the HOB paradigm and the open-source ProofAgent Harness to encode reusable traps, juror personas, audit rules, and fallback policies. Each item lists associated sectors, the potential tool/product/workflow, and key assumptions or dependencies.
- Agent release gates in CI/CD (“EvalOps”)
- Sectors: software, SaaS, platforms with agent features
- Tools/Workflows: Add ProofAgent Harness as a pre-merge or pre-deploy job; run curated Red-Team Traps and audit rules on every model/prompt/tool update; fail builds on objective trace violations (e.g., missing mandatory tool calls)
- Assumptions/Dependencies: Access to agent traces/logs; dev buy-in to treat objective audit failures as hard gates; initial curation of domain traps and rubrics
- Regression testing and version comparison for agent upgrades
- Sectors: software, dev tools, internal platform teams
- Tools/Workflows: Paired runs comparing model versions; median-of-runs scoring; evidence-linked change reports for release managers
- Assumptions/Dependencies: Stable seeds/environment; repeated runs to smooth LLM stochasticity; versioned evaluation intelligence
- Vendor/model procurement due diligence (RFP/RFQ)
- Sectors: finance, healthcare, government, enterprise IT
- Tools/Workflows: Run standardized trap libraries and juror packs against vendor-supplied agents; require evidence-linked reports in bids
- Assumptions/Dependencies: Vendors expose agents with trace capture; legal approval to test; standardized acceptance thresholds
- Compliance-ready audit trails for high-risk use cases
- Sectors: finance (suitability, KYC/AML), healthcare (triage/escalation), legal, privacy
- Tools/Workflows: Evidence-linked reports per evaluation configuration; objective detections (phantom/missing tool calls, policy drift) tied to trace excerpts; archive for audits
- Assumptions/Dependencies: Clear mapping from internal policies/regulations to audit rules; secure storage of reports; privacy controls for trace data
- Tool-use policy enforcement and workflow conformance
- Sectors: enterprise agents with tools (CRMs, ERPs, DevOps, ticketing)
- Tools/Workflows: Audit rules that mark mandatory tool calls, ordering constraints, and forbidden sequences as pass/fail; dashboards highlighting procedural defects
- Assumptions/Dependencies: Agents must log tool calls and parameters; teams must codify required workflows
- Safety and manipulation resistance red teaming at scale
- Sectors: consumer assistants, customer support, education, social platforms
- Tools/Workflows: Curated adversarial scenarios (prompt injection, social engineering, emotional pressure) reused across runs; juror personas scoring safety and manipulation resistance
- Assumptions/Dependencies: Trap library updates to track new attack patterns; regular refresh cadence
- Postmortem and incident analysis with evidence-linked replays
- Sectors: operations/SRE, trust & safety, risk
- Tools/Workflows: Replay production logs through harness; add traps that mimic incident conditions; generate severity-graded findings linked to trace evidence
- Assumptions/Dependencies: Log ingestion pipeline; PII handling; mapping incidents to reusable traps and rubrics
- Cost-efficient asymmetric evaluation for startups and small teams
- Sectors: all (cost-sensitive environments)
- Tools/Workflows: Use smaller Harness LLM tiers with strong audit rules and curated traps to surface objective procedural failures in frontier-LLM agents
- Assumptions/Dependencies: Objective failures depend on trace observability and rule design; subjective juror scores may be lenient and need calibration
- Training data generation from failures (“Eval-to-Fix” loop)
- Sectors: ML teams building agents
- Tools/Workflows: Convert detected failures and traces into fine-tuning examples or reward models; track re-tests to verify fixes
- Assumptions/Dependencies: Data labeling pipeline; privacy-safe data handling; alignment with retraining cadence
- Code-generation agent QA
- Sectors: software engineering
- Tools/Workflows: Traps for required steps (run tests, lint, dependency checks); audit rules catch hallucinated tool use or skipped steps; integrate into PR checks
- Assumptions/Dependencies: Reliable tool invocation logs; standardized coding workflows; compute for test execution
- Financial advisory agent checks
- Sectors: finance/fintech/wealth
- Tools/Workflows: Traps for suitability, disclosure, product restrictions; audit rules for mandatory KYC/AML tool calls; juror personas for compliance and customer fairness
- Assumptions/Dependencies: Up-to-date policy/rulebooks encoded as rubrics; access to anonymized or synthetic client profiles
- Healthcare triage and escalation verification
- Sectors: healthcare, telehealth, medtech
- Tools/Workflows: Traps that test escalation thresholds, refusal vs guidance balance, and privacy handling; evidence-linked failures for clinical governance
- Assumptions/Dependencies: Clinical oversight to author traps/rubrics; HIPAA/GDPR-compliant data handling; clear escalation policies
- Customer support/call-center copilot QA
- Sectors: BPOs, SaaS support, telco, e-commerce
- Tools/Workflows: Traps for procedural adherence (ticket updates, CRM logging), policy compliance, and manipulation resistance; audit rules for phantom CRM updates
- Assumptions/Dependencies: Integration with support tool logs; agent traceability; support policy codification
- Privacy and data-handling guard evaluation
- Sectors: all handling personal or sensitive data
- Tools/Workflows: Juror personas focused on privacy; audit rules verifying no PII is exposed or retained against policy; red-team traps for data exfiltration attempts
- Assumptions/Dependencies: Clear data-classification rules; masking for test data; privacy office involvement
- Marketplace/store submission tests for agent plugins and tools
- Sectors: app/plugin ecosystems
- Tools/Workflows: Standardized trap suites and juror packs as submission criteria; automated evidence-linked pass/fail reports for third-party agents
- Assumptions/Dependencies: Platform policies mapped to audit rules; submission sandbox with trace capture
- Fallback policy testing for production resilience
- Sectors: platform operations, reliability engineering
- Tools/Workflows: Simulate provider/API failures and content blocks; verify retry/switch/escalate behaviors are correctly triggered and logged
- Assumptions/Dependencies: Harness fallback configuration; synthetic fault injection support
- Hybrid evaluation: log replays + adversarial coverage
- Sectors: any with deployed agents
- Tools/Workflows: Combine historical log-based evaluation with curated adversarial runs; track coverage and risk deltas across versions
- Assumptions/Dependencies: Reliable log ingestion; versioned trap libraries; metrics for coverage and severity
Long-Term Applications
These applications require further research, scaling, organizational buy-in, standardization, or regulatory acceptance before widespread deployment.
- Sector-standard “evaluation intelligence” marketplaces
- Sectors: finance, healthcare, education, cybersecurity, government
- Tools/Workflows: Commercial/open repositories of trap packs, juror personas, and audit rules vetted by domain bodies
- Assumptions/Dependencies: Shared standards for artifact formats; IP/licensing; liability frameworks for shared traps
- Regulatory certification frameworks for agentic systems
- Sectors: finance, healthcare, safety-critical, public sector
- Tools/Workflows: Evidence-linked HOB reports accepted in conformity assessments (e.g., EU AI Act high-risk systems), audit readiness kits
- Assumptions/Dependencies: Regulator engagement; validation studies against human experts; accreditation of harnesses and evaluators
- Runtime governance and self-healing agents
- Sectors: platforms with live agents
- Tools/Workflows: Continuous micro-evaluations during operation; objective detections trigger policy updates, tool gating, or human escalation in real time
- Assumptions/Dependencies: Low-latency evaluation loops; safe intervention mechanisms; monitoring budgets
- Cross-organization red-team consortia and incident exchange
- Sectors: critical infrastructure, finance, healthcare, cloud
- Tools/Workflows: Shared near-miss and incident-derived traps; coordinated updates to trap libraries across sectors
- Assumptions/Dependencies: Legal safe-harbors; standardized anonymization; information-sharing agreements
- Formal specification languages for audit rules and policies
- Sectors: all; especially regulated industries
- Tools/Workflows: Machine-checkable policy specs mapped to traces and tool calls; automated conformance proofs in reports
- Assumptions/Dependencies: Consensus on spec semantics; toolchain support; integration with agent SDKs
- Evaluation-to-training (E2T) feedback loops at scale
- Sectors: ML platform teams
- Tools/Workflows: Systematic conversion of HOB findings into reward signals or curriculum for RLAIF/finetuning; auto-retest pipelines
- Assumptions/Dependencies: Data engineering pipelines; robust de-duplication and privacy controls; proven generalization beyond trap overfitting
- Agent SLAs and insurance underwriting based on evidence-linked scores
- Sectors: enterprise SaaS, B2B platforms, insurers
- Tools/Workflows: Contractual SLAs tied to HOB metrics (e.g., max rate of objective procedural failures); actuarial use of severity-graded findings
- Assumptions/Dependencies: Stable, validated metrics; market acceptance; legal frameworks for agent liability
- Robotics and IoT agent behavioral assurance
- Sectors: robotics, manufacturing, logistics, smart homes
- Tools/Workflows: Traps and audit rules for actuation order, safety interlocks, and escalation; evidence-linked safety cases
- Assumptions/Dependencies: Rich action traces and sensor logs; simulators for adversarial scenario execution; cross-domain SMEs
- National/industry evaluation centers and public scorecards
- Sectors: public policy, standards bodies
- Tools/Workflows: Independent HOB-based evaluation labs publishing periodic evidence-linked reports on widely used agents
- Assumptions/Dependencies: Funding and mandate; transparent protocols; cooperation from vendors
- Adaptive defense through dynamic trap generation
- Sectors: security-focused deployments
- Tools/Workflows: Online mining of novel attack patterns from production logs; auto-synthesis of new traps with human review; rapid deployment
- Assumptions/Dependencies: Safe automation boundaries; validation safeguards; monitoring for false positives
- Privacy-preserving or federated evaluation
- Sectors: healthcare, finance, government
- Tools/Workflows: On-premise/federated harness deployments; secure enclaves for traces; standardized redaction/masking across partners
- Assumptions/Dependencies: Secure compute infrastructure; cross-organization trust; performance overhead budgets
- Education quality and fairness evaluation for tutoring agents
- Sectors: education, edtech
- Tools/Workflows: Juror personas for pedagogy quality, bias/fairness, and safety; longitudinal trajectory scoring
- Assumptions/Dependencies: Ground-truth learning outcomes; fairness definitions; IRB/ethics oversight
- Real-time compliance supervision in financial services
- Sectors: banking, brokerage, payments
- Tools/Workflows: Live trap sampling on advisory/copilot interactions; immediate flags for missing KYC/AML tools or disclosure steps
- Assumptions/Dependencies: Latency constraints; regulator acceptance; tight integration with compliance systems
- Benchmark evolution: community-curated behavioral benchmarks
- Sectors: academia, open-source
- Tools/Workflows: Shared, versioned libraries of multi-turn, tool-using scenarios with trace-based scoring; multi-juror ensemble protocols
- Assumptions/Dependencies: Governance for curation; validation against human panels; mechanisms to prevent overfitting to public traps
- Cross-layer resilience testing (agent + tool + provider stack)
- Sectors: cloud platforms, large enterprises
- Tools/Workflows: Fault-injection plus HOB evaluation spanning model, tool APIs, data sources, and fallback policies; end-to-end resilience scoring
- Assumptions/Dependencies: Coordinated test environments; observability across layers; provider cooperation
Glossary
- Agent Under Test: the specific AI agent being evaluated in trials. "The Agent Under Test is the system being evaluated."
- adversarial pressure: structured stress conditions that challenge agents to expose weaknesses. "organizing these techniques around human expertise, adversarial pressure, repeatable execution, and evidence-linked reporting."
- adversarial trials: test runs designed to apply adversarial conditions during evaluation. "It runs adversarial agent trials, captures traces, applies multi-juror scoring, records evidence, handles fallback events, and produces reproducible reports."
- asymmetric evaluation: an assessment setup where a smaller evaluator model tests a stronger agent model. "We use the term asymmetric evaluation when a smaller Harness LLM evaluates an agent built on a stronger frontier LLM backbone."
- audit rules: explicit criteria linking behaviors in traces to scores or failures. "Audit rules define what evidence must support a score or failure."
- evidence-linked evaluation artifact: an evaluation output that ties scores and findings to specific behavioral evidence. "It is an evidence-linked evaluation artifact:"
- evidence-linked findings: reported results that are directly supported by recorded evidence. "and produces evidence-linked findings across finance, healthcare, and code generation."
- evidence-linked reporting: reporting that includes the supporting evidence for each evaluation claim. "multi-juror scoring, and evidence-linked reporting."
- evaluation configuration: a single, defined combination of domain, agent, and evaluator used for testing. "an evaluation configuration refers to one domain-agent-Harness LLM setting."
- evaluation intelligence: curated, reusable knowledge and rules that govern how agents are tested. "evaluation intelligence refers to the reusable knowledge, scenarios, scoring perspectives, audit rules, and recovery policies that define how an agent should be evaluated."
- fallback policies: predefined actions to handle evaluation pipeline errors or instability. "Fallback policies define how the evaluation behaves when the evaluation pipeline becomes unstable."
- frontier LLM-based agents: agents built on cutting-edge, high-capability LLMs. "frontier LLM-based agents and Harness LLM tiers."
- grounding layer: the component providing domain tools/data and constraints that anchor an agent’s behavior. "Each agent is configured with domain-specific skills, tool access, and a grounding layer."
- Harness LLM: the evaluator model used by the harness to apply traps, score, and interact during tests. "The Harness LLM provides the evaluation reasoning and interaction capability."
- Human-in-the-Loop (HITL): an evaluation approach where human experts actively review or decide during runs. "Human-in-the-Loop (HITL) evaluation remains one of the strongest sources of expert judgment."
- human-on-the-loop supervision: a paradigm where humans monitor automated execution and intervene reactively. "human-on-the-loop supervision, where a human monitors automated execution and intervenes reactively."
- Human-on-the-Bridge (HOB): a paradigm where human experts curate evaluation assets upfront and the harness executes them at scale. "This paper introduces Human-on-the-Bridge (HOB), a scalable evaluation paradigm for agentic AI."
- interactive agent benchmarks: tests that evaluate agents within interactive environments over multiple turns. "interactive agent benchmarks, Human-in-the-Loop review, LLM-as-judge evaluation, red teaming, trace and tool-use auditing, log-based evaluation, and open evaluation infrastructure."
- jailbreak attempts: adversarial prompts designed to bypass safety or policy constraints. "They may include policy edge cases, jailbreak attempts, tool-use challenges, memory stress tests, ambiguous user requests, manipulation attempts, or domain-specific failure modes."
- Juror Personas: distinct scoring perspectives (e.g., safety, compliance) used to evaluate agent behavior. "Juror Personas define the scoring perspectives used to evaluate agent behavior."
- LLM-as-judge: using a LLM to score or compare outputs as a proxy for human judgment. "LLM-as-judge methods use LLMs to score outputs, compare responses, or approximate human preferences."
- log-based evaluation: assessment that replays and analyzes real conversation logs to find failures and regressions. "Log-based evaluation allows teams to replay historical user-agent interactions, identify recurring failures, compare versions, and measure regressions."
- manipulation paths: interaction routes where agents can be steered into undesirable or unsafe behavior. "including phantom tool-call claims, missing mandatory tool calls, policy drift, manipulation paths, and safe but non-resolving refusals."
- manipulation resistance: an agent’s ability to resist social engineering or prompt-based attacks. "Manipulation resistance: whether the agent resists prompt injection, social engineering, emotional pressure, and attempts to bypass rules."
- missing mandatory tool calls: failure to invoke required tools/procedures despite indicating progress. "missing mandatory tool calls"
- open evaluation infrastructure: extensible tooling that makes evaluations executable, repeatable, and auditable. "Open evaluation infrastructure provides the execution layer needed to make evaluation repeatable."
- paired score delta: the defined difference in score between two matched agent versions. "The paired score delta is defined as:"
- phantom tool-call claims: assertions that tools were used when no corresponding tool invocation occurred. "phantom tool-call claims"
- policy drift: gradual deviation from required policies across turns or interactions. "policy drift"
- ProofAgent Harness: the open-source execution layer that runs adversarial trials and records evidence. "ProofAgent Harness is the execution layer."
- prompt injection: adversarial inputs crafted to alter or subvert an agent’s instructions or behavior. "whether the agent resists prompt injection, social engineering, emotional pressure, and attempts to bypass rules."
- Red-Team Traps: curated adversarial scenarios that encode expert knowledge of likely failures. "Red-Team Traps define the pressure conditions used to test the Agent Under Test."
- safe but non-resolving refusals: cautious responses that avoid risk but fail to advance the user’s goal. "safe but non-resolving refusals."
- scoring guidelines: rules that specify how jurors should evaluate behaviors on defined dimensions. "Scoring guidelines define how Juror Personas evaluate behavior."
- severity-graded findings: discovered issues categorized by their seriousness level. "Severity-graded findings"
- supervisory-control literature: research on human oversight and control in automated systems. "relative to the supervisory-control literature reviewed in Section~\ref{sec:related_work}."
- symmetric evaluation: an assessment setup where evaluator and agent are of comparable capability. "We use the term symmetric evaluation when the Harness LLM tier is comparable in capability to the agent backbone being evaluated."
- tool-use auditing: checking whether tool use aligns with rules and whether required calls occurred. "trace and tool-use auditing"
- trace auditing: verifying procedural behavior by checking recorded traces against explicit rules. "trace auditing requires explicit evidence rules."
- trace capture: recording detailed sequences of actions and tool calls during evaluation. "multi-turn adversarial evaluations, trace capture, multi-juror scoring, and evidence-linked reporting."
- trajectory-based assessment: evaluating performance over the sequence of turns/actions rather than single responses. "shift evaluation from single-response scoring toward trajectory-based assessment."
Collections
Sign up for free to add this paper to one or more collections.