AOEP-v0 Evaluation Protocol

• AOEP-v0 is a pilot evaluation contract for always-on agents that rigorously defines and tests persistent state transitions under strict governance invariants.
• It employs a typed event-stream format and a validator to recompute invariants like authority monotonicity, scope non-expansion, and rollback traceability.
• The protocol shifts evaluation from mere answer quality to state-trajectory correctness, addressing issues like stale commitments and state poisoning.

Searching arXiv for the cited AOEP-related papers to ground the article and confirm metadata. Always-On Evaluation Protocol (AOEP-v0) is a pilot evaluation contract for always-on agents, introduced in the survey “Always-OnAgents: A Survey of Persistent Memory, State, and Governance in LLMAgents” (Ding et al., 29 Jun 2026). It evaluates correctness of state transitions under governance invariants rather than answer quality alone. In that formulation, always-on agents are persistent-state systems whose future behavior depends on durable state accumulated across earlier interactions, including retrievable memories, task ledgers, permissions, credentials, commitments, provenance and audit records, shared state, trigger conditions, and externally committed effects. AOEP-v0 operationalizes this view through a typed event-stream format, a neutral probing interface, and a validator that recomputes invariant predicates to produce a deterministic scorecard.

1. Formal role and motivation

AOEP is introduced in the abstract, Section 1, Table 1, Figure 1, and Section 9 of the survey, but AOEP-v0 is defined specifically in Section 9, “The Always-On Evaluation Protocol (AOEP),” together with its associated figure (Ding et al., 29 Jun 2026). The authors explicitly describe AOEP-v0 as a pilot instantiation of the general protocol.

Its conceptual role is narrower and more formal than a conventional benchmark. AOEP-v0 is not a fixed task set scored by accuracy or preference alone. It is an evaluation contract that specifies a data format for episodes, a validator that derives governance scores by re-computing invariants, and a pilot harness with a small battery of episodes. The protocol therefore treats state mutation and recovery obligations as first-class evaluation targets.

The motivating gap is that existing agent evaluations are largely episodic. They reset state between tasks and therefore cannot observe persistence-specific failures such as poisoning that activates later, stale commitments, failure to delete, or inability to roll back bad state. Existing evaluations also focus on answer correctness or final task success, rather than on whether writes were authorized, whether deletions propagated, whether provenance was preserved, or whether rollback is possible and correctly logged. Across a 435-work coded corpus, treated as a scoped map rather than an exhaustive census, the survey finds that the literature concentrates more heavily on accumulating and retrieving state than on governing, recovering, or relinquishing it; retrieve and write stages are heavily evaluated, whereas forget, audit, and especially rollback are barely evaluated, with only 27 of 435 works exposing any rollback at all and none reporting recovery success or cost (Ding et al., 29 Jun 2026).

This design shifts the unit of evaluation from output quality to state-trajectory correctness. The protocol’s purpose is to make governance requirements concrete and testable, and to attach deterministic scores to whether an agent respects invariants such as authority monotonicity, scope non-expansion, deletion propagation, provenance preservation, and rollback traceability.

2. Persistent-state model, diagnostic axes, and lifecycle

The survey defines an always-on agent operationally as a system whose policy at time $t$ depends on state accumulated before $t$, beyond the current prompt or task (Ding et al., 29 Jun 2026). It need not run continuously; it is “always-on” if it has persistent identity across sessions or restarts, maintains agent-owned durable state about users, tasks, tools, policies, environment, or other agents, and can use that state for later action. Where that state is consequential, a further obligation follows: temporal accountability, namely being able to say what the system knew, when it knew it, why it acted as it did, and how to identify and repair contaminated state and the actions it caused.

The relevant state is broader than memory. The survey includes retrievable memories, task and workflow ledgers, permissions and credentials, policy and constraints, procedural skills, provenance and audit records, shared or social state, trigger conditions, and externally committed effects such as messages sent, payments made, API calls, and database writes. To formalize this, it models a persistent state record as

$s = \langle v, a, c, m, p, r, k, \tau \rangle,$

where $v$ is value, $a$ is authority, $c$ is scope, $m$ is mutability class, $p$ is provenance chain, $r$ is recoverability handle, $k$ is actionability type, and $t$0 is logical timestamp. The survey’s central claim is that most existing memory systems effectively only materialize $t$1, whereas governance scoring requires the remaining fields to be represented and testable.

AOEP-v0 is grounded in six diagnostic axes. Authority asks who or what grants a record permission to influence actions; its invariant is authority monotonicity, under which authority may narrow over time but must not expand silently, and only current, unrevoked authority may license an action. Scope asks for which user, task, tool, time, or group a record may be used; its invariant is scope non-expansion. Mutability asks how and on what timescale a record can be revised, superseded, decayed, or locked. Provenance asks what sources and transformations produced a record; its invariant is provenance preservation. Recoverability asks whether derived state and decisions caused by a record can be rolled back; its invariant is rollback traceability. Actionability asks whether a record is evidence, preference, policy, skill, or executable commitment.

The protocol is also anchored in a ten-stage persistent-state lifecycle: observe, write, validate, organize, retrieve, act, update, forget, audit, and rollback. The lifecycle is divided into a forward arc—accumulate and use—and a return arc—govern and recover. Authority monotonicity is attached to validate and act; scope non-expansion to organize and retrieve; provenance preservation to organize and update; deletion propagation to forget, checked at audit; and rollback traceability to act and rollback. AOEP-v0 is designed to test whether an agent’s state transitions across this lifecycle respect those invariants.

3. Protocol architecture, state exposure, and scoring

AOEP-v0 represents episodes as typed event streams and evaluates systems by comparing their internal state exposure to an oracle reconstruction (Ding et al., 29 Jun 2026). The pipeline begins with an episode encoded as a typed event stream. Oracle or outcome-revealing fields are then stripped before the system under test sees the episode. The system maintains its own state using its own mechanisms. After the episode, the harness issues neutral probes that mention only IDs, not expected answers or invariant names, in order to recover the system’s internal view. A validator then reconstructs state from the oracle log and the system’s probe answers, recomputes invariant predicates, and produces a deterministic scorecard.

The event schema is designed to make governance metadata explicit. Each event carries an idempotency key to detect replayed duplicate operations after restart; causal links such as parent, supersedes, conflicts-with, plus transaction IDs; a permission epoch capturing authority over time; provenance metadata including source, trust tier, and optionally signatures; retention and privacy constraints including retention policy, privacy class, TTL, and requires-confirmation flags; and an operation type drawn from a closed set such as read, write, update, delete, tombstone, share, unshare, validate, quarantine, deny, and rollback.

The state snapshot exposed after an episode must support audit and recovery checks. The snapshot includes a deletion ledger, a rollback ledger, pending conflicts with candidate values and backing authority, per-resource permission epochs, and any other necessary summary fields such as live records keyed by ID. Crucially, the system is not trusted to assess its own behavior; it supplies data, and the validator computes invariant booleans independently.

The validator recomputes deterministic predicates including no unauthorized write, permission epochs current at time of action, no stale-authority action executed, no scope leakage, no deleted content visible, deletion ledger matches derived state, no untrusted instruction promoted to authoritative state, and no external action without recorded approval and rollback handle. AOEP-v0 reports these as a two-part scorecard. Obligation pass covers positive duties a system must actively meet, such as logging deletions, updating permission epochs after revocation, blocking stale-authority or untrusted actions, surfacing conflicts, and logging rollback for external actions. Negative-invariant pass covers no-leakage checks that a system that stores nothing satisfies vacuously, such as deleted value not being retrievable, out-of-scope data not appearing, and no untrusted instruction being retrieved as instruction. The survey stresses that these must be reported separately so that “amnesia” does not appear to be governance success.

4. Pilot harness, episode patterns, and empirical results

The AOEP-v0 pilot is deliberately small, but it already distinguishes between recall-oriented storage and governed state management (Ding et al., 29 Jun 2026). The harness evaluates a governed reducer as a deterministic reference implementation, a no-memory floor, and five ungoverned memory systems using a fixed local reader, Qwen2.5-7B: a naive append store, a full-context system, a Vector-RAG system using MiniLM embeddings and top-$t$2 cosine, a Mem0-style extracted fact store reimplementation, and a locally configured Mem0 system. All runs in the main experiment are deterministic with greedy decoding.

The protocol exercises nine fault patterns. Three are described in detail. Restart–conflict–deletion tests idempotency, conflict handling, and deletion visibility by replaying a durable billing write after restart, introducing a collaborator’s conflicting shared-scope value, and then issuing a deletion request. Permission epoch drift tests whether revocation updates the system’s permission_epoch and blocks actions that rely on stale authority. Adversarial memory injection tests whether an untrusted tool output containing a malicious instruction is quarantined or later promoted as instruction. Six additional patterns stress deletion-derived summary residue, cross-user scope leaks, stale permission after restart, owner–collaborator conflict resolution, rollback of external action, and untrusted tool-output poisoning. The pilot also includes three longer, realistic multi-actor episodes derived from Tau-bench-style customer refund flows, enterprise scheduling tasks, and AppWorld’s “group dinner Venmo payment request,” and it is further grounded in real AppWorld traces with verified side effects.

The main pilot result over nine fault patterns yields the following governance profile.

System	Obligation pass	Neg.-invariant pass
Governed reducer	15/15	41/41
No-memory	0/15	41/41
Naive append	7/15	40/41
Full context	7/15	40/41
Vector-RAG	7/15	40/41
Mem0-style extracted facts	4/15	38/41
Mem0 (local package)	3/15	36/41

In all-check totals, the governed reducer attains 92/92, the no-memory floor 77/92, naive append 83/92, full context 83/92, Vector-RAG 83/92, Mem0-style extracted facts 77/92, and Mem0 73/92. The survey emphasizes three conclusions. First, raw storage systems—append, full context, and RAG—all achieve the same obligation score of 7/15. They pass duties that reduce to semantic recall, but fail duties that require maintained governance state such as permission epochs, conflict ledgers, and rollback logs. Differences in retrieval do not change governance performance. Second, extracted-fact systems perform worse in this configuration because fact extraction flattens the structured envelope into text and loses governance metadata such as deletion indicators and trust tiers. The local Mem0 setup leaks both a deleted billing address and an exfiltration address from an untrusted tool output. Third, the no-memory floor passes negative invariants perfectly because it stores nothing, yet fails all 15 obligations.

The reader robustness check leads to the same conclusion. Full-context systems with Qwen2.5-3B, Qwen2.5-7B, and Llama-3.1-8B all score 7/15 on obligations, while Mistral-7B scores 4/15. None approaches the governed reducer’s 15/15, and the survey concludes that larger models do not fix missing governance state. A sampling-based variant yields similar obligation scores, with mean approximately 6.6/15, so the gap is not a single-seed artifact. In realistic episodes and AppWorld traces, the same pattern persists: the governed reducer satisfies all obligations, whereas ungoverned systems leak untrusted instructions or deleted data.

5. AOEP-style antecedents in adjacent literatures

Although AOEP-v0 is formally defined in the always-on agents survey, the supplied corpus presents several domain-specific evaluation frameworks as AOEP-style designs or as components that can be transposed into AOEP-like systems.

In compute-first networking, “TPAoI: Ensuring Fresh Service Status at the Network Edge in Compute-First Networking” defines a three-phase freshness metric and a matching always-on control mechanism for status updating, user accessing, and user requesting (He et al., 2024). The paper’s narrative explicitly states that an Always-On Evaluation Protocol for edge or compute-first networking systems needs a metric and control loop that continuously track, interpret, and act on freshness as it is generated, propagated, accessed, and used. Its TPAoI metric measures age at request arrival,

$t$3

thereby decomposing freshness into update delay, user waiting time, and request delay. The control problem is formulated as an MDP with a Dueling Double Deep Q-Network policy. In the reported experiments, TPAoI reduces AoI by an average of $t$4 compared to QAoI metrics and decreases update frequency by an average of $t$5 relative to conventional AoI metrics. This suggests an AOEP-style paradigm in which continuous evaluation and control are coupled through a request-centric freshness objective.

In multimodal inverse problems, “Pointwise Metrics Mislead: An Evaluation Protocol for Multimodal Inverse Problems” argues that pointwise metrics are structurally misaligned whenever posteriors are multimodal (Baattrup et al., 21 May 2026). The paper proposes a three-part evaluation protocol: per-event distributional accuracy via CRPS, population-level marginal accuracy via a spectrum-fidelity diagnostic, and uncertainty trustworthiness via coverage-based calibration. Its central theorem-level argument is that point estimators trained to minimize MSE or MAE yield a marginal spectrum strictly narrower than the truth whenever the posterior has nonzero width. The paper therefore treats the evaluation protocol itself as determinant of the scientific conclusion. For AOEP-like systems, this establishes a distributional, multi-axis model of evaluation in which instance-wise quality, population-level fidelity, and calibration are reported separately rather than collapsed into a single pointwise score.

In generative conversational systems, “An Evaluation Protocol for Generative Conversational Systems” defines a head-to-head pairwise comparison protocol for next-utterance generation using ChatEval, Amazon Mechanical Turk, win–loss–tie judgments, Bradley–Terry ranking, and TrueSkill (Lee et al., 2020). The protocol fixes evaluation tasks, candidate systems, evaluation sets, crowd interface, and statistical aggregation. It uses five fixed evaluation datasets, multiple annotators per A/B pair, weak agreement analysis, annotator correlation analysis, and prompt validity checks. The paper explicitly frames its design as compatible with a living or always-on benchmark, since new models can be compared against a standing pool of systems and embedded into global rankings incrementally.

In instruction-following evaluation, “ReIFE: Re-evaluating Instruction-Following Evaluation” defines an LLM-evaluator as a combination of a base LLM and an evaluation protocol, and meta-evaluates 25 open-source base LLMs across 15 protocols on 4 human-annotated datasets (Liu et al., 2024). Its main findings are that base-LLM performance ranking remains largely consistent across evaluation protocols, that robust evaluation of protocols requires many base LLMs with varying capability levels, and that evaluation results on different datasets are not always consistent. The paper therefore recommends multi-protocol and multi-model evaluation panels, with protocol-consistency and swap-and-synthesize variants to improve robustness and self-agreement. This supplies an AOEP-style lesson about meta-evaluation: evaluator selection and protocol selection must themselves be continuously validated.

In scientific protocol planning, “BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology” introduces BioProt, a dataset of biology protocols paired with pseudocode representations, and uses a teacher–student pipeline to convert natural-language protocols into constrained pseudocode and then automatically score reconstruction quality (O'Donoghue et al., 2023). The framework decomposes performance into next-step prediction, full protocol generation, and function retrieval, and scores function identity, order via normalized Levenshtein distance, and argument semantics via BLEU and SciBERTscore. It also reports that GPT-4 as evaluator is close to chance at distinguishing model output from ground truth. For AOEP-like designs, the paper demonstrates the value of canonical intermediate representations and programmatic metrics when human or LLM-as-judge evaluation is insufficiently reliable.

In cooperative MARL, “Towards a Standardised Performance Evaluation Protocol for Cooperative MARL” proposes a standardised protocol built around fixed environment sets, explicit training budgets, repeated independent runs, periodic evaluation, and robust uncertainty quantification (Gorsane et al., 2022). Its recommended defaults are $t$6 timesteps for off-policy algorithms, $t$7 for on-policy algorithms, $t$8 independent runs, $t$9 evaluation episodes per interval, and evaluation every 10,000 timesteps. It further recommends return plus environment-specific metrics, normalised absolute performance, interquartile mean with stratified bootstrap confidence intervals, probability of improvement, performance profiles, and sample efficiency curves. This provides an AOEP-style template for versioned benchmark suites, fixed resource budgets, and statistically standardized reporting.

6. Limitations, open problems, and broader significance

AOEP-v0 is explicitly a pilot, and the survey is careful about its current limits (Ding et al., 29 Jun 2026). The pilot covers nine synthetic fault patterns plus a small number of realistic episodes and real traces; it is not a broad benchmark across many domains or vendors. The results for Mem0-style systems are configuration-level findings rather than proofs about all extracted-fact systems in principle. The survey also identifies a missing ablation: adding a minimal governance envelope—such as permission epochs, deletion ledgers, and trust tiers—to an extracted-fact store and rerunning AOEP.

The scope of properties is also deliberately limited. AOEP-v0 does not test liveness properties such as progress or absence of deadlock, does not attempt to score overall safety policies beyond the five invariants, and does not evaluate proactive governance at scale. It focuses on whether the required governance state exists and whether state transitions preserve specified invariants. That leaves open questions about cross-surface enforcement across memory, tools, logs, and shared state; shared-memory rollback in multi-agent settings; prospective governance, including when an agent should surface or enforce obligations unprompted; and integration with broader formal frameworks.

A further practical assumption is observability. AOEP-v0 assumes that the system under test can be probed sufficiently to reconstruct which records it believes are live or deleted, current permission epochs, pending conflicts, and rollback ledgers. Parametric memory in model weights is difficult to probe in this way, so AOEP-v0 is more naturally applied to explicit external state. The protocol also does not yet report recovery cost metrics such as mean time to recovery or state loss, even though the survey identifies those as open needs.

Its broader significance lies in changing what is counted as correct behavior in persistent-state systems. Instead of asking only whether an answer is good, AOEP-v0 asks whether state was written under valid authority, organized without scope expansion, preserved with provenance, forgotten with deletion propagation, and acted upon with rollback traceability. The survey therefore connects always-on agents to databases, distributed systems, formal methods, capability security, and machine unlearning. In that sense, AOEP-v0 is both an evaluation protocol and a governance specification: a concrete mechanism for testing whether an always-on system’s state evolves in a way that remains auditable, recoverable, and normatively controlled over time.