ArgusFleet: Evaluation Harness for Governed Shared Memory

• ArgusFleet is an open-source, reproducible evaluation harness designed to test governed shared memory in multi-agent LLM systems by exercising live REST API endpoints.
• It operationalizes four governance dimensions—scope, temporal supersession, provenance, and propagation—to rigorously measure memory correctness and policy compliance.
• The harness exposes critical issues like unauthorized leakage and pipeline-ordering conflicts, providing empirical metrics to validate multi-agent memory governance.

ArgusFleet is a reproducible evaluation harness for governed shared memory in multi-agent LLM systems, introduced in “Governed Shared Memory for Multi-Agent LLM Systems” (Margalit et al., 23 Jun 2026). It is not the memory service itself and is not described as a deployment environment; rather, it is an “open-source Python 3.12 evaluation harness” designed to exercise a live REST API and to test whether a governed memory service actually satisfies the governance properties its architecture claims. In the paper’s framing, ArgusFleet exists because once memory is shared across many LLM agents, memory ceases to be just a retrieval-quality issue and becomes a governed distributed-systems problem whose correctness depends on scope-sound visibility, temporal supersession of contradictions, provenance-preserving storage, and policy-bounded propagation (Margalit et al., 23 Jun 2026).

1. Identity and scope

ArgusFleet is introduced as the paper’s empirical instrument for turning the abstract “fleet-memory” problem into something testable against a live system. The paper is explicit that it is “a reproducible harness” and more precisely an “open-source Python 3.12 evaluation harness” containing four experiments, an async REST client, a typed domain model, structured event logging, and a reproducible reporting pipeline (Margalit et al., 23 Jun 2026). It therefore functions as an evaluation suite, but the paper’s own preferred term is “harness.”

Its target is governed shared memory for multi-agent LLM systems rather than generic retrieval benchmarking. The central claim is that in multi-agent deployments, correctness depends not only on whether a memory is semantically retrievable, but on whether the right agent can see it, whether stale or contradictory state is resolved, whether facts can be traced to their source, and whether knowledge propagates across agent boundaries according to policy. The paper argues that these properties cannot be established by architectural description alone; they must be exercised against running API paths where implementation asymmetries, latency windows, and pipeline-ordering interactions can create failures that the design does not predict away (Margalit et al., 23 Jun 2026).

ArgusFleet is therefore an external measurement layer. In the reported study it tests MemClaw, a multi-tenant production memory service with API operations for writing memories, searching, entity upsert, GET-by-id, and redistribution. The evaluation is “a measurement of one production service rather than a comparison against baselines,” which is methodologically important because the paper foregrounds real production behavior, including negative results, rather than comparative leaderboard-style claims (Margalit et al., 23 Jun 2026).

2. Formalization of fleet memory and governance invariants

The paper defines a fleet-memory system as

$F = (A, M, G, P, T),$

where $A$ is a set of interacting agents, $M$ is a shared memory substrate, $G$ is the governance and policy layer, $P$ is provenance metadata, and $T$ is temporal ordering and supersession semantics (Margalit et al., 23 Jun 2026). This formalization is the conceptual basis for ArgusFleet: shared memory must preserve correctness across interacting reads and writes by multiple autonomous actors over time.

A memory write is modeled as

$w_i = (a_i, c_i, s_i, t_i, p_i),$

with writer $a_i$, content $c_i$, visibility scope $s_i$, temporal ordering $A$0, and provenance $A$1. A retrieval is modeled as

$A$2

where $A$3 is the query, $A$4 the requesting agent, $A$5 governance constraints, and $A$6 temporal correctness conditions (Margalit et al., 23 Jun 2026). Retrieval correctness is therefore not just similarity-based; it additionally depends on authorization, freshness, contradiction resolution, provenance validity, and synchronization.

One governance invariant is made explicit and is central to ArgusFleet’s leakage tests. Let $A$7 mean that agent $A$8 is entitled under governance $A$9 to view a memory with scope $M$0. Retrieval is scope-sound iff

$M$1

In words, if a memory is returned to an agent, that agent must be authorized for that memory’s scope. The unauthorized-leakage failure mode is exactly a violation of this invariant (Margalit et al., 23 Jun 2026).

The paper organizes the problem into four governance dimensions—scope, time, provenance, and propagation—which correspond operationally to four failure modes: unauthorized leakage, stale propagation, contradiction persistence, and provenance collapse. In the system overview these answer who may read a memory, which version is current, where a memory came from, and how it moves across agent boundaries (Margalit et al., 23 Jun 2026).

Dimension	Failure mode	Operational question
Scope	Unauthorized leakage	Who may read a memory
Time	Contradiction persistence	Which version is current
Provenance	Provenance collapse	Where a memory came from
Propagation	Stale propagation	How it moves across agent boundaries

This four-part decomposition is what ArgusFleet operationalizes. A plausible implication is that the harness is best understood not as a retrieval benchmark, but as a governance benchmark for shared memory systems.

3. Harness architecture and reproducibility model

All four ArgusFleet experiments share the same lifecycle. A deterministic plan is generated from a random seed; writes are issued against the live MemClaw REST API; the harness waits through a settle delay to allow asynchronous enrichment or contradiction handling; then it performs verification against plan-derived ground truth (Margalit et al., 23 Jun 2026). The workloads are seeded so the fleet layout, agents, fact subjects, and chain structures are reproducible, while each run also includes a unique nonce in content to avoid collisions when rerunning against a stateful production service.

The harness is written in Python 3.12, uses an async client based on httpx, and uses tenacity for exponential backoff on transient transport errors, $M$2 responses, and $M$3 rate limits. Other $M$4 responses are surfaced unchanged because they are often the governance signal being measured (Margalit et al., 23 Jun 2026). Concurrency is capped per experiment to stay below the server’s documented rate limits, particularly $M$5 requests per minute on /search.

Its output is event-based. ArgusFleet emits canonical JSONL traces, plus CSV summaries and Matplotlib figures derived from those traces. The paper stresses that every number in the study is backed by committed traces in the repository, and that analysis can be rerun without hitting the server again (Margalit et al., 23 Jun 2026). The domain model uses plain dict events in traces but ships typed Pydantic models where useful, notably MemoryScope and PolicyDecision. Translation between harness events and MemClaw’s wire schema is kept in the client layer so that the harness can be retargeted to any “wire-compatible” governed-memory implementation.

These design choices make reproducibility independent of a fresh database snapshot. The deterministic plan and post-hoc trace analysis support exact reruns at the analysis layer, while the nonce mechanism avoids persistent-state collisions at the service layer. This suggests that the paper treats reproducibility as a property of workload specification and evidence preservation rather than as full environmental reset.

4. Experiments and workload design

ArgusFleet operationalizes the four governance dimensions through four concrete workloads (Margalit et al., 23 Jun 2026).

The leakage experiment creates $M$6 fleets with $M$7 agents each inside one tenant. The first agent in each fleet writes two secrets, one fleet-scoped and one agent-scoped. Each secret has a unique canary token in metadata and distinct prose in content. Every agent then probes every secret using semantic search with the secret’s prose as query. This creates $M$8 probes, with $M$9 expected-deny and $G$0 expected-allow cases. The harness records two separate signals: a leak, meaning the server returned a row the probe was not entitled to see, and a miss, meaning the server failed to return a row the probe was entitled to see. To isolate API-path behavior from the authentication model, the authors add a focused adversarial case with agent-scoped keys at different trust levels, where a trust-1 agent and a trust-2 agent bound to one fleet attempt direct GET-by-id on a row from another fleet (Margalit et al., 23 Jun 2026).

The contradiction experiment builds $G$1 facts from $G$2 subject-predicate templates. For each fact it runs two scenarios: sequential contradictory writes separated by a small delay, and concurrent contradictory writes dispatched together. In each pair, both writes share the same $G$3 and predicate but provide different $G$4. The workload deliberately uses RDF triples so the write should route into the server’s structural contradiction detector, not only into semantic deduplication. Verification re-fetches both row ids by GET after a settle window and checks whether supersession occurred: either the newer row carries $G$5 or the older row becomes non-active/outdated (Margalit et al., 23 Jun 2026).

The provenance experiment constructs $G$6 synthetic derivation chains of depth $G$7, for $G$8 total writes. Each chain follows an incident-style narrative with four stages: observation, hypothesis, mitigation, and verification. Each step writes a memory whose metadata contains $G$9 pointing to the parent memory. After the writes settle, the harness fetches the leaf and walks backward via GET to the root (Margalit et al., 23 Jun 2026).

The propagation experiment has two phases. In the rate phase, one writer agent writes $P$0 fleet-scoped facts, each using substantively distinct operational prose from $P$1 templates. Then $P$2 sibling readers in the same fleet and $P$3 non-fleet readers in the same tenant query for those facts, generating $P$4 total probes. Same-fleet readers poll for up to $P$5 seconds; foreign-fleet readers try only once, and any hit counts as a cross-fleet leak. In the separate window phase, the harness measures write-to-visible latency on $P$6 dedicated facts, each polled immediately and continuously from write completion at $P$7 ms intervals, one fact at a time, to stay below the per-tenant search rate limit. The propagation experiment also includes a redistribution test: a deterministic subset of facts is passed through POST /memories/redistribute to another agent, and the harness re-fetches the row to verify that $P$8 changed (Margalit et al., 23 Jun 2026).

5. Metrics and evaluation semantics

ArgusFleet’s metrics are partly formalized and partly defined operationally. For leakage, the paper gives

$P$9

A leak is an unauthorized return; a miss is an availability failure on an entitled read (Margalit et al., 23 Jun 2026). The distinction is important because the same retrieval path can fail as a security mechanism and as a recall mechanism.

For propagation, the harness records three tuple-level outcomes for each $T$0: visibility, time-to-visibility, and $T$1. The paper does not give an explicit symbolic formula for time-to-visibility, but operationally it is

$T$2

For provenance, the graded metrics are completeness, accuracy, and depth fidelity. These are given as criteria and can be written as

$T$3

$T$4

$T$5

For contradiction, the main metric is detection rate:

$T$6

with supersession evidence defined as

$T$7

The stale-read metric is

$T$8

These metrics reflect the paper’s broader claim that governed shared memory requires first-class empirical treatment of policy compliance, temporal supersession, provenance integrity, and synchronization behavior, rather than retrieval quality alone (Margalit et al., 23 Jun 2026).

6. Empirical findings, discovered failures, and significance

The strongest positive result is provenance. All $T$9 planned writes in the provenance workload succeeded, and all $w_i = (a_i, c_i, s_i, t_i, p_i),$0 depth-4 chains reconstructed completely. Chain completeness, writer-identity accuracy, and depth fidelity were all $w_i = (a_i, c_i, s_i, t_i, p_i),$1, with reported 95% Wilson intervals of $w_i = (a_i, c_i, s_i, t_i, p_i),$2. Per-hop fetch latency was p50 $w_i = (a_i, c_i, s_i, t_i, p_i),$3 ms, p95 $w_i = (a_i, c_i, s_i, t_i, p_i),$4 ms, p99 $w_i = (a_i, c_i, s_i, t_i, p_i),$5 ms. The authors note that this required throttling writes below MemClaw’s per-tenant write limit of 10/s; pacing at 8/s eliminated rate-limit induced chain loss (Margalit et al., 23 Jun 2026).

Propagation also produced strong results. In the rate phase, all $w_i = (a_i, c_i, s_i, t_i, p_i),$6 writes succeeded and all $w_i = (a_i, c_i, s_i, t_i, p_i),$7 visibility probes executed. Fleet-sibling visibility was $w_i = (a_i, c_i, s_i, t_i, p_i),$8, with 95% Wilson interval $w_i = (a_i, c_i, s_i, t_i, p_i),$9. Cross-fleet leakage was $a_i$0, with 95% Wilson interval $a_i$1. The dedicated window-phase measurement showed that under write_mode=strong, write-to-visible latency was effectively immediate: all 8 facts were visible on the very first poll after write completion, with p50 $a_i$2 s, p95 $a_i$3 s, and max $a_i$4 s. The paper interprets this as “one search round-trip” (Margalit et al., 23 Jun 2026).

Contradiction handling was more nuanced. Overall detection rate across $a_i$5 fact-runs was $a_i$6, but conditional on both contradictory writes actually being admitted by the write path, detection was $a_i$7, with 95% Wilson interval $a_i$8. The detector worked: after settle and re-fetch, the newer row superseded the older one and the older row flipped to outdated, typically within about six seconds. The low overall rate was instead driven by upstream write admission failures: $a_i$9 writes were rejected with HTTP $c_i$0 by the synchronous deduplication gate before the asynchronous contradiction detector could inspect them. Stale-read rate was $c_i$1, and strong-mode write latency in this experiment was p50 $c_i$2 ms, p95 $c_i$3 ms, p99 $c_i$4 ms across the $c_i$5 writes that completed (Margalit et al., 23 Jun 2026).

The paper’s most important negative result was leakage. On the semantic search path, leak rate over expected-deny cases was $c_i$6, and miss rate over expected-allow cases was $c_i$7. More serious was direct GET-by-id behavior. Using a tenant-scoped key, GET exposure on cross-scope rows was $c_i$8, with in-scope miss rate $c_i$9. The decisive result came from the focused agent-scoped adversarial probe: a trust-1 agent in fleet 0 could still GET a fleet-1 row even though the documented trust ladder says cross-fleet read requires trust $s_i$0. The handler checked only tenant equality and ignored fleet, agent, row scope, and trust level, thereby violating Inv-Scope (Margalit et al., 23 Jun 2026).

ArgusFleet localized this issue architecturally because it probed both search and GET paths and included the trust-level agent-key test. The gap was disclosed and remediated during the study. A scaled re-probe on the next day showed that trust-1 cross-fleet GETs were denied $s_i$1 times, trust-2 cross-fleet GETs were allowed $s_i$2, same-fleet controls were allowed $s_i$3, and total leaks in the re-probe were $s_i$4. The service now returns HTTP $s_i$5 on denied GETs so as not to disclose row existence (Margalit et al., 23 Jun 2026).

The second major negative result was the pipeline-ordering conflict between deduplication and contradiction resolution. The contradiction detector itself was sound, but semantic near-duplicate detection runs synchronously before commit and returns $s_i$6 when similarity is high, so the second write in a natural contradiction pair was often blocked before the contradiction detector could see it. The suggested remediation is either to reorder the pipeline so RDF-structured contradiction detection runs first, or to widen the dedup threshold for RDF-bearing writes because structure is the more informative signal (Margalit et al., 23 Jun 2026).

The paper is also candid about limitations. It is self-evaluation; the same group built MemClaw and ArgusFleet. The sub-tenant GET enforcement issue was established with a focused agent-scope probe rather than an exhaustive adversarial matrix. The experiments are single-tenant, workload scale is moderate, write-to-visible latency was measured on only $s_i$7 facts at $s_i$8 ms granularity, and there is no baseline comparand such as a long-context-only system or another memory service (Margalit et al., 23 Jun 2026).

Taken together, the reported study positions ArgusFleet as a harness for demonstrating that long-context retrieval alone is insufficient for production multi-agent memory, and that live evaluation is vital to expose enforcement and pipeline-ordering failures missed by design-only treatments. Its core contribution lies in making governed shared memory empirically falsifiable at the level of actual API behavior rather than architectural intent (Margalit et al., 23 Jun 2026).