AgentWatcher: Monitoring Autonomous Agents

• AgentWatcher is a role-based monitoring framework that observes agent states and behaviors to support coordination, safety, and attribution.
• It employs a multi-stage architecture combining observation, interpretation, and decision-making to derive actionable insights from diverse agent interactions.
• Applications range from browser-based co-workers that build organizational memory to security systems that intercept threats with high precision.

Searching arXiv for papers mentioning “AgentWatcher” and closely related monitoring frameworks. AgentWatcher is a name that has been used in several adjacent strands of agent research to denote a monitoring layer placed around, beside, or above an autonomous agent system. Across these usages, the common denominator is not a single canonical implementation but a recurring architectural role: an AgentWatcher observes agent state, behavior, or environment-facing effects; derives structured judgments about coordination, safety, provenance, or attribution; and can support intervention, diagnosis, or persistent organizational memory. In the recent browser-agent literature, the term denotes a browser-based co-worker that both executes long-horizon web tasks and learns organizational context by observing users’ browsing behavior (Oyamada et al., 4 May 2026). In security-oriented work, AgentWatcher appears as a prompt-injection monitor (Wang et al., 1 Apr 2026), a decoupled runtime watcher for OpenClaw agents (Liu et al., 25 Mar 2026), a guardrail for tool-use interception inspired by runtime safety layers (Yang, 6 May 2026), and a Theory-of-Mind monitoring design for covertly malicious trajectories (Ahmed et al., 22 May 2026). In observability and web-security settings, the name is also used for telemetry-centric tracing and for server-side fingerprinting and attribution of autonomous web agents (AlSayyad et al., 7 Feb 2026, Kang et al., 18 Jun 2026). An earlier multi-agent lineage formalized AgentWatcher as socially-attentive monitoring for failures in team coordination (Kaminka et al., 2011).

1. Conceptual scope and recurring definition

The term “AgentWatcher” refers to a monitoring construct whose primary function is not task completion but supervision, interpretation, or augmentation of agent behavior. Depending on the system, the monitored object may be a browser agent’s interaction trajectory, an autonomous tool-use runtime, a multi-agent team, or a website visitor that may itself be an autonomous web agent. This suggests that “AgentWatcher” is best understood as a role-based designation for an oversight layer rather than as a single universally standardized framework.

Several distinct research programs instantiate that role differently. In “cotomi Act,” AgentWatcher embodies a browser-based co-worker with two complementary subsystems: an agent scaffold for reliable multi-step automation and a behavior-to-knowledge pipeline that watches user browsing and converts it into editable organizational artifacts (Oyamada et al., 4 May 2026). In “Robust Agent Teams via Socially-Attentive Monitoring,” the same label is used for an agent that monitors peer agents for breakdowns in social relationships such as teamwork agreement, mutual exclusion, role similarity, and formations (Kaminka et al., 2011). In “AgentTrace,” the name is applied to an observability-oriented monitoring framework grounded in structured logging over operational, cognitive, and contextual surfaces (AlSayyad et al., 7 Feb 2026). In ClawKeeper, AgentWatcher is the third-layer “watcher” process, fully decoupled from the task agent, responsible for real-time diagnosis and interruption of risky execution (Liu et al., 25 Mar 2026).

Across these variants, a common decomposition recurs. First, an observation mechanism captures evidence: DOM state, prompts, tool calls, file events, logs, or network and browser-interaction traces. Second, an interpretation mechanism transforms raw traces into higher-level state: plan hypotheses, risk scores, attributed context spans, behavioral knowledge artifacts, or classification features. Third, a decision mechanism emits outputs such as failure declarations, structured verdicts, knowledge updates, attribution labels, or execution-control signals. A plausible implication is that AgentWatcher denotes a general pattern for externalizing agent governance into explicit monitoring interfaces.

2. Browser co-worker and organizational knowledge accumulation

In the browser-agent setting, AgentWatcher is presented as the embodiment of cotomi Act’s central thesis: a browser-based co-worker can both execute complex, long-horizon web tasks and learn tacit organizational context by passively observing a user’s everyday browsing (Oyamada et al., 4 May 2026). The system has two major subsystems: an agent scaffold for execution and a behavior-to-knowledge pipeline for persistent organizational grounding.

The execution scaffold is a ReAct-style loop of Observation → Reasoning → Action augmented by four mechanisms. Adaptive Lazy Observation constructs a viewport view $\mathcal{V}_t \subset \mathcal{D}_t$ instead of always sending the full DOM tree, while permitting enrichment through inspect_full_page() and natural-language verbalDiff change descriptions. The reported effect is a reduction in per-step token cost by up to 2.6×. Verbal-Diff History Compression stores diffs rather than complete historical DOMs, with the history defined as $$H_t = \bigcup_{k=1}^{t-1} \text{verbalDiff}(\Delta_k)$$, yielding comparable accuracy at a 3.8× reduction in per-step token cost, from approximately 23.9 K tokens to approximately 6.3 K, alongside a reported accuracy change of +1.1 pp. Coarse-Grained Action Space replaces fine-grained primitives with six abstractions—click(element), type(element, text), scrollInto(element), selectOption(element, value), hover(element), and executeScript(code)—covering 98 % of common web actions and collapsing typical scroll sequences from approximately 10–20 fine steps into a single coarse action. Test-Time Best-of-$N$ Action Selection samples candidate actions and executes the majority-vote action, with

$$a_t^*=\underset{a}{\arg\max}\sum_{i=1}^N \mathbf{1}\bigl[a_t^{(i)}=a\bigr],$$

and, together with dynamic decomposition into parallel sub-agents, raises long-horizon task success by +3.9 pp (Oyamada et al., 4 May 2026).

The behavior-to-knowledge pipeline supplies the system’s organizational memory. Opt-in passive observation logs site-level events, tab changes, and periodic screenshots, with a diff-based filter removing repeated idle events. Every hour, or upon user idle, an LLM-orchestrated ETL pipeline performs segmentation into episodes $\{E_j\}$, summarization into structured records $(\text{actions},\text{screen},\text{outcome})$, and routing into three artifact classes: Task Board items, Wiki Pages, and Activity Timeline tags. These artifacts populate a shared browser-based workspace editable by both user and agent. When the agent needs context, it calls search_workspace(query) and retrieves top-$k$ matches from the typed knowledge store

$$\mathcal{W}=\{(\text{type},\text{id},\text{content},\text{metadata})\}.$$

Every agent-initiated update appears as a pending change requiring user confirmation, which the source explicitly frames as keeping agent knowledge aligned with reality (Oyamada et al., 4 May 2026).

This design connects reliable web automation with persistent, user-editable memory. The key technical claim is that watching user behavior is not merely a source of demonstrations for action imitation; it is a source of organizational abstractions that can later condition autonomous execution.

3. Monitoring paradigms: coordination, observability, and attribution

One major lineage of AgentWatcher concerns monitoring in multi-agent coordination. In the socially-attentive monitoring framework, AgentWatcher’s task is to infer when agents in a monitored team disagree about a team plan, using partial and uncertain information (Kaminka et al., 2011). The central formal problem is the Monitoring Selectivity Problem: how much peer-modeling effort is needed for reliable detection of teamwork failures while minimizing the number of monitored agents and the size of the team-modeling hypothesis set. The framework comprises four modules: a relationship-model knowledge base, agent/team modeling, failure detection, and diagnosis. The paper distinguishes a centralized scheme, which monitors all teammates and trades soundness against completeness via OPTIMISTIC and PESSIMISTIC disambiguation policies, from a distributed scheme in which each agent monitors only key agents and, under an observably-partitioned plan set, achieves both soundness and completeness with the OPTIMISTIC policy (Kaminka et al., 2011).

A second paradigm emphasizes structured telemetry and introspection. In AgentTrace, described as the framework behind a hypothetical AgentWatcher, the monitored agent is instrumented at runtime across three surfaces: operational, cognitive, and contextual (AlSayyad et al., 7 Feb 2026). Operational events capture method invocation lifecycles; cognitive events record prompts, Chain-of-Thought segments, plans, reflections, and token-level metadata; contextual events capture outbound HTTP, database, cache, and vector-store interactions. All are normalized into a common envelope

$$R \triangleq \{ id, surface, trace\_id, span\_id, timestamp, payload \},$$

allowing unified trace reconstruction. The framework relies on decorator-based wrappers, OpenTelemetry auto-instrumentation, JSONL sinks, and OTel spans, with reported per-call added latency on the order of a few hundred microseconds and a latency model $T_{\log}(n)=\alpha n+\beta n+\gamma$ where $$H_t = \bigcup_{k=1}^{t-1} \text{verbalDiff}(\Delta_k)$$0 ms (AlSayyad et al., 7 Feb 2026). Here AgentWatcher denotes observability as a prerequisite for accountability, debugging, risk analysis, and trust calibration.

A third paradigm treats AgentWatcher as an attribution system for web traffic. In the multi-layer fingerprinting work, AgentWatcher is a server-side monitoring framework that fingerprints visitors at the network protocol layer and the browser interaction layer, then classifies them as humans, legacy crawlers, or specific AI web agents via an Extra Trees classifier (Kang et al., 18 Jun 2026). The feature space includes approximately 70 TLS/HTTP features and approximately 108 browser-behavior features, spanning JA3/JA4 hashes, H2 prelude values, header ordering, Sec-Fetch logic violations, request timing statistics, event counts, mouse trajectories, keyboard patterns, scroll behavior, and tab-navigation style. On 330 labelled trials under 30-fold leave-one-trial-out cross-validation, the combined-feature model achieves accuracy $$H_t = \bigcup_{k=1}^{t-1} \text{verbalDiff}(\Delta_k)$$1, precision $$H_t = \bigcup_{k=1}^{t-1} \text{verbalDiff}(\Delta_k)$$2, recall $$H_t = \bigcup_{k=1}^{t-1} \text{verbalDiff}(\Delta_k)$$3, and $$H_t = \bigcup_{k=1}^{t-1} \text{verbalDiff}(\Delta_k)$$4, with averaged one-vs-rest ROC AUC greater than 0.98 for all classes (Kang et al., 18 Jun 2026).

These paradigms share a methodological core—hypothesis formation from traces—but differ in the object of monitoring. Socially-attentive monitoring observes peer coordination; observability frameworks observe the internals and side effects of an agent runtime; attribution frameworks observe traffic emitted by agents interacting with third-party websites.

4. Security-oriented AgentWatcher systems

A substantial 2026 usage of AgentWatcher is explicitly security-oriented. One such system is the rule-based prompt-injection monitor that wraps an LLM-based agent in two phases: attribution and monitoring (Wang et al., 1 Apr 2026). The agent state is formalized as

$$H_t = \bigcup_{k=1}^{t-1} \text{verbalDiff}(\Delta_k)$$5

and, before action $$H_t = \bigcup_{k=1}^{t-1} \text{verbalDiff}(\Delta_k)$$6, AgentWatcher identifies a short attributed substring $$H_t = \bigcup_{k=1}^{t-1} \text{verbalDiff}(\Delta_k)$$7 from the accumulated context and passes $$H_t = \bigcup_{k=1}^{t-1} \text{verbalDiff}(\Delta_k)$$8 together with an explicit rule set $$H_t = \bigcup_{k=1}^{t-1} \text{verbalDiff}(\Delta_k)$$9 to a monitor LLM. The attribution step computes per-token importance from attention weights and selects top-$N$0 non-overlapping windows using a sliding-window score. The rules enumerate what does and does not constitute prompt injection, including instruction diversion, system-prompt override, secret exfiltration, malicious tool use, forced output changes, direct attention calls, unrelated queries, redirection to untrusted sources, and system-level messages masquerading in context. Tool-use-specific rules additionally define benign cases. The output is a reasoning trace plus a binary verdict, and a “Yes” verdict halts the agent and returns the injected instruction (Wang et al., 1 Apr 2026).

Another security architecture is the ClawKeeper watcher layer, in which AgentWatcher is a separate OpenClaw process connected to the task agent through a WebSocket channel (Liu et al., 25 Mar 2026). It monitors streamed JSON events containing timestamps, prompts and outputs, tool invocations, and file-access metadata. Its state update is written as $N$1, and a risk function $N$2 combines matched malicious patterns and tool-chaining penalties. Decisions are thresholded into CONTINUE, ASK_USER, or STOP, with the task agent plugin enforcing the watcher’s control signal. The source stresses strict architectural separation, continuous oversight of internal state evolution, real-time intervention, human-in-the-loop enforcement, and self-evolving safety knowledge (Liu et al., 25 Mar 2026).

A related tool-interception design appears in the AgentTrust-inspired recipe. There AgentWatcher is a drop-in guardrail placed between an agent runtime and tool APIs, organized as a linear pipeline: ShellNormalizer → ActionAnalyzer → PolicyEngine → Interceptor, with optional SafeFix, SessionTracker (RiskChain), and LLMJudge components (Yang, 6 May 2026). The ShellNormalizer applies nine text-level normalization strategies to shell payloads; the ActionAnalyzer extracts risk factors via approximately 42 regexes; the PolicyEngine evaluates approximately 170 configurable rules; SafeFix suggests safer alternatives without changing the verdict; RiskChain matches order-aware multi-step attack patterns; and the LLM-as-Judge is invoked only for ambiguous cases. Final verdicts are ordered as block > review > warn > allow, risks as none < low < medium < high < critical, and the confidence is assigned by a step function based on which subsystems fired (Yang, 6 May 2026).

The enterprise ADR system uses yet another AgentWatcher framing, now tied to MCP-mediated enterprise agents (Li et al., 17 May 2026). Its ADR Sensor reconstructs agent sessions from local tool caches into JSON-L telemetry; ADR Detector uses a two-tier online pipeline with a fast triage stage and a deeper context-aware reasoning stage using dynamic MCP providers such as get_source_code(tool), get_threat_framework(), and assess_policy_violations(); and ADR Explorer performs offline evolutionary red-teaming over seed attacks with fitness

$N$3

Here AgentWatcher denotes an enterprise-scale detection-and-response architecture integrating observability, online defense, and hard-example generation (Li et al., 17 May 2026).

Taken together, these systems show that security-oriented AgentWatcher designs can intervene at different semantic layers: context interpretation, runtime event monitoring, pre-execution tool interception, or enterprise telemetry analysis.

5. Theory-of-Mind and learning-to-monitor formulations

A more explicitly cognitive formulation appears in Agent-ToM, where AgentWatcher is described as a monitoring framework grounded in Theory-of-Mind reasoning for detecting covert malicious behavior in full agent trajectories (Ahmed et al., 22 May 2026). The monitored object is a full trajectory

$N$4

and the system comprises four tightly coupled modules: a Belief Inference Module, an Intent Hypothesis Generator, an Expected-Action Modeler, and a Behavior Deviation Detector.

Belief-state estimation maintains a posterior over a belief space $N$5 via a Bayes-style recursion. Intent hypotheses $N$6 receive calibrated confidences $N$7 via softmax or logistic calibration. Expected actions are predicted under each $N$8 pair, and the observed action is compared against the expected benign action through a deviation metric $N$9, aggregated into a trajectory-level deviation score

$$a_t^*=\underset{a}{\arg\max}\sum_{i=1}^N \mathbf{1}\bigl[a_t^{(i)}=a\bigr],$$0

At inference time, AgentWatcher runs a two-call Reason–Verify–Refine pipeline: the Reason stage infers beliefs, intents, expected actions, and initial threat labels; the Verify stage re-evaluates the same trajectory with “second-pair-of-eyes” logic; and the Refine stage fuses outcomes into a final label, confidence, and justification (Ahmed et al., 22 May 2026).

Training adds a persistent semantic guardrail memory built from critique-driven distillation. Critiques identify which components led to success or failure; recurring patterns are turned into reusable abstract rules; and memory can be predefined or self-structured, with additive growth over training episodes. This is a significant departure from per-trajectory monitors that do not learn. A plausible implication is that AgentWatcher in this formulation is not merely a filter or rule checker but a continually improving monitor whose state includes reusable knowledge about deception strategies and malicious objectives.

6. Quantitative results across major AgentWatcher instantiations

The literature associated with the AgentWatcher name reports quantitatively strong results, although the benchmarks and target tasks differ substantially.

System / setting	Reported result	Context
cotomi Act scaffold +TTS	80.4 %	WebArena 179-task human subset (Oyamada et al., 4 May 2026)
cotomi Act base scaffold	76.5 %	Same benchmark (Oyamada et al., 4 May 2026)
Human baseline	78.2 %	Reported WebArena baseline (Oyamada et al., 4 May 2026)
OpAgent	74.9 %	Prior state of the art on same subset (Oyamada et al., 4 May 2026)
Prompt-injection AgentWatcher	ASR reductions to 0.01, 0.00, 0.02 in benchmark-specific settings	AgentDojo / InjecAgent / WASP (Wang et al., 1 Apr 2026)
Prompt-injection AgentWatcher	all long-context ASRs reduced to $$a_t^*=\underset{a}{\arg\max}\sum_{i=1}^N \mathbf{1}\bigl[a_t^{(i)}=a\bigr],$$1	Six long-context datasets (Wang et al., 1 Apr 2026)
ClawKeeper overall	85–90% DSR in each threat category	140 adversarial instances across seven categories (Liu et al., 25 Mar 2026)
ClawKeeper watcher self-evolution	DSR from ~90% to ~95%	100 novel adversarial examples (Liu et al., 25 Mar 2026)
Agent-ToM-style AgentWatcher	$$a_t^*=\underset{a}{\arg\max}\sum_{i=1}^N \mathbf{1}\bigl[a_t^{(i)}=a\bigr],$$2	SHADE-Arena, Stage 1+2, 2 calls (Ahmed et al., 22 May 2026)
Agent-ToM-style AgentWatcher	$$a_t^*=\underset{a}{\arg\max}\sum_{i=1}^N \mathbf{1}\bigl[a_t^{(i)}=a\bigr],$$3	OOD transfer to CUA-SHADE-Arena (Ahmed et al., 22 May 2026)
Web-agent fingerprinting AgentWatcher	0.971 accuracy, 0.972 precision, 0.971 recall, 0.971 $$a_t^*=\underset{a}{\arg\max}\sum_{i=1}^N \mathbf{1}\bigl[a_t^{(i)}=a\bigr],$$4	Combined features, 330 trials (Kang et al., 18 Jun 2026)
AgentTrust-inspired AgentWatcher	95.0% verdict accuracy	300-scenario internal benchmark (Yang, 6 May 2026)
AgentTrust-inspired AgentWatcher	96.7% verdict accuracy	Independent 630-scenario benchmark (Yang, 6 May 2026)
ADR	Precision 1.000, Recall 0.667, $$a_t^*=\underset{a}{\arg\max}\sum_{i=1}^N \mathbf{1}\bigl[a_t^{(i)}=a\bigr],$$5	ADR-Bench (Li et al., 17 May 2026)
ADR	Precision $$a_t^*=\underset{a}{\arg\max}\sum_{i=1}^N \mathbf{1}\bigl[a_t^{(i)}=a\bigr],$$6, Recall 1.000, $$a_t^*=\underset{a}{\arg\max}\sum_{i=1}^N \mathbf{1}\bigl[a_t^{(i)}=a\bigr],$$7	AgentDojo (Li et al., 17 May 2026)

In the browser co-worker line, the reported result of 80.4 % on the WebArena 179-task human-evaluation subset is notable because it exceeds the reported 78.2 % human baseline, and the proxy study shows task success increasing from approximately 51 % at zero domain coverage to approximately 61 % at full coverage, with a concrete “service catalog” slice improving from 75 % to 99 % after adding 48 scripts (Oyamada et al., 4 May 2026). In prompt-injection monitoring, the emphasis is on low Attack Success Rate with minimal clean-utility loss, summarized in the source as near-zero ASR and at most a 2% utility drop (Wang et al., 1 Apr 2026). In Theory-of-Mind monitoring, the key trade-off is precision–recall balance per LLM call budget; the two-call memory-enhanced version outperforms several higher-cost baselines (Ahmed et al., 22 May 2026).

The metrics are not directly comparable across papers because they evaluate different threat models, substrates, and operational points. Nonetheless, the aggregate picture is that AgentWatcher-style monitoring can be competitive both as a capability augmenter and as a safety or security layer.

7. Limitations, design tensions, and open problems

Despite these strong results, the underlying papers identify a consistent set of unresolved issues. In the browser co-worker formulation, limitations include domain generalization, because the coarse-action inventory and diff heuristics were tuned on enterprise-style UIs; privacy and compliance, since even opt-in PII-masked recording requires provenance, encryption at rest, and audit trails in regulated settings; noise and drift in user logs; long-term memory and forgetting as $$a_t^*=\underset{a}{\arg\max}\sum_{i=1}^N \mathbf{1}\bigl[a_t^{(i)}=a\bigr],$$8 grows; and collaborative and multi-agent extensions involving shared workspaces and conflict resolution (Oyamada et al., 4 May 2026).

In the prompt-injection monitor, the main limitations are runtime cost, false positives when benign instructions resemble malicious ones, and incomplete robustness against adaptive attackers such as PAIR and TAP; the text explicitly notes open questions around worst-case ASR bounds under adaptive multi-step manipulation and around the trade-off between rule granularity and interpretability (Wang et al., 1 Apr 2026). In the OpenClaw watcher architecture, limitations include dual-agent overhead, privacy concerns in cloud deployment, reliance on plugin instrumentation with possible blind spots, and the need for careful tuning of $$a_t^*=\underset{a}{\arg\max}\sum_{i=1}^N \mathbf{1}\bigl[a_t^{(i)}=a\bigr],$$9 and $\{E_j\}$0 (Liu et al., 25 Mar 2026). In the AgentTrust-inspired guardrail, the hybrid rules-plus-LLM path lowers false negatives but increases false positives and latency, illustrating the general tension between strictness and operational burden (Yang, 6 May 2026).

The observability and enterprise-detection papers identify additional scaling constraints. AgentTrace notes potential log volumes of millions of spans per day, fragility in cognitive extraction when LLM outputs do not conform to expected markers, and unresolved trace linking for multi-agent or long-running sessions (AlSayyad et al., 7 Feb 2026). ADR addresses scale through a two-tier detector but still incurs average latencies measured in seconds rather than milliseconds for deeper reasoning, and its production deployment emphasizes the importance of zero or near-zero false positives for analyst workflows (Li et al., 17 May 2026). The web-agent fingerprinting system, while highly accurate, explicitly anticipates evasion through randomized TLS features, header-order obfuscation, synthetic human-like interaction traces, and masked request timing; its countermeasures therefore involve continual drift monitoring, retraining, challenge elements, and adversarial augmentation (Kang et al., 18 Jun 2026).

An enduring conceptual tension is whether AgentWatcher should remain external and decoupled, as in ClawKeeper and socially-attentive monitoring, or become deeply integrated with agent internals, as in observability and Theory-of-Mind designs. Decoupling improves tamper resistance and separation of concerns; deep integration improves semantic access. Another unresolved question is whether the strongest future AgentWatcher systems will be rule-based, telemetry-based, learned, or hybrid. The available evidence suggests that all four paradigms remain active, and that the most capable systems already combine explicit structure with learned reasoning and memory (Ahmed et al., 22 May 2026).