SentinelBench: A Benchmark for Long-Running Monitoring Agents

Abstract: AI agents are increasingly asked to carry out work that spans minutes, hours, or longer. Yet the default model of agent behavior is continuous action: issuing tool calls, refreshing pages, searching for alternatives, or otherwise trying to force progress. This is the wrong approach for many long-running tasks, which are better served by a strategy of sustained attention. Instead, agents should monitor an environment, notice when an external event makes progress possible, then respond promptly without wasting resources while waiting. To measure progress on this class of tasks, we introduce SentinelBench, an open-source benchmark for time-evolving monitoring tasks. SentinelBench contains 100 tasks across 10 synthetic web environments, including email, calendars, finance, professional networking, and entertainment. Each environment exposes a live web interface and replays a scripted sequence of events, requiring agents to navigate and reason about web pages whose state shifts underfoot. SentinelBench measures task completion, reaction time, and resource use, exposing the tradeoff between responsiveness and cost. We report results across three models and two browser-agent harnesses, establishing performance baselines for future comparison and demonstrating how agent design choices can dramatically impact key metrics. Together, these results show that SentinelBench distinguishes meaningful differences in agent behavior.

• The paper introduces SentinelBench, a new benchmark for evaluating monitoring agents with a focus on reaction time, success rates, and resource costs.
• The benchmark employs synthetic web environments that simulate real-world applications such as email, finance, and social networking to rigorously test long-horizon tasks.
• The paper demonstrates that agent strategies like wait_for significantly reduce API costs and improve task performance compared to naive polling methods.

SentinelBench: Evaluating Long-Running, Monitoring-Oriented Agents in Synthetic Web Environments

Motivation and Problem Formulation

Current agent benchmarks typically evaluate performance in environments where agent actions exclusively influence environment state. However, many real-world tasks require agents not simply to act continuously, but to monitor for critical condition changes over extended time intervals, then respond efficiently and cost-effectively. SentinelBench directly targets this monitoring paradigm by introducing a benchmark for agents tasked with responding to time-evolving, externally-triggered events within synthetic but realistic web applications.

The authors delineate the mismatch between existing agent evaluation protocols and the requirements for long-running monitoring. Classic polling or continuous reactivity is suboptimal due to increased cost, failure to balance reactivity with efficiency, and decreased task completion rates as horizon length grows. SentinelBench provides the first standardized framework for quantifying these tradeoffs, supporting systematic measurement of agent performance along the axes of success rate, reaction time, and inference/resource cost.

Benchmark Design and Synthetic Environment Construction

SentinelBench comprises 100 tasks distributed across 10 synthetic web environments emulating major classes of productivity and consumer applications: email, calendar, finance, professional networking, code hosting, video streaming, music streaming, academic search, team messaging, and photo sharing. All environments are implemented as interactive web applications, populated with data generated from a pipeline combining LLM-based persona/entity construction and open-weight generative media models. The primary user persona is instantiated as a consistent, multimodal identity across all environments.

Figure 1: Four SentinelBench environments, each a synthetic application with a live interface and a monitoring task event sequence.

Tasks within SentinelBench are carefully constructed along two orthogonal axes: action requirement (passive, active, no-op) and criterion type (absolute vs. relative). Passive tasks require monitoring and notification, active tasks demand periodic probing and specific actions, while no-ops test for agent patience by never presenting the success condition. Task triggers are defined as either absolute (“when a star count hits 2,000”) or relative (“when the count increases by 200”), explicitly probing short-term versus accumulative memory and reasoning.

Task scripts define event schedules, enabling precise control over environment state transitions and ensuring experimental repeatability and scalability, including configurable “speed factors” for task duration up to 40 minutes.

Figure 2: SentinelBench leverages 100 synthetic user personas with per-environment coherence, supporting controlled evaluation while avoiding privacy issues and dataset drift.

The data generation pipeline integrates multimodal LLMs to synthesize not only structured catalog data (emails, papers, job listings), but also associated media assets using diffusion and generative audio/video models, ensuring fidelity and intra-persona consistency.

Figure 3: The media-generation pipeline ensures all synthetic users, media assets, and content are environment-coherent and reproducible via open-source generation methods.

Task Anatomy, Evaluation Protocol, and Metrics

Each task encompasses (1) a natural language prompt specifying the monitoring criterion, (2) environment snapshots and event timelines, and (3) a validity SQL program for automatic success adjudication. The evaluation protocol is agent-agnostic: any browser agent accepting prompt/URL input and terminating reliably may be benchmarked.

The simulation life cycle encompasses pre-initialization, ready (agent launch), running (event playback and environment evolution), and completion states, with precise transitions managed via REST APIs. Reaction latency, completion time, and resource metrics (token counts, API cost) are consistently measured, provided cooperative agents report token usage.

Figure 4: Anatomy of a SentinelBench task, with UI context, timeline, and event-trigger conditions directly controlling agent evaluation.

Figure 5: Detailed simulation life cycle schema for SentinelBench, enabling rigorous and repeatable agent interaction.

Agent Strategies and Baseline Analysis

Baseline experiments include three model classes (GPT-5.4, GPT-4o, Qwen 3.5:9B) and two principal agent polling strategies: naive polling using simple sleep, and content-aware polling using a wait_for(condition, timeout) tool. The latter leverages incremental page diffs and LLM-based heuristic event detection to minimize unnecessary tool iterations while maintaining responsiveness.

Figure 6: Side-by-side comparison of agent execution traces using naive polling (sleep) and event-driven waiting (wait_for), illustrating the consequences for efficiency and reactivity.

Comprehensive evaluation reveals several critical findings:

• Wait_for dramatically reduces API cost compared to sleep (up to $9.7\times$ at 40 min tasks), while maintaining or improving completion rates. For GPT-5.4, median per-task cost drops from $4.65 (sleep) to $0.48 (wait_for) at long horizons.
• Completion rates exhibit model dependence. GPT-5.4 achieves up to 75% task completion, substantially outperforming GPT-4o and Qwen 3.5:9B. However, tooling choices remain impactful even for state-of-the-art LLMs.
• Reaction times for wait_for are largely comparable or superior to naive polling for smaller models, with a small median penalty for GPT-5.4, but notable speedups for lower-throughput agents.
• Passive tasks are consistently easier than active tasks, and relative prompt phrasing increases agent error rates, particularly for smaller models.

Figure 7: API cost exhibits a linear relationship with event-wait duration for naive polling, in contrast to stable cost for wait_for, indicating substantial efficiency gains.

Figure 8: For wait_for, task completion times closely track the nominal event trigger, producing minimally delayed and seldom premature reactions in successful cases.

Theoretical and Practical Implications

These results offer direct empirical support for the proposition that agentic tool design has a first-order effect on the cost-effectiveness and reliability of long-horizon monitoring tasks, orthogonal to improvements in base model reasoning or action generation. The significant cost differential will likely shape production deployment architectures for ambient monitoring agents, as naive polling becomes increasingly impractical with rising environment time horizons and model inference pricing.

The naturalistic, synthetic environments and persona-driven tasks also facilitate direct transfer to simulation-based agent training, with repeatability and controlled event scheduling supporting reinforcement learning and robustness analysis. By decoupling environmental evolution from agent action, SentinelBench uniquely supports the investigation of when-to-act and how-to-wait, rather than just what-to-do, addressing a critical gap in current benchmarks.

From a theoretical perspective, SentinelBench provides a foundation for research directions in algorithmic patience, event-driven scheduling, cost-reward optimization over exogenous process monitoring, and compositional tool use. The benchmark exposes performance cliffs associated with action misalignment, context memory errors, and premature or excessively delayed reactions.

Future Work

SentinelBench represents only an initial step in benchmarking agentic patience and environment-driven event detection. Extensions should incorporate more realistic temporal distributions, introduce subjective and ephemeral triggering criteria, and enable scale-out for RL and continual learning in monitoring-centric scenarios. Additional challenge axes may include multi-agent coordination, adversarial event schedules, or occluded state dynamics.

Conclusion

SentinelBench provides a rigorous, controlled, and extensible benchmark for evaluating long-horizon monitoring agents in dynamic synthetic web environments (2606.05342). By systematically exposing the tradeoffs inherent in polling strategies and agent-environment synchronization, SentinelBench facilitates robust comparison across models and tool architectures. Its synthetic, reproducible framework establishes a reference for progress in agentic monitoring and promises to inform both research and deployment of cost-effective, responsive AI monitors in a wide variety of domains.