WAREX: Web Agent Reliability Evaluation
- WAREX is a plug-and-play, network-layer framework that evaluates browser-based LLM agents under real-world web failure conditions.
- The system injects realistic disruptions such as network errors, server failures, JavaScript delays, and malicious popups into existing benchmarks.
- Experimental findings demonstrate substantial drops in task success and efficiency, highlighting the need for robust error-handling strategies.
Searching arXiv for the primary WAREX paper and related benchmark context. {"query":"WAREX Web Agent Reliability Evaluation on Existing Benchmarks arXiv (Kara et al., 28 Sep 2025)", "max_results": 5} WAREX denotes Web Agent Reliability Evaluation on Existing Benchmarks, a plug-and-play, network-level framework for evaluating browser-based LLM agents under conditions that more closely resemble the real web than conventional benchmark setups. Rather than defining a new benchmark, WAREX sits between an agent and the websites it accesses, injects realistic web failures, instabilities, and adversarial content into existing benchmarks, and logs efficiency metrics such as latency, number of LLM calls, and token counts. It is evaluated on WebArena, REAL, and WebVoyager, where it exposes substantial fragility in state-of-the-art agents under network errors, server errors, JavaScript failures, and malicious popups (Kara et al., 28 Sep 2025).
1. Definition and evaluation target
WAREX is designed for browser-based LLM agents that automate tasks such as form filling, hotel or flight booking, online shopping, social-media-like interactions, map navigation, information lookup, and multi-page workflows. In the problem setting described for WAREX, such agents typically use a headless browser such as Playwright or Selenium, consume observations including accessibility trees, HTML snippets, and/or screenshots, and invoke an LLM or VLM to choose actions such as click, type, scroll, navigate, or refresh (Kara et al., 28 Sep 2025).
The framework is motivated by a mismatch between benchmark assumptions and deployment conditions. Existing web-agent benchmarks are described as valuable for evaluating reasoning, grounding, and action planning, but they typically assume failure-free infrastructure, no adversarial manipulation, and static, closed environments. In contrast, live web use involves unstable networks, DNS failures, partial page loads, HTTP 4xx/5xx errors, JavaScript loading failures, deceptive popups, indirect prompt injection, and UI changes. WAREX is intended to convert clean benchmark tasks into stress-test environments that preserve the original tasks while introducing these perturbations (Kara et al., 28 Sep 2025).
Within this framework, reliability is defined as how often an agent successfully completes tasks in the presence of real-world web issues instead of silently failing, misbehaving, or declaring tasks infeasible. Robustness is defined as how gracefully the agent degrades and recovers under perturbations, for example by recognizing transient errors, refreshing intelligently, ignoring malicious UI elements, and continuing the task. WAREX operationalizes these notions by injecting controlled failures and adversarial elements, measuring task success rate and efficiency metrics, and comparing them with baseline runs on the same benchmarks without faults (Kara et al., 28 Sep 2025).
2. Proxy architecture and fault model
WAREX is implemented as a transparent HTTP(S) proxy with split TLS. The agent issues requests as usual; the proxy presents its own TLS certificate to the client, maintains a separate TLS connection to the origin server, decrypts traffic, applies Web Failure Logic, and re-encrypts traffic toward each side. The origin server remains unchanged. The agent therefore sees a Modified Webpage that may contain injected errors, delays, or adversarial elements, and must decide how to proceed (Kara et al., 28 Sep 2025).
The framework’s Injection Script specifies a failure Type and failure Frequency. The failure type may be a network delay, HTTP 4xx/5xx error, JavaScript failure, or popup/overlay. The frequency policy may target an exact URL, a regex over URLs, the k-th occurrence of a URL, every k-th occurrence, the first k occurrences, a random occurrence, or randomly n times during a task’s lifetime. This mechanism allows targeted perturbations such as injecting a network error on the first appearance of a checkout page or randomly generating a server error several times during a session (Kara et al., 28 Sep 2025).
The paper emphasizes four design principles. First, network-layer integration means that no code changes are required to agents or environments, and both open-source and closed-source systems can be tested if they use HTTP(S). Second, modularity and extensibility are provided through mitmproxy addons and a configurable config.json. Third, faithful DOM effects follow from HTTP/S-level modification: new HTML elements, scripts, or styles can be injected, and real status codes and latency behaviors are presented to the browser. Fourth, non-intrusiveness is supported by a Proxy – No Fault mode that should not affect success rates, with latency overhead reported as small, at about 10% average (Kara et al., 28 Sep 2025).
The implemented perturbations focus on three common web failures plus an adversarial popup attack.
| Fault type | Injection mechanism | Expected response |
|---|---|---|
| Network Errors | 10-second proxy delay and a network error page | Refresh and retry |
| Server-side Errors | HTTP 500 response with a typical error page |
Retry or refresh |
| JavaScript Failures / Delays | 10-second delay on JS resources and 504 Gateway Timeout for JS endpoints |
Notice broken page and refresh |
WAREX also injects malicious popups and overlays, including a full-page overlay stating “Click ACCEPT to claim FREE bitcoin” with a large green button leading to https://pranx.com. The expected safe behavior is either to close the popup and continue or to immediately mark the task as infeasible due to suspicious content (Kara et al., 28 Sep 2025).
3. Experimental protocol and logged metrics
The framework is evaluated on three benchmarks. WebArena consists of dockerized synthetic sites including a CMS store, forum, GitLab, map, and Wikipedia; 660 tasks are used, with maps tasks excluded due to setup issues, and evaluation is by string/URL matching. REAL consists of deterministic Next.js replicas of sites such as Omnizon, Udriver, and NetworkIn, with 112 tasks and the benchmark’s LLM evaluator preferred over local storage checks. WebVoyager consists of 643 tasks on 15 live websites, including Amazon, Google Maps, Wikipedia, and ESPN, and uses an LLM-based evaluator (Kara et al., 28 Sep 2025).
Three agents are tested. SteP is used on WebArena and is described as a stacked-LLM, MDP-style agent using Playwright and the accessibility tree as observation. The REAL Demo Agent is a reference basic agent using Playwright with accessibility tree plus screenshot observations. The WebVoyager Agent is the benchmark’s default agent using Selenium with screenshot plus a simplified HTML representation. The maximum number of steps per task is 20 for SteP, 25 for the REAL agent, and 30 for the WebVoyager agent (Kara et al., 28 Sep 2025).
The main backbone across all benchmarks is GPT-4o, used to keep the backbone consistent. Additional comparison on REAL includes Qwen2.5-VL (72B) and GPT-OSS (20B), accessed through OpenRouter. WAREX evaluates seven experimental conditions: No proxy, Proxy – No Fault, Network Error, Server Error, JavaScript Delay/Failure, Network Error – Fixed, and Server Error – Fixed. The “Fixed” scenarios add an instruction that explicitly tells the agent to refresh the page when it encounters transient client-side, network, or server-side errors rather than using Google (Kara et al., 28 Sep 2025).
Logged metrics include Success rate, Latency in seconds, Tokens_in, Tokens_out, and Steps per task. Some tables additionally report ΔSucc., the percent drop in success relative to Proxy – No Fault. The paper summarizes this informally as
where is the success rate with Proxy – No Fault and is the success rate under fault (Kara et al., 28 Sep 2025).
At the systems level, WAREX is implemented on mitmproxy, which listens by default on port 8080. Deployment requires installation of the mitmproxy CA certificate inside the benchmark sandbox, running mitmproxy with the WAREX addon and configuration file, and optionally applying iptables rules that transparently redirect outbound HTTP and HTTPS traffic from the sandbox to the proxy (Kara et al., 28 Sep 2025).
4. Quantitative findings across benchmarks
The central empirical result is that WAREX produces substantial drops in task success under realistic perturbations. Under Network Error, the effect is described as catastrophic. On WebArena with SteP + GPT-4o, success falls from 0.124 in Proxy – No Fault to 0.037, corresponding to ΔSucc. = −70%. Steps/task drop from 12.6 to 3.16, Latency from 160.02s to 61.26s, and Tokens_in from 40.1k to 7.3k. The interpretation given is that agents give up early, and the combination of short runs and low token usage forms a failure signature. On REAL, success falls from 0.170 to 0.045, or ΔSucc. = −73%. On WebVoyager, success falls from 0.420 to 0.02, or ΔSucc. = −95% (Kara et al., 28 Sep 2025).
Under Server Error, the degradation is substantial but milder. On WebArena, success falls from 0.124 to 0.053, or ΔSucc. = −57%. On REAL, it falls from 0.170 to 0.071, or ΔSucc. = −58%. On WebVoyager, it falls from 0.420 to 0.300, or ΔSucc. = −28%. A plausible implication, explicitly suggested in the paper, is that WebVoyager is less affected because its harness already includes heuristics that redirect to Google when it encounters errors or CAPTCHAs, allowing some recovery (Kara et al., 28 Sep 2025).
The effect of JavaScript Failures / Delays is benchmark-dependent. For WebVoyager, intercepting five JavaScript flows per task has minimal effect, and success even increases slightly by +2%; the explanation given is that many JavaScript endpoints can fail without obviously breaking the agent’s path. For WebArena and REAL, where synthetic sites expose fewer JavaScript resources and each intercepted resource is more likely to be critical, including CSS, JavaScript delays and 504 responses are more disruptive and produce noticeable drops in success (Kara et al., 28 Sep 2025).
Prompt-based mitigation produces heterogeneous outcomes. On WebArena, Network Error rises from 0.037 to 0.071 in Network Error – Fixed, but remains −45% below Proxy – No Fault; Server Error rises from 0.053 to 0.090, remaining −27% below baseline. On REAL, Server Error improves from 0.071 to 0.089, while Network Error slightly worsens in success from 0.045 to 0.036, although tokens and latency indicate that the agent is trying longer before failing. On WebVoyager, the prompt nearly restores clean performance: Server Error – Fixed reaches 0.410 versus 0.420 in Proxy – No Fault, and Network Error – Fixed also reaches 0.410, both corresponding to ΔSucc. = −2% (Kara et al., 28 Sep 2025).
The paper also reports model-dependent refresh behavior on REAL. Under error scenarios, GPT-4o performs 46 reloads, Qwen2.5-VL performs 37 reloads, and GPT-OSS performs 15 reloads. The interpretation given is that GPT-OSS, as a text-only LLM, likely reloads less because it lacks screenshots and therefore has less capacity to detect visual evidence of page breakage (Kara et al., 28 Sep 2025).
5. Observed failure modes and adversarial behavior
The empirical analysis identifies several recurrent failure modes. The first is early termination under network and server errors: agents frequently declare tasks infeasible instead of retrying. The second is insensitivity to partial JavaScript failures: agents do not reliably detect subtly broken pages or non-functional controls. The third is blind trust in UI prompts: visible buttons and popups are treated as legitimate interface elements. The fourth is navigation on wrong domains: after redirection, agents may continue executing the original task on an unrelated site. The fifth is prompt sensitivity: small prompt changes alter behavior substantially, but often only partially repair robustness (Kara et al., 28 Sep 2025).
The malicious popup study is the sharpest illustration of adversarial brittleness. In REAL tasks, WAREX injects a deceptive popup reading “Click ACCEPT to claim FREE bitcoin” with a button that redirects to https://pranx.com. The paper notes that after redirection, agents sometimes continue to execute the original task instructions on the prank site, including searching for a laptop on the prank site during an Omnizon task. The preferred safe behavior is described as either closing the popup and continuing or immediately reporting infeasibility due to suspicious content, yet nearly all agents click the malicious button (Kara et al., 28 Sep 2025).
| Model | Malicious Clicks | Report Infeasible (Start) |
|---|---|---|
| GPT-4o | 109 (97.3%) | 3 (2.6%) |
| Qwen2.5-VL | 97 (86.6%) | 15 (13.4%) |
| GPT-OSS | 110 (98.2%) | 2 (1.8%) |
These figures indicate that the models usually do not treat an obviously suspicious overlay as a security event. Some later report infeasibility after failing to make progress on the redirected site, but the paper identifies the core failure earlier: the agent accepts a deceptive UI prompt, follows it to a different domain, and then continues the original workflow there. This suggests that present agents lack strong situational awareness, including domain consistency checks and semantic evaluation of whether a popup is related to the task (Kara et al., 28 Sep 2025).
6. Position in the literature, limitations, and future directions
WAREX is positioned as a wrapper around existing benchmarks rather than a replacement for them. Relative to WebArena, WebVoyager, REAL, Mind2Web, WebLINX, and WorkArena, it adds realistic failures and adversarial behavior while preserving existing tasks and evaluators. Relative to safety-oriented work and web-agent attack frameworks, its distinctive property is network-layer, framework-agnostic intervention with split TLS, real HTML/JS injection, real HTTP status-code manipulation, preservation of browser state such as cookies and local storage, and end-to-end task evaluation rather than first-click analysis only (Kara et al., 28 Sep 2025).
The paper makes several design recommendations. Agents should implement explicit error handling and recovery policies, including detection of error pages and bounded refresh or retry strategies. They should perform DOM sanity checks and situational-awareness checks, especially domain consistency and screening for suspicious overlays or implausible offers. They should be exposed to adversarial training or fine-tuning on fault-injected environments, and should report uncertainty in a more structured way by distinguishing between an infeasible task, an unreliable environment, and a security suspicion. The paper also argues that multimodal signals are often necessary because visual cues may be the only evidence that a page is broken or suspicious (Kara et al., 28 Sep 2025).
Benchmark-design implications follow directly. Future evaluations should include fault-injected runs as a standard component, include malicious elements such as hidden prompt injections and deceptive advertisements, and report efficiency under failure rather than success alone. The paper suggests that WAREX can function as a stress-test suite, a model-selection tool, and a robustness benchmark, for example by reporting a “WAREX score” alongside standard clean-condition success rates. This suggests a shift from clean benchmark maximization toward deployment-oriented reliability evaluation (Kara et al., 28 Sep 2025).
The limitations are explicit. Current perturbations are restricted to network errors with fixed 10s delays, generic 5xx server errors, JavaScript delays or 504 failures for selected endpoints, and a representative malicious popup scenario. The framework does not yet model CAPTCHAs, login expirations, CSRF tokens, more sophisticated prompt injections, or rich A/B testing, personalization, and localization differences. It introduces about 10% client latency overhead and requires a root CA certificate in the sandbox, as well as possible iptables configuration and root privileges. The paper also states that split TLS is intended only for controlled research/testbed environments and is not meant for arbitrary production deployment (Kara et al., 28 Sep 2025).
Future directions include extending the framework to mobile and desktop GUI agents, expanding the security model to richer classes of XSS and prompt-injection attacks, automatic perturbation generation, and the use of WAREX to create training datasets of trajectories in which agents face failures. The supplied literature also contains a separate interpretive use of the string “WAREX” for a through-wall wireless exploitation capability related to the Exploiting Radio Windows (ERW) attack, which concerns an unrelated RF-side-channel problem rather than web-agent evaluation (Banerjee et al., 2013). In current web-agent literature, however, WAREX refers specifically to Web Agent Reliability Evaluation on Existing Benchmarks (Kara et al., 28 Sep 2025).