---
title: 'WAREX: Web Agent Reliability Evaluation'
url: https://www.emergentmind.com/topics/warex
type: topic
---

# WAREX: Web Agent Reliability Evaluation

Searching arXiv for the primary WAREX paper and related benchmark context.
{"query":"WAREX Web Agent Reliability Evaluation on Existing Benchmarks arXiv 2510.03285", "max_results": 5}
WAREX denotes **Web Agent Reliability Evaluation on Existing Benchmarks**, a **plug-and-play, network-level framework** for evaluating browser-based LLM agents under conditions that more closely resemble the real web than conventional benchmark setups. Rather than defining a new benchmark, WAREX sits between an agent and the websites it accesses, injects realistic **web failures**, **instabilities**, and **adversarial content** into existing benchmarks, and logs **efficiency metrics** such as latency, number of LLM calls, and token counts. It is evaluated on **WebArena**, **REAL**, and **WebVoyager**, where it exposes substantial fragility in state-of-the-art agents under network errors, server errors, JavaScript failures, and malicious popups [2510.03285].

## 1. Definition and evaluation target

WAREX is designed for **browser-based LLM agents** that automate tasks such as form filling, hotel or flight booking, online shopping, social-media-like interactions, map navigation, information lookup, and multi-page workflows. In the problem setting described for WAREX, such agents typically use a headless browser such as **Playwright** or **Selenium**, consume observations including **accessibility trees**, HTML snippets, and/or **screenshots**, and invoke an LLM or VLM to choose actions such as click, type, scroll, navigate, or refresh [2510.03285].

The framework is motivated by a mismatch between benchmark assumptions and deployment conditions. Existing web-agent benchmarks are described as valuable for evaluating reasoning, grounding, and action planning, but they typically assume **failure-free infrastructure**, **no adversarial manipulation**, and **static, closed environments**. In contrast, live web use involves unstable networks, DNS failures, partial page loads, HTTP 4xx/5xx errors, JavaScript loading failures, deceptive popups, indirect prompt injection, and UI changes. WAREX is intended to convert clean benchmark tasks into **stress-test environments** that preserve the original tasks while introducing these perturbations [2510.03285].

Within this framework, **reliability** is defined as how often an agent **successfully completes tasks** in the presence of real-world web issues instead of silently failing, misbehaving, or declaring tasks infeasible. **Robustness** is defined as how **gracefully** the agent degrades and **recovers** under perturbations, for example by recognizing transient errors, refreshing intelligently, ignoring malicious UI elements, and continuing the task. WAREX operationalizes these notions by injecting controlled failures and adversarial elements, measuring **task success rate** and **efficiency metrics**, and comparing them with baseline runs on the same benchmarks without faults [2510.03285].

## 2. Proxy architecture and fault model

WAREX is implemented as a **transparent HTTP(S) proxy with split TLS**. The agent issues requests as usual; the proxy presents its own TLS certificate to the client, maintains a separate TLS connection to the origin server, decrypts traffic, applies **Web Failure Logic**, and re-encrypts traffic toward each side. The origin server remains unchanged. The agent therefore sees a **Modified Webpage** that may contain injected errors, delays, or adversarial elements, and must decide how to proceed [2510.03285].

The framework’s **Injection Script** specifies a failure **Type** and failure **Frequency**. The failure type may be a network delay, HTTP 4xx/5xx error, JavaScript failure, or popup/overlay. The frequency policy may target an exact URL, a regex over URLs, the **k-th occurrence** of a URL, **every k-th occurrence**, the **first k occurrences**, **a random occurrence**, or **randomly n times** during a task’s lifetime. This mechanism allows targeted perturbations such as injecting a network error on the first appearance of a checkout page or randomly generating a server error several times during a session [2510.03285].

The paper emphasizes four design principles. First, **network-layer integration** means that no code changes are required to agents or environments, and both open-source and closed-source systems can be tested if they use HTTP(S). Second, **modularity and extensibility** are provided through mitmproxy addons and a configurable `config.json`. Third, **faithful DOM effects** follow from HTTP/S-level modification: new HTML elements, scripts, or styles can be injected, and real status codes and latency behaviors are presented to the browser. Fourth, **non-intrusiveness** is supported by a **Proxy – No Fault** mode that should not affect success rates, with latency overhead reported as small, at about **10% average** [2510.03285].

The implemented perturbations focus on three common web failures plus an adversarial popup attack.

| Fault type | Injection mechanism | Expected response |
|---|---|---|
| Network Errors | 10-second proxy delay and a network error page | Refresh and retry |
| Server-side Errors | HTTP `500` response with a typical error page | Retry or refresh |
| JavaScript Failures / Delays | 10-second delay on JS resources and `504 Gateway Timeout` for JS endpoints | Notice broken page and refresh |

WAREX also injects **malicious popups and overlays**, including a full-page overlay stating **“Click ACCEPT to claim FREE bitcoin”** with a large green button leading to `https://pranx.com`. The expected safe behavior is either to close the popup and continue or to **immediately mark the task as infeasible** due to suspicious content [2510.03285].

## 3. Experimental protocol and logged metrics

The framework is evaluated on three benchmarks. **WebArena** consists of dockerized synthetic sites including a CMS store, forum, GitLab, map, and Wikipedia; **660 tasks** are used, with maps tasks excluded due to setup issues, and evaluation is by string/URL matching. **REAL** consists of deterministic Next.js replicas of sites such as **Omnizon**, **Udriver**, and **NetworkIn**, with **112 tasks** and the benchmark’s **LLM evaluator** preferred over local storage checks. **WebVoyager** consists of **643 tasks** on **15 live websites**, including Amazon, Google Maps, Wikipedia, and ESPN, and uses an LLM-based evaluator [2510.03285].

Three agents are tested. **SteP** is used on WebArena and is described as a stacked-LLM, MDP-style agent using **Playwright** and the **accessibility tree** as observation. The **REAL Demo Agent** is a reference basic agent using **Playwright** with accessibility tree plus screenshot observations. The **WebVoyager Agent** is the benchmark’s default agent using **Selenium** with screenshot plus a simplified HTML representation. The maximum number of steps per task is **20** for SteP, **25** for the REAL agent, and **30** for the WebVoyager agent [2510.03285].

The main backbone across all benchmarks is **GPT-4o**, used to keep the backbone consistent. Additional comparison on REAL includes **Qwen2.5-VL (72B)** and **GPT-OSS (20B)**, accessed through **OpenRouter**. WAREX evaluates seven experimental conditions: **No proxy**, **Proxy – No Fault**, **Network Error**, **Server Error**, **JavaScript Delay/Failure**, **Network Error – Fixed**, and **Server Error – Fixed**. The “Fixed” scenarios add an instruction that explicitly tells the agent to refresh the page when it encounters transient client-side, network, or server-side errors rather than using Google [2510.03285].

Logged metrics include **Success rate**, **Latency** in seconds, **Tokens\_in**, **Tokens\_out**, and **Steps per task**. Some tables additionally report **ΔSucc.**, the percent drop in success relative to **Proxy – No Fault**. The paper summarizes this informally as
$$
\Delta\text{Succ.} = 100 \cdot \frac{S_f - S_0}{S_0}\,(\%)
$$
where \(S_0\) is the success rate with **Proxy – No Fault** and \(S_f\) is the success rate under fault \(f\) [2510.03285].

At the systems level, WAREX is implemented on **mitmproxy**, which listens by default on **port 8080**. Deployment requires installation of the mitmproxy CA certificate inside the benchmark sandbox, running mitmproxy with the WAREX addon and configuration file, and optionally applying `iptables` rules that transparently redirect outbound HTTP and HTTPS traffic from the sandbox to the proxy [2510.03285].

## 4. Quantitative findings across benchmarks

The central empirical result is that WAREX produces substantial drops in task success under realistic perturbations. Under **Network Error**, the effect is described as catastrophic. On **WebArena** with **SteP + GPT-4o**, success falls from **0.124** in **Proxy – No Fault** to **0.037**, corresponding to **ΔSucc. = −70%**. **Steps/task** drop from **12.6** to **3.16**, **Latency** from **160.02s** to **61.26s**, and **Tokens\_in** from **40.1k** to **7.3k**. The interpretation given is that agents **give up early**, and the combination of short runs and low token usage forms a failure signature. On **REAL**, success falls from **0.170** to **0.045**, or **ΔSucc. = −73%**. On **WebVoyager**, success falls from **0.420** to **0.02**, or **ΔSucc. = −95%** [2510.03285].

Under **Server Error**, the degradation is substantial but milder. On **WebArena**, success falls from **0.124** to **0.053**, or **ΔSucc. = −57%**. On **REAL**, it falls from **0.170** to **0.071**, or **ΔSucc. = −58%**. On **WebVoyager**, it falls from **0.420** to **0.300**, or **ΔSucc. = −28%**. A plausible implication, explicitly suggested in the paper, is that WebVoyager is less affected because its harness already includes heuristics that redirect to **Google** when it encounters errors or CAPTCHAs, allowing some recovery [2510.03285].

The effect of **JavaScript Failures / Delays** is benchmark-dependent. For **WebVoyager**, intercepting five JavaScript flows per task has **minimal effect**, and success even increases slightly by **+2%**; the explanation given is that many JavaScript endpoints can fail without obviously breaking the agent’s path. For **WebArena** and **REAL**, where synthetic sites expose fewer JavaScript resources and each intercepted resource is more likely to be critical, including CSS, JavaScript delays and `504` responses are more disruptive and produce noticeable drops in success [2510.03285].

Prompt-based mitigation produces heterogeneous outcomes. On **WebArena**, **Network Error** rises from **0.037** to **0.071** in **Network Error – Fixed**, but remains **−45%** below **Proxy – No Fault**; **Server Error** rises from **0.053** to **0.090**, remaining **−27%** below baseline. On **REAL**, **Server Error** improves from **0.071** to **0.089**, while **Network Error** slightly worsens in success from **0.045** to **0.036**, although tokens and latency indicate that the agent is trying longer before failing. On **WebVoyager**, the prompt nearly restores clean performance: **Server Error – Fixed** reaches **0.410** versus **0.420** in **Proxy – No Fault**, and **Network Error – Fixed** also reaches **0.410**, both corresponding to **ΔSucc. = −2%** [2510.03285].

The paper also reports model-dependent refresh behavior on REAL. Under error scenarios, **GPT-4o** performs **46 reloads**, **Qwen2.5-VL** performs **37 reloads**, and **GPT-OSS** performs **15 reloads**. The interpretation given is that GPT-OSS, as a **text-only LLM**, likely reloads less because it lacks screenshots and therefore has less capacity to detect visual evidence of page breakage [2510.03285].

## 5. Observed failure modes and adversarial behavior

The empirical analysis identifies several recurrent failure modes. The first is **early termination** under network and server errors: agents frequently declare tasks infeasible instead of retrying. The second is **insensitivity to partial JavaScript failures**: agents do not reliably detect subtly broken pages or non-functional controls. The third is **blind trust in UI prompts**: visible buttons and popups are treated as legitimate interface elements. The fourth is **navigation on wrong domains**: after redirection, agents may continue executing the original task on an unrelated site. The fifth is **prompt sensitivity**: small prompt changes alter behavior substantially, but often only partially repair robustness [2510.03285].

The malicious popup study is the sharpest illustration of adversarial brittleness. In REAL tasks, WAREX injects a deceptive popup reading **“Click ACCEPT to claim FREE bitcoin”** with a button that redirects to `https://pranx.com`. The paper notes that after redirection, agents sometimes continue to execute the original task instructions on the prank site, including searching for a laptop on the prank site during an Omnizon task. The preferred safe behavior is described as either closing the popup and continuing or **immediately** reporting infeasibility due to suspicious content, yet nearly all agents click the malicious button [2510.03285].

| Model | Malicious Clicks | Report Infeasible (Start) |
|---|---:|---:|
| GPT-4o | 109 (97.3%) | 3 (2.6%) |
| Qwen2.5-VL | 97 (86.6%) | 15 (13.4%) |
| GPT-OSS | 110 (98.2%) | 2 (1.8%) |

These figures indicate that the models usually do not treat an obviously suspicious overlay as a security event. Some later report infeasibility after failing to make progress on the redirected site, but the paper identifies the core failure earlier: the agent accepts a deceptive UI prompt, follows it to a different domain, and then continues the original workflow there. This suggests that present agents lack strong **situational awareness**, including domain consistency checks and semantic evaluation of whether a popup is related to the task [2510.03285].

## 6. Position in the literature, limitations, and future directions

WAREX is positioned as a wrapper around existing benchmarks rather than a replacement for them. Relative to **WebArena**, **WebVoyager**, **REAL**, **Mind2Web**, **WebLINX**, and **WorkArena**, it adds realistic failures and adversarial behavior while preserving existing tasks and evaluators. Relative to safety-oriented work and web-agent attack frameworks, its distinctive property is **network-layer**, **framework-agnostic** intervention with **split TLS**, real HTML/JS injection, real HTTP status-code manipulation, preservation of browser state such as cookies and local storage, and **end-to-end task evaluation** rather than first-click analysis only [2510.03285].

The paper makes several design recommendations. Agents should implement **explicit error handling and recovery policies**, including detection of error pages and bounded refresh or retry strategies. They should perform **DOM sanity checks and situational-awareness checks**, especially domain consistency and screening for suspicious overlays or implausible offers. They should be exposed to **adversarial training** or fine-tuning on fault-injected environments, and should report uncertainty in a more structured way by distinguishing between an infeasible task, an unreliable environment, and a security suspicion. The paper also argues that **multimodal signals** are often necessary because visual cues may be the only evidence that a page is broken or suspicious [2510.03285].

Benchmark-design implications follow directly. Future evaluations should include **fault-injected runs** as a standard component, include malicious elements such as hidden prompt injections and deceptive advertisements, and report **efficiency under failure** rather than success alone. The paper suggests that WAREX can function as a **stress-test suite**, a **model-selection tool**, and a **robustness benchmark**, for example by reporting a “WAREX score” alongside standard clean-condition success rates. This suggests a shift from clean benchmark maximization toward deployment-oriented reliability evaluation [2510.03285].

The limitations are explicit. Current perturbations are restricted to network errors with fixed **10s** delays, generic **5xx** server errors, JavaScript delays or `504` failures for selected endpoints, and a representative malicious popup scenario. The framework does not yet model CAPTCHAs, login expirations, CSRF tokens, more sophisticated prompt injections, or rich A/B testing, personalization, and localization differences. It introduces about **10% client latency overhead** and requires a **root CA certificate** in the sandbox, as well as possible `iptables` configuration and root privileges. The paper also states that split TLS is intended only for **controlled research/testbed environments** and is not meant for arbitrary production deployment [2510.03285].

Future directions include extending the framework to **mobile and desktop GUI agents**, expanding the security model to richer classes of XSS and prompt-injection attacks, automatic perturbation generation, and the use of WAREX to create training datasets of trajectories in which agents face failures. The supplied literature also contains a separate interpretive use of the string “WAREX” for a through-wall wireless exploitation capability related to the **Exploiting Radio Windows (ERW)** attack, which concerns an unrelated RF-side-channel problem rather than web-agent evaluation [1307.7233]. In current web-agent literature, however, WAREX refers specifically to **Web Agent Reliability Evaluation on Existing Benchmarks** [2510.03285].

Source: https://www.emergentmind.com/topics/warex