Automated Web GUI Testing (AWGT)

• Automated Web GUI Testing (AWGT) is the automated exploration and validation of web applications through GUI interactions, leveraging formal models and modern learning methods.
• AWGT methodologies include model-based, reinforcement learning, and LLM-driven explorations to traverse complex page states and optimize fault detection.
• Key challenges in AWGT involve effective state abstraction, precise oracle construction, and cost-aware test execution, guiding future research directions.

Automated Web GUI Testing (AWGT) is the automated exploration and validation of web applications through GUI actions such as clicks and text inputs. In the recent literature, AWGT is framed both as an engineering problem of generating and maintaining executable GUI interactions and as a sequential decision problem over web states represented by DOM trees, screenshots, embeddings, or higher-level abstractions. Contemporary work spans model-based exploration, reinforcement learning, visual-language and LLM agents, multi-agent testing pipelines, and abstraction-based visual regression, with evaluation commonly centered on coverage, execution success, and defect revelation (Liu et al., 31 May 2025, Liu et al., 15 Jun 2026).

1. Formal problem setting

A recurrent formalization models AWGT as a Partially Observable Markov Decision Process $\mathcal{M}=(S,O,A,T,R)$, where the hidden state may consist of the full DOM tree plus a screenshot image, the observation includes a screenshot image and a DOM or accessibility tree, the transition function is unknown, and the reward is designed to encourage defect discovery. In the minimal action set adapted from GUI Testing Arena, the agent can emit Click(e), Type(e, "text"), Scroll(dir), Enter, Stop, and a no-op $\varnothing$ used in data-model tuples (Zhao et al., 2024).

RL-oriented work uses the closely related MDP formulation $$\langle \mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$$. In WebRLED, each state is a simplified, canonicalized HTML document encoded by the pre-trained Doc2Vec-based WebEmbed model into a fixed-length vector, the action space is derived from DOM-based controls and later augmented by a learned action discriminator, and the objective is to learn a policy maximizing the discounted return $G_t=\sum_{k=0}^{\infty}\gamma^k r_{t+k}$ with $\gamma=0.95$ (Gu et al., 27 Apr 2025).

A second common abstraction is the transition tuple $\tau=(s_{\rm before},a,s_{\rm after})$, which records each GUI interaction and its outcome. This tuple-level view supports stepwise task execution, localized defect detection, and benchmark construction in which episodes are sequences of such tuples rather than only final pass/fail outcomes (Zhao et al., 2024).

2. Exploration paradigms and state abstraction

The current literature distinguishes three high-level exploration paradigms. Model-based exploration constructs an explicit state-transition graph $G=(\mathcal{S},\mathcal{A},\delta)$ and systematically traverses unexplored GUI actions. RL-based exploration also maintains an explicit graph but learns a value function $Q:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$, typically with $\epsilon$-greedy action selection and rewards tied to novelty. LLM-based exploration treats the model as a policy $\pi_\theta$ over current page context plus a history representation, producing the next GUI action directly from multimodal prompt state (Liu et al., 15 Jun 2026).

State abstraction is central because it determines when two raw pages are treated as the same testing state. The empirical study across exploration strategies and state abstractions evaluates six techniques—String comparison (StringCmp), Gestalt pattern matching, Robust Tree Edit Distance (RTED), Perceptual Difference (PDiff), WebEmbed, and Judge—and reports that no single strategy excels across all dimensions; instead, categories exhibit complementary strengths in code coverage, state coverage, and failure discovery. It further finds that strict, fine-grained abstractions favor model-based strategies, compact ones better support RL-based strategies, and concise, functionality-level history performs best for LLM-based strategies. The same study also reports that code coverage is weakly correlated with failure-revealing ability, with Spearman’s $\varnothing$0 and $\varnothing$1 (Liu et al., 15 Jun 2026).

A source-level variant of model-based AWGT appears in LLMVue, where the target abstraction is an executable Page Transition Graph (PTG). The PTG is extracted from Vue Router ASTs and single-file components in three stages: static graph extraction, LLM-based semantic refinement, and graph normalization. The resulting graph $\varnothing$2 uses route paths as nodes and edges $\varnothing$3 with Playwright locator selector $\varnothing$4, event type $\varnothing$5, and optional guard condition $\varnothing$6, and is then used to guide Playwright-based exploration (Li et al., 26 Jun 2026).

3. Representing GUI state, layout, and navigation

AWGT systems differ markedly in how they represent the current GUI. PIRLTest defines a continuous embedding function $\varnothing$7 with $\varnothing$8 for a raw GUI screenshot $\varnothing$9. The embedding concatenates a widget-level representation derived from VGG-16 image and mask features plus a 14-class widget type vector and a layout embedding produced by serializing a four-level tree and passing it through an LSTM and FC layer. Because PIRLTest never reads Android XML layouts or the HTML DOM and detects widgets purely by computer vision on screenshots, it is explicitly platform-independent across mobile and web (Yu et al., 2022).

WebRLED uses a different representation regime. It simplifies the HTML by removing head, script, and style tags and duplicate subtrees, then embeds the resulting canonicalized DOM with WebEmbed. To address action misalignment, it partitions the rendered page into an $$\langle \mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$$0 grid with $$\langle \mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$$1, learns Q-values over cells rather than page-specific action indices, and scores a concrete action by summing nearby cell values within an upsampling radius set to $$\langle \mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$$2 cell width (Gu et al., 27 Apr 2025).

For visual regression and structural differencing, the AGS methodology introduces the Abstract GUI State as a tree-structured representation. Formally, an AGS $$\langle \mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$$3 is a list of elements $$\langle \mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$$4, where each element is a pair of an attribute set and a child-element list, and each attribute is a string key–value tuple. Equality and inequality are defined by structural recursion, and fairly-similar relations are used to pair elements across runs. This enables golden-master visual testing that reports attribute-level changes such as text, type, background-color, href, or onclick, rather than only pixel masks (Kraus et al., 2020).

The following table summarizes representative state or navigation abstractions reported in recent AWGT work.

Abstraction	Core representation	Reported role
PTG	Routes and navigation edges	Guided exploration in Vue.js apps
GUI embedding	Screenshot-derived vector $$\langle \mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$$5	Platform-independent exploration
WebEmbed + grid	Simplified DOM embedding plus $$\langle \mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$$6 grid	DQN-based action-value learning
AGS	Tree of GUI elements and attributes	Visual checkpoint comparison
History representation	No history, action history, state history, functionality history	Prompt state for LLM-based exploration

4. Test generation, execution, and oracle construction

A major line of work decomposes AWGT into specialized stages. SpecOps provides a four-phase pipeline consisting of test case generation, environment setup, test execution, and validation, each handled by a distinct LLM-based specialist agent. In the generation phase, a Test Architect produces an environment recipe, a natural-language test prompt, and an oracle specification; a Test Analyst then reflects on prompt completeness, feasibility of environment setup, and oracle generality. The Infrastructure Manager translates the environment recipe into API calls or database scripts and validates each setup step. The Engineer Specialist drives a real browser instance through universal UI primitives exposed via the MCP interface, verifying text appearance before each keystroke and disambiguating duplicate labels before each click. The validation phase combines an Investigator, which probes the live application or backend, with a Judge, which correlates visual evidence, log traces, and backend findings against the oracle and reports a bug when behavior deviates unreasonably from the oracle, the UI reports a misleading or incorrect status, the intended outcome is incomplete or of poor quality, or the user would need undue intervention to complete the task. To reduce hallucinations, the Judge is prompted with a Meta-Chain-of-Thought that first generates targeted sub-questions and then answers them (Ahmed et al., 10 Mar 2026).

SpecOps also formalizes end-to-end metrics. For a given approach over $$\langle \mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$$7 test cases, planning accuracy is measured by incorrect-step rate $$\langle \mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$$8 and missing-step rate $$\langle \mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$$9. Execution success rate is $G_t=\sum_{k=0}^{\infty}\gamma^k r_{t+k}$0. Bug-detection effectiveness uses $G_t=\sum_{k=0}^{\infty}\gamma^k r_{t+k}$1, $G_t=\sum_{k=0}^{\infty}\gamma^k r_{t+k}$2, and $G_t=\sum_{k=0}^{\infty}\gamma^k r_{t+k}$3 (Ahmed et al., 10 Mar 2026).

WebTestPilot addresses the oracle problem more directly. It introduces a symbolization layer that extracts strongly typed symbols from screenshots and DOM trees through schema-based extraction calls, maintains a global symbol set $G_t=\sum_{k=0}^{\infty}\gamma^k r_{t+k}$4, and translates a natural-language requirement into a sequence of steps each with condition_NL, action_NL, and expectation_NL. For each step $G_t=\sum_{k=0}^{\infty}\gamma^k r_{t+k}$5, the framework defines preconditions $G_t=\sum_{k=0}^{\infty}\gamma^k r_{t+k}$6 and postconditions $G_t=\sum_{k=0}^{\infty}\gamma^k r_{t+k}$7 over the current symbol state, allowing temporal, causal, and data dependencies to be expressed explicitly. Step success is defined by

$G_t=\sum_{k=0}^{\infty}\gamma^k r_{t+k}$8

This formulation is designed to capture implicit requirements that cannot be validated from isolated page states alone (Teoh et al., 12 Feb 2026).

VETL uses a different oracle-construction pathway centered on LVLM scene understanding. It detects an empty text widget, frames it in red, extracts page title, nearest visible text, widget attributes, and HTML constraints, then prompts LLaVA-7B to generate a single context-valid input string. It subsequently frames every clickable element in blue boxes, asks the LVLM to select one numbered target via visual question answering, and delegates action prioritization to a curiosity-driven multi-armed bandit whose reward is the number of newly discovered interactive elements on the resulting page (Wang et al., 2024).

Temac places these ideas inside a staged testing campaign. It first runs an existing AWGT approach broadly, then, when coverage stagnates, constructs a knowledge base from screenshots, HTML, state transitions, low-coverage files, and application-specific facts. A Summarizer produces natural-language state descriptions, a Reviser infers not-covered functionalities as ranked testing tasks, a Navigator selects the most relevant key state, and an Executor uses a planner–actor decomposition to carry out the task from that state toward deeper functionality (Liu et al., 31 May 2025).

5. Evaluation regimes and reported results

Recent AWGT papers use heterogeneous but increasingly formalized metrics. GUI Testing Arena defines Coverage for test intention generation, Type Match (TM), Exact Match (EM), and Success Rate (SR) for task execution, and Accuracy plus recall on defect and no-defect cases for defect detection. LLMVue evaluates PTG quality with Precision, Recall, $G_t=\sum_{k=0}^{\infty}\gamma^k r_{t+k}$9, and Graph Edit Distance (GED). WebTestPilot reports Task Completion (TC), Correct Trace (CT), and step-level bug-detection precision and recall. SpecOps reports planning accuracy, execution success, and bug-detection effectiveness (Zhao et al., 2024, Li et al., 26 Jun 2026, Teoh et al., 12 Feb 2026, Ahmed et al., 10 Mar 2026).

System	Evaluation setting	Reported result
SpecOps (Ahmed et al., 10 Mar 2026)	Five diverse real-world agents	164 true bugs; Precision $\gamma=0.95$0; Recall $\gamma=0.95$1; $\gamma=0.95$2; $\gamma=0.95$30.73$\gamma=0.95$4t<8$$minutes</td> </tr> <tr> <td>WebTestPilot (<a href="/papers/2602.11724" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Teoh et al., 12 Feb 2026</a>)</td> <td>Four bug-injected web apps</td> <td>Task completion rate 0.99; bug detection precision 0.96; recall 0.96</td> </tr> <tr> <td>LLMVue (<a href="/papers/2606.27665" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Li et al., 26 Jun 2026</a>)</td> <td>Ten open-source Vue.js projects</td> <td>Avg. Precision 94.57$$\gamma=0.95$5 91.25%; Avg. GED 16
PIRLTest (Yu et al., 2022)	Five web apps	After 1.5 h: 29.40% line, 13.25% branch; after 500 actions: 31.04% line, 19.18% branch
VETL (Wang et al., 2024)	Four benchmark sites and ten commercial sites	8.1 visited states vs 3.7; 52.05 discovered actions vs 41.65; mean state coverage 0.11 vs 0.07
WebRLED (Gu et al., 27 Apr 2025)	12 open-source apps and Alexa-50	64.2% branch coverage; 49.5% line coverage; 695 unique failures
Temac (Liu et al., 31 May 2025)	Six complex web apps and Top 20 sites	Average line coverage 49.71%; improvements 12.5% to 60.3%; 445 unique faults
AGS visual testing (Kraus et al., 2020)	20 websites with 8 simulated changes each	Overall precision 79.5%; overall recall 92.1%

The reported results also illustrate that “testing effectiveness” is not a single quantity. WebRLED emphasizes branch and line coverage plus console failures, WebTestPilot emphasizes requirement completion and oracle correctness, LLMVue emphasizes model quality of the extracted PTG, and AGS emphasizes reliable detection of structural GUI changes. This suggests that cross-paper comparisons are informative primarily at the level of methodological trade-offs rather than as a single leaderboard.

6. Cost, maintenance, limitations, and research directions

Cost and runtime now appear as first-class AWGT metrics. SpecOps defines per-test model cost as

$\gamma=0.95$6

and reports $\gamma=0.95$70.73$\gamma=0.95$8t<8$\gamma=0.95$9\$\tau=(s_{\rm before},a,s_{\rm after})$0 per 1 h run on GPT-4o. VETL notes that LVLM queries remain costly and that parallelism and batching are unexplored. PIRLTest reports that embedding every page via VGG and LSTM is slower than pure random. WebRLED reports training cost of approximately 4 minutes offline for the discriminator and notes that manual login and input value specification are still required. LLMVue identifies LLM cost and latency as limitations and recommends caching prompts and responses and batching calls (Ahmed et al., 10 Mar 2026, Liu et al., 31 May 2025, Wang et al., 2024, Yu et al., 2022, Gu et al., 27 Apr 2025, Li et al., 26 Jun 2026).

Longer-term economics depend on maintenance. The ROI study on CANEA ONE compares Selenium and EyeAutomate using recorded implementation and maintenance time over $\tau=(s_{\rm before},a,s_{\rm after})$1 selected versions. It defines implementation cost $\tau=(s_{\rm before},a,s_{\rm after})$2, cumulative AGT cost $\tau=(s_{\rm before},a,s_{\rm after})$3, and manual cost $\tau=(s_{\rm before},a,s_{\rm after})$4. Selenium required 2,284.9 min for initial implementation and 467.5 min cumulative maintenance; EyeAutomate required 1,194.4 min implementation and 682.8 min maintenance. Under weekly manual testing, EyeAutomate reached breakeven after approximately 25 versions and Selenium after approximately 43 versions. The same study concludes that implementation time is the leading cost for introducing AGT and that exploratory/manual GUI testing remains necessary for new features and usability issues (Dobslaw et al., 2019).

Several limitations recur across the literature. WebTestPilot identifies the implicit oracle inference challenge and the probabilistic inference challenge, both arising from the fact that LLM-based testers must often act as their own oracle under probabilistic reasoning (Teoh et al., 12 Feb 2026). VETL reports occasional misformatted outputs from an unfine-tuned LVLM and a stateless MAB that does not model temporally extended testing policies (Wang et al., 2024). PIRLTest points to false positives and false negatives in Canny-based widget detection and to state-similarity assumptions that may over-merge visually similar pages with different dynamics (Yu et al., 2022). Temac notes LLM inference latency, sequential knowledge-base construction, and the less-than-100% success rate of LLM actions (Liu et al., 31 May 2025). GUI Testing Arena reports that even the most advanced models struggle to perform well across all sub-tasks of automated GUI testing, with closed-source VLLMs outperforming open-source models by large margins and defect detection accuracy remaining low, which the authors interpret as a significant gap between current autonomous GUI testing capabilities and practical, real-world applicability (Zhao et al., 2024).

Proposed future work is correspondingly diverse: source-level PTG extraction for frameworks beyond Vue.js, stronger grounding of DOM element references, synthetic defect generation at scale, modern object detectors for widget extraction, video-oriented embeddings for dynamic pages, multi-agent or hierarchical RL, and tighter integration of visual and structural abstractions (Li et al., 26 Jun 2026, Zhao et al., 2024, Yu et al., 2022, Gu et al., 27 Apr 2025, Kraus et al., 2020). Collectively, these directions indicate that AWGT is evolving from isolated script automation toward a broader research area in which exploration policy, state abstraction, oracle inference, visual understanding, and cost-aware deployment are treated as co-equal design variables.