Understanding Automated Web GUI Testing: An Empirical Study Across Exploration Strategies and State Abstractions

Abstract: Automated web GUI testing (AWGT) relies on exploration strategies that exercise web applications through GUI actions to maximize code coverage, spanning traditional model-based, reinforcement learning (RL)-based, and emerging LLM-based approaches. State abstraction, which detects pages with the same functionality to avoid repeated testing, has long been recognized as critical to guiding exploration. However, how exploration strategies and state abstractions jointly affect testing effectiveness remains underexplored. We present an empirical study analyzing both factors from the perspectives of code coverage and failure revelation. We compare representative model-based, RL-based, and LLM-based approaches; investigate how six state abstractions influence model-based and RL-based approaches; examine LLM-based approaches under different history representations, which act as a form of state abstraction; and compare the failures exposed by different approaches. Our results show that no single strategy excels across all dimensions; instead, categories exhibit complementary strengths in code coverage, state coverage, and failure discovery. State abstraction is a key factor: strict, fine-grained abstractions favor model-based strategies, while compact ones better support RL-based strategies. History representation substantially affects LLM-based strategies, where concise, functionality-level context performs best. We also find that code coverage is weakly correlated with failure-revealing ability, underscoring the need for multi-dimensional evaluation. These findings offer practical guidance for selecting exploration strategies and designing effective state abstractions for AWGT.

• The paper demonstrates that no single exploration strategy dominates AWGT, with RL-based methods achieving higher coverage while LLM-based systems reveal unique failures.
• The paper reveals that precise state abstractions benefit model-based approaches, whereas compact abstractions are crucial for effective RL-based testing.
• The paper finds that using concise functionality-level history in LLM-driven testing improves coverage and failure detection by mitigating context overflow.

Empirical Analysis of Automated Web GUI Testing Across Exploration Methods and State Abstractions

Introduction

This paper conducts a rigorous empirical study examining how exploration strategies and state abstraction techniques impact the effectiveness of automated web GUI testing (AWGT). It systematically compares model-based, reinforcement learning (RL)–based, and LLM–based AWGT systems, integrating various state abstraction mechanisms and history representations. Evaluation dimensions include code coverage, state exploration, and failure revelation across six representative open-source web applications. The findings provide fine-grained insights into the dependencies and complementarity between exploration paradigms and abstraction strategies, as well as the crucial role of history representation in LLM-driven testing.

Figure 1: The AWGT workflow, illustrating cyclic exploration with state abstraction and action selection driven by the exploration strategy.

Exploration Strategies: Comparative Effectiveness

The comparative analysis targets three principal exploration modalities:

• Model-based: Exemplified by Crawljax and FragGen, which incrementally construct explicit state transition graphs, utilizing deterministic traversal algorithms (e.g., DFS).
• RL-based: Typified by WebExplor and WebRLED, which maintain state transition graphs and operate with Q-learning or DQN-driven action selection.
• LLM-based: Represented by GPTWeb (a port and extension of GPTDroid), leveraging LLMs for high-level understanding and interaction planning.

Quantitative results demonstrate that no single paradigm dominates across all coverage and fault discovery metrics. RL-based approaches achieve the best average code coverage across the suite, but model-based methods excel for certain application structures, notably those requiring robust shallow exploration and systematic component traversal. LLM-based GPTWeb lags in overall coverage due to low execution efficiency (substantial token and API overhead) and inherent LLM error modes (repetition, improper grounding), despite exhibiting competitive performance in reaching functionally deep states.

Figure 2: State transition graphs for each application, with coverage status visualized for each AWGT approach, highlighting differential exploration depth and breadth.

Notably, the RL-based methods frequently outperform on state reachability and dynamic interaction, but can suffer from reward bias and fail on highly combinatorial or form-centric interfaces (e.g., over-prioritization due to early rewards, combinatorial explosion in Pagekit setting screens). Model-based methods, due to their systematic exploration, cover broad functional areas but are constrained in sequential task complexity. LLM-based approaches uniquely access semantically deep or contextually complex states, given high-level function understanding, but are currently bottlenecked by efficiency and accuracy limitations.

The Impact of State Abstraction

Six abstraction mechanisms are evaluated, spanning string comparison, Gestalt pattern matching, robust tree edit distance (RTED), perceptual diff (PDiff), and two deep learning–driven methods (WebEmbed, Judge). Integration of these with Crawljax and WebExplor leads to several key observations:

• Model-based (e.g., Crawljax): Effectiveness is maximized with fine-grained and precise abstractions—Judge consistently produces the best coverage, closely followed by RTED—since over-fragmentation (e.g., via stringent string comparison) can yield redundancy, and over-aggregation (coarse abstraction) leads to premature exploration termination.
• RL-based (e.g., WebExplor): Superior results arise with compact, lower-dimensional state abstractions (Gestalt, WebEmbed), enabling robust Q-value learning, Q-table sharing, and stable policy updates. Excessive state space cardinality (fine-grained abstractions) is detrimental, mirroring the “curse of dimensionality” recognized in RL literature.

  Figure 3: Venn diagrams depicting the unique and overlapping failures detected by AWGT approaches and abstraction strategies, demonstrating their behavioral complementarity.

The empirical results concretely demonstrate the absence of a universally optimal abstraction: effectiveness is tightly coupled to the interaction between exploration policy and abstraction granularity. Moreover, abstraction method strongly modulates failure-discovery capability, with coarse abstractions (e.g., WebEmbed) exposing otherwise undetected failures due to broader behavioral grouping.

History Representation in LLM-based AWGT

For LLM-driven testing, state abstraction manifests as history contextualization—what execution trace and context are prepended to the LLM at each interaction. Four history strategies are rigorously assessed: no history, action history, state history, and functionality history, each with variable context window sizes.

• Functionality history: Compact, high-level summaries of covered functionalities drive maximal code coverage and expanded failure detection capacity, outperforming stateless or verbose (step/state-dense) configurations.
• History window size: There exists an optimal range—excessively long context (full histories) degrades LLM effectiveness and dramatically increases computational/inference cost, reflecting LLMs' limitations for long-sequence reasoning as noted in "Lost in the Middle" [LostInTheMiddle].

The results highlight that semantically rich, but succinct, context windows enable the LLM to avoid repetitive cycles and adaptively seek unexplored behavioral paths.

Failure-Revealing Capability

A core result is that code coverage and unique failure revelation are not strongly correlated. RL-based methods (especially with compact state abstractions) exhibit the broadest unique failure discovery, but LLM-based approaches uncover notably unique failures inaccessible to classical methods. This is most pronounced when GPTWeb is paired with functionality-level history, underpinning the value of semantic exploration in uncovering deep or rare bugs.

State abstraction selection fundamentally shifts the spectrum of revealed failures—the intersection between approaches is modest, and each paradigm exposes approach-specific failures. This validates a multi-strategy ensemble testing methodology for maximizing field bug discovery.

Implications and Future Trajectories

• Practical Guidance: For model-based AWGT, strict and effective state abstraction should be pursued, ensuring traversal depth and minimization of redundancy. RL-based testers must emphasize abstraction compactness for stable and effective value function learning. For LLM agents, concise, functionality-level historical context is critical—overly detailed per-step state representations induce inefficiency and context overflow.
• Tool Design: Architectural decoupling (planner-actor style) and multimodal input (screenshot-grounded) are affirmed as essential for scaling LLM-driven agents to web GUIs. The integration of abstraction "plug-ins" enables dynamic adjustment to application and exploration context.
• Theoretical Significance: The lack of direct correspondence between code coverage and failure-discovery further motivates development of multi-dimensional AWGT effectiveness metrics. The observed behavioral complementarity encourages future work on hybrid AWGT systems employing mixed strategies and abstraction schemas.

Prospective research should investigate LLMs with improved short-term memory and better context handling ("context chunking") and analyze exploration-abstraction co-adaptation via meta-learning regimes. The continued evolution of LLMs—especially in fast “vision-language-action” integration—may fundamentally shift current empirical trends.

Conclusion

This study provides a comprehensive empirical foundation for understanding the intertwined effects of exploration strategy and state abstraction in automated web GUI testing. There is no universally best approach; instead, optimality is context-dependent and best achieved through strategic integration and configuration, informed by application requirements and empirical evidence. State abstraction—both explicit and as realized through history representation in LLMs—emerges as a critical determinant for both coverage and failure discovery. The findings shape actionable guidance for AWGT tool designers and highlight open directions for advancing testing effectiveness in the presence of increasingly complex web interfaces and intelligent exploration agents.