SeeAct-ATA: LLM-Powered ATA Web Testing

• The paper introduces SeeAct-ATA, an LLM-powered autonomous test agent designed to execute, verify, and deliver strict web test case verdicts.
• It employs chain-of-thought reasoning and multimodal feedback from browser states to dynamically control test step execution and assertion verification.
• Evaluations on benchmark web applications reveal performance gaps with modular systems like PinATA, highlighting both its innovation and limitations.

SeeAct-ATA refers to a baseline implementation of an Autonomous Test Agent (ATA) for web application testing, where LLMs are used to autonomously execute, verify, and deliver verdicts for manual test cases on real applications. Developed as an adaptation of the SeeAct Autonomous Web Agent (AWA) architecture, SeeAct-ATA is distinguished by its direct prompt-driven strategy, specialized for strict test case conformity and assertion verification. It represents a new paradigm in low-maintenance test automation, aiming to alleviate the fragility of classic test scripts through natural language and multimodal LLM-powered reasoning.

1. Architecture and Operation

SeeAct-ATA is constructed as a prompt-centric agent utilizing a LLM to control browser interactions for test case execution. Its key components are:

• Prompt Engineering: The agent is instantiated with a prompt that:
  • Sets the "manual tester" role.
  • Incorporates the full natural language test case as the input task.
  • Contains explicit instructions for "Test Case Progress" (tracking step status: DONE, CURRENT, TODO) and "Test Step Assertion Control" (verifying each atomic assertion).
• Chain-of-Thought Reasoning: The LLM receives the latest browser state (screenshot and DOM), past actions, and the upcoming test step. It determines the next interaction, checks assertions, and provides step-by-step rationale.
• Interaction Loop: Browser automation frameworks (e.g., Playwright) execute LLM-specified actions, returning feedback to the LLM. Progress is tracked until all steps are completed or a failure condition occurs.
• Verdict Generation: The final verdict is assigned when test steps are completed and assertions pass, or when assertion or action failure is detected.

This design replaces conventional script fragility (which depends on brittle selectors) with semantic, multimodal reasoning based on natural language and observable states.

2. Evaluation Methodology and Performance Metrics

The evaluation of SeeAct-ATA was conducted using a benchmark composed of three offline web applications ("Classified," "Postmill," "OneStopShop") and a set of 113 manual test cases (62 passing, 51 failing), developed by expert testers.

Several quantitative metrics were computed, both standard and specialized for autonomous test agents:

Metric	Formula	Interpretation
Accuracy	$\frac{TP + TN}{TP + TN + FP + FN}$	Rate of correct verdicts
Sensitivity (Recall)	$\frac{TP}{TP + FN}$	Detecting failing test cases
Specificity	$\frac{TN}{TN + FP}$	Avoidance of false failure alerts
True Accuracy (TruAcc)	Defined in text	Correct verdict for correct reason

SeeAct-ATA achieved average accuracy of 55% with a True Accuracy of 40%. These outcomes were compared to PinATA—a multi-module, advanced ATA—which attained 71% accuracy and 61% True Accuracy, corresponding to a 50% improvement. Notably, PinATA demonstrated enhanced sensitivity to failing cases (0.88 vs. 0.48 for SeeAct-ATA) and similar specificity (up to 94% on certain test types).

Metrics like Automation Error Rate (AER), Hallucination Error Rate (HER), and Sum of Mismatch Errors (SMER) quantified sources of autonomous agent failure, including instances where correct verdicts were reached for incorrect procedural reasons (Type-III errors).

3. Limitations and Error Categories

The singular prompt-driven design of SeeAct-ATA renders it susceptible to several error modalities:

• Action Capacity/Versatility: Misidentification of interactive elements; failure to execute appropriate actions.
• User-Interface Observability: Inability to adequately track dynamic UI changes, resulting in outdated decisions.
• Assertion Verifiability: Errors in assessing assertions, especially in complex or ambiguous visual contexts.
• Test Case Conformity: Execution of steps out of prescribed order or premature anticipation, leading to sequence and logical errors.
• Type-III Errors (Editor's term): Cases where the agent delivers a correct verdict for the wrong underlying reasons, challenging the reliability of automation.

These limitations resulted in reduced accuracy, reliability, and generalizability, as evidenced by both quantitative outcomes and analysis of failure cases.

4. Role of LLMs in SeeAct-ATA

LLMs are the computational nucleus of SeeAct-ATA, orchestrating the entire reasoning and action process:

• Interpretation: The LLM receives screens, DOM states, action histories, and the test script, synthesizing these inputs to determine intent and requirements.
• Reasoning: Chain-of-Thought methods enable multi-step planning, continuous progression assessment, and assertive verification.
• Resilience: By grounding decisions in semantic understanding of test requirements, LLMs mitigate test script fragility and allow more tolerance to application layout modifications—the latter being a primary cause of failure in selector-driven paradigms.
• Maintenance: The design leverages generalizable natural language instructions and real-time perception, which reduces the necessity for brittle script maintenance.

A plausible implication is that the continued evolution of LLMs—particularly with multimodal and longitudinal context memory—may further reduce the need for manual interventions in practical automation scenarios.

5. Comparative Analysis and Improvement Strategies

Analysis exposes clear performance and robustness gaps between SeeAct-ATA and advanced ATAs such as PinATA:

• Modularity: PinATA divides labor between an orchestrator (enforcing sequence), actor (precise browser interaction), and assertor (specialized assertion verification). This modular decomposition addresses many failure modalities inherent in monolithic design.
• Action Planning: Incremental entry strategies and refined grounding mechanisms reduce action and assertion errors.
• Assertion Specialization: Dedicated modules or multimodal analysis (combining screenshot and DOM parsing) enhance assertion accuracy.
• Memory and Sequence Control: Maintaining longer-range state or expectation memory may enforce stricter adherence to test step order.

These improvement strategies are substantiated in the research findings, which suggest modular, specialized systems outperform prompt-only approaches by significant margins.

6. Future Directions and Research Opportunities

The SeeAct-ATA implementation and corresponding benchmark highlight several research tracks to improve autonomous testing:

• Enhanced Modularity and Role Separation: Refined architectures partition planning, action, and assertion responsibilities for greater reliability.
• Grounding and Visual Perception: Improved techniques for element identification and dynamic UI adaptation are needed; incorporating incremental marking and multimodal contextualization can address these gaps.
• Longitudinal Memory: Mechanisms to "remember" pending test steps can prevent premature actions and maintain strict test case conformity.
• Advanced Prompt Engineering: Exploring different LLM prompting paradigms, including multimodal DOM and visual cues integration.
• Benchmark Extension: The published benchmark invites community contributions for additional web applications and test suites, facilitating reproducible evaluation and progress.

A plausible implication is that continued interdisciplinary research—spanning LLMs, browser automation, and software testing—will drive robust, resilient, and low-maintenance test agent development.

7. Impact and Significance

SeeAct-ATA, as the baseline ATA implementation, encapsulates the transition from brittle, manually scripted automation toward resilient, LLM-driven test agents. By formalizing evaluation metrics and establishing a reproducible benchmark, this work provides a foundation for objective measurement and iterative improvement of autonomous testing methodologies. Although current limitations constrain reliability and domain generalization, advancements in modularity, multimodality, and LLM capabilities substantively address these gaps, advancing the state of test automation toward greater robustness and minimal maintenance (Chevrot et al., 2 Apr 2025).