---
title: SeeAct Baselines in Autonomous Testing
url: https://www.emergentmind.com/topics/seeact-baselines
type: topic
---

# SeeAct Baselines in Autonomous Testing

SeeAct baselines refer to a class of autonomous end-to-end testing agents designed by minimally adapting the SeeAct Autonomous Web Agent framework for the purpose of benchmarking Large Language Model (LLM)-powered Autonomous Test Agents (ATAs). The SeeAct-ATA baseline, as introduced in “Are Autonomous Web Agents Good Testers?” [2504.01495], represents a monolithic, prompt-centric approach for web application test script execution and serves as a point of comparison for subsequent, more modular ATA architectures.

## 1. Architectural Foundation of SeeAct-ATA

SeeAct-ATA originates from SeeAct’s single-prompt design, collapsing the classical AWA modules—profile, memory, planning, and action—into a single prompt-driven LLM invocation. The execution workflow comprises several stages:

- **Initialization:** The agent loads a natural-language test case (comprising a sequence of ordered steps with optional explicit assertions) and resets the application browser to the home page.
- **Observation:** After each agent action, a screenshot and optionally the DOM are captured for perceptual context in subsequent LLM prompts.
- **Prompting:** The LLM receives a prompt embedding the entire test case, a running history of previous actions, the most recent screenshot, and explicit, structured instructions. These instructions enforce step tracking, assertion enumeration and verification ({VERIFIED, NOT VERIFIED}), and request the next action (including element selection via multiple-choice, action type among {CLICK, TYPE, SELECT, PRESS ENTER, TERMINATE, NONE}, and optional value input).
- **Execution:** Actions proposed by the LLM are executed via a Playwright-based interaction engine that attempts to ground the referenced element literal in the DOM, then the success or failure of each is recorded.
- **Loop:** The cycle repeats until the LLM outputs TERMINATE or selects NONE, or cannot locate a required DOM element.

All scenario logic, including next-action selection, assertion control, and test step progress tracking, is mastered by the LLM within the prompt. There is no separation between planning, grounding, or assertion modules.

## 2. Decision Process and Control Flow

The control logic for SeeAct-ATA can be abstracted by the following high-level pseudocode (as stated in [2504.01495]):

```python
load(TEST_CASE)
previous_actions ← []
while True:
    screenshot ← capture_screenshot()
    prompt   ← build_prompt(TEST_CASE,
                            previous_actions,
                            screenshot)
    response ← LLM(prompt)
    (element, action, value) ← parse(response)
    if action == TERMINATE or element == NONE:
        verdict ← PASS if all assertions VERIFIED else FAIL
        break
    success ← execute_action(element, action, value)
    previous_actions.append((element, action, value, success))
    if current_step_assertion NOT VERIFIED:
        verdict ← FAIL; break
return verdict
```

Logical constraints such as enforcing ordered steps and verifying intermediate outcomes are governed entirely through LLM prompt instructions, without recourse to explicit external planners or assertion modules.

## 3. Benchmark Composition and Evaluation Criteria

For empirical assessment of SeeAct-ATA, a comprehensive benchmark was constructed consisting of three offline web applications, each containerized via Docker to ensure reproducibility:

| Application     | Domain                    | Passing Tests | Failing Tests |
|-----------------|--------------------------|--------------|--------------|
| Classifieds     | Marketplace (~65k items) | 15           | 15           |
| Postmill        | Social forum              | 18           | 16           |
| OneStopShop     | E-commerce                | 29           | 20           |

A total of 62 passing test cases were generated by four ISTQB-certified testers using a two-column step/assertion format. Failing cases (n=51) were created by single-point perturbations in passing scripts (e.g., targeting a non-existent element), yielding 113 test scenarios in total. Success requires strict in-order action execution and correct, complete assertion verification; omission or mis-verification constitutes failure.

## 4. Quantitative Performance Metrics

Test outcomes were compared to human verdicts and classified as true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). The principal metrics are:

- $\mathrm{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$
- $\mathrm{Specificity} = \frac{TN}{TN + FP}$
- $\mathrm{Sensitivity} = \frac{TP}{TP + FN}$
- Step Mismatch Error Rate: $\mathrm{SMER}$
- True Accuracy: $\mathrm{TrueAcc}$

SeeAct-ATA (GPT-4o, temperature=0) achieves the following averaged results:

| Metric        | Classifieds | Postmill | OneStopShop | Average |
|---------------|-------------|----------|-------------|---------|
| Accuracy      | 0.50        | 0.56     | 0.59        | 0.55    |
| Specificity   | 0.33        | 0.67     | 0.76        | 0.59    |
| Sensitivity   | 0.67        | 0.44     | 0.35        | 0.48    |
| SMER          | 0.45        | 0.20     | 0.17        | 0.28    |
| TrueAcc       | 0.20        | 0.47     | 0.53        | 0.40    |

A plausible implication is that monolithic LLM approaches, while functional for simple scenarios, have significant reliability limitations in assertion validation and step order enforcement.

## 5. Comparative Analysis: SeeAct-ATA vs. PinATA

PinATA, a more advanced modular ATA, decomposes agent responsibilities into orchestrator, actor, and assertor components:

- **Planning with feedback** enables explicit step ordering and the possibility of action retries.
- **Set-Of-Marks grounding** and direct DOM/visual inspection provide richer action-to-element mapping.
- **Dedicated assertion module** re-invokes the LLM as an independent judge over screenshots.

PinATA's metrics (also on GPT-4o) reflect substantial improvements:

- Accuracy = 0.71 (+29 percentage points)
- Sensitivity = 0.88 (+40 points)
- SMER = 0.11 (–60%)
- TrueAcc = 0.61 (+53%)

On Postmill, PinATA achieves 94% sensitivity and approximately 60% overall true accuracy—a relative improvement of nearly 50% compared to SeeAct-ATA.

## 6. Fundamental Limitations and Recommendations for Robustness

SeeAct-ATA’s monolithic prompt architecture is characterized by:

- **Single-prompt reasoning** leads to frequent hallucination (HER ≈ 0.07), and erroneous step execution (SMER ≈ 0.28).
- **No explicit grounding** of action elements, increasing susceptibility to DOM mis-alignment errors.
- **Passive visual assertion control** resulting in mis-verification, particularly where content is dynamic or subtly altered.
- **Lack of recovery mechanisms**: Actions are executed in a single shot, precluding retry or rollback in the presence of execution failures.

To address these deficiencies and approach robust, low-maintenance test automation, recommended architectural enhancements include decomposition of planning/actuation/assertion roles, enhancement of DOM grounding, addition of assertion modules with action capabilities (e.g., scrolling, hover, direct DOM querying), and the adoption of structured scenario memory to enforce strict step-wise conformity and reduce premature side-effects [2504.01495]. This suggests that future research should prioritize modular agent designs and rich perception-grounded action mechanisms for more reliable LLM-powered autonomous test agents.

Source: https://www.emergentmind.com/topics/seeact-baselines