ABot-Explorer Research Framework

Updated 4 July 2026

ABot-Explorer is a research designation for automated exploration across interactive environments, including desktops, web agents, and embodied systems.
It leverages methods like screen similarity, action matching, and semantic navigational affordance to enable dynamic data collection and learning.
Variants of the system focus on GUI automation, large-scale web trajectory synthesis, and real-time embodied navigation with standardized benchmarking.

ABot-Explorer is a designation used in several recent lines of research on automated exploration of interactive environments. In one usage, it refers to Explorer, an end-to-end framework for collecting, modeling, and automating interactions with existing GUIs on desktop browsers and Android phones (Chaimalas et al., 12 Apr 2025). In another, it denotes an exploration-driven framework for synthesizing multimodal web trajectories and training web-task agents (Pahuja et al., 17 Feb 2025). A later work uses ABot-Explorer as the name of an active exploration framework for embodied agents that unifies memory construction and exploration into an online, RGB-only process (Chen et al., 21 Apr 2026). Related work on UI-exploration benchmarking further supplies a formal evaluation setting for such systems (Nica et al., 21 Jun 2025).

1. Nomenclature and research scope

Across the cited literature, the label appears in multiple but related contexts centered on exploration, state discovery, and interaction over complex interfaces or environments. The term is therefore not tied to a single canonical architecture.

Usage	Domain	Principal focus
Explorer	Existing GUIs	Interactable collection, Screen Similarity, Action Matching, trace recording and replication
Explorer	Web agents	Exploration-driven multimodal web trajectory synthesis and agent training
UIExplore-Bench	UI evaluation	Benchmarking exploration via UFO and hUFO
ABot-Explorer	Embodied agents	Online SNA distillation, SG-Memo construction, and greedy exploration

In the GUI-automation line, the system is explicitly described as “sometimes referred to as ‘ABot-Explorer’ in our internal notes,” and is organized around data collection plus GUI automation (Chaimalas et al., 12 Apr 2025). In the web-agent line, ABot-Explorer is used as a designation for “Explorer,” an exploration-driven framework for synthesizing large-scale multimodal web trajectories (Pahuja et al., 17 Feb 2025). In embodied navigation, ABot-Explorer is the published name of a framework that prioritizes Semantic Navigational Affordances and constructs a hierarchical SG-Memo online (Chen et al., 21 Apr 2026). This suggests that the most stable meaning of the term is methodological rather than product-like: exploration is treated as the primary mechanism for acquiring actionable structure.

2. GUI collection and automation

In the GUI setting, Explorer is an end-to-end framework for collecting, modeling, and automating interactions with existing GUIs on desktop browsers (Windows/Mac/Linux) and Android phones (Chaimalas et al., 12 Apr 2025). Its stated goals are to streamline high-quality data collection of “interactable” widgets on a live target application; train lightweight but accurate ML models that detect such widgets and identify when two screenshots belong to the same GUI state; record user demonstrations as sequences of discrete states plus actions; and support a proof-of-concept voice-driven interface with the commands “show,” “click <1–N>,” and “close.”

The data-collection pipeline has three acquisition modes. On Web pages it runs via Selenium; on Android via a Kotlin Accessibility Service; and on “general” platforms via pixel-difference hover sampling. The generalized hover-based procedure moves the mouse in a coarse grid of step size $h$ , captures “before” and “after” screenshots, forms a pixel-difference image, groups non-zero pixels by hash, and fits bounding boxes per hash. Website mode uses Selenium to extract each element’s XPath and CSS bounding box, then filters invisible or out-of-parent widgets and automatically labels buttons, inputs, and related elements. Android mode uses a foreground Accessibility Service in Kotlin that listens to taps and scrolls, waits $4\,\mathrm{s}$ for the screen to settle, screenshots the display, and traverses the Accessibility Tree via breadth-first search to get bounding boxes.

The modeling stack comprises three principal subsystems. The Interactable Detector is an FCOS anchor-free detector with a ResNet backbone using $C3$ – $C5$ , a Feature Pyramid Network using $P3$ – $P7$ , and centerness heads $Ctr3$ – $Ctr7$ . Because widget sizes vary by app and platform, fixed anchors are avoided. The Screen Similarity model is a Siamese network $G_W$ trained with the contrastive loss

$\mathcal{L} = (1-Y)\,\max\{0,\,m_n - d(E_1,E_2)\} + Y\,\max\{0,\,d(E_1,E_2)-m_p\}, \qquad Y\in\{0,1\}.$

Its input is not raw pixels but upsampled and concatenated FPN feature maps $4\,\mathrm{s}$ 0– $4\,\mathrm{s}$ 1 plus the corresponding centerness maps $4\,\mathrm{s}$ 2– $4\,\mathrm{s}$ 3, followed by a light-weight self-attention submodule. Action Matching combines Screen Similarity with an optional OCR-based text embedding:

$4\,\mathrm{s}$ 4

Explorer also records user demonstrations as traces over a directed graph of steady GUI screenshots. A trace is captured through high-frame-rate video plus low-level events such as clicks and keystrokes. The video is segmented into contiguous frame blocks whose pairwise Screen Similarity distance remains below a threshold, and the first frame after the triggering user action is retained as the representative screenshot. Loop and overlap detection compare each new representative screenshot against all existing state-nodes; if the distance is below $4\,\mathrm{s}$ 5, the system identifies it as the same node and adds an edge rather than a new node. Once the state graph is built, navigation reduces to shortest-path search, such as BFS or Dijkstra if transitions are weighted by click-cost.

The empirical setup targeted five GUIs: KhanAcademy, Wolfram Alpha, Wikipedia, Spotify on Android, and the built-in Android Telephone app. The collection produced “a few thousand annotated screens per app,” including 2.8 K for Khan, 1.1 K for Wolfram, and 1.2 K for Spotify, and several hundred same-state groups for Screen Similarity, including 378 for Khan. Reported interactable-detection results include 0.971 mAP on KhanAcademy for a scratch-trained FCOS, compared with 0.65 mAP for web7k $4\,\mathrm{s}$ 6Khan and 0.003 mAP for web7k-only; the Telephone app allowed VINS $4\,\mathrm{s}$ 7Phone to reach 0.965 mAP. For Screen Similarity, the FPN+centerness Siamese attained Test Accuracy 0.992 and F1 0.912 on Khan, and 0.963 and 0.778 on Spotify. In a proof-of-concept voice-navigation benchmark over seven multi-step KhanAcademy tasks, the detector fully completed 67% of steps without user intervention, while an OCR-only baseline reached 93% because nearly all targets were text-buttons. Trace replication from MacOS to Windows 11 matched 5/9 total steps, or 55.6% accuracy, with failures concentrated on tiny or heavily changed widgets. Limitations include the assumption that GUIs visually react to mouseover, heuristic pruning of occluded Android widgets, and the fact that downsampling to 320 px can break small-widget detection, while full-resolution mode runs at 7–12 FPS. The codebase and non-personal datasets are released openly.

3. Exploration-driven web trajectory synthesis

In the web-agent setting, Explorer addresses the shortage of diverse and large-scale trajectory-level datasets by synthesizing successful multimodal web trajectories through extensive web exploration and refinement (Pahuja et al., 17 Feb 2025). The released dataset contains over 94K successful multimodal web trajectories spanning 49K unique URLs, 720K screenshots, and 33M web elements. The reported average cost is 28 cents per successful trajectory.

The synthesis pipeline is multi-agent. A Task Proposer reads a homepage screenshot and accessibility tree and outputs a high-level intent together with a first atomic action. A Task Refiner conditions on all previous task-action pairs and current percepts, then proposes the next action and refines the task description. A Task Summarizer compresses the full action sequence and percepts into a human-readable task description. A Task Verifier checks whether the agent actually achieved that task on the real site; only trajectories marked “success” are stored. The perceptual substrate includes screenshots, HTML, accessibility trees, and set-of-mark annotated images, captured through Playwright-rendered interactions.

The data pipeline converted 175 K raw runs into 94 K successful, verified trajectories over 49 K unique real URLs. The successful trajectories have an average of 7.7 steps. Seed sites include the top 100 transactional sites from SimilarWeb plus 49 K Tranco sites, with filtering of pages that trigger CAPTCHA, logins, payments, or disallowed content. The cost model is explicitly decomposed as about 3.6 K tokens per step at \$4\,\mathrm{s}$80.0028; over an average 7.7-step trajectory this gives a raw-run cost of approximately \$4\,\mathrm{s}$90.28 per successful trajectory at a 53.1% success rate.

For model training, Explorer selects a subset of approximately 30 K runs with at most two scrolls and fine-tunes two small LMM backbones: Phi-3.5V, producing “Explorer-4B,” and Qwen2-VL-7B, producing “Explorer-7B.” The training loss is given as

$C3$0

with typical weights $C3$1 and $C3$2. Evaluation is reported on Mind2Web-Live, Multimodal-Mind2Web, and MiniWob++. On Mind2Web-Live, Explorer-7B reaches Step SR 45.3, Completion 40.2, Task SR (±1) 34.9, and Full SR 19.3; Explorer-4B reaches 44.0, 39.4, 31.3, and 18.1 respectively. On Multimodal-Mind2Web, Explorer-7B (syn+M2W) reports Cross-Task SR 53.2%, Cross-Web SR 56.7%, Cross-Dom SR 53.0%, and Avg Step SR 54.3%. On MiniWob++, Explorer-7B reaches 53.26% accuracy. The paper additionally reports a data-scaling ablation in which 25% of the data yields a 20 percentage-point Step SR improvement over the base LMM, 50% yields a further 5 percentage-point improvement, and 100% yields peak performance. A plausible implication is that, in this usage of ABot-Explorer, exploration is primarily a data-generation mechanism for downstream policy learning rather than a runtime planner alone.

4. Benchmarking UI exploration

UIExplore-Bench provides the first benchmark explicitly dedicated to UI exploration and is directly relevant for evaluating ABot-Explorer-like systems whose objective is coverage rather than only task completion (Nica et al., 21 Jun 2025). The benchmark formalizes exploration as maximizing the set of actionable UI components discovered in a GitLab sandbox environment, over a single 2,000-step trajectory without resets mid-episode.

The environment is instantiated in two modes. In Structured mode, the observation is a full DOM or AXTree snapshot with a unique ID such as bid, semantic metadata, and geometric information; the action space contains 11 discrete parameterized actions, including click(bid), fill(bid,v), goto(URL), scroll_up, and go_back. In Screen mode, the observation is a 1280×720 RGB screenshot of the browser viewport, and the action space contains 18 low-level operations such as mouse_move(x,y), mouse_click(x,y), keyboard_type(t), scroll(dx,dy), go_forward, and go_back. Three sandbox levels are defined: Abundant, Moderate, and Sparse.

The core metric is UI-Functionalities Observed:

$C3$ 3

where $C3$ 4 is the set of functionalities exposed in state $C3$ 5. Because raw UFO depends on UI complexity, the benchmark introduces human-normalized UFO (hUFO) using human reference performance on the Abundant level: $C3$ 6 and $C3$ 7. At $C3$ 8 steps, UIExplore-AlGo reaches mean hUFO 77.2% in Structured mode and 59.0% in Screen mode, with level-specific values of 80.4%, 74.9%, and 76.4% for Structured, and 59.3%, 60.1%, and 57.6% for Screen. A human expert reaches approximately 114% in Structured mode and approximately 117% in Screen mode.

The benchmark also releases a Dockerized GitLab environment, an exploration dataset containing trajectories from humans, UIExplore-AlGo, and baselines, and an evaluation suite that computes UFO, hUFO, and UFT. The paper recommends that an ABot-Explorer system use the benchmark environment to measure raw exploration coverage, pretrain or fine-tune on the exploration dataset, evaluate via hUFO, extend the suite to additional UIs or stricter Screen-only constraints, and use UFT to measure efficiency. This establishes a standardized interface between algorithm design and empirical comparison.

5. Embodied exploration and SG-Memo construction

In embodied navigation, ABot-Explorer is a “novel active exploration framework” that unifies memory construction and exploration into an online, RGB-only process (Chen et al., 21 Apr 2026). The system is organized into four interacting modules: Perception & SNA Distillation, BEV Lifting & SG-Memo Updater, High-Level Planner (Greedy Reasoning), and Low-Level Navigation Policy.

The perceptual front end takes a set of $C3$ 9 synchronized RGB images $C5$ 0 and uses a fine-tuned Vision-LLM to output a local 2D scene graph $C5$ 1. The vertices encode Semantic Navigational Affordance detections, and the edges encode local traversability on the floor plane. Each SNA node is defined as

$C5$ 2

where $C5$ 3 is the SNA type, $C5$ 4 is the room category, $C5$ 5 is the set of visible navigation-relevant objects, and $C5$ 6 is the 3D floor position. The VLM is trained with the autoregressive SFT loss

$C5$ 7

Lifting from image space to 3D is performed through inverse perspective mapping:

$C5$ 8

The lifted local graph is merged into a global hierarchical graph $C5$ 9 called SG-Memo. New nodes are spatially matched under threshold $P3$ 0 or appended, and edges record navigable links. To enforce sparsity, a connectivity-aware NMS groups mutually reachable nodes and retains the one closest to the current robot pose. After each merge, room labels are reassigned by majority vote and object sets by union over clustered detections.

Exploration is optimized with respect to geometric and topological coverage. The paper defines occupancy coverage and SNA coverage as $P3$ 1 and $P3$ 2, and reports efficiency through $P3$ 3 and $P3$ 4 over path length. Rather than reinforcement learning with a hand-crafted reward, ABot-Explorer uses a greedy VLM-based planner that serializes the current SG-Memo as JSON, ranks unvisited SNA candidates first by alignment with current heading and then by global distance, and returns the next subgoal node $P3$ 5. An off-the-shelf low-level controller, such as ABot-N0, navigates to the subgoal while handling obstacle avoidance.

The data contribution extends InteriorGS with SNA and SG-Memo annotations. The resulting dataset comprises 1,179 scenes—1,000 InteriorGS, 145 HM3D, and 34 MP3D—and approximately 500k VQA-style pairs for VLM training. Experimental evaluation uses 50 seen and 50 unseen InteriorGS test scenes with 10 random starts each, 9 held-out HM3D multi-floor scenes, and 34 held-out MP3D scenes. Sensors consist of four RGB cameras with 90° yaw offsets, 720×640 px resolution, and 110°×104° FOV. Reported results state that ABot-Explorer achieves the highest $P3$ 6—up to 96%—and dominates on $P3$ 7; on unseen InteriorGS, SG-Memo quality reaches Room Coverage 92%, Room-Type Accuracy 90%, and Object Recall 81%; downstream reasoning with the full SG-Memo yields 70% Room Identification accuracy, 80% Node Grounding, 65% ObjectNav Success Rate, and 62% ObjectNav SPL. On HM3D multi-floor scenes, SNA stairs detection raises cross-floor $P3$ 8 from 33% to 71%. Limitations noted by the paper include planar IPM, VLM latency from offboard inference, a four-type SNA taxonomy, the use of a greedy planner, and incomplete downstream integration into end-to-end VLN or manipulation pipelines.

The most common conceptual error is to treat ABot-Explorer as a single, universally defined software artifact. The primary literature instead presents several systems that share an exploration-centered orientation but differ sharply in substrate, observation space, action space, and evaluation target. In GUI automation, the substrate is live desktop-browser or Android UI state (Chaimalas et al., 12 Apr 2025). In web-trajectory synthesis, it is the live Web plus LMM-based verification (Pahuja et al., 17 Feb 2025). In embodied navigation, it is RGB-only robot perception with online semantic memory (Chen et al., 21 Apr 2026). In benchmark design, it is a GitLab sandbox instrumented for functionality discovery (Nica et al., 21 Jun 2025).

Two adjacent works help delimit the term. BotExpTest is “a chatbot, called BotExpTest, designed to support testers while performing exploratory tests of software applications,” implemented on top of Discord with functions for bug reporting, time management, guidelines, and presentation of exploratory testing strategies (Copche et al., 2023). It is exploratory-testing support rather than an ABot-Explorer system in the GUI, web-agent, or embodied sense. Likewise, Malware Sight-Seeing introduces Points-of-Interest beacons for reverse engineering and proposes how that POI-beacon approach can be used to build “an automated botnet explorer such as ABot-Explorer,” but the published contribution itself is POI discovery, confidence scoring, and peer extraction over botnets such as ZeroAccess, Sality, Nugache, and Kelihos (See et al., 2021). This suggests that the label has also functioned as an integration target for exploration-oriented automation beyond UI and navigation.

A broader interpretation, clearly marked as inference, is that the name denotes a research pattern: exploration is not treated as a by-product of task execution but as a primary computational objective. In the GUI system, exploration yields app-specific interactable labels and trace graphs. In the web-agent system, it yields successful multimodal trajectories at scale. In UIExplore-Bench, it is formalized as maximizing the discovered set of functionalities. In the embodied framework, it drives online SG-Memo construction through Semantic Navigational Affordances. Under that interpretation, ABot-Explorer names a convergent agenda rather than a single stack.