---
title: 'WebProber: Exploring Web Probing Systems'
url: https://www.emergentmind.com/topics/webprober
type: topic
---

# WebProber: Exploring Web Probing Systems

WebProber is a label used in several strands of systems and security research for platforms that probe web-related behavior, infrastructure, or user interaction. In the arXiv literature, the name spans at least five distinct designs: a remote website-detection attack based on router queueing delay, a distributed WiFi probe-request collection system for MAC-address de-anonymisation, a browser-API platform for gathering keystroke and mouse data, an automated pipeline for generating censorship probe lists, and an AI agent-based web testing framework [1109.0097] [1805.07613] [1910.07455] [2407.08185] [2509.05197]. This suggests that “WebProber” is not a single canonical architecture, but a recurring designation for systems that observe, infer, or test web-adjacent phenomena through active probing or structured collection.

## 1. Scope and recurring design pattern

Across these usages, WebProber denotes systems that begin from an observable signal, convert that signal into structured traces or candidate items, and then infer a higher-level property such as website identity, device ownership, censorship status, or usability defects.

| Variant | Primary purpose | Core mechanism |
|---|---|---|
| Remote website detection [1109.0097] | Determine whether a target site was visited | ICMP probing of router RTTs; queueing-delay recovery; DTW matching |
| MAC de-anonymisation [1805.07613] | Link people to device MAC addresses | Distributed WiFi probe-request capture; central database; co-occurrence matching |
| Browser behavioural platform [1910.07455] | Collect keystroke and mouse data | Chrome extension; DOM event capture; server-side storage |
| Censorship probe-list generation [2407.08185] | Generate fresh URLs for censorship measurement | Topic extraction; topic expansion; search-engine crawling; global testing |
| AI-agent web testing [2509.05197] | Find user-facing bugs and usability issues | Testing prompt generation; VLM-guided interaction; bug report generation |

The commonality is methodological rather than domain-specific. Each system treats web activity as something that can be measured indirectly or exercised systematically, then analyzed with a separate inference or reporting stage. The differences lie in the observation layer: router queues, 802.11 management traffic, browser APIs, search-engine outputs, or live webpage screenshots.

## 2. Remote traffic analysis as an early WebProber-style formulation

The earliest explicitly “WebProber-style” usage in this corpus appears in research on remote website detection through traffic analysis. In that model, Alice is a home broadband user on a DSL line, while Bob is a remote attacker who knows Alice’s IP address and sends frequent ICMP echo requests to Alice’s router. Bob does not observe Alice’s packets directly; instead, he measures the RTT of each ping and exploits queueing delay at Alice’s bottleneck last hop. Because HTTP download bursts build a FIFO queue on the slow DSL link, the RTT series leaks a coarse time series of Alice’s traffic activity [1109.0097].

The paper formalizes the observation as
$$
RTT_i = \sum_{l \in \text{links on path}} q^l_i + t^l_i + p^l_i
$$
and then approximates the non-bottleneck component by a constant baseline:
$$
\sum_{l \in \text{links on path}} t^l_i + p^l_i \approx \min_j RTT_j.
$$
This yields an RTT signal dominated by queueing delay, from which a recovered traffic pattern is estimated using
$$
t_i = t_{ping}\cdot i,\qquad
t'_i = RTT_i - RTT_{min} + t_i,\qquad
\widehat{s_i}= t'_i-\max(t'_{i-1},t_i).
$$
Because the recovered sequence is noisy and time-warped, the attack uses Dynamic Time Warping (DTW) rather than pointwise comparison. Given sequences \(A\) and \(B\), the DTW distance is
$$
D(A,B)=\min_F\left\{\frac{\sum_{k=1}^{K} d(c(k))w(k)}{\sum_{k=1}^{K} w(k)} \right\}.
$$

The evaluation uses two settings: a real DSL setup with a laptop on a 3 Mbps down / 512 Kbps up DSL line and a remote ping server in Quebec, Canada, and a VM-based training setup using Dummynet to emulate DSL. The dataset contains 1,000 Alexa top sites with 12 traces per site from each setup. For a target false-positive rate of 0.5%, about 75 sites work very well in the VM→DSL case and about 320 sites work very well in the DSL→DSL case; for 65 tested sites, the authors could not get any true positives even at a 5% target false-positive rate in the best-case scenario. The paper also emphasizes a deanonymization scenario in which repeated matches between a home user’s remotely inferred traffic pattern and a message board fingerprint can tie that user to pseudonymous posting activity [1109.0097].

This formulation is significant because it expands website fingerprinting beyond direct packet capture. The observable is not encrypted traffic itself, but latency perturbations induced by that traffic.

## 3. Identity inference from local radio and browser signals

A second lineage uses WebProber for identity inference from local observation rather than remote traffic analysis. In "MAC Address De-Anonymisation" [1805.07613], WebProber is a distributed system for collecting WiFi probe requests and linking device MAC addresses to people. Each detector is a Unix laptop running a cron job every 15 minutes; the script places the wireless card into monitor/promiscuous mode, uses Dumpcap to capture packets for about 45 seconds, parses the resulting `.pcap` with Scapy, and uploads JSON via Requests to a Django-backed server with an SQLite database hosted on AWS EC2. The backend creates or updates MAC, sighting, and SSID records and stores time and location metadata.

The associated identification algorithm is evaluated on simulated lecture-attendance data. It adapts an occupancy grid mapping style approach by maintaining a matrix with rows as names and columns as detected MAC addresses. The update logic is qualitative but explicit: person present / MAC present increases likelihood; person present / MAC absent decreases likelihood; person absent / MAC present decreases likelihood more strongly; neither present causes no change. The runtime is stated as
$$
O(n \cdot m), \quad n := |\text{Lectures}|,\; m := |\text{People}|.
$$
Over about 468 simulations, the regression analysis reports Multiple R = 0.88036614, R Square = 0.77504454, Adjusted R Square = 0.77359009, and Standard Error = 0.03119968; observation number, probe request probability, and average attendance all had statistically significant \(p\)-values below 0.05. The real-world capture deployment ended with 17,511 probe requests from 6,926 unique MAC addresses, manufactured by 120 different manufacturers, with over 500 unique SSIDs observed and 87 SSIDs geolocated [1805.07613].

A related but browser-centric system appears in "An experimental platform for gathering user behavioural data via browser APIs" [1910.07455]. There, WebProber is a Chrome extension plus server-side repository designed to collect keystroke and mouse activity via standard DOM APIs. The content script registers six event listeners:
$$
\texttt{mousemove},\ \texttt{mousedown},\ \texttt{mouseup},\ \texttt{wheel},\ \texttt{keydown},\ \texttt{keyup},
$$
forwards events to the background script through Chrome message passing, and the background script sends them to the server using `XMLHttpRequest`. The raw keystroke record is
$$
K = (\text{code}, \text{key}, t_{\downarrow}, t_{\uparrow}, c, a, s, \ell),
$$
while the raw mouse record is
$$
M = (\text{type}, x, y, t).
$$
The paper further describes segmentation of keystroke streams into sets when \(\Delta t > 1\) second or the key is Space, and extraction of bigram timing features and mouse speed features:
$$
d = \sqrt{(x_2-x_1)^2 + (y_2-y_1)^2},\qquad
\Delta t = t_2 - t_1,\qquad
v = \frac{d}{\Delta t}.
$$

Taken together, these systems show that WebProber can target identity-relevant signals far removed from packet traces: 802.11 management traffic in one case and browser event APIs in another.

## 4. Automated censorship probe-list generation

In "Automatic Generation of Web Censorship Probe Lists" [2407.08185], WebProber denotes an automated pipeline for discovering URLs and domains that should be tested by censorship-measurement systems. The paper treats probe lists as the bottleneck in censorship measurement, since stale or biased lists limit what censored content can be detected.

The source corpus combines Weinberg et al.’s lists, the Wikileaks Internet Censorship page, and Citizen Lab test lists, totaling 150,005 URLs with duplicates and 139,957 unique URLs across 106,878 unique domains. After dead-page filtering, the set falls to 84,451 URLs; content-free filtering removes 18,911 more, leaving 65,540 URLs; excluding pages with fewer than 300 characters yields a final seed corpus of 51,313 live URLs. Language detection with Lingua and CLD3 identifies the language of 99.29% of pages across 103 languages. Topic assignment then proceeds through BERTopic, LDA, and Top2Vec, producing 257 BERTopic clusters and 232 Top2Vec topics, while the LDA branch yields 53 potential topics out of 64 from the prior model [2407.08185].

To avoid simple snowball bias, the system expands topics using `gpt-3.5-turbo` for LDA topics and Google Trends for Top2Vec topics. Query construction uses 4 to 9 keywords, with keyword order permuted because order affects Google results. This stage produces 14,450 query combinations; Google Search crawling returns 160,981 URLs, deduplicated to 119,272 URLs and then cleaned to a final probe list of 119,255 unique URLs across 35,147 pay-level domains. Of these, 71,960 URLs across 32,543 domains come from domains not present in the original source list [2407.08185].

The generated list is then tested from 13 vantage points in 11 cities, spanning North America, Europe, and Asia, between November 2023 and March 2024. Most locations repeat measurement 50 times; two U.S. commercial-network vantage points repeat 8 times. A URL is considered consistent if it returns the same code in more than 95% of runs, and the baseline is formed from five high-freedom vantage points in the US, UK, and France. The main result is 1,490 unique domains that are potentially blocked, with over 1,400 not present in the original source list. For China specifically, 1,068 unique domains show anomalous OONI results in both Beijing and Shanghai, and 1,355 domains are affected in at least one of those two cities [2407.08185].

This version of WebProber is not an observation platform in the narrow sense; it is a discovery and refresh mechanism for what should be probed next.

## 5. AI-agent web testing

A substantially different usage appears in "AI Agents for Web Testing: A Case Study in the Wild" [2509.05197], where WebProber is a prototype AI agent-based web testing framework. Instead of inferring hidden properties from side channels, it actively interacts with a website to discover user-facing bugs and usability issues. The architecture has three stages: testing prompt generation, VLM-guided interaction, and bug report generation.

The system begins from a URL and a targeted prompt. In the reported case study, the prompt instructs the agent to explore personal websites, click links, pause to judge coherence and appropriateness, and look for problems such as broken links, UI glitches, illogical design, and textual errors. The interaction loop is built on the Browser-Use Python package: the agent receives a screenshot of the current browser state, asks the VLM for an action, executes that action, records the resulting state, and repeats until a step limit is reached or the target feature has been tested. The full trajectory stores screenshots, reasoning traces, and actions. Bug report generation then asks the model to identify unexpected behaviors, missing or incorrect functionality, visual glitches, UI inconsistencies, and other anomalies, and to output a summary, steps, patterns, and recommendations. The implementation uses Claude-3.7 Sonnet for all three stages [2509.05197].

The evaluation covers 120 academic personal websites crawled from OpenReview author profiles. Across the full set, WebProber identifies 29 usability issues verified by the authors. In a manually inspected subset of 80 websites, 32 bugs are manually identified and WebProber detects 19, yielding 59.4% coverage. At the same time, 85% of all reported bugs across the 120 sites are false positives. The paper attributes many false positives to browser automation limitations, especially PDF access problems caused by security settings, and attributes many misses to deeply nested bugs requiring long-horizon exploration or dynamic content rendering issues [2509.05197].

This formulation turns WebProber into a guided usability tester. The probe is no longer a network request or radio capture, but a model-mediated interaction trajectory over live webpages.

## 6. Broader methodological context

The broader literature shows that WebProber-like systems sit within a wider family of web probing, auditing, and side-channel measurement. "CrowdSurf" proposes a crowd-sourced privacy-auditing framework positioned right below HTTP so it can inspect clear-text data even when HTTPS is used on the wire; "Client-Side Web Proxy Detection from Unprivileged Mobile Devices" detects carrier web proxies from ordinary phones by comparing HTTP and HTTPS behavior; "Web View" performs automated browsing sessions on popular websites in realistic residential environments and records 84 parameters; and "PerfWeb" infers visited websites from Google Chrome in Incognito mode and Tor Browser by observing hardware performance events such as instructions retired, cache activity, and bus cycles [1502.07106] [1511.04493] [2007.05238] [1705.04437].

These adjacent systems clarify two recurrent points. First, observability is layered: probing can occur at the pre-encryption application layer, at the transport and middlebox layer, at the browser instrumentation layer, at the hardware layer, or through live UI interaction. Second, common privacy assumptions are limited. The literature explicitly rejects the view that HTTPS alone solves privacy leakage, shows that private browsing and Tor still leave a measurable microarchitectural signature, and demonstrates that proxy behavior can be inferred without root access or server cooperation [1502.07106] [1705.04437] [1511.04493].

The cumulative significance of WebProber research is therefore not a single implementation, but a recurrent research program: use probing to expose hidden structure in web systems. In some papers the target is user activity, in others censorship, infrastructure behavior, or usability defects. What unifies them is the conversion of low-level observations or exploratory actions into higher-level claims about web behavior.

Source: https://www.emergentmind.com/topics/webprober