WebCompass: Navigation & Web Benchmark

• WebCompass is a dual-use concept, serving as both a browsing assistant that maps visited paths to ease disorientation and a benchmark for evaluating web coding agents.
• The 2013 system employs a proxy-based architecture to capture HTTP events and generate interactive, time-annotated maps, reducing cognitive load during navigation.
• The 2026 benchmark tests front-end model capabilities across generation, editing, and repair tasks using text, image, and video inputs with execution-grounded judging.

WebCompass denotes two distinct research artifacts in the arXiv record. In one usage, it is an accessibility-oriented navigation assistant for web browsing that externalizes a user’s traversal history as a visited site map in order to mitigate disorientation and cognitive overhead during hypertext navigation (Herrouz et al., 2013). In a later and unrelated usage, it is a multimodal benchmark for evaluating web coding agents across generation, editing, and repair tasks under text, image, and video inputs, with execution-grounded judging in a real browser (Lei et al., 20 Apr 2026). The shared name is motivated by orientation in web space in both cases, but the two systems operate at different technical layers: the earlier work addresses human browsing assistance, whereas the later work evaluates model capability in modern web engineering.

1. Dual usage of the term

The term “WebCompass” has been used for two different systems in web research. The earlier system is a browsing assistant implemented around a local proxy-cache server and an interactive graphical map of visited pages. The later system is a benchmark and evaluation framework for code LLMs and coding agents targeting front-end web development workflows (Herrouz et al., 2013, Lei et al., 20 Apr 2026).

WebCompass usage	Publication	Core object
Navigation assistant	2013	Visited site map for browsing orientation
Multimodal benchmark	2026	Lifecycle evaluation of web coding agents

This naming overlap is substantively meaningful. In the 2013 system, the “compass” metaphor refers to bearings within a browsing session: where one is, where one has been, and how to return. In the 2026 benchmark, the term signals broad navigational coverage over the web engineering lifecycle: multimodal specification, iterative code modification, and repair under realistic browser execution. A plausible implication is that the name persists because both works treat the web as a structured space in which orientation—human or agentic—must be made explicit.

2. WebCompass as a browsing-orientation assistant

The 2013 WebCompass begins from a specific accessibility claim: “the ability of users with disabilities to navigate over the Web is not dependent on the graphical complexity, but on the markup used to create the structure of the website” (Herrouz et al., 2013). On that basis, the system targets two classic hypertext problems identified by the authors: disorientation, understood as not knowing where to go, how to get there, or where one is relative to the whole; and cognitive overhead, understood as the burden of making and remembering many link choices on a single screen.

Its central mechanism is a continuously constructed visited site map that gives an explicit representation of virtual space. The map is modeled by a directed graph in which each node represents a visited page and each directed edge represents traversal from one page to the next during the session. The implemented visualization is tree-style, chosen for readability, visibility of nodes, minimal edge crossings, and ease of scanning (Herrouz et al., 2013).

Node attributes include the page address, page topic or title, and time spent connected to that page. Time is not merely a log variable; it functions as a landmark for reorientation. The system records elapsed time information at the proxy layer and surfaces connection duration in the map and URL sequence views, thereby combining structural and temporal breadcrumbs. The result is a route map of browsing history that supports recall, backtracking, and path comprehension.

The assistant was designed as an add-on to standard browsers rather than as a replacement browser. It was described as intended to satisfy “guidelines of accessibility of the W3C recommendation for disabled authors and learners especially with mobility impairments,” and as a tool that “helps people with disabilities to use and contribute to the Web” (Herrouz et al., 2013). In this respect, WebCompass belongs to a strand of accessibility work that treats orientation aids and externalized session memory as assistive infrastructure rather than mere usability enhancements.

3. Architecture, data flow, and interaction model of the 2013 system

The system was implemented in Delphi and designed to work with any browser, including Firefox and Internet Explorer, by interposing a local proxy server rather than a browser-specific extension (Herrouz et al., 2013). This browser independence is one of its distinguishing architectural decisions, separating capture and visualization from the browser’s native UI.

The architecture comprises four main components. First, a local proxy-cache server installed on the user’s machine intercepts all HTTP requests, optionally serves cached responses, and records browsing events. Second, a URL/event collector extracts the address requested and the elapsed time since the last time this address was requested, then persists these data in a log file. Third, a cartographer reads logged events and generates an interactive graphical map while also managing visualization, interaction, and navigation time tracking. Fourth, the user-facing interface presents multiple windows concurrently: one for the browser, one for the URL list or sequence, and one for the navigation map. An auxiliary dictionary/help window provides frequently used Internet terms and system help (Herrouz et al., 2013).

The data flow is explicit. A browser issues an HTTP request; the local proxy intercepts, caches, and forwards it if needed, while logging URL and timing data; the map builder consumes the logs to incrementally construct or update the visited site map; and interaction can loop back when the user double-clicks a node in the map to reopen the corresponding document in the browser (Herrouz et al., 2013).

The map generation procedure is described operationally rather than in formal algorithmic notation. On each intercepted request, the proxy records URL and timing information. The cartographer reads the event sequence, creates a node for each unique URL, creates a directed edge from the previously visited page to the current one, and accumulates time spent at the node level. The paper explicitly does not provide LaTeX formulas, pseudocode, or complexity analysis. A formal graph notation is therefore only implicit in the text, not part of the stated implementation (Herrouz et al., 2013).

Interaction design is organized around multi-level visualization. The authors explicitly list “General view of the information,” “Zooming,” and “Getting the details” as design considerations. Users can switch the node attribute displayed—titles, URLs, or thumbnails—use mouse-over to reveal title and URL, create or delete links, reorganize the graph, or operate in a view-only mode. They can also save, print, and reopen maps, inspect a log report for self-evaluation, and produce daily interaction and per-site time reports. In group-learning settings, maps can be shared so that others benefit from previously explored paths (Herrouz et al., 2013).

4. Accessibility rationale, related tools, and empirical observations

The accessibility rationale of the 2013 WebCompass is grounded in the claim that structural markup and link topology matter more than graphical flourish for navigability, especially for users with disabilities. The system’s response is not deep semantic parsing of headings or ARIA landmarks; the paper explicitly notes that the implementation focuses on capturing titles and URLs through HTTP transactions rather than detailed HTML structural analysis (Herrouz et al., 2013). This is an important boundary condition. WebCompass addresses orientation through session modeling and visualization, not through automatic extraction of semantic landmarks from page markup.

The design intent spans several disability-related dimensions. For visual and cognitive support, the clear tree map, attribute switching, overview/zoom/details affordances, and time annotations help users who struggle with scanning complex pages or remembering paths. For cognitive support in particular, the externalized map is meant to reduce disorientation and cognitive overhead. For motor support, proxy-based independence from browser UIs allows the system to provide its own consolidated windows and controls, although the paper does not enumerate keyboard shortcuts or screen reader integration features (Herrouz et al., 2013).

The work also situates itself against prior navigation tools. NESTOR is described as a browser that draws interactive web maps of visited space, supports user-added concepts and collaborative sharing, but is Windows/Internet Explorer dependent. Broadway is described as a server that tracks document requests for a group and advises next pages via case-based reasoning, but does not include navigation time as a parameter. Footprints, Letizia, Hypercase, and MAWA are acknowledged as related systems. The distinctive contribution claimed for WebCompass is the combination of proxy-based browser independence, visited-path visualization, time landmarks, and concurrent displays intended to reduce cognitive load (Herrouz et al., 2013).

Evaluation in the paper is qualitative. Approximately 100 users aged 19–22 participated, all with some prior browser experience; equipment was limited and the network was not suitable for heavy use. Tasks included free web access and guided access to course materials and links prepared by a supervisor. Participants were asked to collect information about a topic from the Internet, structure it into a document individually or in groups, and submit it using the browsing map as part of the deliverable (Herrouz et al., 2013).

The reported outcomes are observational rather than statistical. The proxy architecture enabled simultaneous display of browser, sequence of sites, and navigation map, which the authors state “helps to reduce the cognitive overload of the users.” Supervisors used maps to analyze teaching content, structure programs and manuals, and build course plans. Guided tours supported simple browsing without limiting exploration freedom. The paper does not report quantitative task completion rates, times, error rates, or statistical analyses, and no disabled participants were included in that phase. The authors therefore explicitly plan a later evaluation “in a Web accessibility environment” to measure simplification of navigation procedures for people with disabilities (Herrouz et al., 2013).

5. WebCompass as a multimodal benchmark for web coding agents

The 2026 WebCompass addresses a different problem: evaluation of LLMs and interactive coding agents for web development (Lei et al., 20 Apr 2026). Its starting point is that existing benchmarks concentrate on narrow slices of coding capability, typically text-conditioned generation with static-correctness metrics, and therefore leave visual fidelity, interaction quality, and codebase-level reasoning insufficiently measured. The benchmark is designed around three stated gaps: a modality gap, because text prompts and single screenshots do not capture appearance and behavior grounded in images and videos; a task coverage gap, because generation alone omits editing and repair in existing repositories; and an evaluation fidelity gap, because static artifacts and DOM heuristics do not adequately assess dynamic interactions and multi-step flows (Lei et al., 20 Apr 2026).

WebCompass responds by defining unified lifecycle coverage over three input modalities and three task types. The modalities are text, image, and video. The task types are generation, editing, and repair. These combine into seven task categories rather than a full $3 \times 3$ grid: Text-Guided Generation, Vision-Guided Generation, Video-Guided Generation, Text-Guided Editing, Vision-Guided Editing, Diagnostic Repair, and Visual-Diagnostic Repair. Video-guided editing and repair are excluded by design because patch-based tasks are described as patch-localized and checklist-aligned, with before/after screenshots plus code diffs deemed sufficient for reliable evaluation, whereas generation benefits more from temporal information in videos (Lei et al., 20 Apr 2026).

The dataset is curated through a multi-stage, human-in-the-loop pipeline. For Text-Guided Generation, sources include WebGen-Bench, ArtifactsBench, BigCode Arena, and high-quality web showcases from V0. Queries are embedded with BGE-M3 and deduplicated by k-means clustering. An LLM assigns domain and difficulty with five independent annotations per query followed by majority vote, and stratified sampling yields 123 text-generation queries. Underspecified queries are expanded into structured design documents covering page content, interaction behaviors, and visual appearance (Lei et al., 20 Apr 2026).

Vision-Guided Generation includes multi-page augmentation from visually complex pages such as WebRenderBench, with subpage screenshots captured via Playwright and subpage links highlighted by bounding boxes. Dynamic keyframes are manually extracted from rich interactive pages from V0 and Figma. Video-Guided Generation uses manually selected webpages with rich temporal interactions, for which annotators plan exploration paths and record final interaction videos (Lei et al., 20 Apr 2026).

Editing and repair tasks share a prototype pool constructed from WebRenderBench through length filtering, quality scoring, and human curation. Total characters across files are constrained to 32k–64k, per-file length to at most 48k, GPT-4o code review retains candidates scoring at least 9 on a 10-point scale, and 50 high-quality prototypes are selected and expanded from single-page to multi-page variants. Editing instances introduce new or enhanced requirements across 16 operation types. Repair instances are generated by reverse construction: the clean prototype is treated as the destination, an LLM injects observable front-end defects from 11 defect types to create a faulty source website, and exact inverse search/replace annotations guarantee a uniquely correct, runnable target (Lei et al., 20 Apr 2026).

The taxonomy is broad but explicit. Generation domains total 15, including E-commerce & Fintech, Data Science & Analytics, Simulation & Scientific Modeling, and Location Services & Transit. Editing operation types total 16 and span complex components, frontend–backend integration, advanced animations, and business scenarios. Repair defect types total 11 and are partitioned into Visual Layout, Semantic Correctness, and Interactive Usability (Lei et al., 20 Apr 2026).

6. Evaluation methodology, benchmark statistics, and reported findings

WebCompass evaluates models with task-aware protocols rather than uniform static metrics (Lei et al., 20 Apr 2026). Editing and repair use checklist-guided LLM-as-a-Judge. The pipeline applies predicted patches, discards non-applying blocks, launches the result in headless Chromium, captures before/after screenshots, and collects build and runtime logs. Editing is scored on Instruction Targeting, Feature Integrity, and Style Conformance, each on a 0–10 scale per dimension with structured JSON output. Repair is scored on Root-Cause Targeting, Interaction Integrity, and Reference Fidelity under the same scale (Lei et al., 20 Apr 2026).

Judge reliability is measured against human annotations. Among Claude-family judges, Claude-Opus-4.5 attains the highest agreement, with Pearson $r \approx 0.93$–$0.96$, compared with approximately $0.88$–$0.90$ for Claude-Sonnet-4.5 and approximately $0.76$–$0.81$ for Claude-Haiku-4.5. Automatic rankings are reported to mostly differ from human rankings by at most one rank position, and Opus-4.5 is adopted as the default judge given the cost–quality trade-off (Lei et al., 20 Apr 2026).

Generation is evaluated by a distinct Agent-as-a-Judge paradigm. Claude Code v2.0.67 orchestrates execution through the Chrome DevTools MCP Server v0.19.0 controlling a headless Chromium browser. The agent first produces a structured checklist with 10–16 items summing to 100 points, with one universal Runnability item, 6–10 Spec Implementation items, and 1–3 Design Quality items. It then launches the site, executes browser interactions such as clicks, typing, scrolling, and navigation, and collects evidence. In the third stage it synthesizes executable JavaScript tests to verify DOM states, CSS properties, and functional logic, allowing selector-only adaptation when elements differ while keeping behavioral assertions fixed. In the final stage it assigns evidence-grounded scores, discarding incomplete or unevidenced judgments. Safeguards include checklist immutability, selector-only adaptation, and mandatory evidence in the form of screenshots, test results, or console logs (Lei et al., 20 Apr 2026).

Task-level aggregation uses the harmonic mean over normalized item scores with $\epsilon = 1$ smoothing for zero-scored items:

$s_{\text{task}} = \frac{n}{\sum_{i=1}^{n} 1 / r_i}$

where $r_i = s_i / m_i$, except that zero-scored items are replaced by $r \approx 0.93$0. The stated purpose is to penalize imbalance so that strong performance on some items does not mask failure on others (Lei et al., 20 Apr 2026).

The benchmark contains 1,526 tasks in total: 123 Text-Guided Generation, 109 Vision-Guided Generation, 94 Video-Guided Generation, 300 Text-Guided Editing, 300 Vision-Guided Editing, 300 Diagnostic Repair, and 300 Visual-Diagnostic Repair. Difficulty is labeled Easy, Medium, or Hard based on functional complexity, number of interactive components, and visual sophistication (Lei et al., 20 Apr 2026).

Representative closed-source and open-source models are evaluated. Closed-source models include Claude-Opus-4.5, Gemini-3-Pro-Preview, Gemini-3-Flash-Preview, GPT-5.2, and Claude-Sonnet-4.5. Open-source models are drawn from the Qwen3-VL series. Overall scores place Claude-Opus-4.5 first with 67.40, followed closely by Gemini-3-Pro-Preview with 66.68, Gemini-3-Flash-Preview with 65.42, GPT-5.2 with 61.90, and Claude-Sonnet-4.5 with 57.12. The strongest open-source model, Qwen3-VL-235B-A22B-Instruct, reaches 41.14, described as more than 26 points behind the top closed-source model (Lei et al., 20 Apr 2026).

Several empirical patterns are highlighted. Closed-source models remain “substantially stronger and more balanced.” Editing and repair show different difficulty profiles: editing follows the ordering executability $r \approx 0.93$1 functional $r \approx 0.93$2 visual for closed-source systems, whereas repair shows Interaction Integrity $r \approx 0.93$3 Reference Fidelity $r \approx 0.93$4 Root-Cause Targeting. Aesthetics is identified as the most persistent bottleneck, with visual dimensions consistently lowest in generation and editing. Framework choice materially affects outcomes: Vanilla HTML/CSS/JS dominates generation and editing for the examined models; Vue is consistently challenging; and in repair, React sometimes offers an advantage. Difficulty scaling is pronounced: in Generation Spec Implementation, Gemini-3-Pro-Preview drops from about 89.83 on Easy instances to about 37.64 on Hard instances. Worst-of-$r \approx 0.93$5 stability analyses with $r \approx 0.93$6 show Gemini-3-Pro-Preview retaining about 80% of Pass@1 performance, from 66.96 to 53.56, whereas Qwen3-VL-235B-A22B-Instruct retains about 69.5%, from 39.95 to 27.78 (Lei et al., 20 Apr 2026).

7. Limitations, reproducibility, and broader significance

The limitations of the two WebCompass systems differ because their technical scopes differ. The 2013 assistant does not detail HTML structural parsing, headings analysis, ARIA landmarks, explicit WCAG compliance checks, keyboard navigation specifics, or screen reader integration. Its evaluation lacks quantitative metrics and did not include users with disabilities in the reported phase. Planned future work includes formal evaluation in a web accessibility environment, user annotations on nodes, and the ability to describe preferred paths through the information space (Herrouz et al., 2013).

The 2026 benchmark is explicitly front-end focused and excludes backend, database, API, and deployment tasks. It emphasizes deterministic instruction-following rather than free-form creativity; highly dynamic real-time pages such as games remain difficult to evaluate faithfully; static benchmark contamination is identified as a future risk; and Agent-as-a-Judge is computationally expensive because of browser orchestration and iterative testing (Lei et al., 20 Apr 2026).

Reproducibility is stronger in the benchmark setting than in the earlier assistive prototype. The 2026 work provides a public dataset, code repository, and project page, and specifies an execution environment centered on Linux, headless Chromium, Chrome DevTools MCP Server v0.19.0, Claude Code v2.0.67, Node.js or Playwright for screenshot capture, and Python for orchestration (Lei et al., 20 Apr 2026). By contrast, the 2013 system is described through architecture and figures rather than through a public benchmark distribution (Herrouz et al., 2013).

Taken together, the two usages of WebCompass illustrate a notable shift in what “web navigation” can mean in research. In the earlier work, navigation concerns human orientation in hyperlink space, with explicit emphasis on accessibility, visited-path visualization, and externalized memory. In the later work, navigation concerns model competence across the web engineering lifecycle, with explicit emphasis on multimodality, patch-based modification, repair, and execution-grounded acceptance testing. This suggests a broader continuity beneath the naming collision: both systems treat the web not as a static document collection but as an interactive, structured environment whose critical properties become visible only when paths, states, and transitions are made inspectable (Herrouz et al., 2013, Lei et al., 20 Apr 2026).