Web and Research Agents (WRAs)

• Web and Research Agents (WRAs) are autonomous or semi-autonomous systems that retrieve, navigate, extract, and synthesize information from the web using advanced AI models.
• They integrate modular pipelines—including perception, planning, and execution—to transition from traditional crawling to multimodal deep research and structured extraction.
• Recent advancements focus on trustworthiness, security, and scalability through standardized benchmarks, reward-guided reasoning, and collaborative governance frameworks.

Web and Research Agents (WRAs) are autonomous or semi-autonomous software systems that act on the Web to retrieve, navigate, extract, reason over, and synthesize information on behalf of humans. In earlier literature, the term covered web crawlers, spiders, robots, and related intelligent software agents for information search and retrieval (Bhute et al., 2013). In more recent work, it denotes AI agents and deep-research systems built on LLMs, Vision-LLMs (VLMs), or Large Foundation Models (LFMs), which operate in web environments through planning, retrieval, tool use, and report generation (Ning et al., 30 Mar 2025, Zhang et al., 18 Aug 2025). Across these traditions, WRAs are defined less by a single implementation than by a functional role: autonomous interaction with web-based information and services under conditions of scale, heterogeneity, and continual change.

1. Historical lineage and scope

The earlier web-agent literature treated the Web primarily as a large, rapidly growing information space whose scale exceeded direct human browsing. Within that setting, search engines were presented as central retrieval systems, while web agents were described as programs acting autonomously on some task and already present in the form of spiders, crawler, and robots (Bhute et al., 2013). A web crawler was defined as a software agent that browses the World Wide Web methodically and automatically, with the terms spider, bot, ant, and worm often used interchangeably. The same body of work also distinguished web robots for automated repetitive web tasks and chatterbots for conversational interaction.

A later semantic-web perspective reframed intelligent software web agents as the unrealized fulfillment of the original Semantic Web vision: machine readable web data that could be automatically actioned upon by intelligent software agents (Kirrane, 2021). This line of work emphasized autonomy, reactivity, pro-activeness, social ability, interoperability, communication, security, privacy, trust, ethics, stability, performance, scalability, and verification as requirements for effective web agent operation. It also proposed a hybrid architecture with Interface, Reactive, Deliberative, Learning, and Controller components.

Contemporary work broadens the category further. In the web automation literature, WebAgents are AI Agents designed to complete daily web tasks according to user instructions, and the survey literature organizes them around architectures, training, and trustworthiness (Ning et al., 30 Mar 2025). In the deep-research literature, WRAs are systems that go beyond one-shot retrieval: they orchestrate multi-agent pipelines, iteratively retrieve and gather web content, and synthesize long-form, structured reports with in-line citations (Zhang et al., 22 May 2026). This suggests that “WRA” now covers a continuum from navigation and transaction agents to extraction agents, fact-checking agents, and open-ended research systems.

2. Core architectures and computational organization

A canonical modern WebAgent pipeline consists of three sequential and interdependent modules: Perception, Planning and Reasoning, and Execution (Ning et al., 30 Mar 2025). Perception may be text-based, screenshot-based, or multi-modal. Planning and reasoning may be explicit or implicit, reactive or strategic, and may incorporate short-term and long-term memory. Execution comprises grounding and interacting, either through web browsing-based actions or tool-based actions.

The action-selection process is commonly formalized as

$$a_t = f_{\Theta}(T, s_t, \{a_1, a_2, ..., a_{t-1}\})$$

with environment update

$s_{t+1} = S(a_t).$

Here, $T$ is the user instruction, $s_t$ is the current state, and prior actions provide task-local memory (Ning et al., 30 Mar 2025). This formalization captures the dominant sequential-control view of WRAs: the agent observes a web state, conditions on task and history, emits an action, and receives a new state.

Older web-crawling architectures were more specialized but structurally analogous. Their components included Seeds, Crawl Frontier, Scheduler, Multi-threaded Downloader, Storage, URL Deduplication, and a Parsing Module (Bhute et al., 2013). These systems were organized around policies for selection, re-visit, politeness, and parallelization. Although they lacked contemporary LLM-based reasoning, they already exhibited key WRA properties: autonomous traversal, dynamic state expansion, and large-scale information acquisition.

The semantic-web literature proposed a broader hybrid architecture in which the Reactive Component handled event-condition-action rules, the Deliberative Component maintained a Knowledge Base, Reasoning Engine, Goal Encoding, and Utility Function, the Learning Component supplied adaptation, and the Controller Component coordinated execution and policy enforcement (Kirrane, 2021). Its utility model was written as

$U(a \mid s) = \sum_{i} w_i\, v_i(a, s).$

This model located WRAs within classical agent design rather than solely within prompting-based orchestration.

A distinct research-oriented reformulation appears in recent deep-research systems. One survey organizes deep research into four stages—planning, question developing, web exploration, and report generation—with representative workflows spanning single-agent and multi-agent systems (Zhang et al., 18 Aug 2025). A separate framework, WebResearcher, recasts deep research as a Markov Decision Process in which the state contains the research question, the evolving report, and the most recent tool response or action:

$$s_i = \Big( \text{question}\ q,\, \text{Report}_{i-1},\, \text{Action}_{i-1},\, \text{ToolResponse}_{i-1} \Big).$$

Its central claim is that periodic report synthesis and workspace reset prevent “context suffocation” and “noise contamination” associated with mono-contextual approaches (Qiao et al., 16 Sep 2025).

3. Interfaces, interaction modalities, and execution paradigms

A major line of differentiation among WRAs concerns how they perceive and act on the Web. The recent survey literature distinguishes text-based WebAgents, which consume webpage textual elements such as HTML or accessibility trees; screenshot-based WebAgents, which consume screenshots of the website; and multi-modal WebAgents, which combine textual and visual signals (Ning et al., 30 Mar 2025). This taxonomy reflects a core representational tension in web-agent design: DOM-based inputs are often structurally complete but very large, whereas screenshots are compact but can miss hidden or interactive elements.

The position paper on Agentic Web Interfaces (AWIs) frames this as a fundamental mismatch between human-designed interfaces and LLM capabilities (Lù et al., 12 Jun 2025). It identifies three widely used interaction surfaces—screenshots, DOM trees, and APIs—and argues that each is suboptimal for current web agents: screenshots can miss hidden interface elements, DOM trees can contain excessive and irrelevant details, and APIs bypass user interfaces but expose limited action spaces and are designed for developers rather than agents. In response, it proposes AWIs as a new, standardized class of interface specifically crafted for agent interaction and gives six guiding principles: Standardized, Human-centric, Safe, Optimal Representations, Efficient to Host, and Developer-friendly.

This infrastructural argument is sharpened by the plan-then-execute literature. That work argues that ReAct should not be the default architecture for web agents and that web agents should instead commit to a task-specific program before observing runtime web content (Piet et al., 14 May 2026). Its central distinction is between outcome manipulation and control-flow hijacking: under ReAct, untrusted content can influence next-action selection directly, whereas under plan-then-execute untrusted data may influence values or branches inside a predefined execution graph but cannot redefine the user task or induce new control flow. The paper categorizes WebArena tasks into Safe, Safe With Influence, and Replan-Needed, and reports that all 860 tasks are compatible with plan-then-execute, with 699 categorized as Safe and 161 as Safe+Influence. It concludes that the main barrier is not model capability but infrastructure: browser actions such as click, type, and scroll do not constitute typed, complete, auditable website APIs.

A related empirical specialization appears in large-scale structured extraction. WebLists argues that navigation and transaction tasks have received much more attention than extraction at scale and introduces BardeenAgent, a two-phase executable architecture consisting of a record phase and a replay phase (Bohra et al., 17 Apr 2025). In the record phase, exploration and extraction actions are recorded over the first data instance and mapped to stable, generalizable CSS selectors; in the replay phase, the agent converts the recorded actions into an executable program that loops over all instances and handles pagination. Its selector model is formalized as

$s = \text{SelectorModel}(g, c, H, d),$

where the inputs are the task goal, column specification, HTML of the least common ancestor node, and a list description.

4. Evaluation ecosystems, benchmarks, and measured capability

The current empirical picture of WRAs is shaped by a rapid expansion of benchmarks and standardized environments. BrowserGym was introduced as a unified, gym-like environment with well-defined observation and action spaces for standardized evaluation across diverse benchmarks, and the later BrowserGym ecosystem extends this with AgentLab for agent creation, testing, and analysis (Chezelles et al., 2024). It wraps benchmarks such as MiniWoB(++), WebArena, VisualWebArena, WorkArena, AssistantBench, and WebLINX into a common interface and models web-agent interaction as a Partially Observable Markov Decision Process.

The first large-scale, multi-benchmark BrowserGym experiment compared six state-of-the-art LLMs across six web-agent benchmarks and reported that Claude-3.5 Sonnet led on almost all benchmarks except vision-related tasks, where GPT-4o was superior (Chezelles et al., 2024). At the same time, the same study emphasized that building robust and efficient web agents remains a significant challenge because of real-world web complexity and current model limitations.

A stronger corrective to optimistic reporting appears in Online-Mind2Web. That benchmark consists of 300 diverse and realistic tasks spanning 136 websites, and its accompanying analysis argues that previously reported progress was overstated because existing benchmarks allowed shortcut strategies and used weaker evaluation protocols (Xue et al., 2 Apr 2025). Human evaluation placed OpenAI Operator at 61.3% success, while SeeAct, Agent-E, Browser Use, and Claude Computer Use were all near 28.0–30.7%. The paper’s LLM-as-a-Judge evaluator, WebJudge, achieved around 85% agreement with human judgment, higher than previous methods.

For web research rather than generic navigation, Deep Research Bench introduces 89 multi-step web research task instances across 8 diverse task categories and a frozen RetroSearch environment so that offline agents can be compared reliably over time (FutureSearch et al., 6 May 2025). Its evaluation reports that o3 achieved the highest average score of approximately 0.51, with Gemini 2.5 Pro and Claude 3.7 Sonnet close behind, and that the noise ceiling was approximately 0.8. It also includes automated trace analysis for hallucinations, tool misuse, and forgetting, with forgetting identified as the strongest negative predictor of success.

For extraction tasks, WebLists provides a contrasting capability profile. It reports that LLMs with search capabilities achieved 3% recall, state-of-the-art web agents 31% recall, and BardeenAgent 66% recall overall on 200 extraction tasks across four business and enterprise use-cases (Bohra et al., 17 Apr 2025). It also reports 72.5% precision, 66.2% recall, and 60.0% Q&A accuracy for BardeenAgent, alongside a 3x reduction in cost per correct output row relative to Wilbur.

For multimodal deep-research settings, BrowseComp-VL and WebWatcher extend evaluation beyond text-centric systems. WebWatcher introduces a benchmark that requires complex information retrieval involving both visual and textual information and reports that WebWatcher-32B outperformed proprietary baseline, RAG workflow, and open-source agents across BrowseComp-VL, HLE, LiveVQA, and MMSearch (Geng et al., 7 Aug 2025). This suggests that the transition from web navigation to research-grade web intelligence is increasingly tied to multimodal reasoning and strategic tool orchestration.

5. Human performance, planning behavior, and learning strategies

A recurrent finding in the literature is that humans and current web agents do not solve web tasks in the same way. A think-aloud study comparing people with web agents examined planning, action, and reflection during tasks such as finding products on Amazon or historical posts on Reddit (Son et al., 2024). Humans were found to exhibit flexible, multi-layered planning, ambiguity identification, exploration, cross-site information forays, nuanced reflection on failure causes, and rollback to earlier plans. Existing agents, by contrast, were described as largely linear and limited in knowledge updating and ambiguity handling.

The authors of that study therefore recommend additional WRA modules for knowledge updating, ambiguity handling/clarification, exploration/explorative planning, robust reflection, and plan revision/knowledge integration (Son et al., 2024). This is consistent with survey-level concerns about generalization and trustworthiness, especially under out-of-distribution websites, layouts, and tasks (Ning et al., 30 Mar 2025).

Learning strategies for modern WRAs fall into several broad classes. The WebAgents survey lists training-free prompting, GUI comprehension training, task-specific fine-tuning, and post-training (Self-Improvement) via reinforcement learning (Ning et al., 30 Mar 2025). Multi-stage frameworks combining GUI comprehension, fine-tuning, and reinforcement learning are described as common. In parallel, domain-specific systems have adopted more structured self-improvement mechanisms. Wilbur, for example, uses a differentiable ranking model to select demonstrations for in-context learning, synthesizes instructions from positive and negative examples, and adds intelligent backtracking to recover from mistakes (Lutz et al., 2024). On WebVoyager it achieved 52.6% average success, compared with 44.3% for the previous text-only state of the art and 57.1% for a strong multimodal model.

Deep-research agents introduce a further shift from local step selection to long-horizon report-centered reasoning. WebResearcher’s training data generation engine, WebFrontier, uses tool-augmented complexity escalation to create synthetic research tasks, and the resulting model is reported to achieve state-of-the-art performance across six challenging benchmarks, including 36.7% on Humanity’s Last Exam and 51.7% on BrowseComp-en for WebResearcher-30B-A3B-heavy (Qiao et al., 16 Sep 2025). A plausible implication is that the field is moving from simple action prediction toward the explicit training of persistent research processes.

6. Trustworthiness, governance, security, and the future web

Trustworthiness is a central organizing concern in recent WRA research. The WebAgents survey identifies safety & robustness, privacy, generalizability, and future concerns around fairness and explainability as major axes of evaluation (Ning et al., 30 Mar 2025). Benchmarking work echoes this emphasis: Deep Research Bench analyzes hallucinations, forgetting, and invalid tool use (FutureSearch et al., 6 May 2025), while WebArbiter introduces a principle-guided Process Reward Model that generates structured justifications and a preference verdict for candidate actions (Zhang et al., 29 Jan 2026). On WebPRMBench, WebArbiter-7B achieved 89.19 pairwise accuracy and 74.60 Best-of-N accuracy, outperforming GPT-5 by 9.1 points on BoN accuracy and improving reward-guided trajectory search on WebArena-Lite.

Security concerns have become more specific as research agents increasingly rely on untrusted web content. Deep-research systems that repeatedly retrieve user-generated content can be poisoned through a single frequently retrieved page, an attack formalized as WARP (Web Agent Retrieval Poisoning) (Zhang et al., 22 May 2026). The study reports that for many topics the same user-generated content pages recur across related queries, and that a short crafted addition to a single recurring page can alter citations and mentions across multiple outputs. Defenses such as source-level blocking, input filtering, and output-based detection are analyzed, but the paper concludes that current defenses are either costly, ineffective, or remove legitimate information sources.

A different privacy threat arises even when agents are run locally. Network-Level Prompt and Trait Leakage in Local Research Agents shows that passive network adversaries can infer prompts and user traits from visited domains and their timings (Jeong et al., 27 Aug 2025). WRAs in that study visited 70–140 domains with distinguishable timing correlations, enabling prompt recovery and multi-session trait inference. The OBELS metric decomposes leakage into functional equivalence, domain-type equivalence, semantic equivalence, and entity granularity tolerance:

$$OBELS(\hat p, p) = \left\{ E_{func},\; E_{dom},\; E_{sem},\; T_{ent} \right\}.$$

The paper reports recovery of over 73% of the functional and domain knowledge of user prompts and up to 19 of 32 latent traits in multi-session settings.

Governance research complements these security analyses by treating norms, policies, and preferences as first-class abstractions in Web-based multiagent systems (Kampik et al., 2022). Its conceptual framework comprises three layers: Reactive Things and Services governed by policies, Autonomous Agents governed by preferences, and Normative Organisations governed by norms. The associated deontic operators are

$O(a, \phi), \quad P(a, \phi), \quad F(a, \phi),$

for obligation, permission, and prohibition. This line of work directly addresses heterogeneity, openness, decentralized reasoning, human oversight, and cross-organizational compliance.

The most expansive forward-looking vision appears in infrastructures for scientific collaboration. Clarus reformulates research as an open, auditable, attributable, and resource-aware multi-phase collaboration process over projects, agents, and resources (Guo et al., 29 Jun 2026). Its object model defines a typed graph

$$\mathcal{G} = (V, \mathcal{E}),\qquad V = \mathcal{P} \cup \mathcal{A} \cup \mathcal{R},$$

with project–agent and agent–resource relations. In a controlled paper-generation case study it organized six research phases, 8 agents, 30 DAG tasks, 29 edges, 128 registered artifacts, and 589 trace events. This suggests that the future of WRAs may lie not only in better single-agent web automation, but in auditable collaboration infrastructures that coordinate digital and physical resources under governance constraints.

Taken together, the literature presents WRAs as a field in transition. Earlier work emphasized crawling, retrieval, and semantic interoperability; recent work emphasizes multimodal perception, long-horizon planning, structured extraction, and deep research; and the newest work shifts attention to typed interfaces, reward-guided reasoning, governance, poisoning resistance, privacy, and collaborative scientific workflows. The broad trajectory is from isolated web automation toward open, accountable, and infrastructure-dependent agent ecosystems.