Avenir-Web: Autonomous Multimodal Web Agent

• Avenir-Web is an autonomous multimodal web agent that integrates visual grounding, site-specific planning, and long-horizon task tracking to address common web navigation failures.
• It employs a decoupled architecture with an initialization phase for strategic planning using online resources and a reactive execution loop that updates memory and checklists.
• Evaluations on the Online-Mind2Web benchmark show competitive task success rates, highlighting the impact of adaptive memory, failure reflection, and a hierarchical action grounding mechanism.

Avenir-Web is an autonomous multimodal web agent designed for live, long-horizon interaction on real websites. It is positioned as an open-source system aimed at fixing three recurring failure modes in existing web agents: unreliable element grounding, lack of site-specific procedural knowledge, and weak long-term task tracking and memory. On Online-Mind2Web, the authors report that Avenir-Web reaches a 53.7% task success rate with Gemini 3 Pro as the main action model, which they describe as a new open-source state of the art and as competitive with several proprietary systems (Li et al., 2 Feb 2026).

1. Definition and problem setting

Avenir-Web studies the setting of autonomous multimodal web agents: systems that receive a natural-language instruction and a target URL, observe a web page through browser-accessible modalities, and then execute a sequence of actions such as clicking, typing, selecting options, scrolling, and terminating once the goal is reached. The paper emphasizes that real websites contain nested iframes, complex and dynamic DOMs, shadow DOMs, canvas-rendered widgets, popups, modals, changing layouts, and site-specific workflows that are hard to infer from first principles during a single run (Li et al., 2 Feb 2026).

The system is introduced as a response to three main limitations in prior agents. First, grounding often relies too heavily on a single modality, especially DOM-centric approaches, which break when structural information is incomplete or inaccessible, such as inside iframes or canvas elements. Second, most agents do not exploit external human knowledge about a website’s procedures, so they waste steps on trial-and-error exploration. Third, they tend to be reactive and short-horizon, so after multiple page transitions they lose track of what has been completed, what remains, and which previous failures should alter future decisions (Li et al., 2 Feb 2026).

This framing places Avenir-Web within a broader shift in web-agent research. The position paper on Agentic Web Interfaces argues that current methods face a “fundamental interface mismatch” because browsers and human-facing web UIs are designed for humans, while APIs are designed for developers and services rather than autonomous AI agents (Lù et al., 12 Jun 2025). A plausible implication is that Avenir-Web can be read as a systems-level attempt to improve reliability under the current web substrate, rather than as a redefinition of the substrate itself.

2. Architecture and execution model

Architecturally, Avenir-Web has two phases: Initialization and Execution Loop. In the initialization phase, the system takes the user instruction and target URL and performs two strategic setup tasks. First, the Experience-Imitation Planner searches online resources related to the target website, such as help centers, forums, and user guides, and synthesizes a short site-specific plan. Second, a Checklist Generator decomposes the user goal into 2–6 atomic outcome states that will serve as a task-progress monitor. These two outputs, strategy and checklist, are then injected into the core agent (Li et al., 2 Feb 2026).

In the execution loop, the core agent repeatedly observes the current browser state, decides on the next action intent, grounds that intent into an executable browser operation through MoGE, executes it, and then updates both memory and checklist based on the resulting state. The architecture figure is described as follows: the web environment sends raw state to a visual perception module; perception is passed to the core action agent; the agent exchanges information bidirectionally with Adaptive Memory and the Task-Tracking Checklist; the agent emits an action intent to MoGE; MoGE turns it into an operation in the web environment; and operation outcomes and observed state changes feed back into memory and checklist (Li et al., 2 Feb 2026).

The design is explicitly “decoupled strategic planning and execution”: planning is done up front by EIP, while execution remains reactive but anchored by strategy, memory, and checklist state. The paper does not present a single monolithic end-to-end learned policy. Instead, Avenir-Web is assembled from components using multiple foundation models. The primary action backbone in the best-performing configuration is Gemini 3 Pro. The EIP module uses Claude 4.5 Sonnet with online search capability. The checklist updater uses Qwen-3-VL-8B as a lightweight model for low-latency state synchronization. The paper also reports a fully open-source configuration in which Qwen-3-VL-8B is used as the main action model, producing a 25.7% success rate (Li et al., 2 Feb 2026).

Component	Role	Reported function
Mixture of Grounding Experts	Grounding	Visual-first grounding with structural fallback
Experience-Imitation Planning	Strategy	Retrieves site-specific online guidance and synthesizes a short plan
Task-Tracking Checklist	Progress state	Converts the goal into explicit verifiable milestones
Adaptive Memory	Long-horizon context	Uses recursive summarization and explicit failure reflection

3. Core mechanisms

The Mixture of Grounding Experts, or MoGE, is introduced because no single grounding strategy is robust across the full diversity of web interfaces. The default, prioritized expert is a direct visual grounding expert built on general-purpose multimodal models such as Qwen3-VL and Gemini 3 Pro. This expert treats the viewport as a unified visual canvas and predicts an executable action directly, typically in one model call. A key implementation detail is that Avenir-Web uses coordinate-based interaction in a normalized 1000×1000 screen space, with the prompt instruction “Screen: 1000x1000, origin (0,0) top-left,” and normalized $[x,y]$ coordinates for nearly all actions (Li et al., 2 Feb 2026).

To improve disambiguation, MoGE optionally overlays interactive elements with Set-of-Mark-style annotations, assigning unique identifiers to visible targets so that the model can choose among tightly packed controls. When visual grounding is insufficient, MoGE falls back to semantic structural reasoning. The paper is explicit that it does not provide a formal gating equation, learned router, or probabilistic confidence fusion formula for expert combination; the “mixture” is therefore described operationally as a hierarchical fallback architecture rather than as a mathematically specified mixture model with learned weights (Li et al., 2 Feb 2026).

Experience-Imitation Planning is the second major mechanism. In this paper, “experience imitation” means letting the agent imitate human procedural experience that is encoded in publicly available online knowledge, rather than relying purely on the model’s internal parametric memory or on trial-and-error interaction. The planner uses Claude 4.5 Sonnet with online search capability to retrieve documentation, help pages, community posts, or user guides relevant to the target site and task, then synthesizes these findings into a short plan consisting of 2–4 imperative directives. The appendix summarizes EIP as Exploration, Roadmap Generation, and Strategic Injection (Li et al., 2 Feb 2026).

The Task-Tracking Checklist is the system’s explicit subtask state representation. Its goal is to prevent navigational drift, repetitive loops, and long-horizon context loss by turning the user’s instruction into a small set of observable goal states. The checklist at step $t$ is formalized as

$C_t = \{(m_i, s_{i,t})\}_{i=1}^k$

where $m_i$ is the $i$-th milestone and $s_{i,t} \in \{\text{P, IP, C, F}\}$ denotes whether that milestone is Pending, In Progress, Completed, or Failed. After each interaction, the checklist is updated according to

$C_{t+1} = \mathcal{F}_{\theta}(C_t, a_t, o_t, S_t)$

where $a_t$ is the executed action, $o_t$ is the observed outcome, and $S_t$ is the current page state (Li et al., 2 Feb 2026).

Adaptive Memory addresses the tradeoff between feeding full history into the model and using only a fixed recent-window history. The buffer for chunk $t$0 with window size $t$1 is defined as

$t$2

and the persistent memory state is updated recursively as

$t$3

where $t$4 is the failure reflection buffer for chunk $t$5. The default sliding window size is $t$6. The paper describes this as a mechanism for preserving long-term awareness while ensuring that failures are not lost when history is compressed (Li et al., 2 Feb 2026).

4. Observation space, action space, and grounding practice

In terms of inputs and outputs, Avenir-Web is multimodal. The main observation appears to be the browser viewport screenshot, described as “Visual Perception (Viewport Encoding).” The system also uses page-state text and browser metadata for verification and checklist updating. The failure detector compares state through visible text, interactive elements, focus, URL, scroll position, and popup state. The paper references DOM-related complexity throughout, but the core approach is visual-first rather than DOM-first (Li et al., 2 Feb 2026).

The action space is explicitly shown in the browser tool schema. Supported actions are left_click, hover, keyboard, type, select, press_enter, scroll_up, scroll_down, scroll_top, scroll_bottom, new_tab, close_tab, go_back, go_forward, wait, and terminate. Most actions require a normalized coordinate argument. Typing can include text and optional clear_first or press_enter_after. Select actions are treated specially: the prompt strongly instructs the model to use select directly for <select> elements rather than clicking to open dropdowns. The output is a single browser action per turn enclosed in <tool_call> tags (Li et al., 2 Feb 2026).

Grounding is defined as converting a high-level intended action into a concrete executable operation on the current web page. For point-based actions, the preferred output is a normalized coordinate. For text entry, the paper describes a staged protocol: first attempt direct coordinate-based input via a virtual keyboard; if that fails or verification shows inconsistency, fall back to structural element targeting; and finally perform a global search over candidate input fields ranked by spatial proximity or semantic similarity, with an LLM resolving remaining ambiguities. Dropdown handling is likewise specialized: it begins with direct value assignment through script-level manipulation, then falls back to semantic search over selectable elements if needed (Li et al., 2 Feb 2026).

The state-change verifier is tightly coupled to this design. It checks whether an action had a meaningful effect by comparing page states before and after the action. For actions like click and type, if no change is detected, the action is flagged as failed. For typing, the system additionally reads back the field value and verifies it matches the intended text. Confirmed failures are added to a failure-reflection buffer and used to guide future reasoning (Li et al., 2 Feb 2026).

This observation-action design differs from earlier website-navigation formulations such as WebVLN, which abstracts each state as a full-page screenshot plus a set of clickable buttons and uses a discrete button-selection action space with an [EOA] stop action (Chen et al., 2023). This suggests that Avenir-Web targets a more operationally realistic class of live-web tasks, including dynamic widgets and long multi-step trajectories, whereas WebVLN is organized around question-based navigation over a simulator.

5. Evaluation on Online-Mind2Web

The benchmark is Online-Mind2Web, described as a challenging live-web benchmark with 300 tasks across 136 websites. The paper emphasizes that it is user-centered and realistic, involving dynamic content, complex DOM structures, and state-dependent workflows. The main metric is Task Success Rate, evaluated with an LLM-as-a-Judge setup powered by o4-mini, which the paper says achieves 85.7% agreement with human judgment and only a 3.8% success-rate gap from human evaluation (Li et al., 2 Feb 2026).

The headline result is that Avenir-Web with Gemini 3 Pro achieves 53.7% overall TSR. Broken down by difficulty, it gets 74.1% on Easy, 54.6% on Medium, and 30.3% on Hard tasks. This exceeds the reported open-source baselines: SeeAct gets 30.0% overall, Agent-E 27.0%, and Browser Use 26.0%. The fully open-source Avenir-Web variant with Qwen-3-VL-8B gets 25.7% overall, with 42.0% on Easy, 23.8% on Medium, and 11.8% on Hard (Li et al., 2 Feb 2026).

Relative to proprietary systems, Avenir-Web does not surpass the top performers in the comparison table: Yutori Navigator scores 64.7%, OpenAI Operator 58.3%, and Google Gemini 2.5 Computer Use 57.3%. It does, however, outperform ACT-1-20250814 at 52.7% and Claude Computer Use 3.7 at 47.3%. The authors therefore characterize it as reaching “performance parity with top-tier proprietary models” in the sense of entering the same performance band and beating some important commercial baselines (Li et al., 2 Feb 2026).

The ablation study is performed on a 50-task subset of Online-Mind2Web with Gemini 3 Flash as backbone. The full ablated model scores 48.0%. Removing the Task-Tracking Checklist reduces TSR to 44.0%. Removing Adaptive Memory and replacing it with a fixed sliding window of $t$7 reduces TSR to 42.0%. Replacing adaptive memory with full-context $t$8 drops performance further to 36.0%. Removing EIP also yields 36.0%. Removing MoGE gives 40.0% (Li et al., 2 Feb 2026).

These ablations are used to argue that the biggest degradations come from losing EIP or using poor memory management, with MoGE also contributing strongly. A plausible implication is that Avenir-Web’s reported gains are architectural rather than attributable to a single stronger backbone model.

6. Position in adjacent research and stated limitations

Avenir-Web belongs to a broader research landscape concerned with multimodal perception, agentic browsing, and accessibility-sensitive web interaction. The case study on Agentic Web Browsers describes AWBs as LLM-powered systems that autonomously navigate and interact with the web in response to natural language instructions, and reports that such systems can make navigation notably fluid and flexible while still exhibiting shortcomings in non-visual feedback, control, transparency, and trust (Colazzo et al., 23 Jun 2026). This suggests that Avenir-Web’s emphasis on explicit task tracking, failure reflection, and strategic guidance addresses reliability bottlenecks, but does not by itself resolve the human-control and transparency questions emphasized in accessibility-oriented work.

The paper’s error analysis and limitations are candid but somewhat scattered. Grounding is still imperfect and bounded by backbone capability. Latency and computational cost remain concerns because the framework depends on large MLLMs and several modules. The system may still be brittle on extremely complex interfaces or fine-grained interactions, especially if both visual and structural cues are weak (Li et al., 2 Feb 2026).

A further limitation is dependence on website accessibility and anti-bot infrastructure. The authors intentionally avoid stealth, CAPTCHA bypassing, and evasive header masking, and report that about 10% of tasks in their most successful run—31 out of 300—were blocked before any action could be performed. This means real-world performance is partly constrained by website defenses rather than only by agent intelligence (Li et al., 2 Feb 2026).

The paper also notes ethical and safety limitations, including privacy risks, dangerous actions in domains like finance or official forms, and the need for safety controls in deployment. These concerns align with the AWI position paper’s claim that agent-native web interfaces should preserve human agency, safety and privacy, and should have defense mechanisms against malicious agents, malicious content, and catastrophic failures during trajectories (Lù et al., 12 Jun 2025).

Overall, Avenir-Web’s significance lies less in a single learned algorithm than in a specific systems thesis: long-horizon web automation is improved by integrating visual-first grounding with structural fallback, retrieval-based site-specific planning, explicit checklist tracking, and recursive failure-aware memory. In that sense, it is best understood as an architectural synthesis for reliable live-web agents rather than as a monolithic end-to-end policy (Li et al., 2 Feb 2026).