Embodied Web Agents: Integrated Task Environments

• Embodied Web Agents Task Environments are unified testbeds that blend realistic 3D physical interactions with digital web reasoning.
• They integrate a comprehensive state and action space, enabling agents to fluidly switch between embodied actions and web-based operations.
• Benchmarks spanning cooking, navigation, and geolocation highlight integration challenges and emphasize the need for coordinated cross-domain intelligence.

An embodied web agent task environment is a unified testbed where AI agents must exhibit physical embodiment—such as perceiving and acting in realistic 3D worlds—while also reasoning over and interacting with web-scale information. The Embodied Web Agents paradigm advances this integration, enabling agents to fluidly coordinate between embodied actions and digital knowledge acquisition to solve complex tasks that neither domain alone is sufficient to address. This is operationalized through a new generation of simulation platforms and benchmarks that tightly couple realistic indoor and outdoor physical environments with fully functional, interactable web interfaces, thereby providing a platform for systematic paper of cross-domain intelligence.

1. Cross-Domain Embodied Agent Model

The Embodied Web Agent model combines physical embodiment and web-scale reasoning within a unified formalism. The environment is defined as:

$E = \langle S, A, O, T \rangle$

• $S$: Combined State Space—encapsulates both physical states (e.g., object arrangements, robot pose, kitchen conditions) and digital/web states (e.g., browser history, website content, shopping cart).
• $A$: Action Space—includes both embodied actions (move, interact, pick, cook), and digital/web actions (search, click, navigate tabs, fill forms).
• $O$: Observation Space—joint observation tuple $(o^e_t, o^w_t)$, where $o^e_t$ is the agent's sensor input (RGB-D, language, feedback) in the physical field, and $o^w_t$ comprises current web perceptions (page content, forms, search results).
• $T$: Transition Function—encodes deterministic or stochastic transitions for both domains, capturing the effects of actions in both the 3D and web context.
• $r(a_{1:T}, s_{1:T})$: Reward functions evaluate agent performance against both embodied and digital goal criteria.

Agents in this environment are architected to (1) ground high-level instructions spanning both worlds, (2) plan stepwise actions flexibly across domains, and (3) process multimodal perceptual streams to guide their decisions.

2. Simulation Platform: Physical and Web Integration

The Embodied Web Agents simulation platform fuses realistic physical and digital environments:

• Physical Worlds:
  • Outdoor Environment: Modeled with city-scale navigation graphs built from Google Street View and Earth APIs, supporting detailed multi-city exploration. States are panoramic, geolocated photographic nodes; edges represent realistic traversable routes.
  • Indoor Environment: Based on AI2-THOR, providing photo-realistic, manipulable kitchen scenes with actionable objects and stateful interactions (pick, cook, slice, etc).
• Web Interfaces:
  • Functional, Unconstrained Web UIs: Suite of custom and adapted websites (using React+FastAPI) including recipe search, inventory-integrated shopping carts, OpenStreetMap-based directions, and Wikipedia for entity lookup and multi-modal information retrieval.
  • Tab Management and Context Preservation: Agents maintain context across multiple sites, switching and recalling web pages as required.
• Seamless Switching: The simulation provides explicit actions for agents to shift context—moving from the physical to the web (and vice versa)—mirroring how humans interleave real-world actions with digital search and manipulation.

3. Benchmark: Diverse Cross-Domain Task Suite

A large-scale benchmark (the Embodied Web Agents Benchmark) is constructed to evaluate the breadth and depth of cross-domain agent intelligence. The benchmark comprises approximately 1,500 tasks organized into major categories:

• Cooking Tasks: Agents fetch web recipes, check physical ingredient and tool availability, execute cooking steps, and adjust based on real-time physical feedback.
• Navigation Tasks: Agents receive or seek route guidance from web maps, then physically traverse real (simulated) city environments.
• Shopping Tasks: Agents optimize purchase plans using online comparison and then retrieve items in physical space.
• Travel/Tourism Tasks: Agents explore landmarks, consult web encyclopedias, and resolve complex entity disambiguation.
• Geolocation Tasks: Inspired by GeoGuessr, agents must determine their simulated location through embodied exploration and web queries.

Across all tasks, success is scored both jointly and in subdomains (web-only, embodied-only, and combined), allowing detailed analysis of agent capabilities and bottlenecks.

4. Performance Findings and Key Challenges

Empirical evaluation shows a pronounced performance gap between current state-of-the-art AI systems and human baselines:

• Human performance remains substantially higher: For cooking, humans achieve 77.08% accuracy versus 6.4% for the best LLM agent (GPT-4o).
• Integration failures dominate: While agents often succeed in web or embodied sub-task portions, over 66% of errors occur at domain interfaces (failure to switch, ground instructions, or carry context between realms).
• Partial success: Agents demonstrate competence when operating exclusively in a single domain, but struggle with tasks requiring coordinated reasoning and sequential planning across both.
• Geolocation improvement: Allowing agents to interactively explore and query the web significantly raises their geo-inference accuracy, underscoring the unique value of cross-domain interaction.

This suggests that integration, rather than mastery of any single domain, is currently the paramount technical challenge.

5. Action Spaces, Planning, and Evaluation Protocols

Action design in these environments is modular and granular:

• Embodied actions: Navigation, object manipulation (pick/put/slice/cook), spatial measurements.
• Web actions: Click-element, type, scroll, switch tabs, search, follow links, manage browser state.
• Switch/Mode actions: Explicit steps to shift focus between the embodied and web context.

Planning algorithms (e.g., Dijkstra's for shortest path in map graphs) and deterministic evaluators for kitchen states are integrated into the benchmarking protocol. Reward signals penalize incomplete or incorrect subtask completion and misaligned cross-domain actions.

6. Datasets, Code, and Community Resources

All resources supporting the Embodied Web Agent paradigm are openly released:

• Project Page: https://embodied-web-agent.github.io/
• Environment Access: The complete interactive web environment is available at http://98.80.38.242:1220/.
• Data & Code: Full task templates, web sites, domain graphs, annotation pipelines, and evaluation tools are provided for community use and extension.

Supplementary material includes prompt templates, data statistics, and tools for new task construction and comprehensive evaluation.

7. Broader Impact and Research Directions

The Embodied Web Agents environment catalyzes a new research frontier by:

• Providing the first high-fidelity, cross-domain testbed for integrating physical and digital intelligence in agents.
• Enabling systematic investigation of planning, grounding, perception, and sequential reasoning across both real-world and web domains.
• Highlighting cross-domain integration bottlenecks and quantifying the performance deficit separating current AI from human problem solving in integrated tasks.
• Opening practical avenues for applications in assistive robotics, multi-modal virtual assistants, and AI systems requiring robust, contextualized understanding.

A plausible implication is that progress on this benchmark will accelerate the development of general-purpose AI systems for contexts where digital information must be actively combined with physical action and perception.

---

Summary Table: Key Features

Aspect	Description
Conceptual Model	Unified agent operating across embodied (physical) and web (digital) realms
Core Tasks	Cooking, navigation, shopping, tourism, geolocation; cross-domain required
Action Space	Combined physical and web interaction primitives
Evaluation	Joint and decomposed (web/embodied) success; ground truth from human and deterministic evaluators
Performance	Human > AI by significant margin across all domains
Resources	Full code, task datasets, and online environments—project page

---

The Embodied Web Agents platform and benchmark establish a comprehensive, extensible foundation for studying integrated, multimodal intelligence in AI, making it a central resource for researchers pursuing advances in embodied cognition and web-scale reasoning.