XARP Tools: XR Framework for Humans and AI
- XARP Tools is an extended reality framework offering a server-side Python library and platform-specific XR clients for streamlined human and AI development.
- It reduces accidental complexity and platform fragmentation by decoupling high-level application logic from device-specific execution.
- The API exposes primitives like read, write, see, and head_pose, enabling rapid prototyping and integration with both MCP and agent toolkits.
to=arxiv_search.search 彩票天天乐json {"2query2 OR \2"XARP Tools: An Extended Reality Platform for Humans and AI Agents\"","max_results":5,"sort_by":"relevance"} XARP Tools is an extended reality framework designed for both human and AI developers, comprising a server-side Python library and platform-specific XR clients. Its stated purpose is to address three persistent problems in XR: high accidental complexity, fragmentation across platforms, and a missing bridge to AI ecosystems. The framework can be used in three modes: as a Python library for human XR developers, as a set of callable tools for AI agents, and as a Model Context Protocol server that exposes XR devices to MCP-compatible ecosystems (&&&2query2&&&).
2(Caetano et al., 6 Aug 2025) OR \2. Definition and stated objectives
XARP Tools, expanded as XR Agent-ready Remote Platform, is presented as a two-part framework: a server-side Python library and platform-specific XR clients. The library offers high-level APIs and communicates with clients via a JSON-based protocol over WebSockets, while the clients encapsulate device and runtime specifics and provide responsive, low-latency user interaction (&&&2query2&&&).
The framework is explicitly motivated by three problems. First, high accidental complexity arises because engine-specific expertise, asset pipelines, and frequent redeployment slow iteration. Second, platform fragmentation complicates cross-device development because runtimes and APIs differ across XR systems. Third, XR devices are not readily accessible to LLM agents through standardized tool protocols, creating what the report describes as a missing bridge to AI ecosystems (&&&2query2&&&).
The design principles are correspondingly narrow and functional. XARP separates concerns so that the Python server handles application logic while the XR client handles low-latency rendering and device-runtime specifics. It exposes a minimal, uniform API built around the primitives read, write, see, and head pose. It is also intended to support rapid prototyping by permitting behavior changes on the server without rebuilding or redeploying XR clients, and it includes a web UI to mock XR devices. Finally, it is described as multiplatform and adaptable: clients for different XR platforms can connect as long as they implement the same primitive tools, and capability discovery allows applications and agents to adapt to heterogeneous devices (&&&2query2&&&).
A plausible implication is that XARP is positioned less as a monolithic XR engine than as an orchestration layer over engine-bound runtimes. That interpretation is consistent with the report’s repeated emphasis on keeping application logic in Python while preserving client-side execution for latency-sensitive work.
2. Architecture and communication model
The architecture is divided into a server and a client. On the server side, the report identifies a Python library centered on a class named XRApp, imported in the examples as from xarp import XRApp. The reference server stack integrates FastAPI for WebSocket endpoints, smolagents for agent tooling, fastmcp for MCP server plumbing, and LMStudio for local MCP testing. On the client side, the reference implementation is a Unity client targeting Meta Quest, and a separate web client is provided for local development and rapid testing. The client maps server commands into XR building blocks via a factory pattern (&&&2query2&&&).
Communication occurs through a JSON-based application protocol over WebSockets. Clients declare supported tools and primitives during connection, enabling capability discovery and adaptive behavior. The server owns application logic, including prompting, sequencing tools, and orchestrating agent or MCP integration. It accepts WebSocket connections from XR clients, sends commands such as write, read, see, and head_pose, and receives results or events. The client handles rendering, interaction, voice input, and camera capture, executes server commands through engine- or runtime-specific APIs, and reports results back via JSON messages (&&&2query2&&&).
The threading and event-loop model is based on Python’s asyncio. In the examples, a FastAPI @app.websocket('/ws') endpoint accepts the client connection, the running event loop is obtained through asyncio.get_running_loop(), and XRApp is constructed with the WebSocket and event loop. The report’s examples show xr = XRApp(ws, loop) in Agent and MCP modes and xr = XRApp(ws, loop, session) in the Library example (&&&2query2&&&).
The architectural rationale is explicitly tied to latency. Pose tracking, rendering, camera capture, and voice input are handled on the client side to minimize round trips, while high-level orchestration remains on the server and communicates through WebSockets. The report does not provide latency metrics, but it states that responsiveness benefits from keeping engine-bound work in the client and application logic in Python (&&&2query2&&&).
3. API surface and operating modes
The API surface documented in the report is deliberately small. The examples consistently use the XRApp constructor and the methods write, listen, see, as_tools, and as_mcp_server. The report does not define the full constructor signature, but the examples show XRApp(WebSocket, asyncio event loop[, session]) (&&&2query2&&&).
The methods used in the examples are as follows:
| Method | Reported behavior | Mode(s) shown |
|---|---|---|
xr.write(message: str) |
Display text to the user via client UI | Library, agent workflows |
xr.listen() -> str |
Request text input from the user and return the string | Library, agent workflows |
xr.see() -> image |
Capture an RGB frame from the device | Agent workflows |
xr.as_tools() -> tools |
Return callable tools for agent integration | Agent Toolkit Mode |
xr.as_mcp_server() -> mcp_server |
Construct an MCP-compatible server instance | MCP Mode |
The framework is described as serving three audiences and three modes of use. In Library Mode, human XR developers use the Python API to prototype and sequence XR interactions without engine-specific code. In Agent Toolkit Mode, the same Python API is exposed as callable tools so that LLM agents can orchestrate on-the-fly user interactions using sensing and actuation on XR devices. In MCP Mode, XR capabilities are exposed as Model Context Protocol endpoints so that XR devices plug into AI ecosystems and MCP-compatible UIs such as LM Studio (&&&2query2&&&).
The Library Mode example is a minimal echo application. A FastAPI WebSocket endpoint accepts a connection, constructs XRApp, writes a prompt, listens for user input, and writes a response. The example printed in the report is:
PRESERVED_PLACEHOLDER_2query2^
The report notes that this snippet demonstrates the high-level intent, but it does not provide the implementation of XRApp or session and directs readers to the repository for a fully runnable example and exact signatures (&&&2query2&&&).
4. Agent tooling and MCP integration
In Agent Toolkit Mode, XRApp.as_tools() converts XR capabilities into tools consumable by agent frameworks. The report’s example uses smolagents with OpenAIServerModel and CodeAgent. The illustrated workflow is to construct XRApp, obtain tools via xr.as_tools(), capture an image via xr.see(), create a CodeAgent with tools and model, and then call agent.run('Help me to fix this!', images=[img]) (&&&2query2&&&).
The example printed in the report is:
PRESERVED_PLACEHOLDER_2(Caetano et al., 6 Aug 2025) OR \2^
This usage pattern is summarized in the report as sense, ask, and contextualize. The agent can sense through xr.see(), prompt through xr.write(), and collect user input through xr.listen(). The report states that the agent can invoke tools to interact with the user or device in real time, orchestrated by model outputs. It also notes a constraint in the opposite direction: rate limits, permissions, and guardrails are not specified, although capability discovery is present so that agents can adapt to available tools (&&&2query2&&&).
In MCP Mode, XRApp.as_mcp_server() produces an MCP-compatible server object, and mcp.run_streamable_http_async() starts a streamable HTTP server for MCP integration. The example shown in the report is:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
import asyncio from fastapi import FastAPI, WebSocket from smolagents import OpenAIServerModel, CodeAgent from xarp import XRApp app = FastAPI() @app.websocket('/ws') async def ws_connect(ws: WebSocket): await ws.accept() loop = asyncio.get_running_loop() xr = XRApp(ws, loop) mcp = xr.as_mcp_server() await mcp.run_streamable_http_async() |
LMStudio is identified as a tool for quick local testing through its MCP plugin capabilities, and the architecture figure described in the report explicitly shows LMStudio MCP integration with XARP. The paper does not enumerate MCP endpoints or schemas, and it does not describe handshake or registration details in depth (&&&2query2&&&).
5. Clients, primitives, and capability discovery
The client side currently consists of two reference implementations: a Unity reference client targeting Meta Quest and a web client for rapid testing. The Unity client targets Meta Quest, with Quest 3 referenced in the acknowledgments, while the web client supports local development, mobile usage, and image uploads to simulate the see primitive (&&&2query2&&&).
The report defines a minimal tool set exposed through the uniform API: read, write, see, and head_pose. In the Python-facing examples, listen() corresponds to read in the tool list. On Meta Quest, the read tool uses virtual keyboard input with a voice option to gather user text, and the write tool displays text to the user. The see tool captures an RGB frame from the left-eye egocentric camera. The head_pose tool provides quaternion orientation and position, described as HMD pose (&&&2query2&&&).
The web client’s see tool allows image uploads from the camera on mobile devices for rapid testing. The report uses this to illustrate interchangeability: as long as clients implement the same high-level tools, they can connect to the same server application. That interchangeability is mediated by capability discovery during connection, where each client declares the tools and primitives it supports (&&&2query2&&&).
The protocol description remains intentionally incomplete in the report. It specifies that XARP uses a JSON-based protocol over WebSockets and includes a capability discovery phase, but it does not publish the exact message schema, handshake sequence, versioning, or error codes. The report identifies command messages from server to client for read, write, see, and head_pose, responses from client to server containing results or event notifications, and a capability discovery payload declaring supported tools and primitives on connect. It also sketches a conceptual connection lifecycle of WebSocket accept, capability discovery, and ongoing bidirectional messaging (&&&2query2&&&).
The report gives conceptual JSON examples, but explicitly marks them as illustrative rather than authoritative. This distinction matters because the message schema is not treated as fixed in the text. A plausible implication is that the authoritative protocol specification is intended to reside in the repository rather than in the technical report.
6. Workflow, deployment, and practical scenarios
The report identifies the code repository as https://github.com/HAL-UCSB/xarp and states that code and working examples are released openly there. The development environment includes Python and asyncio on the server, FastAPI for WebSocket endpoints, smolagents for agent integration, fastmcp for MCP server plumbing, LMStudio for local MCP testing, Unity for the Meta Quest client, and a web client for local development and rapid testing (&&&2query2&&&).
The setup path described in the report is procedural rather than packaged. It consists of cloning the repository with git clone https://github.com/HAL-UCSB/xarp, creating and activating a Python environment, installing dependencies listed in the repository, and running a server script that spins up FastAPI, defines a @app.websocket('/ws') endpoint, constructs XRApp on connection, and exercises Library, Agent, or MCP mode as desired. For MCP testing, LMStudio is used with MCP plugin support. For device testing, the Unity Meta Quest client is deployed to a device, or the web client is used for rapid local testing (&&&2query2&&&).
Three demonstration scenarios are identified. The first is the echo application in Library Mode, where the server writes “I am listening,” listens to user input, and echoes it back with a question mark appended. The second is agentic assistance in Agent Toolkit Mode, where an agent receives an image from the device through see and a textual instruction, “Help me to fix this!”, and then uses XARP tools to interact with and guide the user. The third is MCP integration, in which the XR device is exposed as an MCP server and LMStudio connects to it via MCP for local agent interaction (&&&2query2&&&).
The report also describes conceptual message sequences for these scenarios. In the echo flow, the server sends write("I am listening"), the client returns a read_result, and the server sends write(f"{user_text}?"). In the agent flow, the server requests see(), the client returns a see_result, and the agent may then trigger further write, read, see, or head_pose commands (&&&2query2&&&).
7. Positioning, limitations, and unresolved aspects
The report positions XARP relative to engine-centric XR frameworks such as Unity, Unreal, and native OpenXR or WebXR by arguing that XARP abstracts away engine-specific mechanics and asset deployment from application logic, enabling Python-based sequencing and rapid iteration. The XR clients remain engine-specific, but the server-side logic is intended to remain uniform across devices. Relative to existing agent-tool systems and MCP, the report identifies XARP’s novelty as exposing XR devices as first-class tools or endpoints so that agents can sense images and pose and interact with users through read and write operations in immersive contexts, without embedding agent logic into the engine runtime (&&&2query2&&&).
The stated trade-offs are correspondingly explicit. A minimal tool set consisting of read, write, see, and head pose simplifies cross-platform support but may limit advanced interactions until the primitive set is expanded. JSON over WebSockets introduces network dependency, so low-latency tasks must remain in the client (&&&2query2&&&).
The report is equally explicit about what it does not specify. It does not detail authentication or authorization, privacy protections, data handling or sandboxing, permission models beyond capability declaration, error codes, reconnection strategies, resource management, or performance metrics. It also does not provide empirical latency, throughput, or user-study results. These omissions are not incidental: the report distinguishes concrete elements such as the architecture split, the JSON-over-WebSocket transport, capability discovery, the XRApp class and example methods, and the reference stacks from unspecified elements such as exact JSON schema, handshake frames, versioning, complete XRApp signatures, security features, and performance formulas (&&&2query2&&&).
The current functionality is described as minimal, and the present clients are limited to a Meta Quest application and a web chat interface, although broader platform support is said to be planned. The report further states that ongoing case studies with researchers are shaping the feature set and developer experience, and that at least two case studies are under peer review. Planned workshops with researchers and industry practitioners are intended to evaluate benefits and limitations and to explore human–AI collaborative XR development opportunities (&&&2query2&&&).
Taken together, these details indicate that XARP Tools is not presented as a finished XR middleware standard, but as a deliberately thin, extensible interface layer between XR runtimes and Python-based human or agent orchestration. This suggests that its significance lies less in exhaustive XR capability coverage than in its attempt to normalize XR devices as callable resources within emerging AI tool and MCP ecosystems (&&&2query2&&&).
