PRISM-XR: Privacy-Aware XR Collaboration

• PRISM-XR is a privacy-aware, multi-user XR collaboration framework that integrates multimodal large language models by processing sensor data at the edge to avoid raw data exposure.
• It leverages YOLO v11 for object detection and converts detections into structured textual summaries, ensuring secure scene understanding without compromising sensitive details.
• The framework uses efficient registration with AprilTag and ownership-aware synchronization, achieving low latency and dynamic XR task execution while preserving privacy.

PRISM-XR is a privacy-aware, multi-user XR collaboration framework that integrates multimodal LLMs (MLLMs) without exposing raw sensor data to untrusted cloud endpoints. It is designed for XR settings in which head-mounted devices capture real-world visual context, but that same context may contain bystander faces, personally identifying documents, financial cards, medical records, or other sensitive objects. The framework addresses three coupled challenges: image privacy risks from head-mounted cameras, the inefficiency and privacy cost of commercial colocation mechanisms that require cloud-backed environment scanning, and the difficulty of synchronizing dynamic, MLLM-driven scene changes in XR environments (Chen et al., 9 Feb 2026).

1. Definition and scope

PRISM-XR was introduced as a framework for enabling privacy-aware MLLM integration in collaborative XR. Its motivating use cases include requests such as “add a whiteboard here,” “summarize what we just drew,” and “move the cube next to the chair,” where language and visual context must be combined to generate object-creation, animation, or scene-understanding actions (Chen et al., 9 Feb 2026).

The central design decision is architectural minimization. By default, no raw camera frames are sent to the cloud. Instead, PRISM-XR performs object detection on an edge server, converts detections into a structured textual description, and sends that distilled representation to the cloud MLLM. If visual details are necessary, the cloud model can request a cropped image region, but the crop is generated on the edge, sensitive objects are excluded, and an in-XR confirmation dialog is shown before upload. Recorded audio is also kept local or on-edge: keyword activation and audio capture run on the XR device, Whisper transcription runs offline on the edge, and recorded audio is not sent to the cloud (Chen et al., 9 Feb 2026).

This places PRISM-XR at the intersection of XR collaboration, edge-cloud AI orchestration, privacy-aware sensing, and dynamic scene synchronization. A plausible implication is that the system is less a single algorithm than a compositional framework: privacy protection, registration, command grounding, and synchronization are treated as mutually dependent subsystems rather than as separable implementation layers.

2. Architectural organization

PRISM-XR comprises XR clients, an edge server, and a cloud MLLM endpoint, supplemented by content-sharing and consent modules. The reported client platforms are Quest 3 and HoloLens 2. On HoloLens, keyword detection uses MRTK3’s KeywordRecognitionSubsystem; on Quest, it uses Porcupine. Audio capture and silence detection run locally. FoV snapshots and camera metadata are then sent to the edge server over WebSockets, using System.Net.WebSockets; image capture uses PhotoCapture on HoloLens and the Passthrough Camera API on Quest (Chen et al., 9 Feb 2026).

At the edge, Whisper performs offline transcription and Ultralytics YOLO v11 detects objects in the captured frame. The edge converts detections into textual descriptions containing object name, center pixel, bounding box, and confidence. If the text-only scene description is insufficient, the cloud MLLM returns a CropArea; the edge then generates the crop, excludes sensitive objects, presents a confirmation dialog to the client, and uploads the crop only upon consent. The edge also maintains object registries and ownership, parses structured MLLM outputs, and broadcasts XR actions to collaborating users (Chen et al., 9 Feb 2026).

The cloud side is used for two distinct stages. In the initial stage, the model classifies the request, decides whether a crop is needed, and returns either a CropArea or None. In the refined stage, it emits structured JSON for XR tasks such as object creation, animation, or movement, including prefab names, coordinate space, and positions. GPT-4o was selected for the user studies, while o1 and o3-mini were also evaluated; edge-hosted LLaVA 13B and Llama 3.2 Vision 11B via Ollama were evaluated but underperformed (Chen et al., 9 Feb 2026).

The overall dataflow is therefore asymmetric: sensing and privacy filtering are pushed to the edge, while semantic interpretation and generative command synthesis are delegated to the cloud. This suggests that PRISM-XR treats privacy not as post hoc redaction, but as a constraint on what representations are allowed to leave the local trust boundary.

3. Privacy model and frame preprocessing

The privacy mechanism in PRISM-XR is organized around intelligent frame preprocessing on the edge server. YOLO v11 produces object labels, bounding boxes, centers, and confidences; the framework then transforms these detections into concise textual FoV summaries. For spatially grounded tasks, pixel-space locations are mapped into world coordinates by raycasting from the XR device pose into real-world colliders built by the headset (Chen et al., 9 Feb 2026).

The geometric model is expressed using the pinhole camera equation

$x_{\text{img}} \sim K [R \mid t] X_{\text{world}}.$

Given a pixel $x = (u,v,1)^\top$, PRISM-XR computes an unprojection direction in camera coordinates as $d_c \propto K^{-1}x$, normalizes it, and constructs the world ray

$r(t) = c + td,$

where $c$ is the camera center and $d = Rd_c$. The first hit point on the headset’s real-world colliders becomes the initial world position for the generated object, after which pose correction offsets by object size and aligns with the surface normal (Chen et al., 9 Feb 2026).

Sensitive categories considered in the environment and user study include human face, ID card/driver’s license, mail/personal letter/statement, medicine/medical record, and credit cards. Categories such as laptop/phone, monitor, bags, books, desks, chairs, cups, and lab equipment are described as “maybe sensitive” or insensitive. The operational policy is that only textual summaries are sent by default; cropped imagery is uploaded only when required, only after consent, and only after excluding sensitive detections. If sensitive overlap is unavoidable, the system falls back to text-only context or asks the user to reposition the target (Chen et al., 9 Feb 2026).

The paper also presents a formalized risk score for a candidate crop:

$R(C) = \sum_{i \in S_C} w_{L(c_i)} p_i,$

where $S_C$ is the set of detections overlapping the crop, $L(c_i) \in \{0,1,2\}$ is the sensitivity level, and $p_i$ is detector confidence. A conservative upload rule is then

$x = (u,v,1)^\top$0

The implementation, however, uses binary gating: any highly sensitive content triggers a consent check and exclusion, rather than exposing a numerical threshold as a deployed parameter (Chen et al., 9 Feb 2026).

The privacy guarantees are therefore architectural rather than cryptographic. Raw frames never leave the edge server, recorded audio is not sent to the cloud, and crop uploads require explicit user consent. Residual risks remain: textual leakage about sensitive objects, accidental consent “click-through,” and re-identification from insufficient redaction are explicitly acknowledged (Chen et al., 9 Feb 2026).

4. Registration and synchronization in collaborative XR

A second core contribution of PRISM-XR is a lightweight colocation and synchronization mechanism intended to replace privacy-invasive commercial anchor-sharing pipelines. Registration is performed once, using an AprilTag. When a user says “registration,” the system captures a frame, detects tags, and stores the tag pose in that user’s world frame. After that one-time alignment, the tag need not remain visible; native headset tracking is used for subsequent localization (Chen et al., 9 Feb 2026).

AprilTag pose estimation is written in standard PnP form:

$x = (u,v,1)^\top$1

with known tag corners $x = (u,v,1)^\top$2 and image points $x = (u,v,1)^\top$3. If users $x = (u,v,1)^\top$4 and $x = (u,v,1)^\top$5 both register to the same tag $x = (u,v,1)^\top$6, PRISM-XR stores $x = (u,v,1)^\top$7 and $x = (u,v,1)^\top$8 and computes the cross-user transform

$x = (u,v,1)^\top$9

so that an object pose $d_c \propto K^{-1}x$0 in user $d_c \propto K^{-1}x$1’s world is mapped into user $d_c \propto K^{-1}x$2’s world as

$d_c \propto K^{-1}x$3

Spatial inconsistency is measured as

$d_c \propto K^{-1}x$4

Within 3 m registration distances, the reported mean inconsistency is approximately 3.46 cm, and the system is described as not drifting over time because post-registration localization relies on the HMD’s native tracking (Chen et al., 9 Feb 2026).

Synchronization is ownership-aware and tuned for dynamic, MLLM-generated content. PRISM-XR distinguishes world space, local space, and pixel space. The request-issuing user is the owner; owner-side resolution maps local or pixel coordinates into world coordinates, after which commands are broadcast to other users and transformed into their coordinate frames. Object states are serialized into 48-byte packets containing id, position, quaternion rotation, scale, and event flags, and transmitted at 60 Hz over native sockets. The reported interaction synchronization latency is approximately 15.5 ms (Chen et al., 9 Feb 2026).

This design differs from static-scene synchronization models in commercial XR APIs. Rather than synchronizing dense scans or pre-authored scene graphs, PRISM-XR synchronizes compact representations of dynamic object state. A plausible implication is that the framework is optimized not for persistent scene reconstruction, but for rapid propagation of semantically generated XR content.

5. MLLM integration and task grounding

PRISM-XR uses a two-stage prompting pipeline to reduce ambiguity and token cost. In the initial stage, the cloud model receives the transcribed request and the YOLO-derived textual FoV description, then classifies the intent, determines whether image evidence is needed, and returns a CropArea or None. In the refined stage, it produces schema-constrained JSON for actions such as “create object,” “move object,” or “animate,” including coordinate-space tags and pose parameters (Chen et al., 9 Feb 2026).

Structured outputs are central to this design. They constrain the model to a JSON schema in which positions must be [x,y,z] floats and command semantics are explicit. This is intended to reduce post-processing ambiguity and to make the MLLM an upstream planner rather than an unconstrained natural-language generator. GPT-4o achieved the strongest overall performance among the evaluated cloud models, while the tested edge vision-LLMs showed poor accuracy and high fallout (Chen et al., 9 Feb 2026).

The system’s numerical evaluation separates initial-stage and refined-stage performance. In the initial stage, GPT-4o achieved 89.33% classification accuracy, 78.57% crop recall using IoU $d_c \propto K^{-1}x$5, 1.25% crop fallout, and 2.415 s generation time. In the refined stage, GPT-4o reached 97% success for object creation and 99% for animation creation, with generation times of approximately 2.5–3.6 s (Chen et al., 9 Feb 2026).

An important consequence of this staged design is that the cloud MLLM need not reason over full raw images in every case. It reasons first over symbolic scene summaries and only receives pixel data when the symbolic representation is insufficient. This suggests that PRISM-XR positions MLLMs not merely as perception engines, but as structured semantic controllers embedded within a privacy-constrained sensing pipeline.

6. Empirical evaluation and system performance

The paper reports both numerical evaluation and an IRB-approved user study with 28 participants. Numerically, the platform achieves nearly 90% accuracy in fulfilling user requests, less than 0.27 seconds registration time, and spatial inconsistencies of less than 3.5 cm. The same evaluation reports that highly sensitive objects were automatically filtered from more than 90% of scenarios in the user study while maintaining strong overall usability (Chen et al., 9 Feb 2026).

More detailed metrics show that registration took 0.27 s, compared with approximately 6.73 s for the Meta Shared Spatial Anchors baseline. The mean spatial inconsistency was 3.46 cm for registration distances of at most 3 m. The latency breakdown was reported as follows: transcription $d_c \propto K^{-1}x$6 s, initial stage $d_c \propto K^{-1}x$7 s, refined stage $d_c \propto K^{-1}x$8 s, local processing $d_c \propto K^{-1}x$9 s, communication $r(t) = c + td,$0 s, and TTS $r(t) = c + td,$1 s, yielding a total speak-to-action time of approximately $r(t) = c + td,$2 s, excluding user confirmation. Response synchronization was $r(t) = c + td,$3 s, and interaction synchronization was $r(t) = c + td,$4 s. The system was measured at approximately 72.34 FPS (Chen et al., 9 Feb 2026).

The privacy-filtering effect is quantified by the reduction in highly sensitive object presence from 53.57% of original frames to 7.14% of cropped frames. “Maybe sensitive” and insensitive categories also decreased, but were retained when necessary for task context. Usability evaluation used NASA-TLX and Likert-style questionnaires. NASA-TLX means were all below 6 out of 20, task completion times were approximately 2 minutes for object, animation, and scene-understanding tasks and approximately 3.4 minutes for collaborative drawing, and more than 91% of tasks were rated as success or minor issue. Privacy and collaboration received average scores of 5.6/7 and 5.8/7, while responsiveness was the lowest-rated dimension at 5.0/7 (Chen et al., 9 Feb 2026).

The dominant performance bottleneck is cloud inference rather than local synchronization. The paper explicitly notes that bandwidth is minimal because PRISM-XR shares structured commands rather than video streams or dense scans, and that ongoing object manipulation is not impeded by cloud inference delay once objects are instantiated (Chen et al., 9 Feb 2026).

7. Relation to other uses of the name, limitations, and future significance

The name “Prism XR” is not unique in the literature. A separate 2024 system, “Prism XR -- A Curated Exhibition Experience in Virtual Reality with Peer Annotation Features and Virtual Guides for Art and Archaeology Classes,” uses a similar name for an educational VR platform built in Unity3D for Meta Quest, featuring peer annotations, a guestbook, and a scripted curator-avatar implemented with Ink; that system emphasizes asynchronous collaboration in curated exhibitions rather than privacy-aware MLLM collaboration (Zhang, 2024). By contrast, the 2026 PRISM-XR framework is explicitly centered on privacy-aware MLLM integration, edge preprocessing, lightweight registration, and ownership-aware synchronization (Chen et al., 9 Feb 2026).

Within its own scope, PRISM-XR has several acknowledged limitations. Detection errors in YOLO may cause over-filtering or under-filtering; cropping can remove task-relevant context or expose too much irrelevant context; AprilTag registration can fail under occlusion or poor lighting; wake word detection and transcription may misfire under accents or noise; and the approximately 7.7 s speak-to-action latency remains noticeable because it is dominated by cloud inference. The reported study also focuses on dyads in a single lab environment, leaving broader environments and larger groups as open evaluation settings (Chen et al., 9 Feb 2026).

The framework’s significance lies in showing that collaborative XR with cloud MLLMs need not be built on indiscriminate upload of raw sensor streams. Instead, sensing can remain local or on-edge, visual context can be distilled into structured prompts, imagery can be reduced to minimal consented crops, and collaboration can rely on one-time tag-based alignment plus compact state synchronization. This suggests a more general design pattern for MLLM-integrated XR: privacy-aware context distillation at the edge, structured semantics in the cloud, and lightweight coordinate-frame alignment for multi-user scene coherence (Chen et al., 9 Feb 2026).