JARVIS: A Just-in-Time AR Visual Instruction System for Cross-Reality Task Guidance

Abstract: Many everyday tasks rely on external tutorials such as manuals and videos, requiring users to constantly switch between reading instructions and performing actions, which disrupts workflow and increases cognitive load. Augmented reality (AR) enables in-situ guidance, while recent advances in LLMs and vision-LLMs (VLMs) make it possible to automatically generate such guidance. However, existing AI-powered AR tutorial systems primarily focus on physical procedural tasks and provide limited support for hybrid physical and virtual workspaces. To address this gap, we conduct a formative study of cross-reality tasks and identify key requirements for state awareness and cross-reality coordination. We present JARVIS, a VLM-driven AR instruction system that generates contextual, step-by-step guidance from a single prompt, with real-time state verification and adaptive visual feedback. To inform the system design, we conducted a formative study to understand guidance needs across cross-reality tasks, which we categorize into four types, real-to-real (R2R), real-to-virtual (R2V), virtual-to-real (V2R), and virtual-to-virtual (V2V). A within-subjects study (N=14) across four domains shows JARVIS improves usability, workload, success rate, and visualization effectiveness over baselines.

• The paper introduces JARVIS, a just-in-time AR system that generates adaptive, multimodal guidance for tasks spanning real and virtual environments.
• It employs vision-language and large language models to dynamically plan and render cross-reality instructions, reducing error rates and completion times.
• Empirical studies demonstrate improved usability, lower cognitive workload, and enhanced task success compared to traditional AR guidance methods.

JARVIS: Just-in-Time AR Visual Instruction for Cross-Reality Task Guidance

Introduction and Motivation

Task guidance in everyday activities increasingly relies on external tutorials—manuals, videos, digital wizards—yet such resources impose significant workflow friction. Users must alternate between consuming instructions and performing each step, which elevates cognitive load and induces errors. While AR systems mitigate some friction via in-situ overlays, current AI-driven AR guidance platforms focus almost exclusively on physical procedures, neglecting hybrid workflows that traverse both real (physical) and virtual (digital) action spaces.

"JARVIS: A Just-in-Time AR Visual Instruction System for Cross-Reality Task Guidance" (2604.10108) addresses this critical gap. The paper introduces a system that generates stepwise, multimodal guidance for cross-reality tasks using advances in vision-LLMs (VLMs) and LLMs. JARVIS supports unified interaction and feedback across the complete "reality-virtuality continuum," enabling seamless hand-offs between real-world manipulation and digital operations.

Formative Study and Design Space

A formative study systematically evaluated user performance and preferences when following three instructional modalities—text, images, and video—across four representative tasks spanning both physical (e.g., origami, device setup) and virtual (e.g., digital painting, gaming) domains. Quantitative results established that image guidance led to statistically significant reductions in error rate and completion time compared to text, with both image and video outperforming text in subjective user ratings (Figure 1).

Figure 1: Error rate, completion time, and user rating distributions across text, image, and video guidance modalities; image guidance yields the lowest error and time.

Qualitative observations identified endemic challenges in hybrid task settings:

• Cross-reality transitions impose ambiguity, especially real-to-virtual and virtual-to-real steps where feedback is delayed or lacks sufficient grounding.
• State awareness deficits are common; participants often reported confusion about task progress, step relevance, and completion status.
• Modality preferences are task-dependent: users favor concise image/video guidance but demand explicit feedback for nuanced state tracking.

Based on these findings, tasks are decomposed into four atomic step types according to the referent/action locus: Real-to-Real (R2R), Real-to-Virtual (R2V), Virtual-to-Real (V2R), and Virtual-to-Virtual (V2V) (Figure 2).

Figure 2: Taxonomy of cross-reality step types, categorizing each by referent and action domain (real or virtual).

The derived design space incorporates:

• Persistent state cues: explicit panels showing step state, current/next goals (Figure 3)
• Target configuration previews: overlays of post-step object states for outcome comparison
• Multimodal feedback channels: real-time audio cues for step verification
• Action embodiment: guidance via hand/tool representations
• Static and motion cues: overlays for object localization and action dynamics

Figure 3: Persistent AR state cues, displaying the current goal, step, and next step during task execution.

System Architecture

JARVIS operates as a closed-loop AR instructional workflow comprising four principal modules (Figure 4):

1. Pre-Task Planner: Processes user prompts to retrieve online instructional media and generate a structured JSON task plan, including media segmentation and visualization strategy.
2. In-Task Planner: Verifies user state against expected criteria using VLMs; upon error, adapts the plan and invokes sub-planning for local corrections, with correction feedback delivered via multimodal overlays and audio.
3. Visual Renderer: Anchors guidance overlays in 3D space (using predicted 2D keypoints fused with environmental depth sensing) and renders multimodal augmentations (arrows, bounding boxes, gesture animations, previews).
4. User Interaction: Enables in-situ clarifications—users can issue voice questions and receive immediate, context-grounded updates to the AR overlays.

   Figure 4: JARVIS system pipeline, integrating task planning, in-task verification/recovery, multimodal visualization, and interactive voice clarification, shown for the origami folding use case.

The pipeline leverages Gemini 2.5 (Flash/Pro) for VLM inference and SAM3 for object segmentation, with task/scene state mediating all visualizations.

Multimodal Guidance and Cross-Reality Coverage

JARVIS supports rich, context-sensitive visualizations that dynamically adapt as users transition between step types. Figure 5 displays canonical guidance overlays across four benchmark tasks (latte making, digital painting, origami, gaming), illustrating the heterogeneity of visual feedback necessary for cross-reality workflows.

Figure 5: Visualization types produced by JARVIS, including configuration previews, gesture overlays, motion trajectories, bounding boxes, and state panels across four task domains.

Technical and User Study Evaluation

In an end-to-end technical assessment (8 tasks, 51 steps), JARVIS achieved 74.5% correct guidance generation, with component accuracies peaking at 90.2% for key component identification and 88.2% for textual instruction extraction. Visual type selection and image relevance (80.4% and 76.5% respectively) remain key error points, typically due to ambiguity in image retrieval and insufficient diversity of gestures.

User studies (N=14) compared JARVIS with strong AR baselines: static arrow (direction+text) and image reference overlays. Quantitative metrics showed:

• Usability: JARVIS led to higher System Usability Scale (SUS) scores than the arrow baseline and matched the image baseline.
• Workload: NASA-TLX scores were lower for JARVIS and image guidance than for the arrow baseline.
• Step-level task success: JARVIS had a correct completion rate of 96.3%, substantially surpassing both image (86.2%) and arrow baselines (74.3%) (Figure 6).
• Skip, error, and omission rates were lowest under JARVIS across all cross-reality step types.

  Figure 7: Output quality for four representative tasks under reference, arrow, image, and JARVIS conditions; JARVIS results most closely align with the ground truth.

  

  Figure 6: Per-step heatmap of completion outcomes across JARVIS and baseline systems; JARVIS exhibits minimal errors and high consistency across participants and task types.

Analysis of perceived effectiveness for visualization cues revealed that state cues and target configuration previews outperformed localization-based cues, underscoring the importance of state-centric AR representations in cross-reality settings (Figure 8).

Figure 8: Participant rating distributions for six visual cue types, with state cues and target configuration previews receiving highest effectiveness evaluations.

Implications, Limitations, and Future Directions

The results establish that cross-reality AR instruction must prioritize explicit state modeling, outcome previews, and feedback-level adaptation rather than relying solely on spatial localization. Key open challenges highlighted include:

• State verification latency: Current step-state checks are user-triggered; continuous monitoring is desirable but currently limited by VLM compute cost.
• Spatial precision: Localization methods struggle with small GUIs or stylized interfaces; this affects the interpretability and actionability of overlays in certain domains.
• Generalizability and guidance diversity: Over-reliance on certain gesture/actions (e.g., palm) may limit the system's ability to adapt to the full spectrum of real/virtual tasks.

Future development should focus on on-device lightweight state tracking and direct mapping of guidance diversity (via more granular tutorial parsing and dynamic multimodal fusion). Real-time, always-on step monitoring and more sophisticated mapping between digital and physical workspaces are required for further progress.

Conclusion

JARVIS defines a new architecture for AR-driven, AI-powered cross-reality instruction by integrating task-planning, in-task verification, adaptive sub-planning, and multimodal guidance rendering, all mediated by VLM/LLM frameworks. Empirical results demonstrate that it provides measurable improvements in success rate, usability, and cognitive workload over conventional baselines, particularly by elevating the salience of state and outcome representations. The approach signals a shift toward AR instructional platforms that are context- and state-aware by default, directly supporting the growing class of hybrid and virtualized task environments. Continued advancements in VLM inference efficiency, localization robustness, and guidance diversity will further expand the applicability and robustness of such systems.