Situated Visualization Technique

• Situated visualization is a method that embeds interactive data displays within AR environments to ensure contextually relevant, spatially integrated insights.
• It integrates data transformation, visual mapping, and view manipulation in a seamless, live-feedback workflow that supports rapid prototyping and iterative refinement.
• Leveraging liveness, integration, and expressiveness, this approach enhances user engagement while also highlighting challenges in scalability and usability.

Situated visualization is a paradigm within visualization and immersive analytics in which data representations are embedded in direct spatial relation to their physical counterparts—objects, spaces, or users—within an augmented reality (AR) environment. This approach aims to make visualizations contextually relevant, spatially integrated, and immediately actionable by tightly coupling visual data with their referents in the real world. Emphasizing "liveness" (real-time feedback), integration (end-to-end in-situ authoring and manipulation), and expressiveness (support for complex visual and interaction mappings), situated visualization techniques allow for rapid, iterative, and interactive development and use of visualizations in immersive and real-world contexts.

1. Core Principles and System Architecture

Situated visualization techniques, as operationalized in the AVAR toolkit, are organized around three foundational principles:

• Liveness: Visualization scripts can be authored and immediately re-executed within the immersive AR environment. This feedback loop drives engagement, as users observe the immediate effect of code or data changes, measured in the AVAR user paper by a median active use time $\geq$ 1.5 hours.
• Integration: All aspects of the visualization workflow—data transformation, visual mapping, and view transformation—are performed entirely within the AR context. The need to switch to desktop workflows or external tools is eliminated. The underlying framework enables seamless transitions between stages, and the entire process is operated via immersive input (e.g., a Bluetooth keyboard for script entry, hand gestures and head movement for interaction).
• Expressiveness: A rich domain-specific language exposed in the immersive programming environment permits users to specify not only standard visual mappings but also dynamic, custom visual structures (e.g., node-link diagrams, space–time cubes) and transformations. This expressiveness is paramount for expert users requiring control over complex visualizations.

Technically, this design is realized by embedding the Pharo Smalltalk environment directly into AR (on Microsoft HoloLens), interfacing with visualization engines such as Roassal2 (2D) and Woden (3D), all orchestrated via a Unity front-end that manages user interaction and rendering. Key system primitives include:

Engine/Tool	Primitive/Feature	Function
Roassal2	RTTabTable	Data table manipulation
Woden	RWElement, RWAlign	3D element construction, layout
Woden	RWCube, RWCylinder	3D object visualization
AVAR/Pharo	RTScale, RWXZGridLayout, RWView	DSL features for scaling, layout, view definition

2. The Agile SV Workflow

AVAR structures situated visualization as a rapid, iterative loop comprised of three discrete but interlinked stages:

1. Data Transformations: Filtering, formatting, and normalizing raw datasets to construct analysis-ready tables. Users author and apply transformation scripts live within the AR environment, often using RTTabTable to manipulate tabular data.
2. Visual Mappings: Definition of how transformed data is mapped to visual properties (position, color, size, etc.) via embedded DSLs. Visualizations such as space–time cubes are assembled by binding data axes and attributes to three-dimensional space and encodings (e.g., co-authors to $X$ and $Z$, time to $Y$, and color gradients).
3. View Transformations: Interactive adjustment of visualization projection and arrangement in immersive space, accomplished via hand gestures and head movements. Features such as RWXZGridLayout support dynamic rearrangement, while RWView abstracts the viewpoint configuration.

This workflow is uninterrupted, as all interactions, script editing, and 3D manipulations occur without leaving the AR context. Systemic logging supports precise analysis of user navigation between these steps; green/red indicators in session logs represent successful/failed script executions, evidencing continual liveness.

3. Empirical Findings and Performance Characteristics

Empirical findings from the AVAR exploratory user paper are instructive:

• Engagement & Interactivity: All participants (n=7, expert Smalltalk users) engaged in highly interactive sessions, with a median duration of approximately 106 minutes.
• Effect of Liveness: Immediate visual feedback upon code changes resulted in a steady, high frequency of user-script interaction. More experienced participants demonstrated greater efficiency, completing task 1 (e.g., space–time cube construction) with fewer interactions and shorter times.
• Expressive Use: Participants fully leveraged the DSL, employing advanced constructs such as RTScale for data scaling and RWXZGridLayout for 3D arrangement, indicating system expressiveness supported exploration of diverse visual idioms.
• Integration and Immersion: Data and visualization transformations were conducted without leaving AR, confirming the workflow’s integrative nature and potential to streamline data analysis in spatial computing environments.
• Manipulation in AR: Rotations and translations of visualizations, orchestrated via head and gesture input, validated practicality of 3D immersed manipulation in a real-world scenario.
• Usability Assessment: System Usability Scale (SUS) median was moderate (~58), suggesting some ease-of-use limitations. However, participants widely felt they could "learn to use the system quickly," reflecting strengths in engagement rather than conventional usability measures.

4. Implementation Strategies and Scaling Trade-Offs

The architecture’s modularity derives from off-the-shelf components (Pharo, Unity, Bluetooth keyboard, Microsoft HoloLens), permitting rapid prototyping and low-latency feedback. The live interpreted Smalltalk engine eliminates the compile-run cycle, critically supporting immediate execution.

AVAR’s scripting paradigm allows users to encapsulate complex transforms and mappings in succinct, testable routines, which are instantly visualized and can be revised with minimal overhead. The visualization engines’ primitive support (e.g., RWCube, RWElement) scales well to “common visual structures” but may encounter representational limitations in more complex or high-density visualization contexts.

Scaling considerations include:

• Resource Consumption: AR rendering of dense or highly dynamic 3D visualizations may tax HoloLens computational resources, suggesting that optimizations in rendering pipelines or load balancing between Unity front-end and Pharo back-end may be necessary for more complex data.
• User Expertise Dependency: Expressiveness presumes significant programming competency; real-world deployments outside expert settings may require higher-level abstractions or visual authoring interfaces.
• Session Length and Sustained Interactivity: The live-feedback loop is empirically shown to sustain engagement over long periods ($\geq$ 1.5h median), but user fatigue, device ergonomics, and input constraints (e.g., typing on a physical Bluetooth keyboard) could impact scalability in broader deployment.

5. Limitations, Extensions, and Deployment Considerations

While AVAR demonstrates feasibility, certain limitations and future directions are apparent:

• Ergonomics and Workflow Fluency: Typing code via a physical keyboard in AR is effective for experts but lacks accessibility for non-programmers. Integration of alternative input modalities, such as voice, gesture-based scripting, or visual block-based programming, could broaden adoption.
• Error Handling and Debugging: The built-in error notification console provides immediate feedback; however, debugging complex scripts in immersive AR contexts remains challenging.
• Expressiveness vs. Simplicity Trade-Off: The system’s flexibility supports complex, customized visualizations at the expense of required technical fluency.
• Real-World Deployment Strategies: For systematic, large-scale adoption (e.g., enterprise or scientific data analysis), integration with collaborative multi-user environments, combined desktop/AR workflows, and persistent session management becomes essential.
• Potential Extensions: Integration with additional visualization backends, distributed AR rendering, and adaptive interfaces (e.g., context-sensitive code suggestions, reusable script libraries) would further strengthen the agile SV paradigm.

6. Implications for the Evolution of Situated Visualization

AVAR’s design and exploratory evaluation substantiate several fundamental implications for situated visualization:

• Authoring in Context: Situated visualization authoring—whereby all aspects of the analytic pipeline occur in immersive spatial contexts—not only reduces mental context-switching but may accelerate both prototyping and sensemaking cycles.
• Enhanced Engagement via Liveness: The sustained, interactive feedback loop supports deeper engagement and encourages iterative refinement, which is particularly beneficial in complex, high-dimensional data scenarios.
• Path Toward Expressive Spatial Analytics: Offering a rich, extensible authoring language within immersive space suggests future systems could bridge the benefits of desktop programming environments with multimodal, embodied interaction (e.g., head gestures, manipulation of visual artifacts in space).

A plausible implication is that successful situated visualization systems must balance liveness, integration, and expressiveness, while expanding input and accessibility options and optimizing resource use to support persistent, scalable, and collaborative data analysis directly within spatial computing environments. The agile SV technique exemplified by AVAR thus provides a foundational architecture for next-generation immersive analytics tools.