HeedVision: Attention-Aware Analytics & Navigation

• HeedVision is a computational framework that integrates live attention mapping with immersive analytics and audio-based assistive navigation.
• It employs WebXR and 3D visualization techniques to enable multi-user gaze tracking and voxel-level rendering of salient information.
• The system uses advanced audio sonification and efficient 3D segmentation to enhance spatial awareness and safety for visually impaired users.

HeedVision encompasses a suite of computational frameworks and systems that foreground "attention awareness" across both collaborative immersive analytics and assistive spatial perception. The term has denoted (i) a standards-compliant WebXR platform for real-time collaborative gaze visualization in AR/VR environments and (ii) a real-time, audio-based spatial awareness system for the visually impaired. Both systems share a focus on live, multi-modal mapping of user attention or spatial input, supporting highly efficient group or individual interaction by rendering salient information via direct perceptual channels—visual overlays for co-located analytics and musical audio for navigation. Metrics and evaluation underscore significant improvements in task efficiency, awareness, and, in the accessibility context, spatial comprehension and safety.

1. System Architectures and Core Components

1.1 Collaborative AR/VR: Immersive Analytics

HeedVision, as a collaborative immersive analytics system, is implemented as a fully Web-based, standards-compliant WebXR application, leveraging Three.js for 3D rendering, WebXR APIs for device support, and the Spatialstrates/Webstrates/Varv frameworks for real-time multi-user state synchronization. Each client (typically an AR/VR headset such as Meta Quest 3) locally maintains the scene graph, with a message bus (WebRTC/WebSocket) propagating "gaze" events and voxel-level updates across peers (Srinivasan et al., 11 May 2025).

System modules include:

• Attention Capture: Headset orientation is used as a proxy for gaze, casting view rays into the 3D scene.
• Attention Model: Scene is voxelized (configurable grid, e.g., 0.05 m) and accumulates attention via a Gaussian kernel per frame.
• Revisualization: Aggregated user attention is encoded as voxel opacity and color overlays within the world-space visualization.

Attention model state for each user $A_u(v,t)$ is updated per frame according to

$$A_u(v,t) \leftarrow A_u(v,t-\Delta t)\,e^{-\lambda \Delta t} + \Delta A_n(v)$$

where $\lambda \approx 0.1\,\mathrm{s}^{-1}$ models attention decay, and $\Delta A_n(v) = \exp(-d^2/(2\sigma^2))$ distributes attention to a spherical region around the gaze contact point.

1.2 Assistive Sonification: Augmented Perception for the Visually Impaired

HeedVision as an assistive device fuses an Intel RealSense D415 RGB-D depth camera (640×480 at ~30 FPS), Raspberry Pi 4 Model B, and stereo headphones. The hardware stack is mounted as a streamlined wearable: sensor on a forehead strap, computation module at the rear, and headphones for real-time musical feedback (Mehta et al., 2023).

The software stack alternates between Unity (C#) for controlled lab evaluation and Python+PyGame for field use, facilitating both 3D audio rendering and efficient inference on embedded hardware.

2. Computational Models of Attention and Environmental Mapping

2.1 Multi-User Voxel Attention (CAAVs)

Attention in 3D collaborative visualization is tracked via per-user attention vectors, agglomerated through several schemes:

• Sum: $A_\mathrm{sum}(v,t) = \sum_{u=1}^U A_u(v,t)$
• Max: $A_\mathrm{max}(v,t) = \max_{u} A_u(v,t)$
• Count: $A_\mathrm{cnt}(v,t) = |\{u \mid A_u(v,t) > \tau\}|$
• Difference (for dyads): $A_\mathrm{diff}(v,t) = A_1(v,t) - A_2(v,t)$

Coverage and redundancy are essential derived measures. Coverage at time $t$:

$$\mathrm{Coverage}(t) = \frac{|\{v \mid A_\mathrm{sum}(v,t) \geq \tau\}|}{|V_\mathrm{total}|}$$

Redundancy is entropy-based:

$$A_u(v,t) \leftarrow A_u(v,t-\Delta t)\,e^{-\lambda \Delta t} + \Delta A_n(v)$$0

with $$A_u(v,t) \leftarrow A_u(v,t-\Delta t)\,e^{-\lambda \Delta t} + \Delta A_n(v)$$1 the per-voxel normalized observation frequency; redundancy is defined as $$A_u(v,t) \leftarrow A_u(v,t-\Delta t)\,e^{-\lambda \Delta t} + \Delta A_n(v)$$2.

2.2 3D-to-Audio Mapping and Object Segmentation

The assistive mode uses a two-stage pipeline:

• Point Cloud Downsampling: Raw 640×480 depth map is reduced to a 16×12 grid (192 points) to enable tractable sonification.
• Musical-Note Encoding:

$$A_u(v,t) \leftarrow A_u(v,t-\Delta t)\,e^{-\lambda \Delta t} + \Delta A_n(v)$$3

Horizontal ($$A_u(v,t) \leftarrow A_u(v,t-\Delta t)\,e^{-\lambda \Delta t} + \Delta A_n(v)$$4) position is mapped to stereo pan, and vertical ($$A_u(v,t) \leftarrow A_u(v,t-\Delta t)\,e^{-\lambda \Delta t} + \Delta A_n(v)$$5) to temporal ordering, forming $$A_u(v,t) \leftarrow A_u(v,t-\Delta t)\,e^{-\lambda \Delta t} + \Delta A_n(v)$$6, where $$A_u(v,t) \leftarrow A_u(v,t-\Delta t)\,e^{-\lambda \Delta t} + \Delta A_n(v)$$7 is pitch, $$A_u(v,t) \leftarrow A_u(v,t-\Delta t)\,e^{-\lambda \Delta t} + \Delta A_n(v)$$8 is pan, and $$A_u(v,t) \leftarrow A_u(v,t-\Delta t)\,e^{-\lambda \Delta t} + \Delta A_n(v)$$9 is play order.

• Object Segmentation uses a novel $\lambda \approx 0.1\,\mathrm{s}^{-1}$0 flood-fill algorithm:
  • Quantize $\lambda \approx 0.1\,\mathrm{s}^{-1}$1 to chunk keys.
  • Build a hashmap for chunk occupancy.
  • Perform connected-components segmentation via neighborhood flooding in chunk space.

This method reduces the naive $\lambda \approx 0.1\,\mathrm{s}^{-1}$2 cluster analysis to $\lambda \approx 0.1\,\mathrm{s}^{-1}$3, enabling the real-time identification of structure in large 3D environments.

3. Visual and Auditory Encoding Strategies

3.1 Visual Encoding in Immersive Analytics

HeedVision leverages several visual cues:

• Voxel Opacity: Proportional to normalized aggregate attention, $\lambda \approx 0.1\,\mathrm{s}^{-1}$4.
• Color Coding: User-specific colors $\lambda \approx 0.1\,\mathrm{s}^{-1}$5, blended for joint attention $\lambda \approx 0.1\,\mathrm{s}^{-1}$6.
• World-Space Embedding: Overlays appear as semi-transparent cubic voxels anchored to the 3D data space.
• Trigger Toggle: Overlays rendered only on explicit input to avoid occlusion and information overload.

3.2 Audio-Based Spatial Encoding

Each grid cell's depth is mapped to a distinct MIDI pitch (higher = nearer), pan reflects horizontal position, and sequence timing encodes the vertical axis. All notes are played in rapid staccato, permitting full spatial "scans" (~192 notes) within ≈1 second. The mapping compresses far-field depths via a 0.8 power law, preserving spatial discrimination for nearby obstacles (Mehta et al., 2023).

4. Evaluation Methodologies and Empirical Results

4.1 Collaborative Analytics Performance

A controlled user study (8 graduate students; 4 dyads) assessed HeedVision's collaborative visualization mode (Srinivasan et al., 11 May 2025):

• Study Design: 2 × 2 within-subjects; terrain vs. scatterplot data; CAAV (attention overlays) vs. control.
• Metrics: Target Discovery Rate, spatial coverage, coverage efficiency, team synergy gains ($\lambda \approx 0.1\,\mathrm{s}^{-1}$7, $\lambda \approx 0.1\,\mathrm{s}^{-1}$8), redundancy (entropy).
• Findings:
  • CAAV increased target detection (scatter: 81.7% vs. 71.4%; terrain: 29.1% vs. 17.9%).
  • Spatial coverage improved (scatter: 78.9% vs. 72.0%).
  • Redundancy dropped by ~37–38%.
  • Significant main effects of attention overlays on both targets found ($\lambda \approx 0.1\,\mathrm{s}^{-1}$9) and redundancy ($\Delta A_n(v) = \exp(-d^2/(2\sigma^2))$0).

Qualitatively, overlays reduced verbal coordination and enabled implicit task division.

4.2 Assistive Navigation Outcomes

Object segmentation accuracy is evaluated by Pearson $\Delta A_n(v) = \exp(-d^2/(2\sigma^2))$1 (linear agreement of detected vs. true counts) and exact-match percentage:

• Objectdataset benchmark: $\Delta A_n(v) = \exp(-d^2/(2\sigma^2))$2 (10–$\Delta A_n(v) = \exp(-d^2/(2\sigma^2))$3 objects per scene).
• Outdoor night scene: 87.5% perfect-match accuracy.
• Ball dataset: $\Delta A_n(v) = \exp(-d^2/(2\sigma^2))$4, accuracy = 50% (low-quality sensor input) (Mehta et al., 2023).

5. Applications, Limitations, and Future Directions

5.1 Application Domains

• Group analytics: In-situ multi-user data exploration, geospatial disaster management, collaborative medical/CFD visualization (Srinivasan et al., 11 May 2025).
• Assistive technologies: Real-time spatial sonification for independent navigation and object awareness for blind or visually impaired individuals (Mehta et al., 2023).
• Education: Novices exposed to expert attention maps to scaffold exploration strategies.
• User testing: Visualization designers use aggregated gaze data to identify underexplored regions during usability studies.

5.2 Limitations

• Voxel attention models impose performance constraints at high resolution or with large user groups; GPU acceleration or adaptive spatial resolution is required.
• Prototype gaze proxies rely on head orientation rather than precise eye tracking, limiting fine-grained focus discrimination.
• The current evaluative paradigm is limited to dyads and visual search; extension to larger groups and other analytical tasks needs validation (Srinivasan et al., 11 May 2025).
• In assistive mode, spatial-to-audio mapping efficiency is bounded by sensor quality and head-worn device ergonomics.
• Segmentation power law compression reduces depth detail at far-range, which may obscure certain environmental risks.

5.3 Development Trajectories

Anticipated advances include:

• Network-efficient attention summary models for distributed and asynchronous remote collaboration.
• Time-stamped, persistent attention traces supporting asynchronous analytic review.
• Integration of semantic/structural data models for role-based, dimension-aware attention overlays.
• Fine-grained gaze tracking and event-driven overlays for high-resolution analysis.
• In the assistive field, leveraging high-density sensor arrays and adaptive mapping functions for dense, cluttered environments.

6. Significance, Context, and Broader Implications

HeedVision demonstrates that dynamic attention-aware overlays—whether rendered as voxel heatmaps for collaborative analytics or as sonified spatial representations for navigation—measurably boost task efficiency, reduce redundant activity, and improve spatial and team awareness. The system-agnostic approach allows applications across collaborative scientific domains, disaster response, accessibility, and usability engineering.

The efficacy of HeedVision in both collaborative and assistive contexts is substantiated by statistically significant improvements in objective task performance (targets found, coverage, redundancy) and subjective metrics (SUS scores, cognitive load). Its modular frameworks reveal a generalizable principle: explicit, real-time externalization of attentional focus (visual or auditory) enables stigmergic coordination and cognitive offloading, supporting both enhanced collaboration and safer, more independent navigation (Srinivasan et al., 11 May 2025, Mehta et al., 2023).