Vitessce Link: Hybrid 3D Tissue Visualization Platform

• Vitessce Link is a web-native framework combining 3D stereoscopic mixed reality and 2D analytical displays to enable immersive tissue exploration.
• It leverages modern web technologies like React, Three.js, and WebGL while supporting standard data formats such as OME-Zarr, AnnData, and SpatialData for high-resolution spatial omics.
• The platform synchronizes interactions via a bidirectional AWS-hosted WebSocket server, enhancing collaborative analysis and real-time data integration.

Vitessce Link is a web-based hybrid framework designed to enable integrative analysis of three-dimensional (3D) tissue maps, uniting mixed reality stereoscopic visualization and synchronized 2D analytical displays. With the proliferation of spatial omics and high-resolution imaging technologies, the need for tools that can leverage 3D spatial information while retaining robust analytic capabilities has grown acute. Vitessce Link addresses the limitations of prior methods—chiefly, the dichotomy between tools limited to flat 2D displays and those relying on stereoscopic rendering with minimal analytic integration—by constructing a zero-install, open standards platform for immersive and analytic tissue exploration.

1. Hybrid System Architecture

Vitessce Link combines two principal modes: a 3D stereoscopic mixed reality (MR) view and a browser-based 2D display leveraging the Vitessce platform. The core stack is implemented as a web-native Vitessce instance built with React and TypeScript. The 2D interface offers Vitessce’s canonical widget suite, including channel selectors, threshold sliders, segmentation metadata panels, expression-matrix, and embedding plots.

Mixed reality functionality is realized through Three.js and React Three Fiber, with React Three XR providing access to the WebXR API for head-tracked stereoscopic rendering, supporting hardware such as Meta Quest 3, Quest Pro, and HoloLens 2. The visualization pipeline uses WebGL (GLSL shaders) for volume and surface rendering, with no requirement for native/runtime installation—any WebXR-capable browser suffices. Data sources utilize cloud-optimized OME-Zarr for multiresolution volume streaming, and the framework integrates AnnData, MuData, and SpatialData standards for segmentation masks, cell metadata, and gene-expression matrices. All data are fetched on-demand (e.g., from AWS S3), cached in the client, and benefit from incremental mesh-level-of-detail for scalability.

A lightweight AWS-hosted WebSocket server—based on Amazon API Gateway tutorials—mediates bidirectional event traffic. Interaction events encompassing camera pose, channel-visibility changes, entity selection, filter thresholds, and derived-data brushing are serialized and broadcast to all connected sessions. Session joining is streamlined via a four-digit code, eliminating firewall/network configuration barriers.

Component	Technology/Standard	Functionality
2D Display	React, Vitessce widgets	Analytical controls, metadata, expression analysis
Mixed Reality View	Three.js, React Three Fiber/XR	Stereoscopic volumetric navigation, hand gesture interaction
Data Format	OME-Zarr, AnnData, MuData	Multiresolution streaming, segmentation masks, gene expression
Synchronization	AWS WebSocket server	Bidirectional event propagation across devices

2. Mixed Reality Interaction Design

Vitessce Link implements five principal hand-gesture interaction modes within mixed reality. Device-local hand positions and orientations, tracked via WebXR, are mapped to camera transformations and object selections in world coordinates. The gesture mappings are as follows:

1. Pinch to translate/rotate (I₁): Initial pinch point $p_0$ and orientation quaternion $q_0$ anchor the gesture. As the user moves, translation $t(t) = p(t) - p_0$ and rotation $R(t) = q(t)\cdot q_{0}^{-1}$ update the viewpoint.
2. Two-handed pinch to zoom (I₂): The zoom scale $s(t) = \|h_1(t) - h_2(t)\| / \|h_1(0) - h_2(0)\|$ modulates focal distance.
3. Hover (I₃): Ray-casting from the index-finger tip into the scene highlights intersected mesh or voxel elements.
4. Pinch to select (I₄): Pinch-closure emits a “select entity ID” event; selection propagates to the 2D Vitessce panel for linked analytic updates.
5. Two-finger measurement (I₅): Direct calculation of $d(t) = \|h_1(t)-h_2(t)\|$ in world coordinates yields physical distance in microns.

All mappings are handled client-side within the WebXR event loop. Resultant interaction events synchronize analytic updates, such as plot highlights or channel filtering, bidirectionally between MR and 2D views via WebSocket.

3. Data Management and Visualization Pipeline

Vitessce Link’s visualization strategy integrates volumetric imaging and mesh segmentation with derived data analytics, optimized for browser execution.

• Volumetric streaming: 3D volumes are transferred as OME-Zarr textures and sampled in back-to-front order via custom GLSL shaders. Adjustable transfer functions enable nuanced visualization.
• Mesh segmentation: Segmentation outputs (e.g., glomerular masks, cell polygons) are rendered as Three.js meshes with per-vertex color, opacity controls, and clipping planes. Depth buffering and order-independent transparency maintain occlusion fidelity.
• Derived data analytics: The 2D panel incorporates expression-matrix heatmaps, UMAPs for dimensionality reduction, bar charts of marker intensity, and metadata tables. Interactive selection in MR triggers linked highlighting; filtering and brushing actions in 2D update the MR coloration and visibility.
• Performance: Multi-resolution volume pyramids and incremental mesh-level-of-detail implementations keep GPU memory bounded. WebGL texture caching and fetch coalescing mitigate bandwidth and latency.

4. Application Case Studies and Empirical Evaluation

Vitessce Link has been evaluated in nephrology and oncology scenarios:

• Nephrology (human kidney lightsheet microscopy): Employed for validating glomerular segmentations in dense tissue. Stereoscopic depth facilitates identification of misaligned masks and occlusions among vessels and nerves. Hand-based selection enables immediate analytic feedback in the desktop view (size distributions, shape descriptors, adjacency graphs). Two-finger measurement quantifies inter-glomerular distances, supporting hypotheses about community structure.
• Oncology (3D CyCIF of FFPE melanoma): Over 50 protein markers and cell types analyzed in thick tumor sections. MR view clarifies ambiguous cell-to-cell contacts (tumor–immune synapses) that are indistinct in 2D projections. Segmentation masks toggle in stereo MR and are cross-validated with raw channel fluorescence on desktop panels. MR-enabled hand measurements of cell infiltration depth and vessel proximity facilitate rapid spatial analysis.

User feedback was collected from two NIH consortia meetings (58 informal participants, 16 structured sessions) and a 12-month deployment in eight laboratories. The primary findings were notably improved depth perception compared to screen-only 3D viewers, intuitive navigation using hand-tracking, preference for 2D panel’s fine-tuned controls, and strong interest in multi-user collaborative modes and guided walkthroughs.

5. Integration with Computational Workflows and Extensibility

Vitessce Link embeds directly into computational notebook environments via Anywidget for both Python and R in platforms such as JupyterLab and nteract. Session linkage employs a four-digit joining code, enabling rapid synchronization between desktop analytic panels and MR headset visualization.

Extensibility is accomplished via Vitessce’s plugin API, which permits registration of new analytic panels through React view-adapters subscribing to the shared WebSocket state. Custom visualization components, including specialized WebGL shaders or Three.js extensions, can be integrated by augmenting the React Three Fiber component tree.

All views share a canonical JSON-serializable “app state” comprising camera matrices, channel visibilities, filter ranges, and selected entity IDs. Propagation is handled by WebSocket messages—for example, {“event”:“cameraUpdate”, “pose”:M_camera} or {“event”:“filterChange”, “channel”:k, “range”:[min,max]}. This event model supports extension to timed playback or collaborative interaction primitives.

6. Context and Paradigmatic Significance

Vitessce Link establishes a paradigm for integrative, web-native analysis of 3D tissue maps, combining immersive stereoscopic exploration and analytic data integration. By adhering to open standards (WebGL, WebXR, OME-Zarr) and a frictionless browser deployment model, it facilitates cross-platform usage and integration into existing spatial omics workflows. The framework’s capacity for true depth perception, intuitive hand interaction, and device-agnostic bidirectional synchronization invites further development in collaborative analysis, guided “storytelling” tours, and extension to new data modalities. User feedback suggests broad utility and preference for hybrid analytic approaches; a plausible implication is further decentralization of advanced spatial analytics from specialized hardware to web-native interfaces.