--- title: 'EngageSync: Adaptive Engagement in VR' url: https://www.emergentmind.com/topics/engagesync type: topic --- # EngageSync: Adaptive Engagement in VR EngageSync is a context-adaptive, synchronous engagement and re-engagement platform originally introduced and evaluated for immersive virtual reality (VR) meetings. It integrates real-time context-aware transcript delivery, avatar-anchored interface elements, physiological and behavioral engagement analysis, and relevance-driven data synchronization structures. EngageSync advances the state-of-the-art in collaborative systems, grounding its design in empirical results from VR studies, computational engagement modeling, synchronization research in both social and computational domains, and modern client-server synchronization architectures [2503.16739][2404.04394][1506.05573][1804.03889][1110.0381]. ## 1. System Architecture and Context Awareness EngageSync employs a distributed client-server architecture, optimized for VR settings but extensible to other synchronous, real-time group contexts [2503.16739]. Each client (e.g., running on Meta Quest Pro HMD) streams microphone audio, eye/gaze position, and manual interaction events (such as pinch gestures) via a low-latency network stack (Photon Fusion + Photon Voice). Audio is transcribed in real time by a cloud ASR system (Google Speech-to-Text), then relayed to a "Text Server" for summarization and context routing. The Text Server orchestrates both direct transcript streaming and engagement-based adaptations. It receives utterances tagged with speaker identity, batches and prompts an LLM (OpenAI GPT-4-Turbo) for concise summaries, and redistributes both live and summarized text to all clients, tagging each utterance by avatar network ID and context (live, last-utterance, or re-engagement summary). Clients render panels that are spatially anchored above avatars. State transitions (see Section 3) determine which interface element is shown, with additional logic for summary expiration, attention/focus management (panel auto-fade if not gazed at for 2 s), and interaction via gaze and gesture. ## 2. Engagement Detection and Synchronization Continuous engagement detection leverages both behavioral and physiological signals, as shown in VR and online meeting deployments [2503.16739][2404.04394]. EngageSync distinguishes three principal modes using gaze and activity streams: 1. **Focused on Speaker:** Gaze vectors (using Meta Movement SDK) intersect a speaking avatar; the corresponding avatar's panel displays a live, ASR-generated transcript. 2. **Focused on Listener:** Gaze intersects a non-speaking avatar; the panel shows a concise, LLM-generated summary of the avatar's last utterance. 3. **Disengaged/Re-engaging:** Gaze is off all avatars/objects for >2 s, indicating dropout. On gaze return, the Re-engagement Mode is activated, attaching summaries of all missed utterances to the corresponding avatars. Speech activity is detected by envelope thresholding. Pinch-gesture combined with gaze triggers panel summon; summary “read” is detected via gaze linger (>1.5 s). For engagement modeling in online meetings, video-based remote photoplethysmography (rPPG) with unsupervised contrastive learning extracts heart rate variability (HRV) features at 94% accuracy. Behavioral features—facial action units, gaze, head pose—are extracted and fused with HRV, enabling high-precision engagement classification (96% accuracy) updated every few seconds, suitable for group-level synchronization monitoring [2404.04394]. ## 3. Adaptive Transcript and Summarization Mechanism EngageSync’s core UI innovation is the context-driven, avatar-anchored transcript panel [2503.16739]. Unlike conventional table-fixed interfaces that disrupt gaze and social presence, EngageSync adapts its transcript content and presentation: - **Engagement Mode:** When actively participating (gaze is on speaker or listener avatars), users receive either live transcriptions or a summary of the last utterance, attached directly to the relevant avatar. - **Re-engagement Mode:** Upon return from a disruption, EngageSync automatically generates and displays per-avatar summaries (15-word limit) of all missed content, which auto-expire after being read. Summarization is performed by an LLM with empirically optimized prompt templates ("Summarize the following utterance in no more than 10 words", or 15 for re-engagement), yielding 89–90% summary accuracy against manual reference [2503.16739]. End-to-end latency is typically 1.3 s (ASR median 528 ms, LLM 803 ms). Error rates for summary compression are ≈2%. Color-coded badges signal different transcript types: live (none), engagement summary (green), re-engagement summary (orange); all panels are spatialized above avatars for efficient access and minimal distraction. ## 4. Principles of Synchrony and Real-Time Correlation EngageSync draws directly on empirical and theoretical results showing the centrality of synchronicity and mutual adaptation in effective distributed teamwork [1110.0381][1506.05573]. In financial markets, spontaneous, leaderless synchronization—measured by per-person z-score $s_{ij}$ reflecting second-by-second co-occurrence with peer actions—was shown to predict individual success and was tightly coupled to communication behaviors (instant messaging patterns). These principles generalize: synchrony in collective action, not just in communication or attention, yields performance gains. By extension, EngageSync can incorporate real-time group synchronicity metrics: - For each user and second, compute activity alignment scores relative to peers. - Dynamically overlay heat-maps, crowd-ticker signals, or decision overlays when group alignment exceeds chance—enabling real-time feedback on team-level synchrony. In adaptive HCI systems, synchrony and rapport emerge from reciprocal, continuous adaptation of low-level behavioral rhythms (e.g., turn-taking), operationalized via FSMs and hidden Markov models with social attitude variables (liking, dominance) [1506.05573]. ## 5. Data Synchronization and Relevance Filtering The client-server transactional model underpinning EngageSync leverages advanced relevance-driven data synchronization frameworks [1804.03889]. Rather than full-dataset syncs, EngageSync can maintain for each user a minimal, structurally constrained, per-role data slice defined by a grammar of path expressions. The server, upon a sync request including a user identifier and last-sync timestamp, computes which relevant objects/links have changed since the last update and returns only those deltas. Clients then: - Apply deletions, then additions, then updates to their local view. - Queue offline modifications for later upload with locally unique transaction stamps. - Optionally, utilize set-reconciliation protocols (e.g., MinHash, characteristic polynomial) to optimize bandwidth. This architecture keeps per-user storage and network usage minimal. It also inherently supports multi-device, multi-tenant deployments and provides a substrate for fine-grained event and content synchronization (e.g., missed utterances, engagement state changes). ## 6. Evaluation, Empirical Findings, and Design Recommendations In controlled studies of VR meetings, EngageSync significantly increased social presence (assessed by Networked Minds inventory: co-presence and attentional allocation), accelerated information recall and re-engagement after disruption, and increased the proportion of gaze time spent on peer avatars versus UI compared to both table-fixed and always-on avatar-fixed alternatives [2503.16739]. Key statistical findings include: - Social presence: EngageSync outperformed baselines with χ² = 14.13, p < .001 (W = .471) in combined groups. - Re-engagement time: EngageSync fastest across conditions (χ² = 22.20, p < .001, W = .740). - Information recall: higher in mid-sized groups using EngageSync (χ² = 6.88, p = .028, W = .229; recall of less talkative speakers near ceiling in EngageSync, significantly lower in alternatives). Formative and summative studies support three primary design recommendations: 1. Anchor transcripts to avatars to minimize gaze disruption and preserve group social context. 2. Adapt the transcript content and delivery mode based on detected user context (live participation vs. re-engaging after disruption). 3. Rely on on-demand (gaze+gesture) interface element presentation rather than clutter-inducing always-on overlays. ## 7. Limitations and Future Directions Certain limitations have been noted in current implementations. Fixed panel placement may not be optimal for all users or room configurations. ASR/LLM summarization incurs small, but nonzero, error rates. Only a single dropout duration (4 min) has been systematically studied; real-world session dynamics may necessitate policy adaptation. Summarization prompt refinement and more sophisticated context/ranked retrieval may further enhance catch-up efficiency [2503.16739]. In synchronous engagement modeling, rPPG-based HRV is limited by lighting/motion constraints, and behavioral features degrade under occlusions [2404.04394]. Potential future work includes integration of end-to-end deep learning pipelines for engagement estimation, dynamic summary ordering and multimodal fusion, group-level synchrony visualization, and extension to diverse platforms and domains (beyond VR and online meetings) [2404.04394][1804.03889]. --- **References:** [1110.0381], [1506.05573], [1804.03889], [2404.04394], [2503.16739] Source: https://www.emergentmind.com/topics/engagesync