Group Interaction Sensing Toolkit (GIST)

• Group Interaction Sensing Toolkit (GIST) is a deployable, headset-only system that captures speech, gaze, and 6DoF pose to infer group dynamics in mixed reality.
• The system’s two-layer pipeline integrates sociogram aggregation with temporal clustering to reveal both session-level interaction networks and dynamic behavior phases.
• Empirical validation on 48 participants demonstrated GIST’s capability to detect nuanced shifts in coordination, highlighting limitations of proximity-only measures.

Searching arXiv for the cited works to ground the article in current paper records. Group Interaction Sensing Toolkit (GIST) is a deployable framework for passively sensing and analyzing group interaction in mixed reality (MR) from the sensors already built into commodity headsets. Introduced as a headset-only system for collaborative MR, it captures speech, gaze, and spatial proximity, then derives both session-level interaction networks and dynamic, moment-by-moment behavior patterns without relying on external cameras, intrusive wearables, or offline manual coding as the primary analytic substrate (Romero et al., 15 Jul 2025). In a broader research sense, GIST also names a design paradigm in which multimodal sensing, dyadic inference, graph construction, and higher-level group interpretation are organized into a reusable toolkit pipeline; adjacent systems in smartphone sensing, mobility analysis, and robotics instantiate closely related ideas, albeit with different sensing stacks and inferential targets (Sofia et al., 2019, Katevas et al., 2018, Atrey et al., 2023, Nguyen et al., 29 Sep 2025).

1. Conceptual scope

The core problem addressed by GIST is the measurement of how teams coordinate, share work, and negotiate roles during immersive collaboration. The 2025 GIST formulation is explicitly organized around two research questions: how MR headset sensory systems can effectively capture group behavior during collaborative tasks, and what algorithms can process and interpret those data to infer group behavior (Romero et al., 15 Jul 2025). Its answer is a two-layer pipeline: a sociogram aggregation module that summarizes interaction structure, and a temporal clustering module that models short-term behavioral dynamics.

This definition places GIST between raw telemetry collection and full social interpretation. It is not limited to logging headset events, and it is not restricted to a single modality such as speech activity or proximity alone. Instead, it treats group interaction as a multimodal relational process with both structural and temporal dimensions. The structural dimension asks who is connected to whom over a session or sub-session; the temporal dimension asks how those relationships change over short windows as leadership, turn-taking, co-manipulation, or demonstration emerge and subside (Romero et al., 15 Jul 2025).

A recurrent misconception in adjacent literature is that group interaction sensing is reducible to co-location. The broader record argues against that view. Smartphone-based crowd sensing showed that proximity alone is inadequate for detecting stationary social interaction in crowds and that behavioral cues improve inference (Katevas et al., 2018). NSense similarly argued that “nearness” cannot be inferred from distance alone and must incorporate social strength, motion, and ambient sound (Sofia et al., 2019). W4-Groups extended the critique of pure co-location by modeling who, what, when, and where jointly from mobility traces (Atrey et al., 2023). GIST inherits that broader position but re-specifies it for MR collaboration.

2. Passive sensing architecture in mixed reality

GIST passively gathers synchronized data from each headset during an MR session. The system focuses on three modalities: audio, gaze, and position or 6DoF pose (Romero et al., 15 Jul 2025). Audio is used to infer conversational participation through headset microphones and a lightweight voice activity detector. Gaze is used to detect shared attention when gaze rays intersect the same object or region in the virtual environment. Position is used to estimate interpersonal distance and co-presence from headset poses.

The pipeline is synchronized with NTP-based clock alignment, achieving sub-100 ms precision; in the study deployment, offsets were under 50 ms (Romero et al., 15 Jul 2025). This temporal alignment matters because gaze convergence events can occur on the order of 50–100 ms. The processing sequence is defined as: capture multimodal streams, synchronize with global timestamps, clean and calibrate the streams, extract dyadic primitives, build session-level and windowed sociograms, and cluster short-window dyadic behavior into behavioral phases. Sensing is passive during the session, while processing and analysis are performed offline after collection.

A notable design decision is the omission of gesture and facial-expression cues. The paper states that these are omitted because they would require external cameras, which would conflict with the goal of untethered MR sensing (Romero et al., 15 Jul 2025). This establishes an important boundary condition: GIST is intentionally headset-only rather than sensor-complete. A plausible implication is that its design prioritizes deployability and ecological validity over exhaustive social signal capture.

3. Sociograms and network-level representation

GIST represents interaction as sociograms: weighted social networks in which participants are nodes and edge weights quantify interaction strength (Romero et al., 15 Jul 2025). The framework distinguishes modality-specific sociograms for conversation, shared attention, and proximity from a fused multimodal sociogram. This separation is analytically important because it prevents one modality from being silently substituted for another and allows interpretation to continue if a stream is degraded.

The system produces sociograms at two temporal scales. Session-level sociograms summarize the full interaction. Windowed sociograms are computed over sliding 32-second windows with a 16-second stride (Romero et al., 15 Jul 2025). The 32-second window is described as long enough for stable edge estimation, while the overlap preserves sub-minute transitions. Conversation graphs are directed because speech has speaker-listener asymmetry; attention and proximity graphs are undirected.

Edge definitions are duration-based. Conversation edges accumulate total speaking time from participant $p$ to $q$. Shared-attention edges accumulate overlapping gaze fixation duration. Proximity edges accumulate total time within a distance threshold (Romero et al., 15 Jul 2025). The modality-specific thresholds are explicit: speech segments shorter than $0.5\,\mathrm{s}$ are discarded, gaze overlaps must last at least $13\,\mathrm{ms}$, and proximity requires participants to be within $1.5\,\mathrm{ft}$.

After constructing modality-specific adjacency matrices, GIST normalizes and fuses them using PCA-derived weights:

$$W^{(\mathrm{fused})}=\sum_{m\in\{\mathrm{conv},\mathrm{att},\mathrm{prox}\}} \alpha_m\,W^{(m)} .$$

The paper states that the PCA weights $\alpha_m$ sum to 1 (Romero et al., 15 Jul 2025). This fusion is intended to summarize overall interaction volume, not to erase modality distinctions.

To interpret sociograms, GIST computes eigenvector centrality, average clustering coefficient, density, and reciprocity, with metrics classified into low, medium, and high tiers using fixed thresholds or session-relative percentiles (Romero et al., 15 Jul 2025). Eigenvector centrality is used to infer leadership or brokerage; average clustering coefficient indicates local cohesion; density measures overall connectedness; reciprocity, used only for conversation networks, reflects balance in two-way exchange. The framework therefore treats graph metrics not as generic summaries but as operational proxies for leadership, subgroup cohesion, engagement, and conversational balance.

4. Temporal clustering and behavior-phase discovery

The second analytic layer models interaction as a time-varying process rather than a static graph. GIST begins with more than 20 dyadic features computed on 1-second windows, then filters them using low variance, high correlation with $r \ge 0.95$, and silhouette-based feature importance (Romero et al., 15 Jul 2025). The resulting 7-feature representation comprises $entropy\_speaking$, $dominance\_ratio$, $q$0, $q$1, $q$2, $q$3, and $q$4. Features are z-normalized per dyad and aligned to a common temporal grid.

Each dyadic interaction segment is represented as a $q$5 time-series matrix with $q$6. The sequence model is a convolutional-recurrent autoencoder with two 1-D convolutional layers, ReLU activations, max pooling, a bidirectional LSTM encoder, and a mirrored decoder; the latent dimension is 16 (Romero et al., 15 Jul 2025). Embeddings are clustered with K-means.

The optimization objective is described in the paper as a combined reconstruction and clustering loss, presented in the source as

$q$7

with the paper noting formatting issues in the source text; $q$8 is mean squared reconstruction loss and $q$9 is squared Euclidean distance to the assigned centroid (Romero et al., 15 Jul 2025). Cluster selection is guided by inertia, silhouette score, and adjusted Rand index stability. The chosen configuration is 32-second windows, 16-second stride, and $0.5\,\mathrm{s}$0 clusters.

The four clusters are interpreted as “Rhythmic Leader–Follower,” “Animated Collaboration,” “Monotone Focus,” and “Instructor Demonstration” (Romero et al., 15 Jul 2025). Their feature signatures are specific. Cluster 0 combines high dominance ratio, low speaking entropy, and minimal proximity, corresponding to structured, turn-based leadership. Cluster 1 shows high speaking entropy, frequent shared attention, and close proximity, corresponding to animated co-manipulation and rapid interaction. Cluster 2 combines low material diversity with muted speech dynamics, indicating repetitive, narrow focus. Cluster 3 combines high material diversity with low dominance ratio, suggesting instructor-like demonstration behavior. The paper further analyzes entropy of cluster membership across groups, dyads, and actors, finding that Cluster 0 is relatively team-specific, whereas Cluster 1 is more broadly shared across teams.

5. Empirical validation and main findings

GIST was evaluated in a human subject study with 48 participants organized into 12 four-person groups (Romero et al., 15 Jul 2025). The group size was fixed at four for tractability and because it yields six dyads per group. Participants wore Meta Quest Pro headsets in a $0.5\,\mathrm{s}$1 lab space. The MR application was built in Unity with the Meta XR SDK and streamed eye gaze, binaural audio, and 6DoF pose. Groups performed an open-ended image-sorting task using 28 OASIS images, sorting them into six affective categories—angry, bored, relaxed, tense, pleased, and frustrated—grounded in Russell’s circumplex model of affect. There was no time limit, and the average completion time was 32.4 minutes. Validation used time-aligned egocentric video from the headsets rather than subjective questionnaires.

A central empirical result is that session-level graphs can obscure changes that become visible in windowed analysis (Romero et al., 15 Jul 2025). The paper gives specific examples: Group 8 showed reciprocity shifting from monologue-like interaction to balanced exchange; Group 10 showed strong oscillation in reciprocity, indicating alternating dominance rather than a stable leader; Group 12 showed dialogue gradually becoming more balanced over time. An ANOVA across early, middle, and late session thirds for reciprocity yielded $0.5\,\mathrm{s}$2, supporting the claim that interaction structure changes meaningfully over the course of the task.

The fused multimodal graph proved useful but analytically limited for some tasks. In the example analysis, the first principal component captured 54% of total variance, with loadings of 0.708 on proximity, 0.706 on attention, and only 0.025 on conversation (Romero et al., 15 Jul 2025). This shows that the fused graph summarizes overall engagement well but can underweight speech directionality. The ablation analysis reinforces the point: removing conversation did not change density much but reversed the ranking of strongest ties with Spearman $0.5\,\mathrm{s}$3; removing proximity caused a moderate reshuffling with $0.5\,\mathrm{s}$4; removing attention had almost no effect with $0.5\,\mathrm{s}$5, n.s. The paper therefore recommends fused sociograms for lightweight monitoring of overall coordination and conversation-specific or enriched graphs when leadership, dominance, or role shifts are the target.

The temporal model segmented 71,404 dyadic windows into 4 clusters, with silhouette score $0.5\,\mathrm{s}$6, $0.5\,\mathrm{s}$7, and best $0.5\,\mathrm{s}$8 (Romero et al., 15 Jul 2025). Manual coding of 100 sampled windows yielded 71% accuracy, per-class precision roughly 0.75–0.84, recall roughly 0.64–0.73, and macro-averaged precision 0.71, recall 0.74, and $0.5\,\mathrm{s}$9. Cluster 3 showed high recall but lower precision, meaning that the model often over-predicted the expert-demonstration state.

A further result is the alignment between network metrics and temporal clusters. Cross-tabulation of the four clusters against low, medium, and high bins of structural metrics over 131 windows found significant associations for conversation reciprocity $13\,\mathrm{ms}$0, conversation eigenvector centrality $13\,\mathrm{ms}$1, fused eigenvector centrality $13\,\mathrm{ms}$2, proximity eigenvector centrality $13\,\mathrm{ms}$3, and attention eigenvector centrality $13\,\mathrm{ms}$4 (Romero et al., 15 Jul 2025). High reciprocity and elevated eigenvector centrality coincided with more animated and demonstration-like collaboration states, whereas density and clustering coefficient varied little by state. This suggests that higher-order network measures are more sensitive than crude density to interaction shifts.

6. Related systems, variants, and analytical boundaries

The term “group interaction sensing toolkit” is most precisely attached to the MR framework above, but several nearby systems instantiate comparable pipelines in different domains.

System	Modalities	Primary inference target
GIST (Romero et al., 15 Jul 2025)	Audio, gaze, 6DoF pose	Sociograms and temporal behavior phases
NSense (Sofia et al., 2019)	Wi‑Fi, Bluetooth, accelerometer, microphone	Nearness via propinquity and social interaction
“Finding Dory in the Crowd” (Katevas et al., 2018)	BLE, accelerometer, gyroscope	Stationary pairwise and group interactions
W4-Groups (Atrey et al., 2023)	WiFi traces, LBSN check-ins	Who, what, when, where of groups
RGB-D robot framework (Nguyen et al., 29 Sep 2025)	Single RGB-D camera	Interaction zones for navigation

NSense is best understood as a people-centric, non-intrusive opportunistic sensing middleware for contextualizing “nearness,” rather than as a full GIST in the strictest sense (Sofia et al., 2019). Its architecture is modular and smartphone-based, with pipelines for location, proximity, motion, and sound activity detection. Its central contribution is the formalization of propinquity and social interaction from node degree, motion, social strength, relative distance, and ambient sound. Propinquity is defined as

$13\,\mathrm{ms}$5

and social interaction is given in the paper with an awkwardly truncated Gaussian-like expression. The system was evaluated qualitatively on real smartphone traces from four Samsung S3 devices, with data sensed every minute, emphasizing plausibility and robustness rather than accuracy, precision, or $13\,\mathrm{ms}$6 (Sofia et al., 2019). Its relevance to GIST lies in its modular local inference, privacy-conscious retention policy, and explicit relational utilities, but it does not explicitly model full group dynamics, roles, or conversational turns.

“Finding Dory in the Crowd” moved closer to direct group interaction detection in mobile settings (Katevas et al., 2018). Using BLE/iBeacon proximity, accelerometer-derived motion, and gyroscope-related orientation features, it formulated pairwise stationary social interaction detection as a binary classification problem over participant pairs and used XGBoost to predict both 1:1 and group interactions. In a 24-participant, 45-minute networking-style study, it reported $13\,\mathrm{ms}$7, precision $13\,\mathrm{ms}$8, and recall $13\,\mathrm{ms}$9, a 30.2% improvement over a normalized-proximity baseline. It then lifted pairwise probabilities into group structure through a weighted graph $1.5\,\mathrm{ft}$0 and Louvain-style community detection, yielding node-level performance of 71.09% and group-level performance of 75.19% at resolution 0.5 (Katevas et al., 2018). The paper’s strongest lesson, consistent with GIST, is that proximity alone is insufficient and that pairwise inference plus graph analysis provides a more faithful representation of group structure.

W4-Groups extended group interaction sensing into mobility analysis by modeling the who, what, when, and where of group behavior (Atrey et al., 2023). It represents user trajectories as

$1.5\,\mathrm{ft}$1

and sessions as $1.5\,\mathrm{ft}$2, then detects short-term co-occurrence and long-term mobility similarity through spatial, temporal, and social features. The group definition is

$1.5\,\mathrm{ft}$3

and mobility similarity is defined as a weighted combination of spatial, temporal, and social similarity. Evaluated on two WiFi datasets and a location check-in dataset, W4-Groups reported an average of 92% overall accuracy, 96% precision, and 94% recall (Atrey et al., 2023). Relative to GIST, its inferential substrate is mobility regularity rather than fine-grained interaction dynamics, but its who/what/when/where framing broadens the conceptual horizon of group sensing beyond immediate dyadic behavior.

The RGB-D robotics framework of 2025 illustrates yet another variant: real-time recognition of human interactions for socially aware navigation from a single RealSense D435i RGB-D camera (Nguyen et al., 29 Sep 2025). Its pipeline uses MoveNet for 2D keypoints, depth alignment and 3D localization, PCA for body orientation, DBSCAN for spatial clustering, pairwise line intersections, and the shoelace formula for interaction polygon, engagement area, and shared interest point. It reports average runtime of 3.83 ms per frame and publishes the result as a ROS 2 social-costmap layer (Nguyen et al., 29 Sep 2025). This system is outside MR, but it exemplifies the same general architecture: raw sensing, relational primitives, group-level geometric inference, and downstream action.

Across these systems, several analytical boundaries recur. First, proximity is repeatedly shown to be necessary but insufficient. Second, fused representations can improve robustness yet obscure asymmetries, especially in speech. Third, deployable toolkits usually omit some socially relevant channels—gesture, facial expression, speech semantics, or stable identity confirmation—because those channels raise privacy, hardware, or synchronization costs. Fourth, validation regimes vary sharply: GIST emphasizes alignment between network structure and temporal modes in MR (Romero et al., 15 Jul 2025), NSense emphasizes plausible nearness contextualization (Sofia et al., 2019), “Finding Dory” emphasizes supervised detection in crowds (Katevas et al., 2018), W4-Groups emphasizes mobility-derived group construction (Atrey et al., 2023), and the RGB-D system emphasizes geometric interpretability and real-time robotic control (Nguyen et al., 29 Sep 2025). Taken together, this suggests that GIST is best understood not as a single algorithm but as a family of multimodal, graph-centered, group-sensing frameworks whose specific form depends on the sensing substrate and the operational definition of interaction.