IHSE Descriptor for Bimanual Interaction

• IHSE is a compact descriptor that defines 3D spatial relations between two hands through joint distance metrics and hand openness indicators.
• It aggregates per-frame features over time using RS-DACT to yield a 92-dimensional video-level descriptor for real-time interaction recognition.
• When integrated within the I2S framework, IHSE enhances object classification, human-object interaction recognition, and user identification performance.

Inter-Hand Spatial Envelope (IHSE) is a compact, handcrafted descriptor for bimanual interaction analysis that encodes how the two hands relate to each other in 3D space during manipulation, using only 3D hand pose data. It was introduced in the I2S (Interact2Sign) framework presented in "Person Identification from Egocentric Human-Object Interactions using 3D Hand Pose" (Hamza et al., 20 Sep 2025), where it serves as one of the feature groups for object classification, human-object interaction (HOI) recognition, and user identification from egocentric video. In the reported formulation, IHSE combines per-hand thumb–pinky span with cross-hand distances between corresponding joints, and aggregates these quantities over time using Range-Sensitive Dispersion-Aware Central Tendency (RS-DACT), yielding a 92-dimensional video-level descriptor.

1. Conceptual role in bimanual interaction analysis

IHSE is motivated by settings in which HOIR and user identification are deployed in AR-based personalized assistive technologies, including aircraft cockpits, aerospace maintenance, and surgical procedures (Hamza et al., 20 Sep 2025). The paper characterizes these environments as strongly human-centric and security-critical, and emphasizes that the targeted interactions are predominantly bimanual.

The descriptor is intended to capture the spatial “shell” formed by the two hands during coordinated manipulation. In the source formulation, this “envelope” is not a geometric hull in the strict computational-geometry sense; rather, it is the combined pattern of distances between corresponding joints across hands together with a measure of hand openness. The paper states that the spatial relationship between the hands provides important cues about the object’s size, the grip used, and the type of coordination required. This suggests that IHSE is designed to expose relational structure that is not explicit in per-hand descriptors alone.

The intended informational content is threefold. First, inter-hand distances reflect how far apart the two hands are and how symmetrically they are arranged. Second, thumb–pinky span on each hand reflects whether the hands are more open or more compressed. Third, temporal aggregation preserves central tendency, dispersion, and extremes over a clip, so that abrupt approach–separation behavior within a bimanual interaction remains represented.

2. Formal definition and descriptor construction

The input to IHSE is a 3D hand pose sequence in which each frame contains 21 joints per hand for the left and right hands. The paper defines the framewise joint structure as

$$J_i = \{J_i^R, J_i^L\} = \{\{j_{i1}^R, \dots, j_{i21}^R\}, \{j_{i1}^L, \dots, j_{i21}^L\}\}.$$

Here, $i$ is the frame index, and $j_{ik}^R, j_{ik}^L \in \mathbb{R}^3$ are the 3D coordinates $(x,y,z)$ of joint $k$ in frame $i$ for the right and left hands, respectively (Hamza et al., 20 Sep 2025).

Per frame, IHSE contains two components.

The first component is Hand Span (2 features). For each hand, the descriptor measures the Euclidean distance between thumb tip and pinky tip:

$$\text{span}_R(i) = \left\| j_{\text{thumb}}^R(i) - j_{\text{pinky}}^R(i) \right\|_2$$

$$\text{span}_L(i) = \left\| j_{\text{thumb}}^L(i) - j_{\text{pinky}}^L(i) \right\|_2.$$

These two scalars encode how “open” each hand is in the current frame.

The second component is Inter-Hand Distances (21 features). For each joint index $k = 1,\dots,21$, IHSE computes the Euclidean distance between the corresponding right-hand and left-hand joints:

$$d_k(i) = \left\| j_{ik}^R - j_{ik}^L \right\|_2, \quad k = 1, \dots, 21.$$

Thus, the per-frame descriptor has $i$0 scalar features.

The paper then aggregates these 23 framewise quantities over a temporal window using RS-DACT. For each per-frame feature $i$1, where $i$2 indexes frames within a clip, four statistics are computed over time: mean $i$3, standard deviation $i$4, minimum $i$5, and maximum $i$6. The resulting video-level descriptor is

$i$7

Because there are 23 per-frame features and four temporal statistics per feature, the aggregated IHSE descriptor has dimension $i$8. The paper explicitly lists the “IHSE (I) Descriptor” size as 92. No explicit thresholds or additional hyperparameters are introduced; the design choices are the use of thumb–pinky span, distances between corresponding joints across hands, and RS-DACT aggregation over frames.

3. Relation to other feature families

Within I2S, IHSE is one of five semantically organized feature groups: Spatial, Frequency, Kinematic, Orientation, and IHSE itself (Hamza et al., 20 Sep 2025). The distinction between these groups is central to the paper’s analysis.

Spatial (S) features describe per-hand geometry, including joint coordinates and intra-hand distances, and therefore characterize each hand largely independently. Orientation (O) features encode joint angles and palm normals, focusing on local articulation and orientation. Kinematic (K) features capture motion dynamics such as velocity and acceleration. Frequency (F) features capture spectral properties of hand motion trajectories. IHSE differs from all four by being explicitly inter-hand: it represents the relative geometry between the two hands rather than motion spectra, local articulation, or within-hand geometry.

This distinction is operationally important. The paper states that IHSE provides cross-hand coupling information missing from the other categories, and reports that adding IHSE to S, O, K, or F systematically improves performance. It should therefore not be conflated with the Spatial category merely because both use geometric quantities. Spatial features are primarily per-hand; IHSE is explicitly relational.

The reported intuition is that IHSE is especially informative for object size and geometry, grasp type, and bimanual coordination. Large objects such as a laptop, waffle iron, or microwave often require the hands to be farther apart, whereas smaller objects such as a phone, notebook, or ketchup bottle can bring the hands closer together. Similarly, grasp and use interactions can produce different inter-hand arrangements, including asymmetric configurations when one hand stabilizes and the other actuates. The paper further notes that users may exhibit idiosyncratic bimanual styles even for identical tasks, making IHSE a possible secondary source of biometric signal.

4. Integration within the I2S framework

I2S is a three-stage pipeline in which IHSE participates at the feature level (Hamza et al., 20 Sep 2025). In Stage 1, object classification takes a video-level descriptor $i$9 and uses an XGBoost classifier to predict the object class $j_{ik}^R, j_{ik}^L \in \mathbb{R}^3$0. In Stage 2, HOI recognition takes augmented features $j_{ik}^R, j_{ik}^L \in \mathbb{R}^3$1, defined as the original features plus the predicted object label, and uses an XGBoost classifier to predict the HOI class $j_{ik}^R, j_{ik}^L \in \mathbb{R}^3$2. In Stage 3, user identification takes $j_{ik}^R, j_{ik}^L \in \mathbb{R}^3$3 plus the HOI label and uses an XGBoost classifier to predict the user identity $j_{ik}^R, j_{ik}^L \in \mathbb{R}^3$4.

IHSE can be used alone or concatenated with other descriptor groups. The paper reports descriptor sizes of 688 for S, 144 for O, 756 for K, 504 for F, and 92 for I. Feature fusion is performed through simple concatenation of the aggregated descriptor vectors. Examples given in the paper include IHSE alone at 92 dimensions, FI at 596 dimensions, SOI at 924 dimensions, SFI at 1284 dimensions, and SOKI at 1680 dimensions.

The paper does not describe an explicit normalization step. It evaluates configurations in which IHSE is used standalone and in combinations such as KI, FI, OI, KFI, OFI, SOI, and SOKI. The classifiers that consume IHSE in the main pipeline are XGBoost models, although RF and SVM are also mentioned as comparison models later in the study.

5. Empirical behavior and ablation results

IHSE is evaluated on an augmented ARCTIC + H2O bimanual object manipulation set with 12 subjects, 11 articulated objects, and 22 HOI categories defined as object $j_{ik}^R, j_{ik}^L \in \mathbb{R}^3$5 $j_{ik}^R, j_{ik}^L \in \mathbb{R}^3$6, using stratified 5-fold cross-validation (Hamza et al., 20 Sep 2025). The paper reports F1-scores for object classification, HOI recognition, subject identification, and the overall I2S pipeline.

The ablation results show that IHSE alone is already strong for object and HOI recognition: object F1 $j_{ik}^R, j_{ik}^L \in \mathbb{R}^3$7, HOI F1 $j_{ik}^R, j_{ik}^L \in \mathbb{R}^3$8, subject F1 $j_{ik}^R, j_{ik}^L \in \mathbb{R}^3$9, and overall I2S F1 $(x,y,z)$0. When fused with Orientation, performance rises substantially: OI yields object F1 $(x,y,z)$1, HOI F1 $(x,y,z)$2, subject F1 $(x,y,z)$3, and overall I2S F1 $(x,y,z)$4. Other strong combinations include OFI with $(x,y,z)$5, $(x,y,z)$6, $(x,y,z)$7, and $(x,y,z)$8; SFI with $(x,y,z)$9, $k$0, $k$1, and $k$2; SI with $k$3, $k$4, $k$5, and $k$6; and SOI with object F1 $k$7, HOI F1 $k$8, subject F1 $k$9, and overall I2S F1 $i$0. The best overall configuration is SOKI, which reaches object F1 $i$1, HOI F1 $i$2, subject F1 $i$3, and overall I2S F1 $i$4.

The paper further highlights the effect of adding IHSE to temporal descriptors. K alone yields object F1 $i$5 and HOI F1 $i$6, whereas KI reaches $i$7 and $i$8. F alone yields object F1 $i$9 and HOI F1 $$\text{span}_R(i) = \left\| j_{\text{thumb}}^R(i) - j_{\text{pinky}}^R(i) \right\|_2$$0, whereas FI reaches $$\text{span}_R(i) = \left\| j_{\text{thumb}}^R(i) - j_{\text{pinky}}^R(i) \right\|_2$$1 and $$\text{span}_R(i) = \left\| j_{\text{thumb}}^R(i) - j_{\text{pinky}}^R(i) \right\|_2$$2. These comparisons support the paper’s claim that IHSE contributes non-redundant information, especially for object and HOI recognition.

The study also reports a PCA analysis comparing F with FI. The text states that “the combined descriptor, FI, performed significantly better,” and the associated 2D PCA plots show better class separation for FI than for F alone. This suggests that IHSE improves discriminative structure rather than merely increasing descriptor dimensionality.

6. Efficiency profile, interpretive boundaries, and future directions

A direct comparison between the full SOKI framework and an IHSE-only pipeline clarifies the trade-off between accuracy and efficiency (Hamza et al., 20 Sep 2025). SOKI achieves object F1 $$\text{span}_R(i) = \left\| j_{\text{thumb}}^R(i) - j_{\text{pinky}}^R(i) \right\|_2$$3, HOI F1 $$\text{span}_R(i) = \left\| j_{\text{thumb}}^R(i) - j_{\text{pinky}}^R(i) \right\|_2$$4, user F1 $$\text{span}_R(i) = \left\| j_{\text{thumb}}^R(i) - j_{\text{pinky}}^R(i) \right\|_2$$5, and overall $$\text{span}_R(i) = \left\| j_{\text{thumb}}^R(i) - j_{\text{pinky}}^R(i) \right\|_2$$6, with training time $$\text{span}_R(i) = \left\| j_{\text{thumb}}^R(i) - j_{\text{pinky}}^R(i) \right\|_2$$7 s, inference time $$\text{span}_R(i) = \left\| j_{\text{thumb}}^R(i) - j_{\text{pinky}}^R(i) \right\|_2$$8 s, and model size $$\text{span}_R(i) = \left\| j_{\text{thumb}}^R(i) - j_{\text{pinky}}^R(i) \right\|_2$$9 MB. The IHSE-only pipeline achieves object F1 $$\text{span}_L(i) = \left\| j_{\text{thumb}}^L(i) - j_{\text{pinky}}^L(i) \right\|_2.$$0, HOI F1 $$\text{span}_L(i) = \left\| j_{\text{thumb}}^L(i) - j_{\text{pinky}}^L(i) \right\|_2.$$1, user F1 $$\text{span}_L(i) = \left\| j_{\text{thumb}}^L(i) - j_{\text{pinky}}^L(i) \right\|_2.$$2, and overall $$\text{span}_L(i) = \left\| j_{\text{thumb}}^L(i) - j_{\text{pinky}}^L(i) \right\|_2.$$3, with training time $$\text{span}_L(i) = \left\| j_{\text{thumb}}^L(i) - j_{\text{pinky}}^L(i) \right\|_2.$$4 s, inference time $$\text{span}_L(i) = \left\| j_{\text{thumb}}^L(i) - j_{\text{pinky}}^L(i) \right\|_2.$$5 s, and model size $$\text{span}_L(i) = \left\| j_{\text{thumb}}^L(i) - j_{\text{pinky}}^L(i) \right\|_2.$$6 MB. The paper interprets the slightly larger model size of the IHSE-only case as an effect of XGBoost building deeper trees on fewer features.

These results establish an important boundary condition. IHSE alone is not the highest-performing configuration for user identification, but it remains competitive for object and HOI recognition while being much faster to train and faster at inference. A plausible implication is that IHSE is particularly suitable when real-time computation and compact feature engineering are prioritized, whereas maximal identification performance benefits from fusion with Spatial, Orientation, and Kinematic features.

The paper’s limitations also constrain the scope of current claims. The ARCTIC portion of the data uses simulated objects, and the interaction vocabulary is restricted to “grasp” and “use.” Accordingly, the generalization of IHSE to richer interaction vocabularies, deformable objects, or more fine-grained actions remains untested in the reported experiments. The paper further notes that handcrafted features, including IHSE, are interpretable and lightweight but may not capture extremely complex patterns that deep neural representations can. The stated future directions are “developing hybrid approaches combining handcrafted features with deep learning representations” and “investigating real-time processing optimizations for security-focused AR/VR applications.”

In that sense, IHSE occupies a specific methodological position: it is a deliberately compact and interpretable inter-hand descriptor whose strongest demonstrated contribution lies in encoding bimanual geometry and improving object and HOI discrimination, while also serving as a component of the best-performing user-identification pipeline reported in I2S.