EHWGesture: Clinical Multimodal Gesture Dataset

• EHWGesture is a multimodal, multi-view dataset designed for dynamic hand gesture understanding in a clinical context, featuring synchronized RGB, depth, event, and motion capture streams.
• The dataset supports gesture recognition and action quality assessment through controlled execution speeds, with tasks including both dynamic and static clinical gestures.
• It enables advanced research in multimodal fusion and neuromorphic evaluation, offering precise ground-truth annotations for low-power clinical gesture monitoring.

Searching arXiv for papers on EHWGesture and closely related uses. arXiv search query: "EHWGesture dataset clinical gestures multimodal" EHWGesture is a multimodal, multi-view dataset for dynamic hand gesture understanding in a clinical context, designed primarily around hand dexterity assessment tasks. It includes over 1,100 recordings totaling 6+ hours from 25 healthy subjects, captured with two high-resolution RGB-Depth cameras, an event camera, and a motion capture system that provides precise ground-truth hand landmark tracking; the recordings are spatially calibrated and synchronized, and an action quality assessment component is embedded through controlled execution speeds that mirror clinical evaluation of hand dexterity (Amprimo et al., 9 Sep 2025). In downstream neuromorphic evaluation, EHWGesture is also characterized as a primary clinical benchmark with 9,708 samples across 11 clinical gesture classes, supporting both gesture recognition and action quality assessment on event-based data (Ancilotto et al., 4 May 2026).

1. Clinical scope and task definition

EHWGesture is organized around five clinically relevant gestures drawn from the MDS-UPDRS hand sections: finger tapping, hand opening and closing, pronation-supination, finger-to-nose reaching, and arm extension for distal tremor assessment (Amprimo et al., 9 Sep 2025). Four of these are dynamic and one is static. The dataset is therefore not a generic consumer-gesture benchmark; it is explicitly tied to neurological and dexterity assessment, especially the analysis of movement speed, regularity, amplitude, and tremor-related behavior.

A central design choice is the embedding of action quality assessment through execution speed. For finger tapping, hand opening-closing, and pronation-supination, subjects perform at metronome-guided SLOW, NORMAL, and FAST rates of 75 BPM, 115 BPM, and 145 BPM, respectively. The resulting AQA task is formulated as a three-class speed classification problem rather than a continuous quality regression problem. From a combined perspective, the dataset can be treated as having 11 classes: two free gestures without speed labels and three timed gestures each executed at three speeds (Amprimo et al., 9 Sep 2025).

The same structure makes EHWGesture useful for multi-task formulations. In the Elastic Spiking Transformer study, the dataset is used for both gesture recognition and AQA, with the latter described as more temporally demanding. This dual formulation is important because it separates recognition of gesture identity from assessment of execution quality, which are often conflated in conventional gesture datasets (Ancilotto et al., 4 May 2026).

2. Sensor stack, synchronization, and calibration

EHWGesture combines RGB, depth, event, and motion-capture streams within a single calibrated acquisition volume. The hardware configuration is summarized below.

System	Hardware	Stated characteristics
Motion capture	OptiTrack, 6 × Prime13 cameras	$1280 \times 1024$, $120\ \text{Hz}$, approx. $6 \times 4 \times 3\ \text{m}^3$
RGB-D	2 × Azure Kinect	RGB $1920 \times 1080$, depth $640 \times 480$, $30\ \text{fps}$
Event vision	Inivation DVXplorer Lite	$320 \times 240$, temporal resolution up to $100\ \text{MHz}$

Temporal synchronization is handled through OptiTrack eSync2. The OptiTrack system sends synchronized start signals and synchronizes the exposure intervals of the two Kinects. The event camera runs independently, but a trigger event is generated each time an RGB-D frame is captured, allowing event timestamps to be mapped into RGB frame times; in the released processing, events are usually aggregated into 33 ms windows to produce event frames aligned to the 30 fps video streams (Amprimo et al., 9 Sep 2025).

Spatial calibration is performed across the motion-capture coordinate system, the two Kinects, and the event camera. The extrinsic calibration uses a checkerboard placed on a PC screen with three OptiTrack markers attached to known corners, exploiting the fact that the screen’s pixel refreshing pattern generates clear event responses detectable by the DVXplorer. The calibration is expressed through the standard projection model

$\mathbf{x}_c = K [R \mid \mathbf{t}] \mathbf{x}_w,$

with released extrinsics supporting cross-modal fusion and projection of mocap landmarks into camera views (Amprimo et al., 9 Sep 2025).

3. Dataset organization, labels, and ground truth

The acquisition protocol records 25 subjects, aged 24–65, with 7 female and 18 male participants, predominantly Caucasian, each seated comfortably in the center of the capture volume. Full upper body is recorded, with hands positioned between 30 and 100 cm from each camera. Each recording lasts 20 s, gestures are executed with both left and right hands, and each task is recorded twice per hand for augmentation and variability. This yields 44 recordings per subject and $25 \times 44 = 1100$ recordings in total (Amprimo et al., 9 Sep 2025).

At the frame level, the scale of the dataset is substantial. Each Kinect records 600 frames per 20 s clip at 30 fps. Across both Kinects, the dataset contains 2,640,000 synchronized RGB and depth frames, while the event stream yields 660,000 accumulated event frames when binned into 33 ms windows, for a stated total of 3,300,000 RGB + D + E frames (Amprimo et al., 9 Sep 2025).

Annotations combine high-level labels and precise temporal structure. Each recording includes gesture type, speed class when applicable, anonymous subject ID, side, trial index, lighting-condition category, and hand shape measurements. Ground-truth 3D hand landmarks come from the OptiTrack system at 120 Hz, using a minimal marker configuration focused on key landmarks relevant to each gesture. Trigger labels for dynamic gestures are derived semi-automatically from periodic mocap trajectories: finger tapping uses the pinch moment of thumb and index; hand opening-closing uses maximal flexion; finger-to-nose uses outward reach; pronation-supination uses extrema where palm or dorsum faces the camera. No explicit hand bounding boxes are provided, but the dataset paper notes that such annotations can be generated by standard tools and cross-checked through mocap projection (Amprimo et al., 9 Sep 2025).

4. Baseline tasks, models, and empirical behavior

EHWGesture defines three main benchmark tasks: gesture classification over five gestures, action quality assessment over three speed classes, and gesture trigger detection via temporal localization of key gesture events. Baseline classification models are 3D CNNs—PhiNet-3D, 3D ResNet-50, and 3D ResNeXt-152—obtained through inflation from 2D backbones. Inputs can be unimodal or multimodal, and late fusion is used by processing each modality with its own backbone before concatenation and classification (Amprimo et al., 9 Sep 2025).

The dataset paper also introduces multimodal contrastive pretraining in a SimCLR-style formulation. Positive pairs are sequences from the same time index but different camera or modality, while negatives come from different times or different subjects. This is explicitly designed to encourage modality- and viewpoint-invariant representations before supervised fine-tuning (Amprimo et al., 9 Sep 2025).

Several empirical regularities are emphasized. Depth is slightly better than RGB as a single modality; event-only performance is comparable to other single modalities; RGB+Depth from one camera improves gesture recognition by +0.1% and AQA by +1.6% over the best single modality; two-camera data yields +1.2% for gesture recognition and +3.2% for AQA; and using all three modalities gives, on average, +3.3% for gesture recognition and +4.5% for AQA over unimodal baselines. Input duration and temporal sampling matter differently across tasks: gesture classification is almost invariant to window length and benefits from higher frame rates, whereas AQA improves with longer windows and can slightly benefit from downsampling to 7.5 fps in some architectures (Amprimo et al., 9 Sep 2025).

Trigger detection baselines use MediaPipe Hands on RGB, derive gesture-specific 1D signals, fuse the two Kinect views by arithmetic mean, smooth them with a 1D convolution, and detect local extrema. The reported validation results are as follows.

Gesture	MAE / Accuracy	FDR
FT	0.12 s / $120\ \text{Hz}$0	$120\ \text{Hz}$1
OC	0.11 s / $120\ \text{Hz}$2	$120\ \text{Hz}$3
PS	0.27 s / $120\ \text{Hz}$4	$120\ \text{Hz}$5
NOSE	0.48 s / $120\ \text{Hz}$6	$120\ \text{Hz}$7

These results indicate that trigger counting is relatively easy at a gross level, while temporal precision is more difficult, especially for finger-to-nose reaching (Amprimo et al., 9 Sep 2025).

5. Event-based and neuromorphic use

EHWGesture has become a direct benchmark for event-driven and neuromorphic models because it includes event-camera data and clinical gesture labels. In the Elastic Spiking Transformer study, it is the primary clinical benchmark for NESTformer, a runtime-adaptive spiking transformer with nested elasticity across the feature extractor, spiking self-attention, and feed-forward blocks. The model uses a granularity parameter $120\ \text{Hz}$8 to slice network width, attention heads, and convolutional channels at inference time without retraining, and the main EHWGesture comparison is performed at $120\ \text{Hz}$9 timesteps (Ancilotto et al., 4 May 2026).

Configuration	Energy / Params	Accuracy / AQA / Gesture
$6 \times 4 \times 3\ \text{m}^3$0	46.1 $6 \times 4 \times 3\ \text{m}^3$1 / 2.59M	75.98 / 79.45 / 96.40
$6 \times 4 \times 3\ \text{m}^3$2	28.7 $6 \times 4 \times 3\ \text{m}^3$3 / 1.46M	75.86 / 78.92 / 96.32
$6 \times 4 \times 3\ \text{m}^3$4	19.3 $6 \times 4 \times 3\ \text{m}^3$5 / 0.92M	72.38 / 74.66 / 95.66
$6 \times 4 \times 3\ \text{m}^3$6	13.3 $6 \times 4 \times 3\ \text{m}^3$7 / 0.68M	71.01 / 74.04 / 94.78

Within this setting, NESTformer $6 \times 4 \times 3\ \text{m}^3$8 achieves the best reported EHWGesture accuracy and AQA, while $6 \times 4 \times 3\ \text{m}^3$9 is highlighted as the deployment “sweet spot,” essentially matching $1920 \times 1080$0 accuracy with about 40% less energy. Lower granularities $1920 \times 1080$1 and $1920 \times 1080$2 remain comparable to or above the best CNN baseline while using far less energy. The paper further reports that AQA benefits from larger temporal horizons, with best AQA at $1920 \times 1080$3, where AQA reaches 80.69%, while gesture recognition tends to peak at intermediate temporal resolutions. For example, $1920 \times 1080$4 yields gesture accuracy 96.24% with only 10.49 $1920 \times 1080$5, whereas $1920 \times 1080$6 yields gesture accuracy 94.63% but 91.17 $1920 \times 1080$7 (Ancilotto et al., 4 May 2026).

A notable property of this use case is energy scaling with spike activity. Using Intel Loihi’s 23.6 pJ per synaptic operation energy model, the study reports nearly linear scaling of spikes and energy across granularities on EHWGesture. This makes the dataset particularly relevant for low-power clinical monitoring, on-device privacy, and heterogeneous deployment across Loihi, SpiNNaker, and FPGA-based accelerators (Ancilotto et al., 4 May 2026).

6. Interpretation, adjacent research directions, and limitations

The most important interpretive point is that EHWGesture is simultaneously a gesture-recognition benchmark, an action-quality benchmark, and a cross-modal alignment resource. Treating it only as a 5-class video dataset misses the role of metronome-controlled speed classes, synchronized event streams, and mocap-derived temporal triggers. Conversely, treating it only as an event-based dataset overlooks the fact that its original definition is multimodal and multi-view, with RGB, depth, event, and 3D landmark streams released together (Amprimo et al., 9 Sep 2025).

The released dataset also has explicit limitations. Subject diversity is limited to 25 healthy volunteers and is skewed toward Caucasian skin tones; all data are recorded in a single controlled laboratory environment; no pathological subjects are included in this release; and motion-capture constraints restrict acquisition to the lab. These factors make EHWGesture particularly strong for pretraining, multimodal fusion, event-based vision, and method development, but less suited as a standalone proxy for full clinical severity scoring in unconstrained environments (Amprimo et al., 9 Sep 2025).

Adjacent research suggests several technically relevant extensions. TIP- and wrist-centric hand-pose estimation emphasizes distal phalanx tips and the wrist as structural anchors for semantically stable gesture representations, while adversarial identity-invariant modeling shows that identity labels can be used to remove user-specific information from gesture features rather than to reinforce it. A plausible implication is that EHWGesture can support future pipelines that fuse mocap-supervised hand structure with cross-user-invariant gesture dynamics, especially for subtle dexterity assessments and multimodal sequence modeling (Zheng et al., 13 Jul 2025, Tsutsui et al., 2020).