Towards Whole Hand and Wrist Kinematic Tracking with a Wearable A-Mode Ultrasound Probe

Abstract: A-mode ultrasound (US) has emerged as a promising modality for hand and wrist motion tracking. Prior works have mainly addressed static gesture classification or regression of a few degrees of freedom (DoFs), typically relying on non-wearable systems and external computing devices, and highlight the need for strategies to ensure robustness to sensor repositioning. In this work, we propose a framework for robust whole-hand and wrist kinematic tracking via wearable A-mode US using the WULPUS platform, tackling the regression of 23 DoFs directly on the probe. First, we introduce a compact (11285 parameters) multi-output convolutional neural network combined with an incremental training strategy, which improves inter-session generalization and reduces mean absolute error by more than 17% compared to a non-incremental approach. Second, we demonstrate, for the first time, the feasibility of end-to-end hand and wrist kinematic tracking entirely on-device. We deploy the model on the WULPUS nRF52832 microcontroller, achieving 0.73 mJ per inference, 29.1 ms latency, and showing the feasibility of full operation (data acquisition, online inference, and BLE streaming of results) within 33 mW, enabling up to 36 hours of continuous use and an 88% reduction in wireless bandwidth compared to raw data transmission.

• The paper advances whole-hand and wrist tracking by regressing 23 degrees of freedom using a wearable A-mode ultrasound and compact CNN model.
• It implements an embedded system on an nRF52832 MCU, achieving 29.1 ms real-time inference with low power consumption and efficient data streaming.
• Incremental fine-tuning reduces mean absolute error by up to 33% post-repositioning, improving robustness for practical prosthetic and HCI applications.

Whole-Hand and Wrist Kinematic Tracking Using Wearable A-Mode Ultrasound: An Embedded, Incremental Approach

Introduction

Wearable sensing for hand gesture recognition and fine-grained kinematic tracking remains a critical challenge for robust human–machine interfaces, rehabilitation, and neuromotor prosthetic control. A-mode ultrasound (US) has emerged as a compelling alternative to traditional modalities such as sEMG, particularly due to its deeper muscle probing capabilities and efficient integration into low-resource embedded systems. The presented work advances the state of A-mode US-based kinematic tracking by targeting simultaneous regression of 23 degrees of freedom (DoFs) for whole-hand and wrist movements, while also resolving longstanding issues related to sensor repositioning robustness and practical, wearable deployment.

System Architecture and Embedded Implementation

The WULPUS platform constitutes the hardware backbone, integrating four 32-element linear array transducers into an armband form factor, with all embedded electronics implemented on an nRF52832 microcontroller. An aggressive parameter budget (11285 parameters) in the proposed convolutional neural network enables operation within acute flash and RAM constraints, supporting end-to-end inference entirely on the MCU.

Figure 1: WULPUS architecture and detailed view of the data flow within the nRF52 MCU for embedded inference.

Key pipeline characteristics include: (1) frame acquisition at 30 Hz, (2) SPI-based data transfer to the MCU, (3) on-device regression for all 23 DoFs, and (4) result streaming via BLE. On-chip quantization (float32→int8) and deployment via TensorFlow Lite Micro enable real-time inference at 29.1 ms per pass (energy: 0.73 mJ), facilitating over 36 hours of continuous use on a standard 320 mAh Li-Po cell. Wireless bandwidth reduction (88% relative to raw data streaming) substantially improves system scalability and operation under adverse RF conditions.

Data Collection, Training, and Labeling

The dual-subject protocol employs simultaneous capture from the WULPUS armband and Manus Quantum Metaglove for high-res ground-truth joint labels. Annotations include 20 finger joints and 3 wrist Euler angles, computed via quaternion processing and denoised using session-specific Kalman filtering. Three sessions per subject, with armband repositioning and interleaved rest, ensure realistic assessment of session-to-session variability and transducer displacement artifacts. Each dataset sample comprises $400 \times 4$ raw US channels, with downsampled kinematic ground-truth synchronized to achieve precise cross-modal alignment.

Network Design and Methodology

The architecture leverages two convolutional layers (per channel), ReLU, and max pooling, concatenating features for parallel regression of all target DoFs via fully connected heads. The lightweight model is trained with MSE loss. Multiple validation regimes are used:

• Aggregated 5-fold CV (across all sessions)
• Inter-session generalization (train/val on two sessions, test on unseen)
• Incremental fine-tuning (simulate post-repositioning recalibration, with ablations over fine-tuning set size and learning rate)

Performance Analysis

Kinematic Regression Accuracy

Aggregated 5-fold CV results demonstrate mean absolute error (MAE) of $6.1^\circ \pm 0.5^\circ$ (Subject 1) and $6.4^\circ \pm 0.7^\circ$ (Subject 2), and $R^2$ scores near 0.7, notable given the substantial output dimensionality and network compactness. Inter-session performance, emulating realistic use with armband removal/repositioning, yields degraded but competitive accuracy (MAEs of $9.5^\circ$ and $11.2^\circ$, $R^2$ ≈ 0.3), consistent with—but at lower computational cost than—prior art.

Incremental Fine-Tuning for Repositioning Robustness

A significant contribution is the robust mitigation of performance loss caused by armband repositioning via on-demand fine-tuning. Utilizing only ~5 minutes of recalibration data, an incremental update reduces MAE by 25% for Subject 1 ($9.5^\circ \to 7.1^\circ$) and 33% for Subject 2 ($11.2^\circ \to 7.5^\circ$), with corresponding $R^2$ improvements ($6.1^\circ \pm 0.5^\circ$0, $6.1^\circ \pm 0.5^\circ$1).

Figure 2: Subject 1 cross-validation MAE/RMSE/$6.1^\circ \pm 0.5^\circ$2 across acquisition sets; incremental fine-tuning clearly improves generalization after transducer repositioning.

Even a single post-repositioning set yields >17% reduction in MAE, with further improvement from additional sets. Improvements primarily reflect correction of prediction offsets, highlighting the model's capacity for rapid adaptation to session-specific US-channel-to-anatomy mappings.

Figure 3: Joint angle regression outputs for Subject 2, showing better alignment with ground truth following incremental fine-tuning.

Embedded System Performance and Practical Utility

Deployed system memory usage is highly efficient (16.8 kB Flash, 10.1 kB RAM). Latency (29.1 ms/inference) and power consumption ($6.1^\circ \pm 0.5^\circ$333 mW for the entire pipeline) exceed the real-time requirements for prosthetic device control. Transmission of prediction results—rather than raw US data—enables both resilience and scalability in multi-sensor deployments.

Implications, Limitations, and Future Directions

This research establishes both theoretical and practical foundations for high-DoF, wearable hand/wrist motion tracking using A-mode ultrasound. The results substantiate that compact multi-output CNNs, combined with simple user-driven incremental fine-tuning, can deliver robust regression in resource-constrained, embeddable systems. Practical implications extend to improved control for clinical myoelectric prostheses and more reliable multimodal human–machine interfaces, with deployment scenarios no longer restricted by communication or computation bottlenecks.

The main limitation is the small sample size (two subjects), motivating future multi-user validation. Expanding transducer coverage and exploring dynamic, task-specific fine-tuning schemes are likely to further enhance robustness and spatial kinematic fidelity.

Conclusion

The presented work demonstrates that whole-hand and wrist kinematic tracking via wearable A-mode US is feasible at scale, in real time, and under realistic (variable-sensor) conditions. Compact, incrementally fine-tunable neural models deployed on embedded hardware achieve low error, low latency, and drastic communication cost reductions, with immediate applications in prosthetic control and personalized HCI paradigms.