WatchHand: Continuous 3D Hand Pose Tracking on Smartwatches

• WatchHand is a continuous 3D hand pose tracking system that uses inaudible FMCW chirps from smartwatch speakers and microphones to estimate 20 finger joints relative to the wrist.
• It employs a deep-learning model with a wrist-centered regression approach using spatiotemporal echo profiles, achieving a mean per-joint error of 7.87 mm in cross-session tests.
• The system leverages commodity hardware to eliminate custom sensors, enabling continuous tracking across full motion sequences and reducing deployment barriers.

WatchHand is a continuous 3D hand pose tracking system for commercial off-the-shelf smartwatches that uses only the watch’s built-in speaker and microphone, without custom hardware, external sensors, or cameras. It emits inaudible frequency-modulated continuous waves, captures their reflections from the hand, converts the resulting acoustic signals into spatiotemporal echo profiles, and applies a deep-learning model to estimate the 3D coordinates of 20 finger joints relative to the wrist. Its stated significance is that it lowers the barrier to smartwatch-based hand tracking by eliminating additional hardware while enabling continuous prediction across full motion sequences, including transitions into and out of poses (Kim et al., 25 Feb 2026).

1. Commodity smartwatch hand tracking

WatchHand addresses a central limitation of prior wrist-worn hand sensing: continuous hand reconstruction has generally required custom wristbands, cameras, radar, electrode arrays, or other external instrumentation, whereas commercial smartwatches have largely supported only coarse discrete gesture recognition. The system is framed explicitly as the first continuous 3D hand pose tracking system implemented on off-the-shelf smartwatches using only their built-in speaker and microphone (Kim et al., 25 Feb 2026).

The sensing platform is deliberately restricted to mass-market devices. The evaluated watches are the Samsung Galaxy Watch 7, Xiaomi Watch 2 Pro, and Google Pixel Watch 3, all running WearOS and supporting simultaneous 48 kHz, 16-bit PCM playback and recording. In all three devices, the speaker is on one side and the microphone is on the opposite side. WatchHand uses the 18–21 kHz band, which is intended to be inaudible or minimally audible while remaining reproducible by commodity smartwatch audio hardware. At 10 cm, the measured output levels are 61.6 dBA for Galaxy Watch 7, 66.9 dBA for Xiaomi Watch 2 Pro, and 80.8 dBA for Pixel Watch 3 (Kim et al., 25 Feb 2026).

The task formulation is continuous rather than class-based. The system predicts a 3D hand pose for every video-aligned time step, using all frames rather than isolated static moments. Ground truth is derived from MediaPipe Hands’ 21 landmarks, but the wrist is used as the origin and the remaining 20 landmarks are predicted relative to that origin, yielding a 60-dimensional regression target. This design makes WatchHand a wrist-centered hand-state estimator rather than a gesture classifier (Kim et al., 25 Feb 2026).

2. Acoustic sensing and echo-profile construction

WatchHand turns the smartwatch into an active sonar. The watch speaker emits inaudible FMCW chirps, and the microphone records the direct path and the reflections and diffractions from the hand. The paper adopts correlation-based FMCW rather than conventional linear FMCW because the nominal ranging resolution is bounded by sampling rate rather than sweep bandwidth. For a sweep bandwidth $B$, the conventional linear FMCW range resolution is

$R_{L\text{-}FMCW} = \frac{C}{2B}$

and, with the 18–21 kHz sweep, this becomes

$$R_{L\text{-}FMCW} = \frac{343}{2 \times 3000} = 0.0572\,\text{m} = 57.2\,\text{mm}.$$

For C-FMCW, the effective resolution is instead

$$R_{C\text{-}FMCW} = \frac{C}{2F_s} = \frac{343}{2 \times 48000} = 0.00357\,\text{m} = 3.57\,\text{mm},$$

where $C=343\,\text{m/s}$ and $F_s=48000$ Hz (Kim et al., 25 Feb 2026).

Each chirp has length $L=600$ samples, so the chirp duration is

$$T = \frac{L}{F_s} = \frac{600}{48000} = 0.0125\,\text{s} = 12.5\,\text{ms}.$$

This defines one acoustic frame every 12.5 ms (Kim et al., 25 Feb 2026).

The signal-processing front end begins with a 5th-order Butterworth bandpass filter over 18–21 kHz. Synchronization is then performed using the direct path as the earliest reference. The paper defines

$Corr_{\text{sync}} \leftarrow Rx \star Tx$

and identifies the strongest peak as

$$p_{\text{start}} \leftarrow \arg\max \left( \left| Corr_{\text{sync}} \right| \right),$$

which is adjusted to the chirp frame center to align subsequent processing (Kim et al., 25 Feb 2026).

After synchronization, cross-correlation is again applied:

$R_{L\text{-}FMCW} = \frac{C}{2B}$0

The result is reshaped into an echo-profile matrix

$R_{L\text{-}FMCW} = \frac{C}{2B}$1

whose vertical axis is lag or range bin and whose horizontal axis is time over successive chirps. To suppress static clutter, the system computes differential echo profiles:

$R_{L\text{-}FMCW} = \frac{C}{2B}$2

WatchHand then retains the 60 closest vertical pixels, corresponding to 21.42 cm, and uses a temporal window of 1.2 seconds, or 96 frames. The final model input is therefore a normalized 2-channel tensor of size $R_{L\text{-}FMCW} = \frac{C}{2B}$3, where the two channels are the original echo profile and the differential echo profile (Kim et al., 25 Feb 2026).

Commodity smartwatch audio pipelines introduce device-specific timing artifacts. The paper reports peak misalignment across frames and periodic drift in differential profiles, then corrects them through sliding-window peak correction and a median filter on original echo profiles. These calibration steps are presented as hardware-agnostic and are applied across all tested watches (Kim et al., 25 Feb 2026).

3. Pose representation and learning architecture

The output of WatchHand is the continuous 3D pose of 20 hand landmarks relative to the wrist. Ground-truth landmarks come from MediaPipe Hands, and the wrist-centered representation is further normalized to reduce session-to-session variation. The normalization uses a palm plane defined by the wrist-to-index-MCP and wrist-to-little-MCP vectors, then computes a rotation matrix aligning that plane to a reference palm plane. Hand size is scaled so that the wrist-to-little-MCP distance matches a measured physical length. Wrist rotation is intentionally not normalized (Kim et al., 25 Feb 2026).

The acoustic model is based on FastViT-T12, selected after comparison with ResNet-18 and CNN-LSTM. The architecture includes a convolutional stem with initial $R_{L\text{-}FMCW} = \frac{C}{2B}$4 downsampling, four hierarchical stages, RepMixerBlocks, and PatchEmbed layers between stages. A global average pooling layer and a linear layer form the regression head, producing 60 outputs for the 20 landmarks. The model contains 6.62M parameters (Kim et al., 25 Feb 2026).

Temporal information is encoded directly in the spatiotemporal acoustic input rather than through an explicit recurrent architecture. Each prediction is made from a 1.2-second echo-profile window, and inference uses a sliding window aligned to 30 FPS ground truth with stride $R_{L\text{-}FMCW} = \frac{C}{2B}$5 second. The model is trained end-to-end using a composite loss consisting of standard mean squared error on joint coordinates and a velocity loss weighted by $R_{L\text{-}FMCW} = \frac{C}{2B}$6, intended to encourage temporal smoothness (Kim et al., 25 Feb 2026).

This combination of wrist-centered landmark regression, pose normalization, and temporally stacked acoustic features indicates that WatchHand treats hand tracking as dense spatiotemporal acoustic inversion rather than discrete pose classification. A plausible implication is that the system is optimized for continuous interactive state estimation rather than symbolic gesture vocabularies.

4. Evaluation, robustness, and adaptation

The paper evaluates WatchHand across multiple smartwatch models, wearing-hands, body postures, noise conditions, and pose-variation protocols. Its headline result is a mean per-joint position error of 7.87 mm in cross-session tests with device remounting (Kim et al., 25 Feb 2026).

The reported robustness conditions are central to the system’s intended practicality. Cross-session testing with device remounting directly targets one of the main deployment problems for wrist acoustics: the geometry between speaker, microphone, and hand changes when the watch is removed and worn again. The inclusion of multiple smartwatch models further tests whether the approach depends on a single transducer configuration. The inclusion of wearing-hands and body postures expands the evaluation beyond a single laboratory posture, and noise conditions probe whether the 18–21 kHz band and the correlation-based front end remain stable under real-world acoustic interference (Kim et al., 25 Feb 2026).

The paper is explicit that performance drops for unseen users or gestures. It is equally explicit that the model adapts effectively with lightweight fine-tuning on small amounts of data. This establishes personalization as a significant component of WatchHand’s operating regime. The system is therefore not presented as solving user-independent hand tracking completely; rather, it demonstrates that commodity-watch acoustic sensing can provide a viable base model that benefits from efficient user adaptation (Kim et al., 25 Feb 2026).

The main failure mode identified in the visible technical description is interference from objects held in the hand, because those objects create additional reflections within the 21.42 cm sensing region. This is a structurally important limitation: the method senses acoustic returns from whatever occupies the near-hand volume, not only the anatomical hand surface. This suggests that bare-hand pose tracking and hand-object interaction may require different supervision or model designs, even if they share the same sensing front end.

5. Position within wrist-worn hand sensing

WatchHand belongs to a broader class of wrist-worn hand sensing systems, but it occupies a distinct point defined by commodity deployability. The closest acoustic antecedent in the provided literature is "EchoWrist" (Lee et al., 2024), which also uses active acoustics for continuous wrist-worn hand sensing, but does so with a custom wristband containing two speakers and two microphones. EchoWrist reports MJEDE of 4.81 mm for 20-joint 3D hand pose and 97.6% accuracy on 12 naturalistic hand-object interactions, while operating at 57.9 mW. WatchHand, by contrast, removes the need for custom hardware and runs on unmodified smartwatches, at the cost of higher cross-session error at 7.87 mm (Lee et al., 2024).

A second relevant point of comparison is "Cross, Dwell, or Pinch: Designing and Evaluating Around-Device Selection Methods for Unmodified Smartwatches" (Kim et al., 4 Mar 2025), which showed that a commercial Galaxy Watch 6 can support sonar-based around-device input using only its built-in speaker and microphone, but only for fine-grained one-dimensional finger tracking within 10 cm. WatchHand extends this commodity acoustic line from 1D around-device control to continuous 3D estimation of 20 finger joints (Kim et al., 4 Mar 2025).

Wrist-worn hand sensing also includes optical and multimodal systems that pursue richer state estimation through additional hardware. "WristPP: A Wrist-Worn System for Hand Pose And Pressure Estimation" (Xi et al., 28 Feb 2026) uses a wrist-mounted RGB camera prototype with a Vision Transformer backbone and reports MPJPE of 2.9 mm while jointly estimating 3D hand pose and per-vertex pressure. "Wrist2Finger: Sensing Fingertip Force for Force-Aware Hand Interaction with a Ring-Watch Wearable" (Xiao et al., 5 Oct 2025) uses a ring IMU plus wrist EMG and reports an average MPJPE of 0.57 cm together with fingertip force estimation. These systems expose an important contrast: WatchHand accepts the sensing constraints of mass-market smartwatches, whereas other systems trade additional hardware for either lower geometric error or richer outputs such as pressure and force (Xi et al., 28 Feb 2026).

System	Sensing configuration	Representative reported result
WatchHand	Off-the-shelf smartwatch speaker and microphone	7.87 mm MPJPE in cross-session tests with device remounting
EchoWrist	Custom wristband with two speakers and two microphones	4.81 mm MJEDE; 97.6% interaction accuracy
WristPP	Wrist-mounted RGB camera prototype	2.9 mm MPJPE; per-vertex pressure
Wrist2Finger	Ring IMU + wrist EMG	0.57 cm MPJPE; fingertip force estimation

This comparison suggests that WatchHand occupies the commodity-deployment end of the wrist-worn hand-sensing design space. Its distinctive contribution is not the absolute lowest pose error in the paper set, but the elimination of external sensing hardware while preserving continuous 3D hand tracking on existing watches.

6. Limitations, technical implications, and research significance

WatchHand’s limitations follow directly from its sensing assumptions. It relies on a single commodity speaker-microphone pair per watch, on the audio stack’s ability to sustain simultaneous 48 kHz playback and recording, and on stable near-hand acoustic geometry. Performance degradation for unseen users and gestures indicates that the learned mapping from echo profiles to 3D pose remains partly user- and gesture-dependent. The need for calibration against peak misalignment and periodic drift shows that commodity smartwatch audio subsystems are not designed for precision active sensing and must be corrected in software (Kim et al., 25 Feb 2026).

The handling of held objects is another major boundary condition. Because the method senses reflections in the near-watch region rather than directly observing kinematic landmarks, objects within that region can perturb the echo profile in ways that are not simply nuisance noise. This implies that extending WatchHand from bare-hand tracking to hand-object tracking would likely require explicit object-aware training or a different task formulation.

Despite these constraints, the broader significance of WatchHand is clear. It demonstrates that continuous 3D hand pose tracking can be realized on off-the-shelf smartwatches using only built-in audio hardware, and it does so under cross-session remounting and multi-device evaluation rather than under a single custom laboratory setup. In the context of prior work, this establishes a design pattern for commodity wrist computing: exploit ubiquitous sensors already integrated into watches, construct a task-specific signal representation—in this case original and differential C-FMCW echo profiles—and rely on lightweight adaptation when user-independent generalization is insufficient. This suggests a route toward always-available smartwatch interaction that does not depend on cameras, rings, instrumented environments, or custom straps (Kim et al., 25 Feb 2026).