Every Step of the Way: Video-based Parkinsonian Turning Step Counting

Abstract: As a prominent symptom of Parkinson's disease (PD), turning impairment is evaluated through parameters such as turning angle, duration, and particularly, the number of steps required to complete a turn, which directly reflects motor dysfunction. Accurate step counting is challenging due to variability in real-world turning movements and atypical shuffling patterns in parkinsonian gait. Existing methods are predominantly wearable-based, requiring users to wear and manage dedicated devices, which can be inconvenient for continuous daily use. To address this, we propose a passive, video-based framework that estimates step count in a coarse-to-fine manner using diverse motion representations. Specifically, an initial step count is estimated from foot movement signals derived from 3D human mesh recovery, providing high-level motion structures. To incorporate fine-grained motion details, a motion encoder learns complementary gait dynamics from mesh and optical flow to refine the initial estimate. In this process, coarse foot movement signals query the pixel-level motion cues via cross attention to capture subtle parkinsonian gait dynamics. To handle varying video lengths, we partition each video into clips and integrate clip-wise motion embeddings via multiple instance learning (MIL) for step count residual prediction. Extensive experiments show our method consistently outperforms existing step counting methods on real-world PD turning datasets.

• The paper introduces a video-based framework that fuses 3D mesh and optical flow via cross-attention to robustly count parkinsonian turning steps.
• It achieves significant improvements over wearable and pose-based methods with a notable MAE reduction (15.9 vs. 37.7 on PD-FOG) and higher accuracy thresholds.
• The method’s correlation with MDS-UPDRS III validates its potential for scalable, noncontact digital biomarkers in PD assessment.

Video-Based Step Counting for Parkinsonian Turning: Methodology and Results

Introduction and Problem Context

Parkinson’s disease (PD) is characterized by significant motor impairment, with turning deficits being pronounced in early stages and tightly linked to fall risk and disease progression. The number of steps required to complete a turn directly reflects core clinical phenomena such as bradykinesia and rigidity, correlating with the MDS-UPDRS III score. Reliable quantification of step-level turning parameters historically relied on labor-intensive clinical ratings or dedicated wearables (IMUs, pressure insoles), both of which have limitations concerning convenience, sensor placement sensitivity, or ecological validity. This work addresses these constraints by introducing a fully video-based framework for fine-grained automatic step counting during parkinsonian turning. The approach represents a technical advancement in noncontact, passive assessment of heterogeneous real-world turning patterns.

Figure 1: Illustration of heterogeneous turning steps in PD, visualizing differences between coarse structural movement and subtle shuffling or pivoting actions.

Architecture and Methodological Contributions

The proposed method employs a coarse-to-fine estimation pipeline that integrates multiple motion representations. Initially, 3D human mesh sequences are reconstructed per-frame using a state-of-the-art mesh recovery model (PromptHMR). Foot movement signals are extracted from per-joint stationary logits, capturing coarse-grained step events—even in the presence of atypical sliding or shuffling consistent with PD pathology. Using a noisy-OR fusion of toe/heel logits and a differentiable soft-peak detection, an initial count is produced that already encodes clinical relevancy.

The refinement module introduces a cross-modal fusion architecture. Mesh-based foot motion signals are embedded as local temporal sequences, while per-pixel optical flow features (extracted via FlowSeek, encoded with ResNet-TSM) capture fine-grained, temporally sensitive foot dynamics localized to mesh-projected regions of interest. Cross-attention is used to let coarse mesh-based queries retrieve fine flow details, enabling the model to correct for artifacts or missed steps due to mesh-skate, a common failure in kinematic models for pathological movements.

Inference on long and heterogeneous videos is handled by sequential partitioning into non-overlapping clips. Multiple-Instance Learning (MIL) is performed on fused clip-level motion embeddings, with clip saliency learned to focus attention on most informative (potentially error-prone or clinically salient) regions, summarized for final residual correction. The model objective combines squared error loss with a contrastive regularization (InfoNCE) over high- and low-saliency clip embeddings to promote discriminative representations.

Figure 2: The overall framework for step detection, showing foot motion extraction, fusion with optical flow via cross attention, and multiple-instance learning for residual correction.

Experimental Evaluation

Datasets and Protocol

Performance was validated on two clinically annotated datasets: PD-FOG (in-clinic, long-duration turning-in-place by PD patients with parallel IMU recordings) and Turn-REMAP (trimmed, real-world at-home turnings by PD and healthy controls). Step counts were annotated by clinical experts, and leave-one-subject-out cross-validation was used for PD-FOG.

Core Results

On PD-FOG, the proposed framework achieves a mean absolute error (MAE) of 15.9 steps, substantially outperforming the best IMU-based baseline (Lucot et al.; MAE 37.7) and all vision-based step inference methods. The accuracy at a 20% relative error threshold is improved from 0.779 to 0.962. These gains are especially pronounced in subjects with moderate PD severity, indicating robustness to the increased step irregularity typical of advanced disease phenotypes.

For video data from daily living (Turn-REMAP), the model reduces MAE to 0.625, outperforming prior pose- and activity-counting methods, and delivers the highest accuracy at a strict 5% error threshold, a demanding measure for short, low-step-count clips.

Figure 3: Step count estimation results on PD-FOG, comparing the frame-level predictions of several methods and visualizing error distributions against ground truth.

Qualitative analyses show that the cross-attention mechanism reliably attends to mesh/flow discrepancies—successfully recovering subtle shuffling or sliding steps that are often missed by standard joint-based pose estimators.

Figure 4: Cross attention heatmap from the motion encoder, highlighting mesh-to-flow correspondences particularly where the mesh alone fails to isolate subtle steps.

Ablations and Modal Analysis

Removing cross-attention (i.e., using self-attention or direct concatenation of features) consistently degrades performance, highlighting the benefit of enforcing directed inductive bias for step recovery. Saliency-weighted MIL pooling outperforms max-pooling, and the inclusion of contrastive regularization over clips further boosts discriminability in learned embeddings. Assessments with added RGB or IMU signals do not yield improvement, indicating the sufficiency and complementarity of the mesh-flow dual encoding.

Parameter sensitivity analyses—e.g., variation in $\beta$ for peak sharpness and $\lambda$ for regularization weight—support the robustness of the method to design choices.

Clinical Implications and Statistical Analysis

The framework is clinically validated by correlating automatically estimated step-level parameters with the MDS-UPDRS III motor severity scale. Average step duration is positively correlated ($\rho=0.471$, $p=0.019$), and angular displacement per step is negatively correlated ($\rho=-0.409$, $p=0.040$) with clinical severity. These statistical associations confirm that the video-based approach preserves clinically meaningful variation and that its quantification is aligned with expert-derived gold standards.

Figure 5: Spearman correlation between step-level turning metrics (duration, angle per step) and MDS-UPDRS III scores, demonstrating clinical validity.

Theoretical and Practical Implications

This study demonstrates that robust step counting during unconstrained, heterogeneous parkinsonian turning can be performed without wearables, leveraging only passive video and modern cross-modal attention mechanisms. The use of mesh and flow, fused via attention, effectively handles the distinct multi-scale and noisy patterns intrinsic to PD movement. The technical advances pave the way for scalable digital biomarkers deployable via ubiquitous cameras, supporting early diagnosis, progression monitoring, or clinical trial endpoints in ecologically valid settings.

From a theoretical standpoint, modality-agnostic attention architectures introduced here are extensible—potentially informing work in action segmentation, pathological event recognition, or other temporally dense, fine-grained clinical inference settings. The MIL structure provides a template for aggregating highly local information in long, variable-length event streams, relevant for multi-modal behavioral phenotyping broadly.

Conclusion

This paper introduces a robust, clinically validated, and modality-agnostic pipeline for video-based step counting in parkinsonian turning (2606.27918). By fusing 3D pose mesh with localized optical flow using cross-attention, and aggregating evidence via multiple-instance learning, the method sets a new state of the art for passive and fine-grained gait assessment in PD. The results substantiate both practical deployment potential and offer insights for further methodological innovations in multimodal clinical sensing.