- The paper demonstrates that frozen egocentric-video representations contain clinically meaningful FOG information, with V-JEPA 2 reaching 32.6 F1 and 77.2 AUROC under leave-one-subject-out evaluation.
- Temporal video context substantially improves detection, as VideoMAE-v2 achieved 68.6 AUROC versus 53.6 for the single-frame DINOv3 baseline, although video remained weaker than the accelerometer TCN’s 42.3 F1 and 83.0 AUROC.
- The findings support multimodal fusion because vision and IMUs showed complementary error patterns, but voluntary stopping remained a major shared failure mode and vision alone produced more false alarms during stops than leading IMU models.
Automated detection of freezing of gait (FOG) in Parkinson's disease (PD) has relied almost exclusively on kinematic and physiological sensing, most commonly inertial measurement units (IMUs). This reliance leaves a well-known ambiguity unresolved: low-motion IMU patterns are produced not only by pathological freezing but also by intentional stopping and object interaction during activities of daily living (ADLs). Because FOG is strongly triggered by contextual factors—doorways, cluttered pathways, turning, dual-tasking—and patients often report recurrent episodes at specific home "hotspots," the authors argue that disambiguation requires context beyond kinematics. Egocentric vision (ego-vision), which observes the wearer's first-person view of the environment and ongoing task, is a natural candidate modality, yet no prior study had evaluated it for FOG detection. The paper formulates this as a context-aware clinical motion understanding problem and provides the first empirical test of whether frozen foundation model (FM) representations from ego-video carry FOG-relevant information in the home setting.
Dataset and protocol
The study collected synchronized multimodal data from 15 individuals with PD who reported daily FOG; two were excluded for incomplete recordings, leaving 13 subjects with 171.6 minutes of recordings. Each participant completed two home sessions—one OFF-medication after overnight withdrawal and one ON-medication—covering five ADL tasks per session: a doorway passage, two daily-life tasks drawn from a list of fourteen, and two tasks at self-reported FOG hotspots. Five Xsens DOT IMUs (60 Hz) on the pelvis, shins, and feet provided inertial data; Pupil Core smart glasses captured egocentric video (30 Hz); four external cameras supported expert annotation in ELAN following the updated technical definition of FOG. Data synchronization was handled by the HERMES framework (Yudayev et al., 18 Jan 2026).
Data were segmented into overlapping windows with a 0.5 s stride, using 2 s windows as the main configuration and 3 s and 10 s as ablations. A window was labeled FOG if it contained at least 0.5 s of annotated freezing—a threshold chosen because the median episode duration was only 0.9 s. All models were evaluated under leave-one-subject-out cross-validation (LOSO-CV) with metrics averaged over held-out subjects; two subjects contributed no FOG-positive window under this labeling rule, so F1, recall, AUPRC, and AUROC were averaged over 11 subjects while false positive rate (FPR) used all 13. Statistical comparisons used Friedman tests with Wilcoxon signed-rank post-hoc tests and Holm correction.
The cohort exhibited substantial inter-subject heterogeneity in FOG burden (mean 7.2% time frozen, 14.4 episodes per session, both with large standard deviations), which the authors note poses a genuine challenge for subject-independent modeling and inflates the variance of all reported scores.
Models compared
The evaluation covers two families of frozen FM representations probed with an L2​-regularized logistic regression, plus fully trained TCN baselines:
| Modality |
Model |
Design |
| IMU |
UniMTS |
Motion-specialized, SMPL skeletal graph encoder |
| IMU |
Chronos-2 |
General-purpose multivariate time-series model |
| Ego-video |
DINOv3 |
Single-frame image encoder (ViT-B) |
| Ego-video |
VideoMAE-v2 |
Third-person video, 16 frames (ViT-B) |
| Ego-video |
V-JEPA 2 |
Third-person video, 64 frames (ViT-L) |
| Ego-video |
EgoVideo |
Egocentric video, 4 frames (~1B parameters) |
| IMU baseline |
TCN (acc / acc+gyro) |
Fully trained end-to-end |
This selection deliberately varies temporal coverage, pretraining domain, and capacity. DINOv3 versus VideoMAE-v2 forms the closest controlled pair, sharing backbone size and output dimensionality while differing only in frame count.
Main results
At the primary 2 s operating point, the fully trained accelerometer-only TCN achieved the strongest event-detection performance with 42.3 F1 and 83.0 AUROC. Among frozen representations, Chronos-2 led the IMU models (38.7 F1, 82.9 AUROC, 6.1% FPR), while V-JEPA 2 led all video models (32.6 F1, 77.2 AUROC)—surpassing the motion-specialized UniMTS on F1, AUROC, and FPR. Notably, adding gyroscope channels to the TCN lowered F1 (35.5 vs. 42.3) despite reducing FPR, an observation the authors report without a definitive explanation.
Key findings on the visual modality:
- Temporal aggregation matters: moving from one frame to sixteen with matched backbones raised AUROC from 53.6 (DINOv3) to 68.6 (VideoMAE-v2); all three video encoders exceeded the single-frame baseline on AUROC (p<0.01). This establishes that egocentric video carries FOG-relevant information beyond static appearance.
- Capacity does not explain performance: EgoVideo (~1B parameters) underperformed V-JEPA 2 (~300M) on F1 (27.7 vs. 32.6).
- Above-chance but weaker than IMU: ego-video alone did not outperform the trained TCN or the best IMU FM, so the claim that visual context helps must be qualified—it is predictive, not superior.
A window-length ablation revealed a divergent pattern: F1 generally increased from 2 s to 10 s windows (e.g., Chronos-2 from 38.7 to 42.4) while AUROC consistently decreased (Chronos-2 from 82.9 to 71.6). The authors attribute this to longer windows mixing FOG and non-FOG behavior and diluting label precision, and correctly caution that the three lengths constitute separate operating points rather than comparable conditions—rising F1 here does not indicate improved discrimination.
Qualitative analysis of modality-specific errors
Case-level inspection showed that ego-vision and IMU models frequently detected the same episodes at similar timestamps, supporting the interpretation that visual features encode independent predictive information rather than redundant signals. In some recordings, V-JEPA 2 separated prolonged true freezes from voluntary stops better than the IMU models. However, a direct quantitative check undermines any strong claim that vision solves the stopping-ambiguity problem: measured on annotated stop windows, V-JEPA 2 produced more false alarms than Chronos-2 (24.9% vs. 9.7%) and the acc-only TCN (11.0%), and every model false-alarmed more during stopping than on other non-FOG windows. Voluntary stopping remains a shared failure mode across modalities.
Failure cases were informative: one subject with no annotated FOG generated many V-JEPA 2 false positives during object interaction with hands outside the camera view, and another subject's recording suffered degraded visual features under poor lighting. These observations motivate multimodal fusion rather than vision-only deployment, though the authors did not implement fusion in this work.
Limitations and open questions
Several constraints bound the conclusions. The cohort is small (13 analyzed subjects) with short semi-structured ADL sessions; although more ecologically valid than laboratory protocols, robustness across unsupervised free-living conditions remains untested. The framework relies entirely on frozen features, leaving encoders unadapted to FOG—low-rank adaptation may yield gains that this design cannot capture. Whether the ego-vision signal reflects motion, environmental context, or both is left unresolved pending interpretability analysis. The pipeline performs offline classification over pre-segmented windows and is not directly applicable to real-time closed-loop interventions such as on-demand cueing, which would require streaming causal inference and lightweight encoders. Finally, long-term usability and privacy implications of smart-glasses wear in the home were not assessed.
Conclusion
This paper introduces egocentric vision as a contextual modality for FOG detection during home-based ADLs and demonstrates, via linear probes on frozen FM representations under LOSO evaluation, that ego-video features contain clinically meaningful FOG information above chance—with V-JEPA 2 reaching 77.2 AUROC against 83.0 for a fully trained IMU TCN. The evidence supports complementarity between visual context and inertial sensing, but also shows plainly that vision alone neither matches IMU performance nor resolves voluntary-stopping false positives at the cohort level. The open questions the work leaves—multimodal fusion, fine-tuning, interpretability of what the visual signal encodes, and real-time deployment—are concrete and well-scoped follow-ons to the formulation established here.
