Papers
Topics
Authors
Recent
Search
2000 character limit reached

A Gait Foundation Model Predicts Multi-System Health Phenotypes from 3D Skeletal Motion

Published 26 Mar 2026 in cs.AI and q-bio.QM | (2603.25283v1)

Abstract: Gait is increasingly recognized as a vital sign, yet current approaches treat it as a symptom of specific pathologies rather than a systemic biomarker. We developed a gait foundation model for 3D skeletal motion from 3,414 deeply phenotyped adults, recorded via a depth camera during five motor tasks. Learned embeddings outperformed engineered features, predicting age (Pearson r = 0.69), BMI (r = 0.90), and visceral adipose tissue area (r = 0.82). Embeddings significantly predicted 1,980 of 3,210 phenotypic targets; after adjustment for age, BMI, VAT, and height, gait provided independent gains in all 18 body systems in males and 17 of 18 in females, and improved prediction of clinical diagnoses and medication use. Anatomical ablation revealed that legs dominated metabolic and frailty predictions while torso encoded sleep and lifestyle phenotypes. These findings establish gait as an independent multi-system biosignal, motivating translation to consumer-grade video and its integration as a scalable, passive vital sign.

Summary

  • The paper introduces a dual-stream spatio-temporal transformer that leverages self-supervised learning from 3D gait sequences of 3,414 adults to predict 18 distinct health systems.
  • The model outperforms traditional engineered features, achieving high Pearson r values (0.69 for age, 0.90 for BMI) and near-perfect AUC scores in sex classification.
  • Gait embeddings provide region-specific, independent biomarkers that improve clinical diagnosis detection and enable scalable, passive health screening.

Gait-Based Self-Supervised Embeddings as Independent Multi-System Health Biomarkers

Background and Motivation

Traditionally, quantitative gait analysis has focused on identifying deviations associated with specific pathologies or functional deficits. Conventional engineered summary metrics (e.g., cadence, stride length) reduce complex movement patterns to low-dimensional scalar features, thus limiting their sensitivity to systemic health states beyond overt dysfunction. Recent advances in self-supervised deep learning and transformer architectures have enabled richer representation learning from high-dimensional spatiotemporal motion data, yet most prior work addresses single pathologies or isolated biomechanical tasks.

This study leverages the largest 3D skeletal gait dataset by subject count (3,414 adults, 17,589 sequences) from the Human Phenotype Project, encompassing deep phenotyping across 18 physiological systems, to derive self-supervised gait embeddings and systematically evaluate their association with broad multi-system health phenotypes. The paradigm shift proposed herein treats gait not merely as a symptom of dysfunction but as a foundational biosignal encoding information orthogonal to age, body composition, VAT, and height.

Methodological Framework

Data Acquisition and Cohort Structure

Motion was captured using a single Azure Kinect depth camera during five standardized motor tasks, yielding 3D skeletal pose sequences for each participant. Extensive multivariate physiological, clinical, behavioral, and multi-omics data defined ground truth phenotypes, organized into 18 body systems.

Model Architecture

A Dual-Stream Spatio-Temporal Transformer (DSTformer), inspired by the Masked Autoencoder (MAE) paradigm, was deployed for self-supervised learning from raw joint position sequences. Anatomy-adaptive spatial-temporal masking and Gaussian noise augmentation were used to encourage generalizable embeddings. The denoising objective incorporated both Euclidean joint position and velocity losses, achieving an average reconstruction error of 8 mm per joint.

Hierarchical pooling of latent vectors aggregated information across anatomical regions, maximizing physiological interpretability and predictive reliability. The final subject-level representation comprised merged anatomical groups with percentile pooling, yielding robust 1024-dimensional signatures.

Statistical Evaluation

A rigorous nested cross-validation framework was employed, with stratification by sex and adjustment for key confounders (age, BMI, VAT, height). Predictive performance was quantified by Pearson r for continuous outcomes and AUC-ROC for binary/ordinal variables. Critically, models comparing demographic/anthropometric baselines versus baselines augmented with gait embeddings isolated the unique contribution of gait physiology to systemic health prediction.

Numerical Results and Contradictory Claims

Learned Embeddings Outperform Engineered Features

  • Age prediction: Gait Fusion model achieved Pearson r = 0.69, surpassing engineered features (r = 0.46–0.51 with height).
  • BMI prediction: r = 0.90 for males, r = 0.90 for females; VAT area prediction r = 0.82 (males), r = 0.80 (females), both dramatically exceeding conventional gait descriptors with height (VAT: r = 0.52, 0.47).
  • Sex classification: Embedding-based ensemble AUC = 0.997, individual task AUC ≥ 0.962.

Multi-System Phenotype Prediction

  • Embeddings significantly predicted 1,980 of 3,210 phenotypes after FDR correction.
  • After covariate adjustment for age, BMI, VAT, and height, gait embeddings yielded independent gains in all 18 systems for males and 17/18 systems for females, including metabolic, cardiovascular, hematopoietic, psychiatric, bone density, liver, sleep, and lifestyle targets.

Anatomical Attribution

  • Legs dominated prediction of metabolic and frailty phenotypes; torso encoded sleep and lifestyle features.
  • Region-specific importance scores (ablation): Legs most critical for 9/12 body systems; sex-stratified analysis revealed age prediction driven by leg dynamics in males but arm/torso in females; bone mineral density mapped to head region in females, distributed in males.
  • The contradictory claim arises in the context of phenotype prediction: While conventional wisdom held that anthropometric factors and age primarily drive health variation, the study demonstrates that gait contains orthogonal information not captured by these variables.

Clinical Relevance

  • Gait embeddings improved prediction of clinical diagnoses (depression, insomnia, diabetes, IBS, heart valve disease, peptic ulcer disease, glaucoma) and mirrored medication usage patterns (antidepressants, proton pump inhibitors, iron supplements).
  • Prediction of organ-specific outcomes, e.g., liver elasticity, sound speed, lipids, hematopoietic indices, and molecular metabolites (e.g., LysoPE(P-16:0)), supports the assertion that gait encodes systemic physiological states, possibly prior to clinical manifestation.

Implications and Future Directions

Practical Implications

  • Passive, scalable health screening using video-based motion capture is feasible; the single-depth camera setup negates the need for force plates or multi-camera laboratories.
  • Embeddings derived from 2D video pose estimation architectures (e.g., MotionBERT) could be adapted for large-scale, consumer-grade application.
  • Targeted motor protocols (trunk-focused for sleep/lifestyle, lower-limb for metabolism/frailty) may optimize screening sensitivity for specific clinical endpoints.

Theoretical Implications

  • Gait emerges as an independent biosignal, with region-specific kinematic signatures mapping onto distinct physiological domains.
  • The findings suggest biomechanical-molecular links, e.g., sarcopenia-MASLD interplay, neuroinflammatory processes detectable via movement prior to symptomatic phase.
  • Sex-divergent biomechanical signatures underscore the need for stratified analytical approaches in future screening paradigms.

Speculation on AI Developments

  • Transfer learning from foundation gait models to other populations, hardware, and pathologies will require domain-adaptive architectures.
  • Longitudinal prediction of disease onset and progression, leveraging temporally stable individual gait signatures, will be enabled as follow-up data matures.
  • Finer-grained interpretability (joint-level, temporal attribution) and integration of multi-modal data (e.g., wearables, environmental exposures) may produce composite digital biomarkers for whole-body health.

Conclusion

This work establishes self-supervised gait embeddings as robust, reproducible biomarkers of multi-system health, independently predictive of broad phenotypic, molecular, and clinical endpoints beyond demographic and anthropometric confounders (2603.25283). The development of anatomically interpretable foundation models and their translation to scalable, passive video acquisition pave the way for digital gait to become an accessible vital sign and integrative health readout. Continued methodological refinement, external validation, and targeted motor assessment design will be required to fully realize the clinical and research potential of gait-centric digital phenotyping.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

Overview: What is this paper about?

This study shows that the way you move—your “gait”—can reveal a lot about your overall health, not just whether you have a specific problem. The researchers built an AI model that watches simple 3D “stick‑figure” motion of people doing five short movement tasks and learns a kind of movement “fingerprint.” Using only how someone moves, the model can predict many health traits across the body, from bone strength and body fat to sleep patterns and even some mental health indicators. The big idea: gait can be treated like a new vital sign for many parts of health, not just for diagnosing one disease.

What were they trying to find out?

The researchers asked a few simple questions in everyday terms:

  • Can an AI learn from how people move to predict a wide range of health measures, not just movement problems?
  • Does movement tell us something new about health beyond basic facts like age, height, and body size?
  • Which body parts (legs, torso, arms, head) carry the most useful health signals while we move?

How did they do it?

They studied 3,414 adults (ages 20–79) who were generally healthy. Each person did five short movement tasks while facing a single depth camera (like an advanced motion sensor that turns you into a 3D stick figure). Tasks included:

  • Walking on a treadmill (at a fixed speed and at self‑chosen speed)
  • Marching in place
  • Standing still with eyes closed (balance test)
  • Sitting down and standing up (sit‑to‑stand)

Then they trained an AI model in a few steps:

  • Turning people into stick figures: The camera recorded the positions of key joints (like hips, knees, shoulders) in 3D over time—basically a moving stick‑figure.
  • Teaching by a “fill‑in‑the‑blank” game: The AI used a method called a masked autoencoder. Think of it like hiding pieces of the motion and asking the model to guess the missing parts. By practicing this over and over, it learned the patterns of human movement without needing labels. The model got so good it could fill in joint positions with an average error of about 8 millimeters.
  • Making a movement “fingerprint”: From this training, the AI created compact “embeddings”—numbers that summarize a person’s unique movement patterns across space and time.
  • Testing predictions: Using only these movement fingerprints, the researchers tried to predict hundreds of health traits (called “phenotypes”) across 18 body systems (like bones, blood, liver, sleep, and more). They compared their approach to traditional gait measures (like stride length or step speed) and checked whether movement adds information beyond age, height, body mass index (BMI), and visceral fat (VAT, the fat around organs).

Key terms in simple words:

  • Embedding: a short numerical “signature” of someone’s movement style.
  • Correlation (Pearson r): a score from 0 to 1 that tells how closely predictions match real values (higher is better).
  • AUC (for diagnoses): a score from 0.5 (no better than guessing) to 1.0 (perfect).
  • VAT (visceral fat): the fat deep in your belly around organs, linked to health risks.

What did they discover, and why does it matter?

The main takeaways are straightforward and important:

  • Movement alone predicts many health traits well.
    • From gait, the model strongly predicted:
    • Age (correlation about 0.69 when combining all tasks)
    • BMI (about 0.90)
    • Visceral fat area (about 0.82)
    • It found 1,980 significant links out of 3,210 tested traits using only movement.
  • Movement adds new information beyond age and body size.
    • Even after accounting for age, BMI, height, and VAT, movement still improved predictions across nearly all body systems (all 18 in men; 17 of 18 in women). This means gait isn’t just a proxy for “you’re older” or “you have more body fat”—it captures something deeper about health.
  • Better than traditional gait stats.
    • Simple measures like step length and cadence missed a lot. The AI’s learned movement fingerprint did much better, especially for traits that don’t seem obviously connected to walking (like liver measures or blood markers).
  • Clinical relevance showed up, too.
    • Gait helped predict diagnoses and medication use (beyond age and body size). For example, it improved detection of:
    • Depression, insomnia, back pain (men), and conditions like irritable bowel syndrome and diabetes (women)
    • Matching medications (like antidepressants, stomach acid medicines, iron supplements) also became more predictable from how people move.
  • Which body parts matter most?
    • The legs carried the most information for many systems, especially metabolism and frailty (e.g., grip strength, triglycerides).
    • The torso (your trunk) was most informative for sleep and lifestyle habits (like hours of sleep and exercise patterns), suggesting your core stability and sway hold valuable clues.
    • There were some differences between men and women—for example, age prediction relied more on leg movements in men and more on arm/torso movements in women.

Why this matters:

  • It shows your everyday movement reflects whole‑body health in subtle ways.
  • It suggests we could screen health passively—just by recording short, simple movements—without blood tests or expensive equipment.

What could this mean for the future?

  • A new “vital sign”: Gait could become a routine, easy health check—like measuring heart rate or blood pressure—because it touches many body systems at once.
  • Easy to scale: The motions were captured with a single depth camera in a normal clinic setup. In the future, similar models might work with regular video (like a smartphone), making large‑scale, low‑cost health checks possible.
  • Early warning signals: Movement might pick up early hints of problems in metabolism, sleep, mental health, bones, and more—before they’re obvious in tests or symptoms.
  • Smarter movement tests: Because different body parts signal different health areas (legs for metabolism/frailty; torso for sleep/lifestyle), clinics could design short, targeted tasks to focus on what they want to check.

A few caveats to keep in mind

  • The participants were mostly from a specific background (Ashkenazi Jewish adults in Israel). Results need testing in more diverse groups.
  • The data is a snapshot in time, so it shows patterns, not cause and effect.
  • The model learned from one type of camera; using different cameras will need extra work.
  • More studies are needed to confirm how this performs in real‑world medical screening.

Bottom line

This research shows that how you move is like a window into your whole health. An AI trained to “read” 3D stick‑figure motion can predict many health traits, even after accounting for age and body size. Legs tell a lot about metabolism and strength; the torso hints at sleep and lifestyle habits. With simple cameras and short tasks, gait could become a practical, passive vital sign to help spot issues earlier and keep people healthier.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Unresolved knowledge gaps, limitations, and open questions

Below is a consolidated, concrete list of what remains missing, uncertain, or unexplored in the paper, framed to guide future research.

  • External validity across populations: performance on non-Ashkenazi, geographically diverse, and socioeconomically varied cohorts; adults >80 years, adolescents/children, and individuals with disabilities.
  • Generalization to clinical populations: evaluation in patients with prevalent conditions (e.g., Parkinson’s, stroke, COPD, heart failure, musculoskeletal disorders, chronic pain) that were largely excluded from the current healthy cohort.
  • Hardware and pipeline portability: robustness and domain adaptation from Azure Kinect 3D depth to monocular 2D video (smartphones, clinic cameras) and to different pose-estimation pipelines; quantification of performance drops and calibration under domain shift.
  • Free-living and overground validity: replication outside laboratory treadmills for overground walking, variable terrains, stairs, turning, uneven surfaces, and daily-life settings.
  • Recording condition robustness: sensitivity to camera placement and distance, occlusions, lighting, clothing/garments, footwear, walking aids, and multi-person scenes.
  • Longitudinal and causal inference: ability of embeddings to predict incident disease, progression, and functional decline; sensitivity to interventions (exercise programs, medications, weight loss); minimal detectable change and test–retest reliability over months/years.
  • Prospective clinical utility: decision thresholds, calibration, net benefit (decision curve analysis), and cost-effectiveness in real screening or triage workflows; impact on clinical outcomes.
  • External replication: validation on independent cohorts collected by unaffiliated teams with different hardware, tasks, and processing pipelines.
  • Risk of identity leakage in evaluation: confirmation of strictly person-disjoint cross-validation (across all tasks and visits) and quantification of how identity signatures affect predictive performance.
  • Task protocol optimization: identification of the minimal task set and recording duration needed per phenotype; per-phenotype task contributions; interpretability and attribution extended beyond treadmill-only analyses.
  • Finer-grained interpretability: joint- and segment-level, gait-cycle phase, and frequency-domain attribution; validation of masking-based importance against alternative methods (e.g., SHAP, Integrated Gradients, counterfactuals).
  • Mechanistic pathways: mediation analyses to test whether muscle mass, inflammation, autonomic function, or neuropathy mediate associations with liver elasticity, hematopoietic markers, sleep, and mental health.
  • Female renal phenotypes: investigation into why renal function showed no independent gain in females (measurement noise, cohort characteristics, or true absence of signal).
  • Label quality and noise: explicit modeling of uncertainty for questionnaire-derived targets (lifestyle, mental health), including repeated measures, rater variability, and latent-variable approaches to denoise labels.
  • Clinical metrics beyond Pearson r/AUC: reporting of calibration, reclassification (NRI), sensitivity/specificity at clinically relevant cutoffs, and subgroup-specific predictive values.
  • Fairness and bias: subgroup performance stratified by age deciles, BMI strata, height extremes, menopausal status, pregnancy, and intersectional groups; bias mitigation if disparities are found.
  • Confounding beyond age/BMI/VAT/height: assessment and adjustment for comorbidities, pain, sleep deprivation, medications that alter motor function (e.g., sedatives, beta-blockers), and socioeconomic/behavioral confounders.
  • Medication–disease disentanglement: causal modeling to separate disease-related from medication-induced motor signatures; stratified analyses by medication class and dose.
  • Privacy and security: quantification of re-identification risk from embeddings that encode stable identity signatures; development and testing of privacy-preserving representations.
  • Scaling behavior: how performance scales with more subjects, longer sequences, and added task diversity; diminishing returns and optimal data composition for pretraining.
  • Model comparisons and ablations: benchmarking MAE against contrastive/self-distillation/sequence forecasting baselines and supervised models; sensitivity to architecture size, masking ratios, and hyperparameters.
  • Multimodal fusion: added value from integrating inertial sensors, ECG, respiratory signals, or audio; formal synergy and ablation to quantify incremental gains beyond gait alone.
  • Disease-specific utility: evaluation for differential diagnosis (e.g., Parkinson’s vs. essential tremor), staging, and severity scoring; robustness to comorbid conditions.
  • Time-to-event endpoints: survival, hospitalization, falls, and acute exacerbations modeled with embeddings in Cox or deep survival frameworks.
  • Consumer deployment constraints: performance with short, opportunistic recordings on smartphones at home; on-device inference feasibility, bandwidth, and energy requirements.
  • Reproducibility and transparency: availability of code, pretrained models, and standardized benchmarks; harmonized evaluation protocols to enable independent reproduction.
  • Microbiome associations: replication and mechanistic exploration of weak/variable microbiome signals; disentangling common lifestyle or dietary confounders.
  • Measurement coupling: assessment of potential coupling between movement assessments and contemporaneously collected phenotypes (e.g., fatigue, heart rate) that could inflate associations; counterbalancing or temporal separation experiments.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

Below are actionable use cases that can be deployed now, leveraging the study’s findings with currently available hardware (single depth camera), existing clinic workflows, and standard ML tooling.

Healthcare and Clinical Care

  • Gait as a fifth/sixth vital sign kiosk in clinics (primary care, endocrinology, geriatrics)
    • Sector: Healthcare; Software
    • What: Add a short, 5-task gait capture station (as used in the study) to intake. Produce risk scores for age-related decline, frailty, metabolic risk (BMI/VAT proxy), and liver stiffness proxies, plus mental-health flags (depression/insomnia risk).
    • Tools/workflows: Azure-Kinect-class depth camera or equivalent; pose extraction; pretrained embedding encoder; EHR plugin to write scores; sex-stratified calibration.
    • Dependencies/assumptions: Generalization from an Ashkenazi-majority cohort; cross-sectional evidence (screening/triage, not diagnosis); clinic space for a treadmill or standardized tasks; device procurement and maintenance; data privacy consent.
  • Triage for advanced imaging/labs (DXA, liver elastography, lipid panel)
    • Sector: Healthcare; Payers
    • What: Use gait-derived risk to prioritize who gets DXA (bone density), liver elastography, or lab panels (lipids, hematologic markers). Reduces unnecessary testing and accelerates evaluation for high-risk patients.
    • Tools/workflows: Threshold-based decision support; audit trails; quality dashboards showing PPV/NPV versus local standards.
    • Dependencies/assumptions: Local validation thresholds; clinical governance; fairness monitoring across sex/age/ethnicity.
  • Targeted motor assessments informed by anatomical attribution
    • Sector: Healthcare; Rehabilitation; Sleep medicine
    • What: Incorporate trunk-focused tasks (for sleep/lifestyle screening) and lower-limb-focused protocols (for metabolic/frailty) during in-clinic assessments to improve sensitivity for specific domains.
    • Tools/workflows: Protocol updates; brief standing balance/torso sway tasks; EMR-linked templates.
    • Dependencies/assumptions: Staff training; reproducibility of attribution patterns; standardization of task instructions.
  • Mental health and medication side-effect monitoring
    • Sector: Psychiatry; Primary care; Pharmacy
    • What: Use embeddings to flag depressive symptom risk and detect gait changes associated with antidepressants or other meds (e.g., akathisia/parkinsonism signals), prompting clinician review.
    • Tools/workflows: Periodic gait capture; side-effect dashboards; pharmacist-physician communication loops.
    • Dependencies/assumptions: Not diagnostic; requires careful messaging to avoid stigma; clinical follow-up pathways.
  • Diabetes clinic adjunct for neuropathy/fall-risk screening
    • Sector: Endocrinology; Neurology; PT/OT
    • What: Use gait signals to screen for motor consequences of diabetic peripheral neuropathy and to triage fall-risk interventions.
    • Tools/workflows: PT referral triggers; balance training programs; remote follow-up cadence.
    • Dependencies/assumptions: Validation against neuropathy gold standards; fall-risk policy alignment.
  • Rehabilitation progress tracking
    • Sector: Physical therapy; Orthopedics; Cardiac rehab
    • What: Track patient-specific gait embeddings longitudinally to quantify recovery or deconditioning beyond simple speed/cadence.
    • Tools/workflows: Baseline and follow-up captures; trend visualizations; goal setting tied to embedding deltas.
    • Dependencies/assumptions: Stable camera setup; alignment of sessions and tasks; patient adherence.
  • Sleep clinic pre-screening and follow-up
    • Sector: Sleep medicine
    • What: Utilize torso-dominated gait signals to prioritize polysomnography referrals and monitor post-therapy changes (e.g., CPAP adherence effects on postural control).
    • Tools/workflows: Screening score integrated into referral criteria; 3–6 month follow-up captures.
    • Dependencies/assumptions: Moderate effect sizes; local threshold tuning; prevent over-referral.

Software/AI and Products

  • Gait Vital Sign API for clinics and research
    • Sector: Software; Digital health
    • What: Offer an on-prem or cloud SDK that ingests 3D skeleton sequences and returns multi-system risk scores, with sex-stratified models and task-specific embeddings.
    • Tools/workflows: REST API/SDK; model cards; calibration tools; PHI-safe deployment options.
    • Dependencies/assumptions: HIPAA/GDPR compliance; hardware compatibility (depth cameras); MLOps monitoring.
  • Embedding-based patient fingerprinting for re-identification within a clinic
    • Sector: Healthcare IT
    • What: Use stable individual gait signatures to match repeat visits (consent-driven) and detect drifts indicating health changes.
    • Tools/workflows: Hashing/secure linkage; drift detectors; alerting.
    • Dependencies/assumptions: Privacy-by-design; explicit consent; mitigation of re-identification risk outside the care setting.

Employers and Occupational Health

  • Workplace wellness and fatigue screening
    • Sector: Occupational health; HR
    • What: Voluntary, privacy-preserving gait check-ins to flag musculoskeletal strain, deconditioning, or sleep-related impairment (via torso signals).
    • Tools/workflows: Kiosk in onsite clinics; aggregate reporting to employees; individual data ownership.
    • Dependencies/assumptions: Ethical guardrails; non-punitive use; strong consent and data minimization.

Academia and Clinical Research

  • Phenome-wide association studies from movement
    • Sector: Academia
    • What: Use the embedding approach to study multi-system associations, sex differences, and targeted task design in diverse cohorts.
    • Tools/workflows: Open-source pipelines; multi-site protocols; meta-analysis frameworks.
    • Dependencies/assumptions: Access to depth cameras or motion labs; IRB approvals; diverse recruitment.
  • Low-cost surrogate endpoints in trials
    • Sector: Pharma; CROs; Academia
    • What: Include gait embeddings as exploratory biomarkers in metabolic, hepatic, or neuropsychiatric trials to capture functional change.
    • Tools/workflows: Trial SOPs; blinded analysis; pre-registered hypotheses.
    • Dependencies/assumptions: Qualification as exploratory endpoints; site training; device standardization.

Consumer and Daily Life

  • In-gym or clinic-adjacent fitness assessment
    • Sector: Fitness; Wellness
    • What: Quick gait capture to personalize programs emphasizing trunk stability for sleep/energy benefits or lower-limb strength for metabolic/frailty goals.
    • Tools/workflows: Trainer dashboard; program templates; periodic reassessment.
    • Dependencies/assumptions: Clear consumer messaging; avoid medical claims; informed consent.
  • Fall-prevention coaching for older adults (in supervised settings)
    • Sector: Senior care; Community health
    • What: Community centers use gait screenings to recommend balance classes or PT referral.
    • Tools/workflows: Mobile kiosks; referral networks; printouts for caregivers.
    • Dependencies/assumptions: Accessibility; staff training; cultural sensitivity.

Long-Term Applications

These applications require further research, scaling, domain adaptation, or regulatory development before widespread deployment.

Healthcare and Public Health

  • Smartphone/2D camera–based population screening
    • Sector: Healthcare; Public health; Software
    • What: Extract health-relevant embeddings from consumer-grade monocular video for opportunistic screening in clinics, pharmacies, or at home.
    • Tools/workflows: 2D-to-3D pose reconstruction; on-device inference; federated learning.
    • Dependencies/assumptions: Robust domain adaptation from depth to 2D; accuracy under varied lighting/backgrounds; bias audits across populations; privacy and consent frameworks.
  • Longitudinal prediction of disease onset and exacerbations
    • Sector: Healthcare; Payers
    • What: Use trajectory of gait embeddings to forecast incident MASLD/fibrosis progression, anemia, depression relapse, or COPD/asthma exacerbations.
    • Tools/workflows: Risk models with time-series embeddings; integration into care pathways; alert fatigue controls.
    • Dependencies/assumptions: Prospective validation; causal inference; regulatory clearance for clinical decision support.
  • Regulatory-cleared digital biomarker suite
    • Sector: Healthcare; MedTech
    • What: FDA/CE-marked software as a medical device (SaMD) providing validated gait-derived biomarkers for specific indications (e.g., frailty, neuropathy risk, liver disease triage).
    • Tools/workflows: Pivotal studies; locked models; post-market surveillance.
    • Dependencies/assumptions: Clinical utility evidence; safety/effectiveness; cybersecurity.
  • EHR-native “movement phenotyping” layer
    • Sector: Health IT
    • What: Standardize movement data as a structured resource, enabling multi-omic + movement risk models for precision medicine.
    • Tools/workflows: FHIR extensions; terminology mapping; governance.
    • Dependencies/assumptions: Interoperability standards; data stewardship; payer incentives.
  • At-home smart-home monitoring for aging-in-place
    • Sector: Senior tech; IoT
    • What: Passive gait capture from smart TVs or ambient cameras to monitor decline, falls risk, and sleep-related instability.
    • Tools/workflows: Edge processing; privacy-preserving analytics; caregiver portals.
    • Dependencies/assumptions: Strong privacy safeguards; user trust; robust on-device performance.

Software/AI and Platforms

  • Gait foundation model as a service (cross-domain, cross-device)
    • Sector: Software; AI
    • What: Pretrained movement foundation models fine-tuned across devices and tasks, with adapters for clinics, sports, robotics, and AR/VR.
    • Tools/workflows: Domain adaptation (e.g., adapters, synthetic data); evaluation suite; fairness toolkits.
    • Dependencies/assumptions: Access to diverse labeled/unlabeled datasets; licensing; compute.
  • Multi-modal fusion (gait + wearables + voice + text)
    • Sector: Digital health; Research
    • What: Combine gait embeddings with heart rate variability, sleep wearables, and patient-reported outcomes for robust phenotyping.
    • Tools/workflows: Late/early fusion architectures; consent-managed data hubs.
    • Dependencies/assumptions: Data interoperability; missingness handling; privacy.

Robotics and Assistive Technologies

  • Adaptive exoskeletons and prosthetics using health-informed gait embeddings
    • Sector: Robotics; Rehab tech
    • What: Real-time embeddings inform assistance levels personalized to metabolic/frailty states and trunk control.
    • Tools/workflows: Embedded inference; controller adaptation; safety layers.
    • Dependencies/assumptions: Low-latency models; human-in-the-loop validation; regulatory approval for Class II/III devices.
  • Human-robot collaboration tuned to user state
    • Sector: Industrial robotics; Safety
    • What: Robots adjust speed/trajectory based on detected fatigue or instability to reduce accidents.
    • Tools/workflows: On-prem deployment; safety monitoring; compliance with ISO/ANSI standards.
    • Dependencies/assumptions: High-precision, privacy-safe sensing; liability frameworks.

Policy and Ethics

  • Guidelines for gait as a vital sign and reimbursement pathways
    • Sector: Policy; Payers
    • What: Establish clinical practice guidelines and reimbursement codes for gait-based screening and monitoring.
    • Tools/workflows: Health technology assessments; cost-effectiveness models; pilot programs.
    • Dependencies/assumptions: Evidence of outcomes improvement; equity impact assessments.
  • Privacy and bias standards for camera-based health analytics
    • Sector: Policy; Standards bodies
    • What: Create consensus standards for consent, on-device processing, de-identification, and fairness reporting for movement analytics.
    • Tools/workflows: Certification programs; audits; algorithmic impact assessments.
    • Dependencies/assumptions: Multi-stakeholder coordination; enforceability.

Sports, Fitness, and Consumer

  • Personalized training that targets systemic health outcomes
    • Sector: Sports tech; Wellness
    • What: Move beyond “form” toward programs that optimize trunk stability for sleep quality or lower-limb power for metabolic health risk reduction, tracked via embeddings.
    • Tools/workflows: Coach-facing analytics; consumer apps; periodic in-gym scans; AR guidance.
    • Dependencies/assumptions: Non-medical claims; guardrails to prevent over-interpretation; user education.
  • Insurance and financial risk models augmented with movement (with guardrails)
    • Sector: Insurance; Finance
    • What: Use gait-derived functional health indicators for wellness incentives or risk stratification with strict fairness and consent requirements.
    • Tools/workflows: Opt-in programs; third-party audits; transparency reports.
    • Dependencies/assumptions: Regulatory approval; ethical frameworks; avoidance of discrimination.

Notes on cross-cutting assumptions and dependencies:

  • External validity: The cohort is predominantly Ashkenazi Jewish; broad deployment requires validation on diverse populations and environments.
  • Hardware/domain shift: Models were trained on Azure Kinect depth data; translation to other depth sensors or 2D video requires domain adaptation and revalidation.
  • Use as screening/triage: Current evidence is cross-sectional; applications should be positioned as risk estimation or prioritization, not standalone diagnosis.
  • Sex-stratified modeling: Performance and anatomical attributions differ by sex; deploy sex-specific models or calibration.
  • Data protection: Camera-based health analytics demand strong privacy, security, consent, and on-device processing where feasible.
  • Workflow fit: Short, standardized tasks and staff training are essential to reproduce model performance.
  • Regulatory pathway: Clinical decision support and SaMD use cases will require formal validation and regulatory clearance.
View Paper Prompt  View All Prompts 

Glossary

  • ACE inhibitors: Medications that block angiotensin-converting enzyme to lower blood pressure and reduce cardiac workload. "Medications Statins, ACE inhibitors"
  • Adrenergic inhalants: Bronchodilator medications that stimulate adrenergic receptors to relieve airway constriction. "adrenergic inhalants"
  • Alanine transaminase (ALT): A liver enzyme used as a biomarker of hepatocellular injury. "ALT, alanine transaminase"
  • Anatomical ablation: A model interpretability technique that removes (masks) specific body regions to assess their contribution to predictions. "Anatomical ablation revealed that legs dominated metabolic and frailty predictions while torso encoded sleep and lifestyle phenotypes."
  • Anatomical attribution: Analysis assigning importance of predictions to specific anatomical regions or joint groups. "Sex-stratified anatomical attribution for five representative health outcomes per joint group in male (b) and female (c) participants."
  • Apnea-hypopnea index (AHI): A sleep metric counting apneas and hypopneas per hour, used to assess sleep apnea severity. "AHI, apnea-hypopnea index"
  • Aspartate transaminase (AST): A liver enzyme indicating liver or muscle injury. "AST, aspartate transaminase"
  • AUC-ROC: Area under the receiver operating characteristic curve; a measure of classification performance. "Dumbbell plots comparing AUC-ROC for models trained on covariates alone (age, BMI, VAT, height, open circles) versus models combining covariates with gait embeddings (filled circles)"
  • Azure Kinect depth camera: A depth-sensing camera used to capture 3D skeletal pose data. "single Azure Kinect depth camera"
  • Bone mineral content (BMC): The mass of mineral content in bone, reflecting bone strength. "BMC, bone mineral content"
  • Bone mineral density (BMD): A measure of bone mineral concentration, used to assess osteoporosis risk. "BMD, bone mineral density"
  • C-reactive protein (CRP): An inflammatory biomarker measured in blood. "CRP, C-reactive protein"
  • Confounders: Variables that influence both predictors and outcomes, potentially biasing associations if not controlled. "we compared the predictive performance of demographic and anthropometric confounders alone (age, BMI, height 22, VAT) against confounders combined with gait embeddings"
  • Denoising masked autoencoding objective: A self-supervised learning task where a model reconstructs original data from masked and noised inputs. "Model training was performed using a denoising masked autoencoding objective"
  • Domain transfer: Adapting models trained in one data domain or device setup to another. "adapting to other capture systems or public gait datasets 50,51 would require non-trivial domain transfer."
  • Dual-stream spatiotemporal transformer: A transformer architecture modeling spatial and temporal dynamics via two streams. "we employed a representation learning approach based on a dual-stream spatiotemporal transformer architecture"
  • ECG T-wave amplitude: The magnitude of the T-wave in an electrocardiogram, reflecting ventricular repolarization. "ECG T-wave amplitude"
  • Estimated glomerular filtration rate (eGFR): A calculated index of kidney function. "eGFR, estimated glomerular filtration rate"
  • False Discovery Rate (FDR) correction: A multiple-testing procedure controlling the expected rate of false positives. "after False Discovery Rate (FDR) correction."
  • Gait foundation model: A general-purpose model trained on gait data to learn representations transferable across tasks. "We developed a gait foundation model for 3D skeletal motion"
  • Gait Fusion model: An ensemble that integrates predictions across multiple gait activities. "we constructed a Gait Fusion model by averaging predictions derived from each activity-specific embedding."
  • Glycated hemoglobin (HbA1c): A biomarker reflecting average blood glucose over several months. "HbA1c, glycated hemoglobin"
  • Hematopoietic: Pertaining to blood cell formation and related biomarkers. "The largest median improvements were observed in hematopoietic markers"
  • Hemiparesis (post-stroke): Weakness on one side of the body following a stroke. "the asymmetry of post-stroke hemiparesis 7"
  • Hierarchical pooling: An aggregation method that combines features across temporal or structural hierarchies into a compact embedding. "representations from the frozen encoder undergo hierarchical pooling to generate subject-level multi-activity embeddings"
  • Human Phenotype Project (HPP): A longitudinal cohort integrating deep physiological and multi-omics profiling. "We analyzed data from the Human Phenotype Project (HPP), a longitudinal study linking deep physiological profiling with multi-omics data."
  • Irritable bowel syndrome (IBS): A functional gastrointestinal disorder characterized by abdominal discomfort and altered bowel habits. "irritable bowel syndrome (IBS)"
  • LDL-C (low-density lipoprotein cholesterol): A lipid measure associated with cardiovascular risk. "LDL-C, low-density lipoprotein cholesterol"
  • Logistic regression with ridge regularization: A penalized classification method that adds an L2 penalty to reduce overfitting. "All models used logistic regression with ridge regularization"
  • Masked Autoencoder (MAE): A self-supervised model that reconstructs masked parts of input data to learn representations. "we adopted the Masked Autoencoder (MAE) paradigm 15 , extended to skeleton sequences 16"
  • Mean per joint position error: An error metric measuring average distance between predicted and true joint positions in 3D pose reconstruction. "mean per joint position error of 0.008"
  • Metabolic Dysfunction-Associated Steatotic Liver Disease (MASLD): A liver disease characterized by fat accumulation linked to metabolic dysfunction. "Metabolic Dysfunction-Associated Steatotic Liver Disease (MASLD) severity"
  • Metabolomics: The large-scale study of small molecules (metabolites) in biological systems. "including cardiovascular function, serum metabolomics, sleep, bone density, liver health"
  • Monocular 2D video: Single-camera video data without depth, used to infer 3D pose via learning-based methods. "reconstruction of 3D pose from monocular 2D video"
  • Multi-omics: Integration of multiple high-throughput biological data types (e.g., genomics, proteomics, metabolomics). "linking deep physiological profiling with multi-omics data."
  • Nested cross-validation: A model evaluation framework with inner loops for hyperparameter tuning and outer loops for unbiased performance estimation. "evaluated using 5-fold nested cross-validation repeated across 15 random seeds (n = 75 train-test splits)"
  • NMR (Nuclear Magnetic Resonance) profiling (Nightingale): A spectroscopic platform for high-throughput serum metabolite quantification. "Nightingale NMR Fatty acids, Amino acids"
  • Oxylipins: Bioactive oxidized fatty acids involved in inflammation and metabolic signaling. "17 of 42 of tested serum oxylipins correlated with gait speed in older men"
  • Plasmalogen: An ether phospholipid class abundant in brain and myelin; alterations are linked to neurodegeneration. "a plasmalogen-type lysophospholipid enriched in brain and myelin"
  • Pressure-sensitive walkways: Instrumented floors that measure spatiotemporal gait parameters via pressure sensors. "most instrumented gait analysis studies, which rely on multi-camera motion capture systems or pressure-sensitive walkways"
  • Proton pump inhibitors (PPIs): Medications that reduce gastric acid production, used for GERD and ulcers. "including proton pump inhibitors (PPIs), antihistamines, and antidepressants."
  • Respiratory disturbance index (RDI): A sleep metric reflecting abnormal breathing events per hour, including apneas, hypopneas, and related disturbances. "RDI, respiratory disturbance index"
  • Romberg test: A clinical assessment of postural control and balance with eyes closed. "whereas males exhibited larger anteroposterior sway amplitudes during the Romberg test"
  • Sarcopenia: Age- or disease-related skeletal muscle mass and strength loss. "sarcopenia and MASLD share pathophysiological pathways"
  • Self-supervised learning: A learning paradigm where models learn from inherent structure of unlabeled data via proxy tasks. "Self-supervised learning of gait representations has recently shown promise"
  • Sex-stratified analyses: Analytical approach that separates data and modeling by sex to prevent confounding and reveal sex-specific patterns. "Analyses were sex-stratified throughout."
  • UMAP (Uniform Manifold Approximation and Projection): A nonlinear dimensionality reduction technique for visualizing high-dimensional data. "UMAP projection of gait embeddings colored by motor task"
  • VLDL (very-low-density lipoprotein): A class of lipoproteins that transport triglycerides; elevated levels are linked to cardiometabolic risk. "VLDL, very-low-density lipoprotein"
  • Visceral adipose tissue (VAT): Fat stored within the abdominal cavity around internal organs, strongly associated with metabolic risk. "visceral adipose tissue (VAT) area"
View Paper Prompt  View All Prompts 

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 57 likes about this paper.

Sign Up for Free