Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

End-To-End Prediction of Knee Osteoarthritis Progression With Multi-Modal Transformers (2307.00873v1)

Published 3 Jul 2023 in eess.IV and cs.CV

Abstract: Knee Osteoarthritis (KOA) is a highly prevalent chronic musculoskeletal condition with no currently available treatment. The manifestation of KOA is heterogeneous and prediction of its progression is challenging. Current literature suggests that the use of multi-modal data and advanced modeling methods, such as the ones based on Deep Learning, has promise in tackling this challenge. To date, however, the evidence on the efficacy of this approach is limited. In this study, we leveraged recent advances in Deep Learning and, using a Transformer approach, developed a unified framework for the multi-modal fusion of knee imaging data. Subsequently, we analyzed its performance across a range of scenarios by investigating multiple progression horizons -- from short-term to long-term. We report our findings using a large cohort (n=2421-3967) derived from the Osteoarthritis Initiative dataset. We show that structural knee MRI allows identifying radiographic KOA progressors on par with multi-modal fusion approaches, achieving an area under the ROC curve (ROC AUC) of 0.70-0.76 and Average Precision (AP) of 0.15-0.54 in 2-8 year horizons. Progression within 1 year was better predicted with a multi-modal method using X-ray, structural, and compositional MR images -- ROC AUC of 0.76(0.04), AP of 0.13(0.04) -- or via clinical data. Our follow-up analysis generally shows that prediction from the imaging data is more accurate for post-traumatic subjects, and we further investigate which subject subgroups may benefit the most. The present study provides novel insights into multi-modal imaging of KOA and brings a unified data-driven framework for studying its progression in an end-to-end manner, providing new tools for the design of more efficient clinical trials. The source code of our framework and the pre-trained models are made publicly available.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (76)
  1. Cui, A. et al. Global, regional prevalence, incidence and risk factors of knee osteoarthritis in population-based studies. \JournalTitleEClinicalMedicine 29, 100587 (2020).
  2. Neogi, T. et al. Association between radiographic features of knee osteoarthritis and pain: results from two cohort studies. \JournalTitleBmj 339 (2009).
  3. Deveza, L. A. et al. Knee osteoarthritis phenotypes and their relevance for outcomes: a systematic review. \JournalTitleOsteoarthritis and Cartilage 25, 1926–1941 (2017).
  4. Mobasheri, A. et al. Recent advances in understanding the phenotypes of osteoarthritis. \JournalTitleF1000Research 8 (2019).
  5. Roemer, F. W. et al. Structural phenotypes of knee osteoarthritis: potential clinical and research relevance. \JournalTitleSkeletal Radiology 1–10 (2022).
  6. Hunter, D. et al. Change in joint space width: hyaline articular cartilage loss or alteration in meniscus? \JournalTitleArthritis & Rheumatism: Official Journal of the American College of Rheumatology 54, 2488–2495 (2006).
  7. Radiological assessment of rheumatoid arthritis. \JournalTitleAnnals of the rheumatic diseases 16, 485 (1957).
  8. Eckstein, F. et al. Cartilage thickness change as an imaging biomarker of knee osteoarthritis progression–data from the FNIH OA Biomarkers Consortium. \JournalTitleArthritis & rheumatology (Hoboken, NJ) 67, 3184 (2015).
  9. Ajuied, A. et al. Anterior cruciate ligament injury and radiologic progression of knee osteoarthritis: a systematic review and meta-analysis. \JournalTitleThe American journal of sports medicine 42, 2242–2252 (2014).
  10. The role of thigh muscle and adipose tissue in knee osteoarthritis progression in women: data from the Osteoarthritis Initiative. \JournalTitleOsteoarthritis and Cartilage 26, 1190–1195 (2018).
  11. Wang, Y. et al. Knee effusion volume assessed by magnetic resonance imaging and progression of knee osteoarthritis: data from the Osteoarthritis Initiative. \JournalTitleRheumatology 58, 246–253 (2019).
  12. Wang, K. et al. Quantitative signal intensity alteration in infrapatellar fat pad predicts incident radiographic osteoarthritis: the Osteoarthritis Initiative. \JournalTitleArthritis care & research 71, 30–38 (2019).
  13. Hunter, D. J. et al. Evolution of semi-quantitative whole joint assessment of knee OA: MOAKS (MRI Osteoarthritis Knee Score). \JournalTitleOsteoarthritis and Cartilage 19, 990–1002 (2011).
  14. Namiri, N. K. et al. Deep learning for large scale MRI-based morphological phenotyping of osteoarthritis. \JournalTitleScientific reports 11, 1–10 (2021).
  15. Roemer, F. W. et al. Structural tissue damage and 24-month progression of semi-quantitative MRI biomarkers of knee osteoarthritis in the IMI-APPROACH cohort. \JournalTitleBMC Musculoskeletal Disorders 23, 1–20 (2022).
  16. T1ρ𝜌\rhoitalic_ρ and T2 relaxation times predict progression of knee osteoarthritis. \JournalTitleOsteoarthritis and Cartilage 21, 69–76 (2013).
  17. Hunter, D. et al. OARSI clinical trials recommendations: knee imaging in clinical trials in osteoarthritis. \JournalTitleOsteoarthritis and Cartilage 23, 698–715 (2015).
  18. Emanuel, K. S. et al. The relation between the biochemical composition of knee articular cartilage and quantitative MRI: a systematic review and meta-analysis. \JournalTitleOsteoarthritis and Cartilage (2021).
  19. Zarins, Z. et al. Cartilage and meniscus assessment using T1rho and T2 measurements in healthy subjects and patients with osteoarthritis. \JournalTitleOsteoarthritis and Cartilage 18, 1408–1416 (2010).
  20. Eck, B. L. et al. Quantitative MRI for evaluation of musculoskeletal disease: cartilage and muscle composition, joint inflammation, and biomechanics in osteoarthritis. \JournalTitleInvestigative Radiology 58, 60–75 (2023).
  21. Hilbert, T. et al. Accelerated T2 mapping combining parallel mri and model-based reconstruction: GRAPPATINI. \JournalTitleJournal of Magnetic Resonance Imaging 48, 359–368 (2018).
  22. Chalian, M. et al. The QIBA profile for MRI-based compositional imaging of knee cartilage. \JournalTitleRadiology 301, 423–432 (2021).
  23. Razmjoo, A. et al. T2 analysis of the entire osteoarthritis initiative dataset. \JournalTitleJournal of Orthopaedic Research 39, 74–85 (2021).
  24. Eckstein, F. et al. Proposal for a nomenclature for magnetic resonance imaging based measures of articular cartilage in osteoarthritis. \JournalTitleOsteoarthritis and Cartilage 14, 974–983 (2006).
  25. Neogi, T. et al. Magnetic resonance imaging–based three-dimensional bone shape of the knee predicts onset of knee osteoarthritis: data from the Osteoarthritis Initiative. \JournalTitleArthritis & Rheumatism 65, 2048–2058 (2013).
  26. Li, J. et al. MRI-based texture analysis of infrapatellar fat pad to predict knee osteoarthritis incidence. \JournalTitleRadiology 304, 611–621 (2022).
  27. Iriondo, C. et al. Towards understanding mechanistic subgroups of osteoarthritis: 8-year cartilage thickness trajectory analysis. \JournalTitleJournal of Orthopaedic Research 39, 1305–1317 (2021).
  28. Litjens, G. et al. A survey on deep learning in medical image analysis. \JournalTitleMedical image analysis 42, 60–88 (2017).
  29. Deep learning for healthcare: review, opportunities and challenges. \JournalTitleBriefings in bioinformatics 19, 1236–1246 (2018).
  30. Model-based and data-driven strategies in medical image computing. \JournalTitleProceedings of the IEEE 108, 110–124 (2019).
  31. One map does not fit all: evaluating saliency map explanation on multi-modal medical images. \JournalTitlearXiv preprint arXiv:2107.05047 (2021).
  32. Which explanation should i choose? a function approximation perspective to characterizing post hoc explanations. \JournalTitlearXiv preprint arXiv:2206.01254 (2022).
  33. Hirvasniemi, J. et al. The KNee OsteoArthritis Prediction (KNOAP2020) Challenge: an image analysis challenge to predict incident symptomatic radiographic knee osteoarthritis from MRI and X-ray images. \JournalTitleOsteoarthritis and Cartilage (2022).
  34. Tiulpin, A. et al. Multimodal machine learning-based knee osteoarthritis progression prediction from plain radiographs and clinical data. \JournalTitleScientific reports 9, 1–11 (2019).
  35. Total knee replacement prediction using structural MRIs and 3D convolutional neural networks. In International Conference on Medical Imaging with Deep Learning–Extended Abstract Track (2019).
  36. Deep learning predicts total knee replacement from magnetic resonance images. \JournalTitleScientific reports 10, 1–12 (2020).
  37. Predicting knee osteoarthritis progression from structural MRI using deep learning. In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), 1–5 (IEEE, 2022).
  38. Transfer learning on fused multiparametric MR images for classifying histopathological subtypes of rhabdomyosarcoma. \JournalTitleComputerized Medical Imaging and Graphics 65, 167–175 (2018).
  39. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. \JournalTitleNPJ digital medicine 3, 136 (2020).
  40. A deep learning methodology for improved breast cancer diagnosis using multiparametric MRI. \JournalTitleScientific reports 10, 10536 (2020).
  41. Tsai, Y.-H. H. et al. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting, vol. 2019, 6558 (NIH Public Access, 2019).
  42. Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. \JournalTitlearXiv preprint arXiv:2010.11929 (2020).
  43. Multi-view analysis of unregistered medical images using cross-view transformers. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part III 24, 104–113 (Springer, 2021).
  44. CLIMAT: clinically-inspired multi-agent Transformers for knee osteoarthritis trajectory forecasting. In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), 1–5 (IEEE, 2022).
  45. d’Ascoli, S. et al. ConViT: improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning, 2286–2296 (PMLR, 2021).
  46. Medical transformer: universal brain encoder for 3D MRI analysis. \JournalTitlearXiv preprint arXiv:2104.13633 (2021).
  47. The incident tibiofemoral osteoarthritis with rapid progression phenotype: development and validation of a prognostic prediction rule. \JournalTitleOsteoarthritis and Cartilage 24, 2100–2107 (2016).
  48. Guan, B. et al. Deep learning approach to predict pain progression in knee osteoarthritis. \JournalTitleSkeletal radiology 1–11 (2022).
  49. Validation study of WOMAC: a health status instrument for measuring clinically important patient relevant outcomes to antirheumatic drug therapy in patients with osteoarthritis of the hip or knee. \JournalTitleJournal of rheumatology (1988).
  50. Anderson, D. D. et al. Post-traumatic osteoarthritis: improved understanding and opportunities for early intervention. \JournalTitleJournal of orthopaedic research 29, 802–809 (2011).
  51. Master your metrics with calibration. In Advances in Intelligent Data Analysis XVIII: 18th International Symposium on Intelligent Data Analysis, IDA 2020, Konstanz, Germany, April 27–29, 2020, Proceedings 18, 457–469 (Springer, 2020).
  52. Hänninen, N. E. et al. Orientation anisotropy of quantitative MRI parameters in degenerated human articular cartilage. \JournalTitleJournal of Orthopaedic Research 39, 861–870 (2021).
  53. T2 orientation anisotropy mapping of articular cartilage using qMRI. \JournalTitlePhysics in Medicine & Biology 68, 085004 (2023).
  54. Ma, D. et al. Magnetic resonance fingerprinting. \JournalTitleNature 495, 187–192 (2013).
  55. Runhaar, J. et al. How to define subregional osteoarthritis progression using semi-quantitative MRI osteoarthritis knee score (MOAKS). \JournalTitleOsteoarthritis and Cartilage 22, 1533–1536 (2014).
  56. Lee, J. et al. A personalized risk model leverages MRI-based structural phenotypes and clinical factors to predict incidence of radiographic osteoarthritis. \JournalTitleArthritis Care & Research (2022).
  57. Lee, L. S. et al. Artificial intelligence in diagnosis of knee osteoarthritis and prediction of arthroplasty outcomes: a review. \JournalTitleArthroplasty 4, 1–9 (2022).
  58. Modeling and predicting osteoarthritis progression: data from the Osteoarthritis Initiative. \JournalTitleOsteoarthritis and Cartilage 26, 1643–1650 (2018).
  59. An ensemble clinical and MR-image deep learning model predicts 8-year knee pain trajectory: data from the Osteoarthritis Initiative. \JournalTitleOsteoarthritis Imaging 1, 100003 (2021).
  60. Surface spherical encoding and contrastive learning for virtual bone shape aging. \JournalTitleMedical Image Analysis 77, 102388 (2022).
  61. Chu, X. et al. Twins: revisiting the design of spatial attention in vision transformers. \JournalTitleAdvances in Neural Information Processing Systems 34, 9355–9366 (2021).
  62. Arnab, A. et al. ViViT: a video vision transformer. \JournalTitlearXiv preprint arXiv:2103.15691 (2021).
  63. Adapting pretrained vision-language foundational models to medical imaging domains. \JournalTitlearXiv preprint arXiv:2210.04133 (2022).
  64. Moor, M. et al. Foundation models for generalist medical artificial intelligence. \JournalTitleNature 616, 259–265 (2023).
  65. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. In International Conference on Machine Learning, 24043–24055 (PMLR, 2022).
  66. Guidelines and evaluation of clinical explainable AI in medical image analysis. \JournalTitleMedical Image Analysis 84, 102684 (2023).
  67. KNEEL: knee anatomical landmark localization using hourglass networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 0–0 (2019).
  68. A comparative study at 3T of sequence dependence of T2 quantitation in the knee. \JournalTitleMagnetic resonance imaging 26, 1215–1220 (2008).
  69. Pedregosa, F. et al. Scikit-learn: machine learning in Python. \JournalTitleJournal of Machine Learning Research 12, 2825–2830 (2011).
  70. Paszke, A. et al. PyTorch: an imperative style, high-performance deep learning library. \JournalTitleAdvances in neural information processing systems 32 (2019).
  71. Adam: a method for stochastic optimization. \JournalTitlearXiv preprint arXiv:1412.6980 (2014).
  72. Virtanen, P. et al. SciPy 1.0: fundamental algorithms for scientific computing in Python. \JournalTitleNature Methods 17, 261–272, DOI: 10.1038/s41592-019-0686-2 (2020).
  73. The use and reporting of WOMAC in the assessment of the benefit of physical therapies for the pain of osteoarthritis of the knee: findings from a systematic review of clinical trials. \JournalTitleRheumatology 51, 1440–1446 (2012).
  74. Shirley Ryan Ability Lab. Western Ontario and McMaster Universities Osteoarthritis Index (2016).
  75. Comparison of reliability and responsiveness of patient-reported clinical outcome measures in knee osteoarthritis rehabilitation. \JournalTitlejournal of orthopaedic & sports physical therapy 42, 716–723 (2012).
  76. Kokhlikyan, N. et al. Captum: a unified and generic model interpretability library for PyTorch. \JournalTitlearXiv preprint arXiv:2009.07896 (2020).

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now