Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

STREAMLINE: An Automated Machine Learning Pipeline for Biomedicine Applied to Examine the Utility of Photography-Based Phenotypes for OSA Prediction Across International Sleep Centers (2312.05461v1)

Published 9 Dec 2023 in cs.LG and cs.AI

Abstract: While ML includes a valuable array of tools for analyzing biomedical data, significant time and expertise is required to assemble effective, rigorous, and unbiased pipelines. Automated ML (AutoML) tools seek to facilitate ML application by automating a subset of analysis pipeline elements. In this study we develop and validate a Simple, Transparent, End-to-end Automated Machine Learning Pipeline (STREAMLINE) and apply it to investigate the added utility of photography-based phenotypes for predicting obstructive sleep apnea (OSA); a common and underdiagnosed condition associated with a variety of health, economic, and safety consequences. STREAMLINE is designed to tackle biomedical binary classification tasks while adhering to best practices and accommodating complexity, scalability, reproducibility, customization, and model interpretation. Benchmarking analyses validated the efficacy of STREAMLINE across data simulations with increasingly complex patterns of association. Then we applied STREAMLINE to evaluate the utility of demographics (DEM), self-reported comorbidities (DX), symptoms (SYM), and photography-based craniofacial (CF) and intraoral (IO) anatomy measures in predicting any OSA or moderate/severe OSA using 3,111 participants from Sleep Apnea Global Interdisciplinary Consortium (SAGIC). OSA analyses identified a significant increase in ROC-AUC when adding CF to DEM+DX+SYM to predict moderate/severe OSA. A consistent but non-significant increase in PRC-AUC was observed with the addition of each subsequent feature set to predict any OSA, with CF and IO yielding minimal improvements. Application of STREAMLINE to OSA data suggests that CF features provide additional value in predicting moderate/severe OSA, but neither CF nor IO features meaningfully improved the prediction of any OSA beyond established demographics, comorbidity and symptom characteristics.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (96)
  1. An Introduction to Machine Learning Approaches for Biomedical Research. Frontiers in Medicine, 8, 2021. ISSN 2296-858X. URL https://www.frontiersin.org/articles/10.3389/fmed.2021.771607.
  2. An Introduction to Machine Learning. Clinical Pharmacology & Therapeutics, 107(4):871–885, 2020. ISSN 1532-6535. doi: 10.1002/cpt.1796. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/cpt.1796. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/cpt.1796.
  3. Machine Learning and Integrative Analysis of Biomedical Big Data. Genes, 10(2):87, February 2019. ISSN 2073-4425. doi: 10.3390/genes10020087. URL https://www.mdpi.com/2073-4425/10/2/87. Number: 2 Publisher: Multidisciplinary Digital Publishing Institute.
  4. Technical Challenges for Big Data in Biomedicine and Health: Data Sources, Infrastructure, and Analytics. Yearbook of Medical Informatics, 23(1):42–47, 2014. ISSN 0943-4747, 2364-0502. doi: 10.15265/IY-2014-0018. URL http://www.thieme-connect.de/DOI/DOI?10.15265/IY-2014-0018. Publisher: Georg Thieme Verlag KG.
  5. Navigating the pitfalls of applying machine learning in genomics. Nature Reviews Genetics, 23(3):169–181, March 2022. ISSN 1471-0064. doi: 10.1038/s41576-021-00434-9. URL https://www.nature.com/articles/s41576-021-00434-9. Number: 3 Publisher: Nature Publishing Group.
  6. scikit-learn : Machine Learning Simplified: Implement scikit-learn into every step of the data science pipeline. Packt Publishing Ltd, November 2017. ISBN 978-1-78883-152-9. Google-Books-ID: sEFPDwAAQBAJ.
  7. A guide to machine learning for biologists. Nature Reviews Molecular Cell Biology, 23(1):40–55, January 2022. ISSN 1471-0080. doi: 10.1038/s41580-021-00407-0. URL https://www.nature.com/articles/s41580-021-00407-0. Number: 1 Publisher: Nature Publishing Group.
  8. Davide Chicco. Ten quick tips for machine learning in computational biology. BioData Mining, 10(1):35, December 2017. ISSN 1756-0381. doi: 10.1186/s13040-017-0155-3. URL https://doi.org/10.1186/s13040-017-0155-3.
  9. Mohamed L. Seghier. Ten simple rules for reporting machine learning methods implementation and evaluation on biomedical data. International Journal of Imaging Systems and Technology, 32(1):5–11, 2022. ISSN 1098-1098. doi: 10.1002/ima.22674. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/ima.22674. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/ima.22674.
  10. Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research: A Multidisciplinary View. Journal of Medical Internet Research, 18(12):e5870, December 2016. doi: 10.2196/jmir.5870. URL https://www.jmir.org/2016/12/e323. Company: Journal of Medical Internet Research Distributor: Journal of Medical Internet Research Institution: Journal of Medical Internet Research Label: Journal of Medical Internet Research Publisher: JMIR Publications Inc., Toronto, Canada.
  11. Reproducibility standards for machine learning in the life sciences. Nature Methods, 18(10):1132–1135, October 2021. ISSN 1548-7105. doi: 10.1038/s41592-021-01256-7. URL https://www.nature.com/articles/s41592-021-01256-7. Number: 10 Publisher: Nature Publishing Group.
  12. Patrick Riley. Three pitfalls to avoid in machine learning. Nature, 572(7767):27–29, August 2019. doi: 10.1038/d41586-019-02307-y. URL https://www.nature.com/articles/d41586-019-02307-y. Bandiera_abtest: a Cg_type: Comment Number: 7767 Publisher: Nature Publishing Group Subject_term: Mathematics and computing, Software, Research data.
  13. Pitfalls of supervised feature selection. Bioinformatics, 26(3):440–443, February 2010. ISSN 1367-4803. doi: 10.1093/bioinformatics/btp621. URL https://doi.org/10.1093/bioinformatics/btp621.
  14. The Effect of Training and Testing Process on Machine Learning in Biomedical Datasets. Mathematical Problems in Engineering, 2020:e2836236, May 2020. ISSN 1024-123X. doi: 10.1155/2020/2836236. URL https://www.hindawi.com/journals/mpe/2020/2836236/. Publisher: Hindawi.
  15. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1):67–82, April 1997. ISSN 1941-0026. doi: 10.1109/4235.585893. Conference Name: IEEE Transactions on Evolutionary Computation.
  16. Hugo Jair Escalante. Automated Machine Learning—A Brief Review at the End of the Early Years. In Nelishia Pillay and Rong Qu, editors, Automated Design of Machine Learning and Search Algorithms, Natural Computing Series, pages 11–28. Springer International Publishing, Cham, 2021. ISBN 978-3-030-72069-8. doi: 10.1007/978-3-030-72069-8_2. URL https://doi.org/10.1007/978-3-030-72069-8_2.
  17. Automated machine learning: Review of the state-of-the-art and opportunities for healthcare. Artificial Intelligence in Medicine, 104:101822, April 2020. ISSN 0933-3657. doi: 10.1016/j.artmed.2020.101822. URL https://www.sciencedirect.com/science/article/pii/S0933365719310437.
  18. AutoKeras: An AutoML Library for Deep Learning. Journal of Machine Learning Research, 24(6):1–6, 2023. ISSN 1533-7928. URL http://jmlr.org/papers/v24/20-1355.html.
  19. Automated evolutionary approach for the design of composite machine learning pipelines. Future Generation Computer Systems, 127:109–125, February 2022. ISSN 0167-739X. doi: 10.1016/j.future.2021.08.022. URL https://www.sciencedirect.com/science/article/pii/S0167739X21003307.
  20. AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data, March 2020. URL http://arxiv.org/abs/2003.06505. arXiv:2003.06505 [cs, stat].
  21. Ludwig: a type-based declarative deep learning toolbox, September 2019. URL http://arxiv.org/abs/1909.07930. arXiv:1909.07930 [cs, stat].
  22. Efficient and Robust Automated Machine Learning. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper_files/paper/2015/hash/11d0e6287202fced83f79975ec59a3a6-Abstract.html.
  23. Welcome to PyCaret, September 2023a. URL https://github.com/pycaret/pycaret. original-date: 2019-11-23T18:40:48Z.
  24. Evaluating recommender systems for AI-driven biomedical informatics. Bioinformatics, 37(2):250–256, April 2021. ISSN 1367-4803. doi: 10.1093/bioinformatics/btaa698. URL https://doi.org/10.1093/bioinformatics/btaa698.
  25. Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL, April 2021. URL http://arxiv.org/abs/2006.13799. arXiv:2006.13799 [cs, stat].
  26. Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, GECCO ’16, pages 485–492, New York, NY, USA, July 2016. Association for Computing Machinery. ISBN 978-1-4503-4206-3. doi: 10.1145/2908812.2908918. URL https://dl.acm.org/doi/10.1145/2908812.2908918.
  27. TPOT: A Tree-based Pipeline Optimization Tool for Automating Machine Learning. In Proceedings of the Workshop on Automatic Machine Learning, pages 66–74. PMLR, December 2016. URL https://proceedings.mlr.press/v64/olson_tpot_2016.html. ISSN: 1938-7228.
  28. Scaling tree-based automated machine learning to biomedical big data with a feature set selector. Bioinformatics, 36(1):250–256, January 2020. ISSN 1367-4803. doi: 10.1093/bioinformatics/btz470. URL https://doi.org/10.1093/bioinformatics/btz470.
  29. GAMA: A General Automated Machine Learning Assistant. In Yuxiao Dong, Georgiana Ifrim, Dunja Mladenić, Craig Saunders, and Sofie Van Hoecke, editors, Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track, Lecture Notes in Computer Science, pages 560–564, Cham, 2021. Springer International Publishing. ISBN 978-3-030-67670-4. doi: 10.1007/978-3-030-67670-4_39.
  30. RECIPE: A Grammar-Based Framework for Automatically Evolving Classification Pipelines. In James McDermott, Mauro Castelli, Lukas Sekanina, Evert Haasdijk, and Pablo García-Sánchez, editors, Genetic Programming, Lecture Notes in Computer Science, pages 246–261, Cham, 2017. Springer International Publishing. ISBN 978-3-319-55696-3. doi: 10.1007/978-3-319-55696-3_16.
  31. ML-Plan: Automated machine learning via hierarchical planning. Machine Learning, 107(8):1495–1515, September 2018. ISSN 1573-0565. doi: 10.1007/s10994-018-5735-z. URL https://doi.org/10.1007/s10994-018-5735-z.
  32. Machine Learning Made Easy (MLme): A Comprehensive Toolkit for Machine Learning-Driven Data Analysis, July 2023. URL https://www.biorxiv.org/content/10.1101/2023.07.04.546825v1. Pages: 2023.07.04.546825 Section: New Results.
  33. MLJAR Automated Machine Learning for Humans, September 2023b. URL https://github.com/mljar/mljar-supervised. original-date: 2018-11-05T12:58:04Z.
  34. H2O AutoML: Scalable automatic machine learning. 7th ICML Workshop on Automated Machine Learning (AutoML), July 2020. URL https://www.automl.org/wp-content/uploads/2020/07/AutoML_2020_paper_61.pdf.
  35. STREAMLINE: A Simple, Transparent, End-To-End Automated Machine Learning Pipeline Facilitating Data Analysis and Algorithm Comparison. In Leonardo Trujillo, Stephan M. Winkler, Sara Silva, and Wolfgang Banzhaf, editors, Genetic Programming Theory and Practice XIX, Genetic and Evolutionary Computation, pages 201–231. Springer Nature, Singapore, 2023. ISBN 978-981-19846-0-0. doi: 10.1007/978-981-19-8460-0_9. URL https://doi.org/10.1007/978-981-19-8460-0_9.
  36. Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA. Journal of Machine Learning Research, 18(25):1–5, 2017. ISSN 1533-7928. URL http://jmlr.org/papers/v18/16-261.html.
  37. LightAutoML: AutoML Solution for a Large Financial Services Ecosystem, September 2021. URL https://arxiv.org/abs/2109.01528v2.
  38. FLAML: A Fast and Lightweight AutoML Library, 2021. URL https://github.com/microsoft/FLAML. original-date: 2020-08-20T20:46:11Z.
  39. Automated Machine Learning: Methods, Systems, Challenges. The Springer Series on Challenges in Machine Learning. Springer International Publishing, Cham, 2019. ISBN 978-3-030-05317-8 978-3-030-05318-5. doi: 10.1007/978-3-030-05318-5. URL http://link.springer.com/10.1007/978-3-030-05318-5.
  40. TransmogrifAI, September 2023c. URL https://github.com/salesforce/TransmogrifAI. original-date: 2017-11-02T16:15:15Z.
  41. MLBox: Machine learning box for asymptotic scheduling. Information Sciences, 433-434:401–416, April 2018. ISSN 0020-0255. doi: 10.1016/j.ins.2017.01.005. URL https://www.sciencedirect.com/science/article/pii/S0020025517300099.
  42. Reiichiro Nakano. Xcessiv, August 2023. URL https://github.com/reiinakano/xcessiv. original-date: 2017-03-07T18:18:25Z.
  43. Exploring Automated Machine Learning for Cognitive Outcome Prediction from Multimodal Brain Imaging using STREAMLINE. AMIA Summits on Translational Science Proceedings, 2023:544–553, June 2023. ISSN 2153-4063. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10283099/.
  44. Comparing Amyloid Imaging Normalization Strategies for Alzheimer’s Disease Classification using an Automated Machine Learning Pipeline. AMIA Summits on Translational Science Proceedings, 2023:525–533, June 2023. ISSN 2153-4063. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10283108/.
  45. Toward Predicting 30-Day Readmission Among Oncology Patients: Identifying Timely and Actionable Risk Factors. JCO Clinical Cancer Informatics, (7):e2200097, September 2023. doi: 10.1200/CCI.22.00097. URL https://ascopubs.org/doi/abs/10.1200/CCI.22.00097. Publisher: Wolters Kluwer.
  46. A Data-Driven Analysis of Ward Capacity Strain Metrics That Predict Clinical Outcomes Among Survivors of Acute Respiratory Failure. Journal of Medical Systems, 47(1):83, August 2023. ISSN 1573-689X. doi: 10.1007/s10916-023-01978-5. URL https://doi.org/10.1007/s10916-023-01978-5.
  47. A Rigorous Machine Learning Analysis Pipeline for Biomedical Binary Classification: Application in Pancreatic Cancer Nested Case-control Studies with Implications for Bias Assessments, September 2020. URL http://arxiv.org/abs/2008.12829. arXiv:2008.12829 [cs, stat].
  48. A survey on missing data in machine learning. Journal of Big Data, 8(1):140, October 2021. ISSN 2196-1115. doi: 10.1186/s40537-021-00516-9. URL https://doi.org/10.1186/s40537-021-00516-9.
  49. Benchmarking relief-based feature selection methods for bioinformatics data mining. Journal of Biomedical Informatics, 85:168–188, September 2018. ISSN 1532-0464. doi: 10.1016/j.jbi.2018.07.015. URL https://www.sciencedirect.com/science/article/pii/S1532046418301412.
  50. Collective feature selection to identify crucial epistatic variants. BioData Mining, 11(1):5, April 2018. ISSN 1756-0381. doi: 10.1186/s13040-018-0168-6. URL https://doi.org/10.1186/s13040-018-0168-6.
  51. ExSTraCS 2.0: description and evaluation of a scalable learning classifier system. Evolutionary Intelligence, 8(2):89–116, September 2015. ISSN 1864-5917. doi: 10.1007/s12065-015-0128-8. URL https://doi.org/10.1007/s12065-015-0128-8.
  52. Machine Learning for Detecting Gene-Gene Interactions. Applied Bioinformatics, 5(2):77–88, June 2006. ISSN 1175-5636. doi: 10.2165/00822942-200605020-00002. URL https://doi.org/10.2165/00822942-200605020-00002.
  53. Genetic heterogeneity: Challenges, impacts, and methods through an associative lens. Genetic Epidemiology, 46(8):555–571, 2022. ISSN 1098-2272. doi: 10.1002/gepi.22497. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/gepi.22497. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/gepi.22497.
  54. Agreement in the scoring of respiratory events and sleep among international sleep centers. Sleep, 36(4):591–596, April 2013. ISSN 1550-9109. doi: 10.5665/sleep.2552.
  55. Agreement in the Scoring of Respiratory Events Among International Sleep Centers for Home Sleep Testing. Journal of clinical sleep medicine: JCSM: official publication of the American Academy of Sleep Medicine, 12(1):71–77, January 2016. ISSN 1550-9397. doi: 10.5664/jcsm.5398.
  56. Recognizable clinical subtypes of obstructive sleep apnea across international sleep centers: a cluster analysis. Sleep, 41(3):zsx214, March 2018. ISSN 1550-9109. doi: 10.1093/sleep/zsx214.
  57. Defining Extreme Phenotypes of OSA Across International Sleep Centers. Chest, 158(3):1187–1197, September 2020. ISSN 1931-3543. doi: 10.1016/j.chest.2020.03.055.
  58. Facial and Intraoral Photographic Traits Related to Sleep Apnea in a Clinical Sample with Genetic Ancestry Analysis. Annals of the American Thoracic Society, 20(6):880–890, June 2023. ISSN 2325-6621. doi: 10.1513/AnnalsATS.202207-577OC.
  59. Global burden of sleep-disordered breathing and its implications. Respirology, 25(7):690–702, 2020. ISSN 1440-1843. doi: 10.1111/resp.13838. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/resp.13838. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/resp.13838.
  60. Diagnosis and Management of Obstructive Sleep Apnea: A Review. JAMA, 323(14):1389–1400, April 2020. ISSN 0098-7484. doi: 10.1001/jama.2020.3514. URL https://doi.org/10.1001/jama.2020.3514.
  61. Day-night pattern of sudden death in obstructive sleep apnea. The New England Journal of Medicine, 352(12):1206–1214, March 2005. ISSN 1533-4406. doi: 10.1056/NEJMoa041832.
  62. Obstructive sleep apnea as a risk factor for stroke and death. The New England Journal of Medicine, 353(19):2034–2041, November 2005. ISSN 1533-4406. doi: 10.1056/NEJMoa043104.
  63. The economic impact of obstructive sleep apnea. Lung, 186(1):7–12, 2008. ISSN 0341-2040. doi: 10.1007/s00408-007-9055-5.
  64. The impact of obstructive sleep apnea and daytime sleepiness on work limitation. Sleep Medicine, 9(1):42–53, December 2007. ISSN 1389-9457. doi: 10.1016/j.sleep.2007.01.009.
  65. C. F. George. Reduction in motor vehicle collisions following treatment of sleep apnoea with nasal CPAP. Thorax, 56(7):508–512, July 2001. ISSN 0040-6376. doi: 10.1136/thorax.56.7.508.
  66. Obstructive sleep apnoea in the general population: highly prevalent but minimal symptoms. The European Respiratory Journal, 47(1):194–202, January 2016. ISSN 1399-3003. doi: 10.1183/13993003.01148-2015.
  67. Assessing the Adequacy of Obstructive Sleep Apnea Diagnosis for High-Risk Patients in Primary Care. Journal of the American Board of Family Medicine: JABFM, 35(2):320–328, 2022. ISSN 1558-7118. doi: 10.3122/jabfm.2022.02.210296.
  68. Prevalence of symptoms and risk of sleep apnea in primary care. Chest, 124(4):1406–1414, October 2003. ISSN 0012-3692. doi: 10.1378/chest.124.4.1406.
  69. The medical cost of undiagnosed sleep apnea. Sleep, 22(6):749–755, September 1999. ISSN 0161-8105. doi: 10.1093/sleep/22.6.749.
  70. A survey screen for prediction of apnea. Sleep, 18(3):158–166, April 1995. ISSN 0161-8105. doi: 10.1093/sleep/18.3.158.
  71. Digital Morphometrics: A New Upper Airway Phenotyping Paradigm in OSA. Chest, 152(2):330–342, August 2017. ISSN 1931-3543. doi: 10.1016/j.chest.2017.05.005.
  72. Facial phenotyping by quantitative photography reflects craniofacial morphology measured on magnetic resonance imaging in Icelandic sleep apnea patients. Sleep, 37(5):959–968, May 2014. ISSN 1550-9109. doi: 10.5665/sleep.3670.
  73. Craniofacial phenotyping for prediction of obstructive sleep apnoea in a Chinese population. Respirology (Carlton, Vic.), 21(6):1118–1125, August 2016. ISSN 1440-1843. doi: 10.1111/resp.12792.
  74. Craniofacial phenotyping in obstructive sleep apnea–a novel quantitative photographic approach. Sleep, 32(1):37–45, January 2009a. ISSN 0161-8105.
  75. Prediction of obstructive sleep apnea with craniofacial photographic analysis. Sleep, 32(1):46–52, January 2009b. ISSN 0161-8105.
  76. Brian C. Ross. Mutual Information between Discrete and Continuous Data Sets. PLOS ONE, 9(2):e87357, February 2014. ISSN 1932-6203. doi: 10.1371/journal.pone.0087357. URL https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0087357. Publisher: Public Library of Science.
  77. Introduction to learning classifier systems. Springer, 2017.
  78. Martin V. Butz. The XCS Classifier System. In Martin V. Butz, editor, Rule-Based Evolutionary Online Learning Systems: A Principled Approach to LCS Analysis and Design, Studies in Fuzziness and Soft Computing, pages 51–64. Springer, Berlin, Heidelberg, 2006. ISBN 978-3-540-31231-4. doi: 10.1007/3-540-31231-5_4. URL https://doi.org/10.1007/3-540-31231-5_4.
  79. Optuna: A Next-generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, pages 2623–2631, New York, NY, USA, July 2019. Association for Computing Machinery. ISBN 978-1-4503-6201-6. doi: 10.1145/3292500.3330701. URL https://doi.org/10.1145/3292500.3330701.
  80. Leo Breiman. Random Forests. Machine Learning, 45(1):5–32, October 2001. ISSN 1573-0565. doi: 10.1023/A:1010933404324. URL https://doi.org/10.1023/A:1010933404324.
  81. sklearn.linear_model.ElasticNet. URL https://scikit-learn/stable/modules/generated/sklearn.linear_model.ElasticNet.html.
  82. Pedro Abreu Miriam Santos. HCC Survival, 2015. URL https://archive.ics.uci.edu/dataset/423.
  83. GAMETES: a fast, direct algorithm for generating pure, strict, epistatic models with random architectures. BioData Mining, 5(1):16, October 2012. ISSN 1756-0381. doi: 10.1186/1756-0381-5-16. URL https://doi.org/10.1186/1756-0381-5-16.
  84. S. Rao Mallampati. Clinical sign to predict difficult tracheal intubation (hypothesis). Canadian Anaesthetists’ Society Journal, 30(3):316–317, May 1983. ISSN 1496-8975. doi: 10.1007/BF03013818. URL https://doi.org/10.1007/BF03013818.
  85. Simple and Unbiased OSA Prescreening: Introduction of a New Morphologic OSA Prediction Score. Nature and Science of Sleep, 13:2039–2049, 2021. ISSN 1179-1608. doi: 10.2147/NSS.S333471.
  86. Heart rate variability during wakefulness as a marker of obstructive sleep apnea severity. Sleep, 44(5):zsab018, May 2021. ISSN 1550-9109. doi: 10.1093/sleep/zsab018.
  87. Diagnostic Performance of Machine Learning-Derived OSA Prediction Tools in Large Clinical and Community-Based Samples. Chest, 161(3):807–817, March 2022. ISSN 1931-3543. doi: 10.1016/j.chest.2021.10.023.
  88. A Global Comparison of Anatomic Risk Factors and Their Relationship to Obstructive Sleep Apnea Severity in Clinical Samples. Journal of clinical sleep medicine: JCSM: official publication of the American Academy of Sleep Medicine, 15(4):629–639, April 2019. ISSN 1550-9397. doi: 10.5664/jcsm.7730.
  89. Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning. August 2021. URL https://openreview.net/forum?id=mPducS1MsEK.
  90. Benchmark and Survey of Automated Machine Learning Frameworks. Journal of Artificial Intelligence Research, 70:409–472, January 2021. ISSN 1076-9757. doi: 10.1613/jair.1.11854. URL https://www.jair.org/index.php/jair/article/view/11854.
  91. Benchmarking Automatic Machine Learning Frameworks, August 2018. URL http://arxiv.org/abs/1808.06492. arXiv:1808.06492 [cs, stat].
  92. A Comparison of AutoML Tools for Machine Learning, Deep Learning and XGBoost. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8, July 2021. doi: 10.1109/IJCNN52387.2021.9534091. URL https://ieeexplore.ieee.org/abstract/document/9534091. ISSN: 2161-4407.
  93. AMLB: an AutoML Benchmark, July 2022. URL http://arxiv.org/abs/2207.12560. arXiv:2207.12560 [cs, stat].
  94. The occurrence of sleep-disordered breathing among middle-aged adults. The New England Journal of Medicine, 328(17):1230–1235, April 1993. ISSN 0028-4793. doi: 10.1056/NEJM199304293281704.
  95. Increased prevalence of sleep-disordered breathing in adults. American Journal of Epidemiology, 177(9):1006–1014, May 2013. ISSN 1476-6256. doi: 10.1093/aje/kws342.
  96. Fairness in machine learning: A survey. ACM Computing Surveys, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (22)
  1. Ryan J. Urbanowicz (15 papers)
  2. Harsh Bandhey (2 papers)
  3. Brendan T. Keenan (1 paper)
  4. Greg Maislin (1 paper)
  5. Sy Hwang (2 papers)
  6. Danielle L. Mowery (1 paper)
  7. Shannon M. Lynch (3 papers)
  8. Diego R. Mazzotti (1 paper)
  9. Fang Han (57 papers)
  10. Qing Yun Li (1 paper)
  11. Thomas Penzel (6 papers)
  12. Sergio Tufik (1 paper)
  13. Lia Bittencourt (1 paper)
  14. Thorarinn Gislason (1 paper)
  15. Philip de Chazal (7 papers)
  16. Bhajan Singh (1 paper)
  17. Nigel McArdle (1 paper)
  18. Ning-Hung Chen (1 paper)
  19. Allan Pack (1 paper)
  20. Richard J. Schwab (1 paper)
Citations (1)
View on Semantic Scholar

Summary

Overview of STREAMLINE: An Automated Machine Learning Pipeline for Biomedicine

The research paper introduces STREAMLINE, an Automated Machine Learning (AutoML) pipeline designed specifically for binary classification in biomedical data analysis. The paper focuses on evaluating the application of this pipeline to predict obstructive sleep apnea (OSA) using various feature sets, including photography-based phenotypes, across international sleep centers.

Methodology

STREAMLINE automates many aspects of the ML analysis pipeline while allowing for human customization. It integrates exploratory data analysis (EDA), data cleaning, feature engineering, and advanced ML modeling. The pipeline is optimized for structured binary classification datasets prevalent in biomedical research. It includes robust elements like MultiSURF for feature selection and supports extensive hyperparameter optimization via Optuna. STREAMLINE is designed to enhance scalability, reproducibility, and transparency.

Experimental Evaluation

STREAMLINE's performance was validated on both simulated and real-world datasets. The pipeline’s efficacy was previously benchmarked with datasets simulating complex associations, such as epistatic interactions. The real-world application focused on predicting OSA across a dataset of 3,111 participants from the Sleep Apnea Global Interdisciplinary Consortium (SAGIC). The paper assessed the predictive utility of demographics (DEM), self-reported comorbidities (DX), symptoms (SYM), and photography-based craniofacial (CF) and intraoral (IO) measures.

Key Findings

  • STREAMLINE showed increased ROC-AUC when CF data was included in predicting moderate/severe OSA. However, CF and IO features did not significantly improve predictions for any OSA (AHI≥5).
  • Among the feature sets tested, demographics and craniofacial features consistently demonstrated higher predictive value over DX and IO features.
  • The artificial neural network yielded the highest median ROC-AUC for moderate/severe OSA (AHI≥15), and elastic net performed best for any OSA. However, model performance varied depending on specific datasets and metrics.
  • Exploratory data and feature importance analyses consistently ranked BMI, loud snoring, and various craniofacial features as significant predictors.

Implications and Speculation

STREAMLINE's ability to automate and streamline ML processes while maintaining flexibility makes it a valuable tool in biomedical research. The application of photography-based phenotypes in OSA risk prediction exemplifies its potential in integrating novel data types into standard diagnostic processes. Moving forward, the refinement of feature selection strategies and exploration of ensemble methods could enhance its efficacy further. Additionally, deploying STREAMLINE in more diverse settings could establish its broader applicability in clinical diagnostics.

Future Directions

For STREAMLINE, expanding capabilities to handle multi-class and regression tasks is a priority. Enhancing fairness evaluations and integrating ensemble learning could address existing limitations. Comparative benchmarking with other AutoML tools is necessary to validate and enhance the competitiveness of STREAMLINE's approach. It holds promise for future developments in scalable, reproducible machine learning applications in biomedicine.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube