Scaling a Variant Calling Genomics Pipeline with FaaS
Abstract: With the escalating complexity and volume of genomic data, the capacity of biology institutions' HPC faces limitations. While the Cloud presents a viable solution for short-term elasticity, its intricacies pose challenges for bioinformatics users. Alternatively, serverless computing allows for workload scalability with minimal developer burden. However, porting a scientific application to serverless is not a straightforward process. In this article, we present a Variant Calling genomics pipeline migrated from single-node HPC to a serverless architecture. We describe the inherent challenges of this approach and the engineering efforts required to achieve scalability. We contribute by open-sourcing the pipeline for future systems research and as a scalable user-friendly tool for the bioinformatics community.
- European Nucleotide Archive. 2022. Statistics. https://www.ebi.ac.uk/ena/browser/about/statistics.
- Scaling a Variant Calling Genomics Pipeline with FaaS. In Proceedings of the 9th International Workshop on Serverless Computing (Bologna, Italy) (WoSC ’23). Association for Computing Machinery, New York, NY, USA, 59–64. https://doi.org/10.1145/3631295.3631403
- Daniel Barcelona-Pons and Pedro GarcÃa-López. 2021. Benchmarking parallelism in FaaS platforms. Future Generation Computer Systems 124 (2021), 268–284. https://doi.org/10.1016/j.future.2021.06.005
- The European Nucleotide Archive in 2022. Nucleic Acids Res 51, D1 (Jan 2023), D121–D125.
- Challenges and Opportunities of Amazon Serverless Lambda Services in Bioinformatics. 663–668. https://doi.org/10.1145/3307339.3343462
- Nextflow enables reproducible computational workflows. Nature biotechnology 35, 4 (2017), 316–319.
- SiNPle: Fast and Sensitive Variant Calling for Deep Sequencing Data. Genes (2019).
- Serverless Computing: One Step Forward, Two Steps Back. arXiv:1812.03651Â [cs.DC]
- Accessible and interactive RNA sequencing analysis using serverless computing. bioRxiv (2020). https://doi.org/10.1101/576199 arXiv:https://www.biorxiv.org/content/early/2020/10/03/576199.full.pdf
- Evaluation of serverless computing for scalable execution of a joint variant calling workflow. PLOS ONE 16, 7 (07 2021), 1–12. https://doi.org/10.1371/journal.pone.0254363
- Occupy the Cloud: Distributed Computing for the 99%. In Proceedings of the 2017 Symposium on Cloud Computing (Santa Clara, California) (SoCC ’17). Association for Computing Machinery, New York, NY, USA, 445–451. https://doi.org/10.1145/3127479.3128601
- DNAvisualization.org: a serverless web tool for DNA sequence visualization. Nucleic Acids Res 47, W1 (July 2019), W20–W25.
- The GEM mapper: fast, accurate and versatile alignment by filtration. Nature methods 9, 12 (2012), 1185–1188.
- Leveraging Serverless Computing to Improve Performance for Sequence Comparison. 683–687. https://doi.org/10.1145/3307339.3343465
- Shuffling, Fast and Slow: Scalable Analytics on Serverless Infrastructure. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19). USENIX Association, Boston, MA, 193–206. https://www.usenix.org/conference/nsdi19/presentation/pu
- David P. Rodgers. 1985. Improvements in Multiprocessor System Design. SIGARCH Comput. Archit. News 13, 3 (jun 1985), 225–231. https://doi.org/10.1145/327070.327215
- Toward Multicloud Access Transparency in Serverless Computing. IEEE Software 38, 1 (2021), 68–74. https://doi.org/10.1109/MS.2020.3029994
- numpywren: serverless linear algebra. arXiv:1810.09679Â [cs.DC]
- FaaSter, Better, Cheaper: The Prospect of Serverless Scientific Computing and HPC. In High Performance Computing, Esteban Mocskos and Sergio Nesmachnow (Eds.). Springer International Publishing, Cham, 154–168.
Paper Prompts
Sign up for free to create and run prompts on this paper.