Toward End-to-End MLOps Tools Map: A Preliminary Study based on a Multivocal Literature Review

Abstract: MLOps tools enable continuous development of machine learning, following the DevOps process. Different MLOps tools have been presented on the market, however, such a number of tools often create confusion on the most appropriate tool to be used in each DevOps phase. To overcome this issue, we conducted a multivocal literature review mapping 84 MLOps tools identified from 254 Primary Studies, on the DevOps phases, highlighting their purpose, and possible incompatibilities. The result of this work will be helpful to both practitioners and researchers, as a starting point for future investigations on MLOps tools, pipelines, and processes.

• The paper provides a systematic mapping of 84 MLOps tools using a rigorous multivocal literature review and a graphical DevOps map.
• It demonstrates tool compatibility across various DevOps phases, emphasizing continuous integration, testing, and operations.
• The study highlights implications for further empirical research and improvements in streamlining MLOps tool integration.

Toward End-to-End MLOps Tools Map: A Preliminary Study

Introduction

The paper "Toward End-to-End MLOps Tools Map: A Preliminary Study based on a Multivocal Literature Review" (2304.03254) embarks on an exploration of the Machine Learning Operations (MLOps) landscape by systematically mapping the existing MLOps tools. The motivation behind this study stems from the rapidly growing importance of MLOps tools in automating and streamlining the processes of building, deploying, and maintaining machine learning models. Given the proliferation of these tools, it becomes imperative to understand their compatibility and applicability across different phases of the DevOps pipeline.

Background and Methodology

MLOps is defined in the context of merging best practices from software engineering and data science, which are crucial for managing the lifecycle of machine learning models. The conventional DevOps process, encompassing phases such as Plan, Code, Build, Test, Release, Deploy, Operate, and Monitor, provides a structured framework that MLOps tools can augment specifically for machine learning models. The study is grounded in a Systematic Multivocal Literature Review (MLR) methodology, which incorporates insights from both academic and gray literature to present a broad understanding of MLOps tools' implementation and adaptability.

The authors identified this literature to assess the tools' coverage across the DevOps phases and their compatibilities. This large-scale review was done to address gaps in prior studies, which suffered from limited scope and lack of detailed tool-phase mapping. This differentiates the current study by presenting an exhaustive examination of tools that cater to each stage of the MLOps lifecycle.

Analysis of MLOps Tools

The paper undertakes a comprehensive classification of 84 MLOps tools identified from 254 primary studies. These tools are further categorized based on their applicability to distinct DevOps phases, such as Continuous Development (CD), Continuous Integration (CI), Continuous Testing (CT), and Operations (OPS). Each category details the tools tailored for specific processes, whether for testing and validating data and models, or for orchestrating and managing deployment infrastructure.

One of the standout contributions of the study is a graphical DevOps map providing visual representation for practitioners and researchers, enabling them to identify suitable tools for each DevOps phase. This map facilitates the understanding of how various tools can integrate seamlessly across different pipeline stages, promoting an efficient MLOps deployment environment.

Discussion and Implications

The study reveals the substantial number of MLOps tools available, primarily developed and disseminated by practitioners rather than the academic community. This indicates a significant opportunity for empirical research to better understand how these tools function in practice and to investigate optimal tool combinations for enhanced pipeline automation.

A critical insight from the study is the inherent compatibility of these tools across various DevOps stages, offering orthogonal support throughout the MLOps pipeline. This suggests opportunities for more intricate studies into information flows between tools and the potential redundancies in using multiple tools within the same phase.

Additionally, implications for tool vendors and developers are substantial. The study's findings provide a cohesive overview of tool capabilities and shortcomings, guiding future enhancements of existing systems or innovations in underserved areas within the MLOps pipeline.

Conclusion

The paper provides a foundation for understanding the breadth of MLOps tools and their integration in end-to-end machine learning pipelines. It advances the discussion in the MLOps domain by offering a detailed tool classification and compatibility analysis, setting the stage for future empirical evaluations and collaborations between practitioners and researchers. The graphical depiction of tool compatibility within the MLOps lifecycle serves as an essential resource for effectively navigating the complex landscape of machine learning operations.