"Project smells" -- Experiences in Analysing the Software Quality of ML Projects with mllint (2201.08246v1)

Abstract: Machine Learning (ML) projects incur novel challenges in their development and productionisation over traditional software applications, though established principles and best practices in ensuring the project's software quality still apply. While using static analysis to catch code smells has been shown to improve software quality attributes, it is only a small piece of the software quality puzzle, especially in the case of ML projects given their additional challenges and lower degree of Software Engineering (SE) experience in the data scientists that develop them. We introduce the novel concept of project smells which consider deficits in project management as a more holistic perspective on software quality in ML projects. An open-source static analysis tool mllint was also implemented to help detect and mitigate these. Our research evaluates this novel concept of project smells in the industrial context of ING, a global bank and large software- and data-intensive organisation. We also investigate the perceived importance of these project smells for proof-of-concept versus production-ready ML projects, as well as the perceived obstructions and benefits to using static analysis tools such as mllint. Our findings indicate a need for context-aware static analysis tools, that fit the needs of the project at its current stage of development, while requiring minimal configuration effort from the user.

• The paper introduces project smells as a comprehensive framework that broadens static analysis to address ML projects' unique quality challenges.
• It presents mllint, an open-source tool that evaluates factors such as dependency management, version control, and testing practices.
• The study assesses mllint in an industrial context, highlighting practical benefits and adoption challenges in moving from proof-of-concept to production-ready systems.

Analyzing Software Quality in ML Projects: The Concept of "Project Smells"

The paper "Project smells Experiences in Analysing the Software Quality of ML Projects with mllint," presents an innovative approach to evaluating software quality in ML projects through the lens of "project smells." Traditional static analysis primarily targets code smells, offering improvements in software attributes like reliability and maintainability. However, ML projects face distinctive challenges, demanding a broader perspective beyond code smells.

The authors introduce "project smells," a comprehensive framework encompassing deficits in project management that negatively impact software quality. They also present "mllint," an open-source static analysis tool developed to detect and address these project smells. The paper was conducted in collaboration with ING, a global financial institution heavily invested in software and data-driven solutions.

Key Contributions and Findings

1. The Concept of Project Smells: The paper posits that focusing solely on code smells is insufficient for ML projects, which demand additional scrutiny due to unique challenges, such as data management and reproducibility. Project smells provide a holistic perspective integrating elements of dependency management, testing practices, and more.
2. mllint Tool Implementation: mllint serves as a tangible application of the project smells concept. It evaluates various aspects of software quality through categories such as version control, dependency management, continuous integration, code quality, and testing strategies. The tool's architecture permits easy extensibility, allowing the incorporation of additional checks and rules as the field evolves.
3. Industrial Context Evaluation: The research evaluates how mllint aligns with the software engineering practices at ING, identifying obstacles and benefits in its application. It highlights contexts where project smells are most prevalent, revealing areas where ML projects often fall short.
4. Proof-of-Concept vs. Production-Ready Projects: The paper discusses varying perceptions of project smells' importance in different development stages. While certain practices are crucial across the board, their priority often shifts when transitioning from proof-of-concept to production-ready stages.
5. Adoption Barriers and Benefits: mllint is recognized for its potential in reinforcing best practices and guiding project quality. Nonetheless, configuration challenges and habitual use of existing tools present significant adoption barriers. The implementation of mllint in the CI pipeline is suggested to enhance code review processes.

Theoretical and Practical Implications

Theoretically, this research extends the discourse on software engineering in ML projects by introducing project smells, a concept aiming to encapsulate the broader set of challenges such projects present. Practically, mllint equips developers with insights and actionable recommendations for improving their software's robustness and maintainability.

Future Directions

Developments in mllint are paramount to its future utility, particularly in expanding its rule set to encompass more ML-specific challenges. Furthermore, the exploration of context-aware static analysis could democratize tools like mllint, tailoring their configurations to adapt to varying technological environments and project maturities.

With the increasing integration of ML into production systems, the work presented in this paper provides a framework for addressing software quality from a more interdisciplinary perspective, reinforcing the foundational practices required for sustainable ML system deployment.