Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps (1802.02312v2)

Abstract: It is common practice for developers of user-facing software to transform a mock-up of a graphical user interface (GUI) into code. This process takes place both at an application's inception and in an evolutionary context as GUI changes keep pace with evolving features. Unfortunately, this practice is challenging and time-consuming. In this paper, we present an approach that automates this process by enabling accurate prototyping of GUIs via three tasks: detection, classification, and assembly. First, logical components of a GUI are detected from a mock-up artifact using either computer vision techniques or mock-up metadata. Then, software repository mining, automated dynamic analysis, and deep convolutional neural networks are utilized to accurately classify GUI-components into domain-specific types (e.g., toggle-button). Finally, a data-driven, K-nearest-neighbors algorithm generates a suitable hierarchical GUI structure from which a prototype application can be automatically assembled. We implemented this approach for Android in a system called ReDraw. Our evaluation illustrates that ReDraw achieves an average GUI-component classification accuracy of 91% and assembles prototype applications that closely mirror target mock-ups in terms of visual affinity while exhibiting reasonable code structure. Interviews with industrial practitioners illustrate ReDraw's potential to improve real development workflows.

Essay: Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

The paper "Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps" by Moran et al. presents an approach to automating the prototyping of Graphical User Interfaces (GUIs) for mobile applications, specifically targeting the Android platform. This approach addresses the time-consuming and error-prone process of translating static design mock-ups into functional GUI code, particularly in industrial settings where design and coding are often handled by different teams.

Overview of the Approach

The authors decompose the prototyping process into three fundamental tasks: detection, classification, and assembly of GUI components. The detection phase involves identifying potential GUI components from mock-up artifacts, either by parsing mock-up metadata or utilizing computer vision (CV) techniques. This allows for the automatic extraction of bounding boxes for GUI components from design mockups or images using CV methods like edge detection and contour analysis.

In the classification phase, a Convolutional Neural Network (CNN) is employed to classify the extracted GUI components into domain-specific types, such as buttons or text fields. The CNN is trained using a dataset generated through large-scale dynamic analysis of popular Android applications, extracting annotated GUI components by leveraging Android's UIAutomator framework. This data-driven approach enables high classification accuracy by learning from existing applications' GUIs, achieving a top-1 classification accuracy of 91%.

Lastly, the assembly phase involves constructing a hierarchical GUI structure using a K-nearest-neighbors (KNN) algorithm. The algorithm matches the detected components to a corpus of real application GUIs to infer a logical hierarchy of components and containers. Additional visual characteristics, such as color and text style, are inferred using color quantization and histogram analysis.

Evaluation and Results

The prototype system, named ReDraw, demonstrates its efficacy through a comprehensive evaluation. The approach is compared against existing methods, such as REMAUI and pix2code, illustrating superior performance in generating both visually and hierarchically accurate prototypes. The generated prototypes closely resemble the target mock-ups and exhibit a coherent code structure, addressing the shortcomings of previous approaches that either oversimplify component classification or produce unrealistic hierarchies. The paper involves quantitative metrics like mean squared error (MSE) and mean average error (MAE) for visual similarity and pure edit distance for hierarchy similarity.

Industrial relevance is further validated through interviews with practitioners from companies like Google and Huawei, who recognize ReDraw's potential to automate and accelerate GUI prototyping in real-world workflows, especially for rapid iteration or evolutionary enhancements.

Implications and Future Directions

The implications of this research extend to enhancing the efficiency and accuracy of GUI design-to-code translation, reducing the disconnect between design and development teams, and alleviating the cognitive load associated with rebuilding GUIs from scratch. From a theoretical perspective, this work underscores the synergy between CV, ML, and software repository mining to derive impactful solutions in software engineering.

Future research could explore adapting ReDraw to other platforms, including iOS or web applications, addressing the limitations of current platform-specific datasets. Extensions to the CNN's architecture to handle a broader spectrum of component types or more nuanced stylistic details are promising avenues. Additionally, integrating ReDraw into popular Integrated Development Environments (IDEs) could facilitate seamless adoption by developers.

In conclusion, this paper presents a rigorous and empirically validated methodology for automated GUI prototyping, merging state-of-the-art ML techniques with practical software engineering needs. Its contributions pave the way for more sophisticated, data-driven design automation tools that can significantly optimize the software development lifecycle.