Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Prototype2Code: End-to-end Front-end Code Generation from UI Design Prototypes (2405.04975v1)

Published 8 May 2024 in cs.SE

Abstract: UI-to-code technology has streamlined the front-end development process, reducing repetitive tasks for engineers. prior research mainly use design prototypes as inputs, with the effectiveness of the generated code heavily dependent on these prototypes' quality, leading to compromised robustness. Moreover, these approaches also exhibit shortcomings in code quality, including issues such as disorganized UI structures and the inability to support responsive layouts. To address these challenges, we introduce Prototype2Code, which achieves end-to-end front-end code generation with business demands. For Prototype2Code, we incorporate design linting into the workflow, addressing the detection of fragmented elements and perceptual groups, enhancing the robustness of the generated outcomes. By optimizing the hierarchical structure and intelligently recognizing UI element types, Prototype2Code generates code that is more readable and structurally clearer. To meet responsive design requirements, Prototype2Code primarily supports flexbox layout model, ensuring code compatibility across various device sizes. To validate the efficacy, we compare Prototype2Code with the commercial code generation platform CodeFun and Screenshot-to-code based on GPT-4 with vision. Employing structural similarity index measure (SSIM), peak signal-to-noise ratio (PSNR), and mean squared error (MSE) for visual similarity assessment, Prototype2Code's rendered UI effects align most closely with the design prototypes, exhibiting the minimal errors. We also conduct a user study with five experienced front-end engineers, inviting them to review and revise code generated by the three methods. As a result, Prototype2Code surpasses other methods in readability, usability, and maintainability, better meeting the business needs of industrial development.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. ‘‘From ui design image to gui skeleton: a neural machine translator to bootstrap mobile gui implementation.’’ Proceedings of the 40th International Conference on Software Engineering: pp. 665–676. 2018.
  2. ‘‘Rico: A mobile app dataset for building data-driven design applications.’’ Proceedings of the 30th annual ACM symposium on user interface software and technology: pp. 845–854. 2017.
  3. ‘‘Learning to denoise raw mobile UI layouts for improving datasets at scale.’’ Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems: pp. 1–13. 2022.
  4. ‘‘Doodle2App: native app code by freehand UI sketching.’’ Proceedings of the IEEE/ACM 7th International Conference on Mobile Software Engineering and Systems: pp. 81–84. 2020.
  5. ‘‘Designing with Language: Wireframing UI Design Intent with Generative Large Language Models.’’ (2023). URL arXiv2312.07755.
  6. ‘‘Automated cross-platform GUI code generation for mobile apps.’’ 2019 IEEE 1st International Workshop on Artificial Intelligence for Mobile (AI4Mobile): pp. 13–16. 2019. IEEE.
  7. ‘‘Figma2Code: Automatic code generation method for Figma design drafts.’’ Journal of Computer-Aided Design & Computer Graphics. (2023): pp. 1–910.3724/SP.J.1089.2023-00336.
  8. ‘‘Intelligent Code Generation for Design Drafts.’’ (2022). URL https://www.imgcook.com/.
  9. ‘‘Codefun platform.’’ (2022). URL https://code.fun/.
  10. ‘‘Sketch software.’’ (2022). URL https://www.sketch.com/.
  11. ‘‘Figma platform.’’ (2023). URL https://www.figma.com/.
  12. ‘‘UI Layers Merger: merging UI layers via visual learning and boundary prior.’’ Frontiers of Information Technology & Electronic Engineering (2022): p. 373–38710.1631/FITEE.2200099.
  13. ‘‘ULDGNN: A Fragmented UI Layer Detector Based on Graph Neural Networks.’’ arXiv preprint arXiv:2208.06658 (2022): p. 1–12URL https://arxiv.org/abs/2208.06658.
  14. ‘‘EGFE: End-to-end Grouping of Fragmented Elements in UI Designs with Multimodal Learning.’’ Proceedings of the 46th IEEE/ACM International Conference on Software Engineering: pp. 1–12. 2024.
  15. ‘‘UI Layers Group Detector: Grouping UI Layers via Text Fusion and Box Attention.’’ CAAI International Conference on Artificial Intelligence: pp. 303–314. 2022. Springer.
  16. ‘‘UI semantic component group detection: Grouping UI elements with similar semantics in mobile graphical user interface.’’ Displays Vol. 83 (2024): p. 102679.
  17. ‘‘Figma to Code.’’ (2023). URL https://www.figma.com/community/plugin/842128343887142055/.
  18. ‘‘builder.io.’’ (2023). URL https://www.builder.io/.
  19. ‘‘yotako.’’ (2023). URL https://yotako.io/.
  20. Ikumapayi, Nathan Adelola. ‘‘Automated Front-End Code Generation Using OpenAI: Empowering Web Development Efficiency.’’ Available at SSRN 4590704 (2023): pp. 1–11.
  21. ‘‘In-IDE Generation-based Information Support with a Large Language Model.’’ arXiv preprint arXiv:2307.08177 (2023): pp. 1–13.
  22. ‘‘Anima: AI Design to Code.’’ (2023). URL https://www.animaapp.com/.
  23. ‘‘Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels.’’ Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems: pp. 1–15. 2021. Association for Computing Machinery, New York, NY, USA. 10.1145/3411764.3445186.
  24. ‘‘Psychologically-inspired, unsupervised inference of perceptual groups of GUI widgets from GUI images.’’ Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering: pp. 332–343. 2022.
  25. ‘‘Deep residual learning for image recognition.’’ Proceedings of the IEEE conference on computer vision and pattern recognition: pp. 770–778. 2016.
  26. ‘‘Cascade rpn: Delving into high-quality region proposal network with adaptive convolution.’’ Advances in neural information processing systems Vol. 32 (2019): pp. 1–11.
  27. ‘‘MMDetection: Open mmlab detection toolbox and benchmark.’’ arXiv preprint arXiv:1906.07155 (2019): pp. 1–13.
  28. ‘‘Object detection for graphical user interface: Old fashioned or deep learning or a combination?’’ proceedings of the 28th ACM joint meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering: pp. 1202–1214. 2020.
  29. ‘‘Multimodal icon annotation for mobile applications.’’ Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction: pp. 1–11. 2021.
  30. ‘‘Do transformers really perform badly for graph representation?’’ Advances in Neural Information Processing Systems Vol. 34 (2021): pp. 28877–28888.
  31. ‘‘Recipe for a general, powerful, scalable graph transformer.’’ Advances in Neural Information Processing Systems Vol. 35 (2022): pp. 14501–14515.
  32. ‘‘Neural message passing for quantum chemistry.’’ International conference on machine learning: pp. 1263–1272. 2017. PMLR.
  33. ‘‘Screenshot-to-Code.’’ (2024). URL https://github.com/abi/screenshot-to-code/.
  34. ‘‘OpenAI Chat.’’ (2023). URL https://chat.openai.com/.
  35. ‘‘Figma API.’’ (2023). URL https://www.figma.com/developers/api/.
  36. ‘‘Microsoft coco: Common objects in context.’’ Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13: pp. 740–755. 2014. Springer.
  37. ‘‘Faster r-cnn: Towards real-time object detection with region proposal networks.’’ Advances in neural information processing systems Vol. 28 (2015): pp. 1–9.
  38. ‘‘Region proposal by guided anchoring.’’ Proceedings of the IEEE/CVF conference on computer vision and pattern recognition: pp. 2965–2974. 2019.
  39. ‘‘Centernet: Keypoint triplets for object detection.’’ Proceedings of the IEEE/CVF international conference on computer vision: pp. 6569–6578. 2019.
  40. ‘‘You only look one-level feature.’’ Proceedings of the IEEE/CVF conference on computer vision and pattern recognition: pp. 13039–13048. 2021.
  41. ‘‘Deformable detr: Deformable transformers for end-to-end object detection.’’ arXiv preprint arXiv:2010.04159 (2020): pp. 1–16.
  42. ‘‘Magic Layouts: Structural Prior for Component Detection in User Interface Designs.’’ 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR): pp. 15804–15813. 2021. 10.1109/CVPR46437.2021.01555.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Shuhong Xiao (9 papers)
  2. Yunnong Chen (11 papers)
  3. Jiazhi Li (18 papers)
  4. Liuqing Chen (16 papers)
  5. Lingyun Sun (38 papers)
  6. Tingting Zhou (14 papers)
Citations (1)
View on Semantic Scholar