Papers
Topics
Authors
Recent
Search
2000 character limit reached

Ivie: Lightweight Anchored Explanations of Just-Generated Code

Published 4 Mar 2024 in cs.HC | (2403.02491v1)

Abstract: Programming assistants have reshaped the experience of programming into one where programmers spend less time writing and more time critically examining code. In this paper, we explore how programming assistants can be extended to accelerate the inspection of generated code. We introduce an extension to the programming assistant called Ivie, or instantly visible in-situ explanations. When using Ivie, a programmer's generated code is instantly accompanied by explanations positioned just adjacent to the code. Our design was optimized for extremely low-cost invocation and dismissal. Explanations are compact and informative. They describe meaningful expressions, from individual variables to entire blocks of code. We present an implementation of Ivie that forks VS Code, applying a modern LLM for timely segmentation and explanation of generated code. In a lab study, we compared Ivie to a contemporary baseline tool for code understanding. Ivie improved understanding of generated code, and was received by programmers as a highly useful, low distraction, desirable complement to the programming assistant.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. Naser Al Madi. 2023. How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilot. In Proceedings of the International Conference on Automated Software Engineering. ACM, Article 205.
  2. Amazon CodeWhisperer. Retrieved July 25, 2023 from https://aws.amazon.com/codewhisperer/
  3. Grounded Copilot: How Programmers Interact with Code-Generating Models. Proc. ACM Program. Lang., Article 78 (April 2023).
  4. Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools. Queue 20, 6 (2022), 35–57.
  5. Qualitative HCI research: Going behind the scenes. Morgan & Claypool Publishers.
  6. Ruven Brooks. 1983. Towards a theory of the comprehension of computer programs. International journal of man-machine studies 18, 6 (1983), 543–554.
  7. Explaining visual changes in web interfaces. In Proceedings of the Symposium on User Interface Software and Technology. ACM, 259–268.
  8. John M Carroll. 1990. The Nurnberg funnel: Designing minimalist instruction for practical computer skill. MIT Press.
  9. Paul Chandler and John Sweller. 1992. The split-attention effect as a factor in the design of instruction. British Journal of Educational Psychology 62, 2 (1992), 233–246.
  10. Kerry Shih-Ping Chang and Brad A. Myers. 2012. WebCrystal: Understanding and Reusing Examples in Web Authoring. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, 3205–3214.
  11. ChatGPT. Retrieved September 15, 2023 from https://chat.openai.com
  12. Codeium. Retrieved July 25, 2023 from https://marketplace.visualstudio.com/items?itemName=Codeium.codeium
  13. William Perry Crichton. 2022. Revisiting Program Slicing with Ownership-Based Information Flow. Ph. D. Dissertation.
  14. Françoise Détienne. 2001. Software Design–Cognitive Aspects. Springer Science & Business Media.
  15. Thomas Dohmke. 2022. GitHub Copilot is generally available to all developers. Retrieved July 25, 2023 from https://github.blog/2022-06-21-github-copilot-is-generally-available-to-all-developers/
  16. Ekwa Duala-Ekoko and Martin P. Robillard. 2012. Asking and answering questions about unfamiliar APIs: An exploratory study. In Proceedings of the International Conference on Software Engineering. IEEE, 266–276.
  17. EasyCode. Retrieved July 25, 2023 from https://www.easycode.ai/
  18. Live Exploration of AI-Generated Programs. (2023). arXiv:2306.09541 [cs.HC]
  19. Using Psycho-Physiological Measures to Assess Task Difficulty in Software Development. In Proceedings of the International Conference on Software Engineering. ACM, 402–413.
  20. Genie AI. Retrieved July 25, 2023 from https://marketplace.visualstudio.com/items?itemName=genieai.chatgpt-vscode
  21. GitHub Copilot X. Retrieved July 25, 2023 from https://github.com/features/preview/copilot-x
  22. GitHub Copilot · Your AI pair programmer. Retrieved July 25, 2023 from https://github.com/features/copilot
  23. GitHub Next — Code Brushes. Retrieved September 10, 2023 from https://githubnext.com/projects/code-brushes/
  24. Visualizing API Usage Examples at Scale. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Article 580.
  25. Tutorons: Generating context-relevant, on-demand explanations and demonstrations of online code. In Proceedings of the Symposium on Visual Languages and Human-Centric Computing. IEEE, 3–12.
  26. Joshua Hibschman and Haoqi Zhang. 2016. Telescope: Fine-tuned discovery of interactive web UI feature implementation. In Proceedings of the Symposium on User Interface Software and Technology. ACM, 233–245.
  27. Augmenting Code with In Situ Visualizations to Aid Program Understanding. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Article 532.
  28. Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics (1979), 65–70.
  29. Understanding How Programmers Can Use Annotations on Documentation. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Article 69.
  30. Using Annotations for Sensemaking About Code. In Proceedings of the Symposium on User Interface Software and Technology. ACM, Article 61.
  31. Through (Tracking) Their Eyes: Abstraction and Complexity in Program Comprehension. ACM Transactions on Computing Education, Article 17 (November 2021).
  32. Information needs in collocated software development teams. In Proceedings of the International Conference on Software Engineering. IEEE, 344–353.
  33. Sorin Lerner. 2020. Projection Boxes: On-the-Fly Reconfigurable Visualization for Live Programming. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Article 367.
  34. A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges. In Proceedings of the International Conference on Software Engineering. IEEE/ACM, Article 52.
  35. CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes. In Proceedings of Koli Calling International Conference on Computing Education Research. ACM, Article 8.
  36. Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming. (2024). To appear.
  37. Quantifying programmers’ mental workload during program comprehension based on cerebral blood flow measurement: A controlled experiment. In Proceedings of the International Conference on Software Engineering. ACM, 448–451.
  38. Using an LLM to Help With Code Understanding. In Proceedings of the International Conference on Software Engineering. To appear.
  39. Jakob Nielsen. 1994. Enhancing the explanatory power of usability heuristics. In Proceedings of the CHI conference on Human Factors in Computing Systems. ACM, 152–158.
  40. OpenAI. 2023. GPT-4 Technical Report. (2023). arXiv:2303.08774 [cs.CL]
  41. OpenCV. Retrieved September 15, 2023 from https://opencv.org/
  42. Martin T Orne. 2017. On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. In Sociological methods. Routledge, 279–299.
  43. Nancy Pennington. 1987. Stimulus structures and mental representations in expert comprehension of computer programs. Cognitive psychology 19, 3 (1987), 295–341.
  44. Contextualized Programming Language Documentation. In Proceedings of the International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software. ACM.
  45. PyGame. Retrieved September 15, 2023 from https://www.pygame.org
  46. Martin P. Robillard and Robert DeLine. 2011. A field study of API learning obstacles. Empirical Software Engineering 16 (2011), 703–732.
  47. How developers search for code: a case study. In Proceedings of the Joint Meeting on Foundations of Software Engineering. ACM, 191–201.
  48. Franklin E. Satterthwaite. 1946. An approximate distribution of estimates of variance components. Biometrics bulletin 2, 6 (1946), 110–114.
  49. Teresa M. Shaft and Iris Vessey. 1995. The Relevance of Application Domain Knowledge: The Case of Computer Program Comprehension. Information systems research 6, 3 (1995), 286–299.
  50. Bonita Sharif and Jonathan I. Maletic. 2010. An Eye Tracking Study on camelCase and under_score Identifier Styles. In Proceedings of the International Conference on Program Comprehension. IEEE, 196–205.
  51. Ben Shneiderman. 1976. Exploratory experiments in programmer behavior. International Journal of Computer & Information Sciences 5 (1976), 123–143.
  52. Ben Shneiderman and Richard Mayer. 1979. Syntactic/semantic interactions in programmer behavior: A model and experimental results. International Journal of Computer & Information Sciences 8 (1979), 219–238.
  53. Understanding understanding source code with functional magnetic resonance imaging. In Proceedings of the International Conference on Software Engineering. ACM, 378–389.
  54. Questions Programmers Ask during Software Evolution Tasks. In Proceedings of the International Symposium on Foundations of Software Engineering. ACM, 23–34.
  55. Elliot Soloway and Kate Ehrlich. 1984. Empirical Studies of Programming Knowledge. IEEE Transactions on Software Engineering SE-10, 5 (1984), 595–609.
  56. IntelliCode Compose: Code Generation using Transformer. In Proceedings of the Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 1433–1443.
  57. An Empirical Study of Developer Behaviors for Validating and Repairing AI-Generated Code. In Proceedings of the Annual Workshop at the Intersection of PL and HCI.
  58. TLX @ NASA Ames - Home. Retrieved September 15, 2023 from https://humansystems.arc.nasa.gov/groups/TLX/
  59. Tobii. Enter the world of eye tracking with Tobii Pro Spark - Tobii. Retrieved September 15, 2023 from https://www.tobii.com/products/eye-trackers/screen-based/tobii-pro-spark
  60. Gias Uddin and Martin P. Robillard. 2015. How API documentation fails. IEEE Software 32, 4 (2015), 68–75.
  61. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. ACM, Article 332.
  62. Visual Studio Code. Retrieved September 15, 2023 from https://code.visualstudio.com/
  63. A. von Mayrhauser and A. M. Vans. 1993. From program comprehension to tool requirements for an industrial environment. In Proceedings of the Workshop on Program Comprehension. IEEE, 78–86.
  64. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1069–1088.
  65. Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In Breakthroughs in Statistics: Methodology and Distribution. Springer, 196–202.
  66. Measuring program comprehension: A large-scale field study with professionals. IEEE Transactions on Software Engineering 44, 10 (2017), 951–976.
  67. In-IDE Code Generation from Natural Language: Promise and Challenges. ACM Trans. Softw. Eng. Methodol. 31, Article 29 (March 2022).
  68. Visualizing Examples of Deep Neural Networks at Scale. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Article 313.
  69. Concept-Annotated Examples for Library Comparison. In Proceedings of the Symposium on User Interface Software and Technology. ACM, Article 65.
Citations (8)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free