Papers
Topics
Authors
Recent
Search
2000 character limit reached

Programmer Visual Attention During Context-Aware Code Summarization

Published 28 May 2024 in cs.SE | (2405.18573v2)

Abstract: Abridged: Programmer attention represents the visual focus of programmers on parts of the source code in pursuit of programming tasks. We conducted an in-depth human study with 10 Java programmers, where each programmer generated summaries for 40 methods from five large Java projects over five one-hour sessions. We used eye-tracking equipment to map the visual attention of programmers while they wrote the summaries. We also rate the quality of each summary. We found eye-gaze patterns and metrics that define common behaviors between programmer attention during context-aware code summarization. Specifically, we found that programmers need to read significantly (p<0.01) fewer words and make significantly (p<0.03) fewer revisits to words as they summarize more methods during a session, while maintaining the quality of summaries. We also found that the amount of source code a participant looks at correlates with a higher quality summary, but this trend follows a bell-shaped curve, such that after a threshold reading more source code leads to a significant (p<0.01) decrease in the quality of summaries. We also gathered insight into the type of methods in the project that provide the most contextual information for code summarization based on programmer attention. Specifically, we observed that programmers spent a majority of their time looking at methods inside the same class as the target method to be summarized. Surprisingly, we found that programmers spent significantly less time looking at methods in the call graph of the target method. We discuss how our empirical observations may aid future studies towards modeling programmer attention and improving context-aware automatic source code summarization.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. J. Cámara, J. Troya, L. Burgueño, and A. Vallecillo, “On the assessment of generative ai in modeling tasks: an experience report with chatgpt and uml,” Software and Systems Modeling, pp. 1–13, 2023.
  2. S. Haque, A. LeClair, L. Wu, and C. McMillan, “Improved automatic summarization of subroutines via attention to file context,” International Conference on Mining Software Repositories, 2020.
  3. A. Bansal, S. Haque, and C. McMillan, “Project-level encoding for neural source code summarization of subroutines,” International Conference on Program Comprehension, 2021.
  4. A. Bansal, Z. Eberhart, Z. Karas, Y. Huang, and C. McMillan, “Function call graph context encoding for neural source code summarization,” IEEE Transactions on Software Engineering, pp. 1–14, 2023.
  5. R. Bednarik, “Expertise-dependent visual attention strategies develop over time during debugging with multiple code representations,” International Journal of Human-Computer Studies, vol. 70, no. 2, pp. 143–155, 2012.
  6. P. Rodeghero, C. McMillan, P. W. McBurney, N. Bosch, and S. D’Mello, “Improving automated source code summarization via an eye-tracking study of programmers,” in Proceedings of the 36th international conference on Software engineering, ser. ICSE ’14, 2014, to appear.
  7. Z. Sharafi, Z. Soh, and Y.-G. Guéhéneuc, “A systematic literature review on the usage of eye-tracking in software engineering,” Information and Software Technology, vol. 67, pp. 79–107, 2015.
  8. R. Bednarik and M. Tukiainen, “Temporal eye-tracking data: Evolution of debugging strategies with multiple representations,” in Proceedings of the 2008 symposium on Eye tracking research & applications, 2008, pp. 99–102.
  9. N. Ouerhani, R. Von Wartburg, H. Hugli, and R. Muri, “Empirical validation of the saliency-based model of visual attention,” ELCVIA Electronic Letters on Computer Vision and Image Analysis, vol. 3, no. 1, pp. 13–24, 2004.
  10. B. M. Hood, J. D. Willen, and J. Driver, “Adult’s eyes trigger shifts of visual attention in human infants,” Psychological Science, vol. 9, no. 2, pp. 131–134, 1998.
  11. C. N. Olivers, F. Meijer, and J. Theeuwes, “Feature-based memory-driven attentional capture: visual working memory content affects visual attention.” Journal of Experimental Psychology: Human Perception and Performance, vol. 32, no. 5, p. 1243, 2006.
  12. M. Paltenghi and M. Pradel, “Thinking like a developer? comparing the attention of humans with neural models of code,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE).   IEEE, 2021, pp. 867–879.
  13. R. Bednarik and M. Tukiainen, “An eye-tracking methodology for characterizing program comprehension processes,” in Proceedings of the 2006 symposium on Eye tracking research & applications, 2006, pp. 125–132.
  14. Z. Karas, A. Bansal, Y. Zhang, T. Li, C. McMillan, and Y. Huang, “A tale of two comprehensions? analyzing student programmer attention during code summarization,” ACM Transactions on Software Engineering and Methodology, 2024.
  15. A. Bansal, B. Sharif, and C. McMillan, “Towards modeling human attention from eye movements for neutral source code summarization,” Proceedings of ACM Human-Computer Interaction, ETRA Vol. 7, 2023.
  16. A. Bansal, C.-Y. Su, Z. Karas, Y. Zhang, Y. Huang, T. J.-J. Li, and C. McMillan, “Modeling programmer attention as scanpath prediction,” in Proceedings of The 38th IEEE/ACM International Conference on Automated Software Engineering ASE 2023 - NIER track, 2023.
  17. Y. Zhang, J. Li, Z. Karas, A. Bansal, T. J.-J. Li, C. McMillan, K. Leach, and Y. Huang, “Eyetrans: Merging human and machine attention for neural code summarization,” in Proceedings of The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2024), 2024.
  18. Y. Braw, M. Ratmansky, and I. Goor-Aryeh, “Integrating the numerical pain rating scale (nprs) with an eye tracker: Feasibility and initial validation,” 2023.
  19. C. Aschwanden and M. Crosby, “Code scanning patterns in program comprehension,” in Proceedings of the 39th hawaii international conference on system sciences.   Citeseer, 2006.
  20. P. Rodeghero, C. McMillan, P. W. McBurney, N. Bosch, and S. D’Mello, “Improving automated source code summarization via an eye-tracking study of programmers,” in Proceedings of the 36th international conference on Software engineering.   ACM, 2014, pp. 390–401.
  21. Y.-G. Guéhéneuc, “Taupe: towards understanding program comprehension,” in Proceedings of the 2006 conference of the Center for Advanced Studies on Collaborative research, 2006, pp. 1–es.
  22. B. Sharif and J. I. Maletic, “The effects of layout on detecting the role of design patterns,” in Proceedings of the 2010 23rd IEEE Conference on Software Engineering Education and Training, ser. CSEET ’10.   Washington, DC, USA: IEEE Computer Society, 2010, pp. 41–48. [Online]. Available: http://dx.doi.org/10.1109/CSEET.2010.23
  23. D. Lübke, M. Ahrens, and K. Schneider, “Influence of diagram layout and scrolling on understandability of bpmn processes: an eye tracking experiment with bpmn diagrams,” Information Technology and Management, vol. 22, pp. 99–131, 2021.
  24. Z. Sharafi, B. Sharif, Y. Guéhéneuc, A. Begel, R. Bednarik, and M. E. Crosby, “A practical guide on conducting eye tracking studies in software engineering,” Empir. Softw. Eng., vol. 25, no. 5, pp. 3128–3174, 2020. [Online]. Available: https://doi.org/10.1007/s10664-020-09829-4
  25. R. Turner, M. Falcone, B. Sharif, and A. Lazar, “An eye-tracking study assessing the comprehension of c++ and python source code,” in Proceedings of the Symposium on Eye Tracking Research and Applications, 2014, pp. 231–234.
  26. S. Haiduc, J. Aponte, and A. Marcus, “Supporting program comprehension with source code summarization,” in Proceedings of the 32Nd ACM/IEEE International Conference on Software Engineering-Volume 2.   ACM, 2010, pp. 223–226.
  27. G. Sridhara, L. Pollock, and K. Vijay-Shanker, “Automatically detecting and describing high level actions within methods,” in Proceedings of the 33rd International Conference on Software Engineering.   ACM, 2011, pp. 101–110.
  28. P. W. McBurney, C. Liu, and C. McMillan, “Automated feature discovery via sentence selection and source code summarization,” Journal of Software: Evolution and Process, vol. 28, no. 2, pp. 120–145, 2016.
  29. B. Zhang, E. Hill, and J. Clause, “Towards automatically generating descriptive names for unit tests,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering.   ACM, 2016, pp. 625–636.
  30. S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016, pp. 2073–2083.
  31. X. Hu, G. Li, X. Xia, D. Lo, S. Lu, and Z. Jin, “Summarizing source code with transferred api knowledge,” in Proceedings of the 27th International Joint Conference on Artificial Intelligence.   AAAI Press, 2018, pp. 2269–2275.
  32. C.-Y. Su and C. McMillan, “Distilled gpt for source code summarization,” Automated Software Engineering, vol. 31, no. 1, p. 22, 2024.
  33. M. Geng, S. Wang, D. Dong, H. Wang, G. Li, Z. Jin, X. Mao, and X. Liao, “Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning,” 2024.
  34. C.-Y. Su, A. Bansal, V. Jain, S. Ghanavati, and C. Mcmillan, “A language model of java methods with train/test deduplication,” arXiv preprint arXiv:2305.08286, 2023.
  35. P. Zhang, Z. Liu, S. Xiao, N. Shao, Q. Ye, and Z. Dou, “Soaring from 4k to 400k: Extending llm’s context with activation beacon,” arXiv preprint arXiv:2401.03462, 2024.
  36. A. Bansal, C.-Y. Su, and C. McMillan, “Revisiting file context for source code summarization,” arXiv preprint arXiv:2309.02326, 2023.
  37. B. Clark and B. Sharif, “itracevis: Visualizing eye movement data within eclipse,” in 2017 IEEE Working Conference on Software Visualization (VISSOFT).   IEEE, 2017, pp. 22–32.
  38. T. R. Shaffer, J. L. Wise, B. M. Walters, S. C. Müller, M. Falcone, and B. Sharif, “Itrace: Enabling eye tracking on software artifacts within the ide to support software engineering tasks,” ser. ESEC/FSE 2015.   Association for Computing Machinery, 2015, p. 954–957.
  39. V. Zyrianov, D. T. Guarnera, C. S. Peterson, B. Sharif, and J. I. Maletic, “Automated recording and semantics-aware replaying of high-speed eye tracking and interaction data to support cognitive studies of software engineering tasks,” in 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2020, pp. 464–475.
  40. D. T. Guarnera, C. A. Bryant, A. Mishra, J. I. Maletic, and B. Sharif, “itrace: Eye tracking infrastructure for development environments,” in Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, 2018, pp. 1–3.
  41. J. Behler, G. Chiudioni, A. Ely, J. Pangonis, B. Sharif, and J. I. Maletic, “itrace-visualize: Visualizing eye-tracking data for software engineering studies,” in 2023 IEEE Working Conference on Software Visualization (VISSOFT), 2023, pp. 100–104.
  42. M. L. Collard, M. J. Decker, and J. I. Maletic, “Lightweight transformation and fact extraction with the srcml toolkit,” in 2011 IEEE 11th international working conference on source code analysis and manipulation.   IEEE, 2011, pp. 173–184.
  43. Tobii, “Reach further with your research: Choose tobii pro fusion,” 2023. [Online]. Available: https://www.tobii.com/products/eye-trackers/screen-based/tobii-pro-fusion
  44. G. Buscher, E. Cutrell, and M. R. Morris, “What do you see when you’re surfing? using eye tracking to predict salient regions of web pages,” ser. CHI ’09.   Association for Computing Machinery, 2009, p. 21–30.
  45. R. W. Booth and U. W. Weger, “The function of regressions in reading: Backward eye movements allow rereading,” Memory & cognition, vol. 41, pp. 82–97, 2013.
  46. P. Rodeghero and C. McMillan, “An empirical study on the patterns of eye movement during summarization tasks,” in 2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 2015, pp. 1–10.
  47. M. Son, J. Lee, and A. Godfroid, “Attention to form and meaning revisited: Insights from eye tracking,” Studies in Second Language Acquisition, vol. 44, no. 3, pp. 788–817, 2022.
  48. C. Treude, J. Middleton, and T. Atapattu, “Beyond accuracy: Assessing software documentation quality,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, pp. 1509–1512.
  49. S. Rastkar, G. C. Murphy, and G. Murray, “Automatic summarization of bug reports,” IEEE Transactions on Software Engineering, vol. 40, no. 4, pp. 366–380, 2014.
  50. A. Wood, P. Rodeghero, A. Armaly, and C. McMillan, “Detecting speech act types in developer question/answer conversations during bug repair,” in Proceedings of the 2018 26th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, 2018, pp. 491–502.
  51. T. Liu and T. Yuizono, “Mind mapping training’s effects on reading ability: Detection based on eye tracking sensors,” Sensors, vol. 20, no. 16, p. 4422, 2020.
  52. N. Nachar et al., “The mann-whitney u: A test for assessing whether two independent samples come from the same distribution,” Tutorials in quantitative Methods for Psychology, vol. 4, no. 1, pp. 13–20, 2008.
  53. J. Starke, C. Luce, and J. Sillito, “Searching and skimming: An exploratory study,” in 2009 IEEE International Conference on Software Maintenance.   IEEE, 2009, pp. 157–166.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 2 tweets with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free