Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

EyeTrans: Merging Human and Machine Attention for Neural Code Summarization (2402.14096v3)

Published 21 Feb 2024 in cs.SE, cs.AI, and cs.HC

Abstract: Neural code summarization leverages deep learning models to automatically generate brief natural language summaries of code snippets. The development of Transformer models has led to extensive use of attention during model design. While existing work has primarily and almost exclusively focused on static properties of source code and related structural representations like the Abstract Syntax Tree (AST), few studies have considered human attention, that is, where programmers focus while examining and comprehending code. In this paper, we develop a method for incorporating human attention into machine attention to enhance neural code summarization. To facilitate this incorporation and vindicate this hypothesis, we introduce EyeTrans, which consists of three steps: (1) we conduct an extensive eye-tracking human study to collect and pre-analyze data for model training, (2) we devise a data-centric approach to integrate human attention with machine attention in the Transformer architecture, and (3) we conduct comprehensive experiments on two code summarization tasks to demonstrate the effectiveness of incorporating human attention into Transformers. Integrating human attention leads to an improvement of up to 29.91% in Functional Summarization and up to 6.39% in General Code Summarization performance, demonstrating the substantial benefits of this combination. We further explore performance in terms of robustness and efficiency by creating challenging summarization scenarios in which EyeTrans exhibits interesting properties. We also visualize the attention map to depict the simplifying effect of machine attention in the Transformer by incorporating human attention. This work has the potential to propel AI research in software engineering by introducing more human-centered approaches and data.

View on arXiv
Definition Search Book Streamline Icon: https://streamlinehq.com
References (77)
  1. 2023. https://go.tobii.com/tobii-pro-fusion-user-manual
  2. Using developer eye movements to externalize the mental model used in code summarization tasks. In Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications. 1–9.
  3. Developer reading behavior while summarizing java methods: Size and context matters. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 384–395.
  4. How Do We Read Formal Claims? Eye-Tracking and the Cognition of Proofs about Algorithms. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 208–220.
  5. A transformer-based approach for source code summarization. arXiv preprint arXiv:2005.00653 (2020).
  6. An empirical study on the importance of source code entities for requirements traceability. Empirical software engineering 20 (2015), 442–478.
  7. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400 (2018).
  8. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).
  9. Towards Modeling Human Attention from Eye Movements for Neural Source Code Summarization. Proceedings of the ACM on Human-Computer Interaction 7, ETRA (2023), 1–19.
  10. Modeling Programmer Attention as Scanpath Prediction. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1732–1736.
  11. Sequence classification with human attention. In Proceedings of the 22nd conference on computational natural language learning. 302–312.
  12. Roman Bednarik. 2012. Expertise-dependent Visual Attention Strategies Develop over Time During Debugging with Multiple Code Representations. International Journal of Human-Computer Studies 70, 2 (Feb. 2012), 143–155. https://doi.org/10.1016/j.ijhcs.2011.09.003
  13. Jean-Francois Bergeretti and Bernard A Carré. 1985. Information-flow and data-flow analysis of while-programs. ACM Transactions on Programming Languages and Systems (TOPLAS) 7, 1 (1985), 37–61.
  14. Birtukan Birawo and Pawel Kasprowski. 2022. Review and evaluation of eye movement event detection algorithms. Sensors 22, 22 (2022), 8810.
  15. Egon Börger and Wolfram Schulte. 1999. A programmer friendly modular definition of the semantics of Java. Formal Syntax and Semantics of Java (1999), 353–404.
  16. Eye movements in code reading: Relaxing the linear order. In 2015 IEEE 23rd International Conference on Program Comprehension. IEEE, 255–265.
  17. srcml: An infrastructure for the exploration, analysis, and manipulation of source code: A tool demonstration. In 2013 IEEE International conference on software maintenance. IEEE, 516–519.
  18. The Roles Beacons Play in Comprehension for Novice and Expert Programmers.. In PPIG. 5.
  19. Zhengcong Fei. 2022. Attention-aligned transformer for image captioning. In proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 607–615.
  20. Code Structure–Guided Transformer for Source Code Summarization. ACM Transactions on Software Engineering and Methodology 32, 1 (2023), 1–32.
  21. Source code summarization with structural relative position guided transformer. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 13–24.
  22. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366 (2020).
  23. Attention mechanisms in computer vision: A survey. Computational visual media 8, 3 (2022), 331–368.
  24. Supporting program comprehension with source code summarization. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 2. 223–226.
  25. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  26. Deep code comment generation. In Proceedings of the 26th conference on program comprehension. 200–210.
  27. Biases and differences in code review using medical imaging and eye-tracking: genders, humans, and machines. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 456–468.
  28. Multi-modal reference resolution in situated dialogue by integrating linguistic and extra-linguistic clues. In Proceedings of 5th International Joint Conference on Natural Language Processing. 84–92.
  29. Summarizing source code using a neural attention model. In 54th Annual Meeting of the Association for Computational Linguistics 2016. Association for Computational Linguistics, 2073–2083.
  30. Marcel A Just and Patricia A Carpenter. 1980. A theory of reading: from eye fixations to comprehension. Psychological review 87, 4 (1980), 329.
  31. Gaze embeddings for zero-shot image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4525–4534.
  32. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
  33. Improving sentence compression by learning to predict gaze. arXiv preprint arXiv:1604.03357 (2016).
  34. Sigrid Klerke and Barbara Plank. 2019. At a glance: The impact of gaze aggregation views on syntactic tagging. In Proceedings of the Beyond Vision and LANguage: inTEgrating Real-world kNowledge (LANTERN). 51–61.
  35. Autopruner: transformer-based call graph pruning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 520–532.
  36. Improved code summarization via a graph neural network. In Proceedings of the 28th international conference on program comprehension. 184–195.
  37. Alexander LeClair and Collin McMillan. 2019. Recommendations for datasets for source code summarization. arXiv preprint arXiv:1904.02660 (2019).
  38. Improving code summarization with block-wise abstract syntax tree splitting. In 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC). IEEE, 184–195.
  39. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81.
  40. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019).
  41. Sun Developer Network. 1999. Code conventions for the Java programming language.
  42. A review on the attention mechanism of deep learning. Neurocomputing 452 (2021), 48–62.
  43. A Survey on the Usage of Eye-Tracking in Computer Programming. ACM Comput. Surv. 51, 1, Article 5 (Jan. 2018), 58 pages. https://doi.org/10.1145/3145904
  44. Anneli Olsen. 2012. The Tobii I-VT fixation filter. Tobii Technology 21 (2012), 4–19.
  45. What drives the reading order of programmers? an eye tracking study. In Proceedings of the 28th International Conference on Program Comprehension. 342–353.
  46. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 (2023).
  47. Exploring human-like attention supervision in visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.
  48. The EZ Reader model of eye-movement control in reading: Comparisons to other models. Behavioral and brain sciences 26, 4 (2003), 445–476.
  49. An eye-tracking study of java programmers and application to source code summarization. IEEE Transactions on Software Engineering 41, 11 (2015), 1038–1054.
  50. Paige Rodeghero and Collin McMillan. 2015. An empirical study on the patterns of eye movement during summarization tasks. In 2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 1–10.
  51. Improving automated source code summarization via an eye-tracking study of programmers. In Proceedings of the 36th international conference on Software engineering. 390–401.
  52. Using gaze data to predict multiword expressions. (2017).
  53. Herbert Schildt. 2007. Java: the complete reference. (2007).
  54. itrace: Enabling eye tracking on software artifacts within the ide to support software engineering tasks. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. 954–957.
  55. Toward an Objective Measure of Developers’ Cognitive Activities. ACM Transactions on Software Engineering and Methodology (TOSEM) 30, 3 (2021), 1–40.
  56. Eye-tracking metrics in software engineering. In 2015 Asia-Pacific Software Engineering Conference (APSEC). IEEE, 96–103.
  57. A practical guide on conducting eye tracking studies in software engineering. Empirical Software Engineering 25 (2020), 3128–3174.
  58. On the evaluation of neural code summarization. In Proceedings of the 44th International Conference on Software Engineering. 1597–1608.
  59. Cast: Enhancing code summarization with hierarchical splitting and reconstruction of abstract syntax trees. arXiv preprint arXiv:2108.12987 (2021).
  60. Improving natural language processing tasks with human gaze-guided neural attention. Advances in Neural Information Processing Systems 33 (2020), 6327–6341.
  61. A human study of comprehension and code summarization. In Proceedings of the 28th International Conference on Program Comprehension. 2–13.
  62. Yusuke Sugano and Andreas Bulling. 2016. Seeing with humans: Gaze-assisted neural image captioning. arXiv preprint arXiv:1608.05203 (2016).
  63. Jerry Chih-Yuan Sun and Kelly Yi-Chuan Hsu. 2019. A smart eye-tracking feedback scaffolding approach to improving students’ learning self-efficacy and performance in a C programming course. Computers in Human Behavior 95 (2019), 66–72.
  64. AST-transformer: Encoding abstract syntax trees efficiently for code summarization. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1193–1195.
  65. AST-trans: Code summarization with efficient tree-structured attention. In Proceedings of the 44th International Conference on Software Engineering. 150–162.
  66. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).
  67. Attention is all you need. Advances in neural information processing systems 30 (2017).
  68. Jtrans: Jump-aware transformer for binary code similarity detection. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 1–13.
  69. Code summarization with structure-induced transformer. arXiv preprint arXiv:2012.14710 (2020).
  70. Gaze-enabled egocentric video summarization via constrained submodular maximization. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2235–2244.
  71. Classifying referential and non-referential it using gaze. arXiv preprint arXiv:2006.13327 (2020).
  72. Supervising neural attention models for video captioning by human gaze data. In Proceedings of the IEEE conference on computer vision and pattern recognition. 490–498.
  73. An extensive study on pre-trained models for program understanding and generation. In Proceedings of the 31st ACM SIGSOFT international symposium on software testing and analysis. 39–51.
  74. Yifan Zhang. [n. d.]. Reproduction Package for the FSE 2024 Paper “EyeTrans: Merging Human and Machine Attention for Neural Code Summarization”. https://doi.org/10.5281/zenodo.10684985
  75. Yingyi Zhang and Chengzhi Zhang. 2019. Using human attention to extract keyphrase from microblog post. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 5867–5872.
  76. QueryFormer: a tree transformer model for query plan representation. Proceedings of the VLDB Endowment 15, 8 (2022), 1658–1670.
  77. kTrans: Knowledge-Aware Transformer for Binary Code Embedding. arXiv preprint arXiv:2308.12659 (2023).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Yifan Zhang (245 papers)
  2. Jiliang Li (10 papers)
  3. Zachary Karas (6 papers)
  4. Aakash Bansal (22 papers)
  5. Toby Jia-Jun Li (57 papers)
  6. Collin McMillan (38 papers)
  7. Kevin Leach (29 papers)
  8. Yu Huang (176 papers)
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now