Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

I see an IC: A Mixed-Methods Approach to Study Human Problem-Solving Processes in Hardware Reverse Engineering (2402.15452v1)

Published 23 Feb 2024 in cs.HC

Abstract: Trust in digital systems depends on secure hardware, often assured through Hardware Reverse Engineering (HRE). This work develops methods for investigating human problem-solving processes in HRE, an underexplored yet critical aspect. Since reverse engineers rely heavily on visual information, eye tracking holds promise for studying their cognitive processes. To gain further insights, we additionally employ verbal thought protocols during and immediately after HRE tasks: Concurrent and Retrospective Think Aloud. We evaluate the combination of eye tracking and Think Aloud with 41 participants in an HRE simulation. Eye tracking accurately identifies fixations on individual circuit elements and highlights critical components. Based on two use cases, we demonstrate that eye tracking and Think Aloud can complement each other to improve data quality. Our methodological insights can inform future studies in HRE, a specific setting of human-computer interaction, and in other problem-solving settings involving misleading or missing information.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (90)
  1. DANA Universal Dataflow Analysis for Gate-Level Netlist Reverse Engineering. IACR Transactions on Cryptographic Hardware and Embedded Systems (TCHES) 2020, 4 (2020), 309–336. https://doi.org/10.13154/tches.v2020.i4.309-336
  2. How can we ensure visibility and diversity in research contributions? How the Contributor Role Taxonomy (CRediT) is helping the shift from authorship to contributorship. Learned Publishing 32, 1 (2019), 71–74. https://doi.org/10.1002/leap.1210
  3. A Survey of Algorithmic Methods in IC Reverse Engineering. Journal of Cryptographic Engineering 11, 3 (2021), 299–315. https://doi.org/10.1007/s13389-021-00268-5
  4. Alan Baddeley. 1992. Working Memory. Science 255, 5044 (1992), 556–559. https://doi.org/10.1126/science.1736359
  5. An Exploratory Study of Hardware Reverse Engineering — Technical and Cognitive Processes. In Sixteenth Symposium on Usable Privacy and Security, SOUPS 2020, August 7-11, 2020. USENIX Association, Berkeley, CA, USA, 285–300. https://doi.org/10.1145/3577198
  6. ReverSim: A Game-Based Approach to Accessing Large Populations for Studying Human Aspects in Hardware Reverse Engineering. https://doi.org/10.48550/ARXIV.2309.05740 arXiv:2309.05740 [cs.CR]
  7. Roman Bednarik. 2012. Expertise-dependent visual attention strategies develop over time during debugging with multiple code representations. International Journal of Human-Computer Studies 70, 2 (2012), 143–155. https://doi.org/10.1016/j.ijhcs.2011.09.003
  8. Detecting personality traits using eye-tracking data. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow, Scotland, GB, 1–12. https://doi.org/10.1145/3290605.3300451
  9. Gabriel Biehal and Dipankar Chakravarti. 1989. The Effects of Concurrent Verbalization on Choice Processing. Journal of Marketing Research 26, 1 (1989), 84. https://doi.org/10.2307/3172671
  10. Triangulating user behavior using eye movement, interaction, and think aloud data. In Proceedings of the Ninth Biennial Symposium on Eye Tracking Research & Applications. ACM, Charleston, SC, USA, 175–182. https://doi.org/10.1145/2857491.2857523
  11. Layout Reconstruction of Complex Silicon Chips. IEEE Journal of Solid-State Circuits 28, 2 (1993), 138–145. https://doi.org/10.1109/4.192045
  12. Gerardo Canfora and Massimiliano Di Penta. 2007. New frontiers of reverse engineering. In Future of Software Engineering (FOSE’07). IEEE Computer Society, Minneapolis, MN, USA, 326–341. https://doi.org/10.1109/FOSE.2007.15
  13. Circuit Camouflage Integration for Hardware IP Protection. In The 51st Annual Design Automation Conference 2014, DAC ’14, San Francisco, CA, USA, June 1-5, 2014. ACM, San Francisco, CA, USA, 153:1–153:5. https://doi.org/10.1145/2593069.2602554
  14. L. Cooke and E. Cuddihy. 2005. Using Eye Tracking to Address Limitations in Think-Aloud Protocol. In IPCC 2005. Proceedings. International Professional Communication Conference, 2005. IEEE, Limerick, IE, 653–658. https://doi.org/10.1109/IPCC.2005.1494236
  15. Simon P. Davies. 1995. Effects of concurrent verbalization on design problem solving. Design Studies 16, 1 (1995), 102–116. https://doi.org/10.1016/0142-694x(95)90649-z
  16. Deep-SAGA: a deep-learning-based system for automatic gaze annotation from eye-tracking data. Behavior Research Methods 55, 3 (2023), 1372–1391. https://doi.org/10.3758/s13428-022-01833-4
  17. Eye Tracking in Retrospective Think-Aloud Usability Testing: Is There Added Value? J. Usability Studies 12, 3 (2017), 95–110. https://doi.org/10.5555/3190862.3190864
  18. Combining Concurrent Think-Aloud Protocols and Eye-Tracking Observations: An Analysis of Verbalizations and Silences. IEEE Transactions on Professional Communication 55, 3 (2012), 206–220. https://doi.org/10.1109/TPC.2012.2206190
  19. K. A. Ericsson and H. A. Simon. 1993. Protocol analysis: Verbal reports as data (Rev. ed.). The MIT Press, Cambridge, Massachusetts, USA. https://doi.org/10.7551/mitpress/5657.001.0001
  20. European Comission. 2022. A Chips Act for Europe – Comission Staff Working Document. https://digital-strategy.ec.europa.eu/en/library/european-chips-act-staff-working-document
  21. Jessica I. Fleck and Robert W. Weisberg. 2004. The use of verbal protocols as data: An analysis of insight in the candle problem. Memory & Cognition 32, 6 (2004), 990–1006. https://doi.org/10.3758/bf03196876
  22. Hardware Reverse Engineering: Overview and Open Challenges. In IEEE 2nd International Verification and Security Workshop, IVSW 2017, Thessaloniki, Greece, July 3-5, 2017. IEEE, Thessaloniki, GR, 88–94. https://doi.org/10.1109/IVSW.2017.8031550
  23. HAL – The Missing Piece of the Puzzle for Hardware Reverse Engineering, Trojan Detection and Insertion. IEEE Transactions on Dependable and Secure Computing 16, 3 (2019), 498–510. https://doi.org/10.1109/TDSC.2018.2812183
  24. Reverse engineering techniques applied to a human skull, for CAD 3D reconstruction and physical replication by rapid prototyping. Journal of Medical Engineering & Technology 30, 2 (2006), 102–111. https://doi.org/10.1080/03091900500131714
  25. Andreas Gegenfurtner and Marko Seppänen. 2013. Transfer of Expertise: An Eye Tracking and Think Aloud Study Using Dynamic Medical Visualizations. Computers & Education 63 (2013), 393–403. https://doi.org/10.1016/j.compedu.2012.12.021
  26. Joseph H. Goldberg and Jonathan I. Helfman. 2010. Comparing information graphics: a critical look at eye tracking. In Proceedings of the 3rd BELIV’10 Workshop: BEyond time and errors: novel evaLuation methods for Information Visualization. ACM, Atlanta, GA, USA, 71–78. https://doi.org/10.1145/2110192.2110203
  27. The validity of the stimulated retrospective think-aloud method as measured by eye tracking. In Proceedings of the SIGCHI conference on Human Factors in computing systems. ACM, Montréal, Québec, CA, 1253–1262. https://doi.org/10.1145/1124772.1124961
  28. Counterfeit Integrated Circuits: A Rising Threat in the Global Semiconductor Supply Chain. Proc. IEEE 102, 8 (2014), 1207–1228. https://doi.org/10.1109/JPROC.2014.2332291
  29. Noise-robust fixation detection in eye movement data: Identification by two-means clustering (I2MC). Behavior Research Methods 49, 5 (2017), 1802–1823. https://doi.org/10.3758/s13428-016-0822-1
  30. Sture Holm. 1979. A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics 6 (1979), 65–70. http://www.jstor.org/stable/4615733
  31. Improving and Analyzing Sketchy High-Fidelity Free-Eye Drawing. In Proceedings of the 2023 ACM Designing Interactive Systems Conference. ACM, Pittsburgh, PA, USA, 856–870. https://doi.org/10.1145/3563657.3596121
  32. Feasibility of Longitudinal Eye-Gaze Tracking in the Workplace. Proceedings of the ACM on Human-Computer Interaction 6, ETRA (2022), 1–21. https://doi.org/10.1145/3530889
  33. Marcel A. Just and Patricia A. Carpenter. 1980. A theory of reading: from eye fixations to comprehension. Psychological Review 87, 4 (1980), 329–354. https://doi.org/10.1037/0033-295x.87.4.329
  34. Alexandros Kafkas and Daniela Montaldi. 2015. The pupillary response discriminates between subjective and objective familiarity and novelty. Psychophysiology 52, 10 (2015), 1305–1316. https://doi.org/10.1111/psyp.12471
  35. Through (Tracking) Their Eyes: Abstraction and Complexity in Program Comprehension. ACM Transactions on Computing Education 22, 2 (2021), 1–33. https://doi.org/10.1145/3480171
  36. Just follow my eyes: The influence of model-observer similarity on Eye Movement Modeling Examples. Learning and Instruction 61 (2019), 126–137. https://doi.org/10.1016/j.learninstruc.2018.10.005
  37. Do prior knowledge, model-observer similarity and social comparison influence the effectiveness of eye movement modeling examples for supporting multimedia learning? Instructional Science 49, 5 (2021), 607–635. https://doi.org/10.1007/s11251-021-09552-7
  38. Klaus Krippendorff. 2019. Content Analysis: An Introduction to Its Methodology. SAGE Publications, Inc., Thousand Oaks,CA, USA. https://doi.org/10.4135/9781071878781
  39. Hannu Kuusela and Pallab Paul. 2000. A Comparison of Concurrent and Retrospective Verbal Protocol Analysis. The American Journal of Psychology 113, 3 (2000), 387. https://doi.org/10.2307/1423365
  40. Medical reverse engineering applications and methods. , 186-196 pages. http://gala.gre.ac.uk/id/eprint/11735/
  41. How prior knowledge affects problem-solving performance in a medical simulation game: Using game-logs and eye-tracking. Computers in Human Behavior 99 (2019), 268–277. https://doi.org/10.1016/j.chb.2019.05.035
  42. N. Y. Louis Lee and P. N. Johnson-Laird. 2013a. Strategic changes in problem solving. Journal of Cognitive Psychology 25, 2 (2013), 165–173. https://doi.org/10.1080/20445911.2012.719021
  43. N. Y. Louis Lee and P. N. Johnson-Laird. 2013b. A Theory of Reverse Engineering and its Application to Boolean Systems. Journal of Cognitive Psychology 25, 4 (2013), 365–389. https://doi.org/10.1080/20445911.2013.782033
  44. Children’s play and problem solving in motion-based educational games: Synergies between human annotations and multi-modal data. In Interaction Design and Children. Association for Computing Machinery, New York, NY, USA, 408–420. https://doi.org/10.1145/3459990.3460702
  45. Howard Levene et al. 1960. Robust tests for equality of variances. Contributions to probability and statistics. Essays in honor of Harold Hotelling 278 (1960), 292. https://doi.org/10.2307/2285659
  46. Hubert W. Lilliefors. 1967. On the Kolmogorov-Smirnov Test for Normality with Mean and Variance Unknown. J. Amer. Statist. Assoc. 62, 318 (1967), 399–402. https://doi.org/10.1080/01621459.1967.10482916
  47. Tracking students’ cognitive processes during program debugging—An eye-movement approach. IEEE Transactions on Education 59, 3 (2016), 175–186. https://doi.org/10.1109/TE.2015.2487341
  48. Päivi Majaranta and Andreas Bulling. 2014. Eye tracking and eye-based human–computer interaction. In Advances in physiological computing. Springer, London, GB, 39–65. https://doi.org/10.1007/978-1-4471-6392-3_3
  49. H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics 18, 1 (1947), 50 – 60. https://doi.org/10.1214/aoms/1177730491
  50. RE-Mind: a First Look Inside the Mind of a Reverse Engineer. In 31st USENIX Security Symposium (USENIX Security 22). USENIX Association, Berkeley, CA, USA, 2727–2745. https://www.usenix.org/conference/usenixsecurity22/presentation/mantovani
  51. Netlist Reverse Engineering for High-Level Functionality Reconstruction. In 21st Asia and South Pacific Design Automation Conference, ASP-DAC 2016, Macao, Macao, January 25-28, 2016. IEEE, Macao, CN, 655–660. https://doi.org/10.1109/ASPDAC.2016.7428086
  52. A survey on the usage of eye-tracking in computer programming. Comput. Surveys 51, 1 (2018), 1–58. https://doi.org/10.1145/3145904
  53. openai.com. 2024. DALL-E 3. https://openai.com/dall-e-3. [Online; accessed 2024-February-21].
  54. Estimating cognitive load using remote eye tracking in a driving simulator. In Proceedings of the 2010 symposium on eye-tracking research & applications. ACM, New York, NY, USA, 141–144. https://doi.org/10.1145/1743666.1743701
  55. Using eye-tracking to unveil differences between kids and teens in coding activities. In proceedings of the 2017 conference on interaction design and children. ACM, New York, NY, USA, 171–181. https://doi.org/10.1145/3078072.3079740
  56. Karl Pearson. 1900. X. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 50, 302 (July 1900), 157–175. https://doi.org/10.1080/14786440009463897
  57. Impact of Think-Aloud on Eye-Tracking: A Comparison of Concurrent and Retrospective Think-Aloud for Research on Decision-Making in the Game Environment. Sensors 20, 10 (2020), 2750. https://doi.org/10.3390/s20102750
  58. Red Team vs. Blue Team: A Real-World Hardware Trojan Detection Case Study Across Four Modern CMOS Technology Generations. In 2023 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, Los Alamitos, CA, USA, 56–74. https://doi.org/10.1109/SP46215.2023.00044
  59. M. G. Rekoff. 1985. On reverse engineering. IEEE Transactions on Systems, Man, and Cybernetics 15, 2 (1985), 244–252. https://doi.org/10.1109/TSMC.1985.6313354
  60. Combination of Eye Tracking and Think-Aloud Methods in Engineering Design Research. In Design Computing and Cognition '14. Springer International Publishing, Cham, DE, 81–97. https://doi.org/10.1007/978-3-319-14956-1_5
  61. Scanning and Deep Processing of Information in Hypertext: An Eye Tracking and Cued Retrospective Think-aloud Study. Journal of Computer Assisted Learning 33, 3 (2017), 222–233. https://doi.org/10.1111/jcal.12152
  62. Dario D. Salvucci and Joseph H. Goldberg. 2000. Identifying fixations and saccades in eye-tracking protocols. In Proceedings of the 2000 symposium on Eye tracking research & applications. ACM, New York, NY, USA, 71–78. https://doi.org/10.1145/355017.355028
  63. Texplained SARL. 2021. Chipjuice IC Reverse Engineering Software. https://www.texplained.com/about-us/chipjuice-software/. [Online; accessed 2024-February-22].
  64. Senate of the United States. 2022. CHIPS and Science Act 2022 (P.L. 117-167). https://www.congress.gov/bill/117th-congress/house-bill/4346/text
  65. Covert Gates: Protecting Integrated Circuits with Undetectable Camouflaging. IACR Transactions on Cryptographic Hardware and Embedded Systems (TCHES) 2019, 3 (2019), 86–118. https://doi.org/10.13154/tches.v2019.i3.86-118
  66. Eyes on Code: A Study on Developers’ Code Navigation Strategies. IEEE Transactions on Software Engineering 48, 5 (2022), 1692–1704. https://doi.org/10.1109/TSE.2020.3032064
  67. Toward an objective measure of developers’ cognitive activities. ACM Transactions on Software Engineering and Methodology 30, 3 (2021), 1–40. https://doi.org/10.1145/3434643
  68. Eye-tracking metrics in software engineering. In 2015 Asia-Pacific Software Engineering Conference (APSEC). IEEE Computer Society, New Dehli,IN, 96–103. https://doi.org/10.1109/APSEC.2015.53
  69. A practical guide on conducting eye tracking studies in software engineering. Empirical Software Engineering 25 (2020), 3128–3174. https://doi.org/10.1007/S10664-020-09829-4
  70. A systematic literature review on the usage of eye-tracking in software engineering. Information and Software Technology 67 (2015), 79–107. https://doi.org/10.1016/j.infsof.2015.06.008
  71. Review of eye tracking metrics involved in emotional and cognitive processes. IEEE Reviews in Biomedical Engineering 16 (2021), 260–277. https://doi.org/10.1109/RBME.2021.3066072
  72. Combining eye tracking, pupil dilation and EEG analysis for predicting web users click intention. Information Fusion 35 (2017), 51–57. https://doi.org/10.1016/j.inffus.2016.09.003
  73. Reverse Engineering Digital Circuits Using Structural and Functional Analyses. IEEE Transactions on Emerging Topics in Computing 2, 1 (2014), 63–80. https://doi.org/10.1109/TETC.2013.2294918
  74. K. Lynn Taylor and Jean-Paul Dionne. 2000. Accessing problem-solving strategy knowledge: The complementary use of concurrent verbal protocols and retrospective debriefing. Journal of Educational Psychology 92, 3 (2000), 413–425. https://doi.org/10.1037/0022-0663.92.3.413
  75. tobii.com. 2023. Tobii Pro Eye Tracker Manager. https://www.tobii.com/products/software/applications-and-developer-kits/tobii-pro-eye-tracker-manager#overview. [Online; accessed 2024-February-21].
  76. Randy Torrance and Dick James. 2009. The State-of-the-Art in IC Reverse Engineering. In Cryptographic Hardware and Embedded Systems - CHES 2009, 11th International Workshop, Lausanne, Switzerland, September 6-9, 2009, Proceedings. Springer, Berlin, DE, 363–381. https://doi.org/10.1007/978-3-642-04138-9_26
  77. Analyzing individual performance of source code review using reviewers’ eye movement. In Proceedings of the 2006 symposium on Eye tracking research & applications. Association for Computing Machinery, New York, NY, USA, 133–140. https://doi.org/10.1145/1117309.1117357
  78. Uncovering the Problem-Solving Process: Cued Retrospective Reporting Versus Concurrent and Retrospective Reporting. Journal of Experimental Psychology: Applied 11, 4 (2005), 237–244. https://doi.org/10.1037/1076-898x.11.4.237
  79. Wearable eye tracking for mental health monitoring. Computer Communications 35, 11 (2012), 1306–1311. https://doi.org/10.1016/j.comcom.2011.11.002
  80. Highway to HAL: Open-sourcing the First Extendable Gate-level Netlist Reverse Engineering Framework. In Proceedings of the 16th ACM International Conference on Computing Frontiers, CF 2019, Alghero, Italy, April 30 - May 2, 2019, Francesca Palumbo, Michela Becchi, Martin Schulz, and Kento Sato (Eds.). ACM, New York, NY, USA, 392–397. https://doi.org/10.1145/3310273.3323419
  81. John B. Watson. 2009. Is thinking merely the action of language mechanisms? British Journal of Psychology 100, S1 (2009), 169–180. https://doi.org/10.1348/000712608x336095
  82. B. L. Welch. 1947. The Generalization of ‘Student's’ Problem When Several Different Population Variances are Involved. Biometrika 34, 1-2 (1947), 28–35. https://doi.org/10.1093/biomet/34.1-2.28
  83. Towards Cognitive Obfuscation: Impeding Hardware Reverse Engineering based on Psychological Insights. In Proceedings of the 24th Asia and South Pacific Design Automation Conference, ASPDAC 2019, Tokyo, Japan, January 21-24, 2019, Toshiyuki Shibuya (Ed.). ACM, New York, NY, USA, 104–111. https://doi.org/10.1145/3287624.3288741
  84. Promoting the Acquisition of Hardware Reverse Engineering Skills. In IEEE Frontiers in Education Conference, FIE 2019, Cincinnati, OH, USA, October 16-19, 2019. IEEE, Cincinnati, OH, USA, 1–9. https://doi.org/10.1109/FIE43999.2019.9028668
  85. Teaching Hardware Reverse Engineering: Educational Guidelines and Practical Insights. In IEEE International Conference on Teaching, Assessment, and Learning for Engineering, TALE 2018, Wollongong, Australia, December 4-7, 2018. IEEE, Wollongong, NSW, AU, 438–445. https://doi.org/10.1109/TALE.2018.8615270
  86. The Anatomy of Hardware Reverse Engineering: An Exploration of Human Factors During Problem Solving. Comput.-Hum. Interact 30, 4 (2023), 62:1–62:44. https://doi.org/10.1145/3577198
  87. Cyber-Physical System Discovery: Reverse Engineering Physical Processes. In Proceedings of the 3rd ACM Workshop on Cyber-Physical System Security (ASIA CCS ’17). ACM, New York, NY, USA, 3–14. https://doi.org/10.1145/3055186.3055195
  88. Daesub Yoon and N. Hari Narayanan. 2004. Mental imagery in problem solving: an eye tracking study. In Proceedings of the Eye Tracking Research & Application Symposium, ETRA 2004, San Antonio, Texas, USA, March 22-24, 2004. ACM, New York, NY, USA, 77–84. https://doi.org/10.1145/968363.968382
  89. Jiliang Zhang. 2015. A practical logic obfuscation technique for hardware security. IEEE Transactions on very large scale integration (VLSI) systems 24, 3 (2015), 1193–1197. https://doi.org/10.1109/TVLSI.2015.2437996
  90. Evaluation of appearance-based methods and implications for gaze-based applications. In Proceedings of the 2019 CHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300646

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets