Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The Metacognitive Demands and Opportunities of Generative AI (2312.10893v3)

Published 18 Dec 2023 in cs.HC

Abstract: Generative AI (GenAI) systems offer unprecedented opportunities for transforming professional and personal work, yet present challenges around prompting, evaluating and relying on outputs, and optimizing workflows. We argue that metacognition$\unicode{x2013}$the psychological ability to monitor and control one's thoughts and behavior$\unicode{x2013}$offers a valuable lens to understand and design for these usability challenges. Drawing on research in psychology and cognitive science, and recent GenAI user studies, we illustrate how GenAI systems impose metacognitive demands on users, requiring a high degree of metacognitive monitoring and control. We propose these demands could be addressed by integrating metacognitive support strategies into GenAI systems, and by designing GenAI systems to reduce their metacognitive demand by targeting explainability and customizability. Metacognition offers a coherent framework for understanding the usability challenges posed by GenAI, and provides novel research and design directions to advance human-AI interaction.

PDF HTML Abstract

Metacognitive Demands and Opportunities of Generative AI: A Comprehensive Analysis

The paper "The Metacognitive Demands and Opportunities of Generative AI" presents an in-depth examination of the intersection between metacognition and generative AI (GenAI) systems, highlighting the psychological challenges and design opportunities those systems introduce. Authored by Lev Tankelevitch et al., this paper leverages extensive research in psychology, cognitive science, and user studies on GenAI to offer a novel perspective on the usability of these technologies.

Central to the argument is the assertion that GenAI places substantial metacognitive demands on users, particularly in the domains of prompting, output evaluation, and workflow integration. Metacognition, defined here as the psychological ability to monitor and control one's own cognitive processes, emerges as a crucial lens for understanding how users interact with and can be better supported in using GenAI systems.

Metacognitive Demands of Generative AI

The authors categorize the metacognitive demands into three main areas:

  1. Prompting: Users must effectively articulate task goals and decompose these into suitable prompts for GenAI systems. This requires significant metacognitive monitoring, where self-awareness of goals and strategies is paramount. Users often struggle due to the models’ non-deterministic nature and flexibility, which mandates adaptive prompting strategies—skills that hinge on metacognitive abilities.
  2. Evaluating and Relying on Outputs: Users should maintain well-calibrated confidence in assessing AI-generated outputs. This depends on recognizing the system’s unpredictability and understanding its limitations. Established findings in AI-assisted decision-making suggest poorly calibrated confidence can hinder users' trust and lead to suboptimal reliance on AI outputs.
  3. Automation Strategy: At a higher level, deciding when and how to integrate GenAI into workflows requires substantial metacognitive oversight. Users must evaluate the applicability of AI to their tasks and adjust their workflows dynamically, which demands ongoing self-awareness and cognitive flexibility.

Addressing Metacognitive Demands

The paper proposes dual strategies to address these demands:

  • Improving Users' Metacognition: This involves integrating metacognitive support strategies into GenAI systems, such as providing planning aids, reflective prompts, and feedback mechanisms for task decomposition and evaluation strategies. These interventions can help users build self-awareness and task-specific metacognitive strategies.
  • Reducing Metacognitive Demands: The authors advocate for systemic design changes to decrease the cognitive load on users. Enhancements in GenAI explainability and customizability can offer users actionable insights into system operation and capabilities. Interactive explanations and user-tailored interface settings can significantly reduce the metacognitive load, enabling more efficient human-AI interaction.

Implications and Future Directions

On a theoretical level, the emphasis on metacognition provides a framework for exploring the cognitive components of human-AI interaction, encouraging interdisciplinary research that could refine our understanding of cognitive processes in digital environments. Practically, the proposed strategies for enhancing GenAI systems could lead to more intuitive and efficient interactions, reducing barriers to adoption and facilitating a broader range of applications in professional and personal domains.

Looking ahead, the paper suggests that advancing metacognitive support within GenAI holds the potential to transform user experiences by aligning system design with human cognitive patterns. Future research might explore optimizing the balance between cognitive load and metacognitive support, as well as leveraging the unique properties of GenAI to enhance system usability further.

In conclusion, by situating metacognition at the core of human-GenAI interaction, this paper not only provides a thorough analysis of current challenges but also points toward innovative directions for future development. Through enhanced support and system design improvements, generative AI has the potential to become a more integrated and effective tool in human cognitive toolkits, ushering in new possibilities for collaboration between humans and AI.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (214)
  1. The Power of Nudging: Exploring Three Interventions for Metacognitive Skills Instruction across Intelligent Tutoring Systems. https://doi.org/10.48550/arXiv.2303.11965 arXiv:2303.11965 [cs].
  2. Rakefet Ackerman. 2014. The diminishing criterion model for metacognitive regulation of time investment. Journal of Experimental Psychology: General 143, 3 (2014), 1349–1368. https://doi.org/10.1037/a0035098 Place: US Publisher: American Psychological Association.
  3. Rakefet Ackerman. 2019. Heuristic Cues for Meta-Reasoning Judgments: Review and Methodology. Psihologijske teme 28, 1 (May 2019), 1–20. https://doi.org/10.31820/pt.28.1.1 Publisher: Filozofski fakultet u Rijeci.
  4. Rakefet Ackerman and Valerie Thompson. 2017. Meta-Reasoning: Shedding meta-cognitive light on reasoning research. 1–15.
  5. Quality Estimation & Interpretability for Code Translation. https://doi.org/10.48550/arXiv.2012.07581 arXiv:2012.07581 [cs].
  6. To Plan or Not to Plan? A Mixed-Methods Diary Study Examining When, How and Why Knowledge Work Planning is Inaccurate. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (Jan. 2021), 222:1–222:20. https://doi.org/10.1145/3432921
  7. Overcoming intuition: metacognitive difficulty activates analytic reasoning. Journal of Experimental Psychology. General 136, 4 (Nov. 2007), 569–576. https://doi.org/10.1037/0096-3445.136.4.569
  8. Amine Amzil. 2013. The Effect of a Metacognitive Intervention on College Students’ Reading Performance and Metacognitive Skills. Journal of Educational and Developmental Psychology 4, 1 (Dec. 2013), p27. https://doi.org/10.5539/jedp.v4n1p27
  9. Heidi Goodrich Andrade. 2000. Using Rubrics To Promote Thinking and Learning. Educational Leadership 57, 5 (2000), 13–18. ERIC Number: EJ609600.
  10. One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques. https://doi.org/10.48550/arXiv.1909.03012 arXiv:1909.03012 [cs, stat].
  11. Roger Azevedo. 2020. Reflections on the field of metacognition: issues, challenges, and opportunities. Metacognition and Learning 15, 2 (Aug. 2020), 91–98. https://doi.org/10.1007/s11409-020-09231-x
  12. Lisanne Bainbridge. 1983. Ironies of automation. Automatica 19, 6 (Nov. 1983), 775–779. https://doi.org/10.1016/0005-1098(83)90046-8
  13. Albert Bandura. 1997. Self-efficacy: The exercise of control. W H Freeman/Times Books/ Henry Holt & Co, New York, NY, US. Pages: ix, 604.
  14. Grounded Copilot: How Programmers Interact with Code-Generating Models. Proceedings of the ACM on Programming Languages 7, OOPSLA1 (April 2023), 78:85–78:111. https://doi.org/10.1145/3586030
  15. Systematic metacognitive reflection helps people discover far-sighted decision strategies: A process-tracing experiment. Judgment and Decision Making 18 (Jan. 2023), e15. https://doi.org/10.1017/jdm.2023.16 Publisher: Cambridge University Press.
  16. The mismeasure of memory: When retrieval fluency is misleading as a metamnemonic index. Journal of Experimental Psychology: General 127, 1 (1998), 55–68. https://doi.org/10.1037/0096-3445.127.1.55 Place: US Publisher: American Psychological Association.
  17. Revisiting Reflection in HCI: Four Design Resources for Technologies that Support Reflection. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 1 (March 2022), 2:1–2:27. https://doi.org/10.1145/3517233
  18. On Selective, Mutable and Dialogic XAI: a Review of What Users Say about Different Types of Interactive Explanations. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–21. https://doi.org/10.1145/3544548.3581314
  19. Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (AIES ’21). Association for Computing Machinery, New York, NY, USA, 401–413. https://doi.org/10.1145/3461702.3462571
  20. A.F. Blackwell. 2002. First steps in programming: a rationale for attention investment models. In Proceedings IEEE 2002 Symposia on Human Centric Computing Languages and Environments. 2–10. https://doi.org/10.1109/HCC.2002.1046334
  21. Monique Boekaerts and Lyn Corno. 2005. Self-Regulation in the Classroom: A Perspective on Assessment and Intervention. Applied Psychology: An International Review 54, 2 (2005), 199–231. https://doi.org/10.1111/j.1464-0597.2005.00205.x Place: United Kingdom Publisher: Blackwell Publishing.
  22. Annika Boldt and Sam J. Gilbert. 2019. Confidence guides spontaneous cognitive offloading. Cognitive Research: Principles and Implications 4, 1 (Dec. 2019), 45. https://doi.org/10.1186/s41235-019-0195-y
  23. On the Opportunities and Risks of Foundation Models. https://doi.org/10.48550/arXiv.2108.07258 arXiv:2108.07258 [cs].
  24. Mimi Bong and Einar M. Skaalvik. 2003. Academic Self-Concept and Self-Efficacy: How Different Are They Really? Educational Psychology Review 15, 1 (March 2003), 1–40. https://doi.org/10.1023/A:1021302408382
  25. Follow the Successful Herd: Towards Explanations for Improved Use and Mental Models of Natural Language Systems. In Proceedings of the 28th International Conference on Intelligent User Interfaces (IUI ’23). Association for Computing Machinery, New York, NY, USA, 220–239. https://doi.org/10.1145/3581641.3584088
  26. How to Support Users in Understanding Intelligent Systems? An Analysis and Conceptual Framework of User Questions Considering User Mindsets, Involvement, and Knowledge Outcomes. ACM Transactions on Interactive Intelligent Systems 12, 4 (Nov. 2022), 29:1–29:27. https://doi.org/10.1145/3519264
  27. Deborah L. Butler. 1998. The strategic content learning approach to promoting self-regulated learning: A report of three studies. Journal of Educational Psychology 90, 4 (1998), 682–697. https://doi.org/10.1037/0022-0663.90.4.682 Place: US Publisher: American Psychological Association.
  28. Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300234
  29. Better beware: comparing metacognition for phishing and legitimate emails. Metacognition and Learning 14, 3 (Dec. 2019), 343–362. https://doi.org/10.1007/s11409-019-09197-5
  30. Domain-General Enhancements of Metacognitive Ability Through Adaptive Training. Journal of Experimental Psychology. General 148, 1 (Jan. 2019), 51–64. https://doi.org/10.1037/xge0000505
  31. Cameron S. Carter and Vincent van Veen. 2007. Anterior cingulate cortex and conflict detection: An update of theory and data. Cognitive, Affective, & Behavioral Neuroscience 7, 4 (Dec. 2007), 367–379. https://doi.org/10.3758/CABN.7.4.367
  32. Cognitive inflexibility and the development and use of strategies for solving complex dynamic problems: effects of different types of training. Theoretical Issues in Ergonomics Science 6, 1 (Jan. 2005), 95–108. https://doi.org/10.1080/14639220512331311599 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/14639220512331311599.
  33. Machine Explanations and Human Understanding. https://doi.org/10.48550/arXiv.2202.04092 arXiv:2202.04092 [cs].
  34. Next Steps for Human-Centered Generative AI: A Technical Perspective. https://doi.org/10.48550/arXiv.2306.15774 arXiv:2306.15774 [cs].
  35. Human confidence in artificial intelligence and in themselves: The evolution and impact of confidence on adoption of AI advice. Computers in Human Behavior 127 (Feb. 2022), 107018. https://doi.org/10.1016/j.chb.2021.107018
  36. David Church and Mark Carroll. 2023. How does metacognition improve decision-making in healthcare practitioners? Journal of Paramedic Practice 15, 3 (March 2023), 113–123. https://doi.org/10.12968/jpar.2023.15.3.113 Publisher: Mark Allen Group.
  37. Timothy J. Cleary and Barry J. Zimmerman. 2004. Self-Regulation Empowerment Program: A school-based program to enhance self-regulated and self-motivated cycles of student learning. Psychology in the Schools 41, 5 (May 2004), 537–550. https://doi.org/10.1002/pits.10177 Publisher: Wiley-Liss Inc..
  38. Supporting Metacognition during Exploratory Search with the OrgBox. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’21). Association for Computing Machinery, New York, NY, USA, 1197–1207. https://doi.org/10.1145/3404835.3462955
  39. David R. Cross and Scott G. Paris. 1988. Developmental and instructional analyses of children’s metacognition and reading comprehension. Journal of Educational Psychology 80, 2 (1988), 131–142. https://doi.org/10.1037/0022-0663.80.2.131 Place: US Publisher: American Psychological Association.
  40. GitHub Copilot AI pair programmer: Asset or Liability? https://doi.org/10.48550/arXiv.2206.15331 arXiv:2206.15331 [cs].
  41. Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–17. https://doi.org/10.1145/3544548.3580969
  42. GANSlider: How Users Control Generative Models for Images using Multiple Sliders with and without Feedforward Information. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3491102.3502141
  43. How to Prompt? Opportunities and Challenges of Zero- and Few-Shot Learning for Human-AI Interaction in Creative Applications of Generative Models. https://doi.org/10.48550/arXiv.2209.01390 arXiv:2209.01390 [cs].
  44. Changing minds about climate change: a pervasive role for domain-general metacognition. Humanities and Social Sciences Communications 10, 1 (Feb. 2023), 1–10. https://doi.org/10.1057/s41599-023-01528-x Number: 1 Publisher: Palgrave.
  45. Long-term effects of metacognitive strategy instruction on student academic performance: A meta-analysis. Educational Research Review 24 (June 2018), 98–115. https://doi.org/10.1016/j.edurev.2018.03.002
  46. Synthesizing Cognitive Load and Self-regulation Theory: a Theoretical Framework and Research Agenda. Educational Psychology Review 32, 4 (Dec. 2020), 903–915. https://doi.org/10.1007/s10648-020-09576-4
  47. The effects of errors on system trust, self-confidence, and the allocation of control in route planning. International Journal of Human-Computer Studies 58, 6 (June 2003), 719–735. https://doi.org/10.1016/S1071-5819(03)00039-9
  48. Victor R. Delclos and Christine Harrington. 1991. Effects of strategy monitoring and proactive instruction on children’s problem-solving performance. Journal of Educational Psychology 83, 1 (1991), 35–42. https://doi.org/10.1037/0022-0663.83.1.35 Place: US Publisher: American Psychological Association.
  49. Subjective Confidence Predicts Information Seeking in Decision Making. Psychological Science 29, 5 (May 2018), 761–778. https://doi.org/10.1177/0956797617744771 Publisher: SAGE Publications Inc.
  50. Design Lab. 2017. What does it mean to be literate in the Age of Google? | Dan Russell | Design@Large. https://www.youtube.com/watch?v=SgOBrYOttZg
  51. Supporting self-regulated learning in computer-based learning environments: Systematic review of effects of scaffolding in the domain of science education. Journal of Computer Assisted Learning 28 (Dec. 2012). https://doi.org/10.1111/j.1365-2729.2011.00476.x
  52. Effectiveness of learning strategy instruction on academic performance: A meta-analysis. Educational Research Review 11 (Jan. 2014), 1–26. https://doi.org/10.1016/j.edurev.2013.11.002
  53. Distributed metacognition: Increased bias and deficits in metacognitive sensitivity when retrieving information from the internet. Technology, Mind, and Behavior 2, 3 (Aug. 2021). https://doi.org/10.1037/tmb0000039
  54. Jacquelynne S. Eccles and Allan Wigfield. 2002. Motivational Beliefs, Values, and Goals. Annual Review of Psychology 53, 1 (2002), 109–132. https://doi.org/10.1146/annurev.psych.53.100901.135153 _eprint: https://doi.org/10.1146/annurev.psych.53.100901.135153.
  55. Anastasia Efklides. 2008. Metacognition: Defining its facets and levels of functioning in relation to self-regulation and co-regulation. European Psychologist 13, 4 (2008), 277–287. https://doi.org/10.1027/1016-9040.13.4.277 Place: Germany Publisher: Hogrefe & Huber Publishers.
  56. Anastasia Efklides and Plousia Misailidi (Eds.). 2010. Trends and Prospects in Metacognition Research. Springer US, Boston, MA. https://doi.org/10.1007/978-1-4419-6546-2
  57. Upol Ehsan and Mark O. Riedl. 2020. Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach. In HCI International 2020 - Late Breaking Papers: Multimodality and Intelligence (Lecture Notes in Computer Science), Constantine Stephanidis, Masaaki Kurosu, Helmut Degen, and Lauren Reinerman-Jones (Eds.). Springer International Publishing, Cham, 449–466. https://doi.org/10.1007/978-3-030-60117-1_33
  58. Emmaline Drew Eliseev and Elizabeth J. Marsh. 2023. Understanding why searching the internet inflates confidence in explanatory ability. Applied Cognitive Psychology 37, 4 (2023), 711–720. https://doi.org/10.1002/acp.4058 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/acp.4058.
  59. K.A. Ericsson and H.A. Simon. 1993. Protocol Analysis. The MIT Press. https://mitpress.mit.edu/9780262550239/protocol-analysis/
  60. Anneli Eteläpelto. 1993. Metacognition and the Expertise of Computer Program Comprehension. Scandinavian Journal of Educational Research 37, 3 (Jan. 1993), 243–254. https://doi.org/10.1080/0031383930370305 Publisher: Routledge _eprint: https://doi.org/10.1080/0031383930370305.
  61. Small-Step Live Programming by Example. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. ACM, Virtual Event USA, 614–626. https://doi.org/10.1145/3379337.3415869
  62. Metacognition: Monitoring and controlling one’s own knowledge, reasoning and decisions. The psychology of human thought: An introduction (2019), 89–111. Publisher: Heidelberg University Publishing: Heidelberg.
  63. Matthew Fisher and Daniel M. Oppenheimer. 2021. Harder Than You Think: How Outside Assistance Leads to Overconfidence. Psychological Science 32, 4 (April 2021), 598–610. https://doi.org/10.1177/0956797620975779
  64. John H. Flavell. 1979. Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist 34, 10 (1979), 906–911. https://doi.org/10.1037/0003-066X.34.10.906 Place: US Publisher: American Psychological Association.
  65. Stephen Fleming. 2023. Metacognition and confidence: A review and synthesis. https://doi.org/10.31234/osf.io/ge7tz
  66. Stephen M. Fleming and Hakwan C. Lau. 2014. How to measure metacognition. Frontiers in Human Neuroscience 8 (2014). https://www.frontiersin.org/articles/10.3389/fnhum.2014.00443
  67. Petros Georghiades. 2004. From the general to the situated: three decades of metacognition. International Journal of Science Education 26, 3 (Feb. 2004), 365–383. https://doi.org/10.1080/0950069032000119401 Publisher: Routledge _eprint: https://doi.org/10.1080/0950069032000119401.
  68. Sam J. Gilbert. 2015. Strategic offloading of delayed intentions into the external environment. Quarterly Journal of Experimental Psychology 68, 5 (May 2015), 971–992. https://doi.org/10.1080/17470218.2014.972963 Publisher: SAGE Publications.
  69. Outsourcing Memory to External Tools: A Review of ‘Intention Offloading’. Psychonomic Bulletin & Review 30, 1 (Feb. 2023), 60–76. https://doi.org/10.3758/s13423-022-02139-4
  70. Exploring Challenges and Opportunities to Support Designers in Learning to Co-create with AI-based Manufacturing Design Tools. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–20. https://doi.org/10.1145/3544548.3580999
  71. Co-audit for copilots: tools to help humans double-check AI-generated content. Technical Report. Microsoft Research. https://aka.ms/co-audit
  72. The Self-Reflection and Insight Scale: A new measure of private self-consciousness. Social Behavior and Personality: An International Journal 30, 8 (2002), 821–835. https://doi.org/10.2224/sbp.2002.30.8.821 Place: New Zealand Publisher: Society for Personality Research.
  73. Metacognition and IT: The influence of self-efficacy and self-awareness. In AMCIS 2002 Proceedings. 147.
  74. How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study. https://doi.org/10.48550/arXiv.2309.10108 arXiv:2309.10108 [cs].
  75. The effectiveness of pair programming: A meta-analysis. Information and Software Technology 51, 7 (July 2009), 1110–1122. https://doi.org/10.1016/j.infsof.2009.02.001
  76. Neville Hatton and David Smith. 1995. Reflection in teacher education: Towards definition and implementation. Teaching and Teacher Education 11, 1 (Jan. 1995), 33–49. https://doi.org/10.1016/0742-051X(94)00012-U
  77. Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–18. https://doi.org/10.1145/3544548.3581025
  78. A Sports Analogy for Understanding Different Ways to Use AI. Harvard Business Review (Dec. 2023). https://hbr.org/2023/12/a-sports-analogy-for-understanding-different-ways-to-use-ai Section: AI and machine learning.
  79. A role for metamemory in cognitive offloading. Cognition 193 (Dec. 2019), 104012. https://doi.org/10.1016/j.cognition.2019.104012
  80. Jessica D. Huff and John L. Nietfeld. 2009. Using strategy instruction and confidence judgments to improve metacognitive monitoring. Metacognition and Learning 4, 2 (2009), 161–176. https://doi.org/10.1007/s11409-009-9042-8 Place: Germany Publisher: Springer.
  81. Sarah Inman and David Ribes. 2019. ”Beautiful Seams”: Strategic Revelations and Concealments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300508
  82. History and future of human-automation interaction. International Journal of Human-Computer Studies 131 (Nov. 2019), 99–107. https://doi.org/10.1016/j.ijhcs.2019.05.006
  83. Exploring the Learnability of Program Synthesizers by Novice Programmers. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST ’22). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3526113.3545659
  84. Survey of Hallucination in Natural Language Generation. Comput. Surveys 55, 12 (Dec. 2023), 1–38. https://doi.org/10.1145/3571730 arXiv:2202.03629 [cs].
  85. Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–19. https://doi.org/10.1145/3491102.3501870
  86. David H. Jonassen. 1997. Instructional design models for well-structured and III-structured problem-solving learning outcomes. Educational Technology Research and Development 45, 1 (March 1997), 65–94. https://doi.org/10.1007/BF02299613
  87. Sucharit Katyal and Stephen Fleming. 2023. Construct validity in metacognition research: balancing the tightrope between rigor of measurement and breadth of construct. https://doi.org/10.31234/osf.io/etjqh
  88. Alan Kay. 1990. User Interface: A Personal View. In The Art of Human-Computer Interface Design. 191–207. http://ui.korea.ac.kr/Board/Upload/a%20personal%20view_n.pdf
  89. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–23. https://doi.org/10.1145/3544548.3580919
  90. Nina Keith and Michael Frese. 2005. Self-Regulation in Error Management Training: Emotion Control and Metacognition as Mediators of Performance Effects. Journal of Applied Psychology 90, 4 (2005), 677–691. https://doi.org/10.1037/0021-9010.90.4.677 Place: US Publisher: American Psychological Association.
  91. Alison King. 1991. Improving lecture comprehension: Effects of a metacognitive strategy. Applied Cognitive Psychology 5, 4 (1991), 331–346. https://doi.org/10.1002/acp.2350050404 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/acp.2350050404.
  92. Reflection Companion: A Conversational System for Engaging Users in Reflection on Physical Activity. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 2 (July 2018), 1–26. https://doi.org/10.1145/3214273
  93. Asher Koriat. 2007. Metacognition and consciousness. In The Cambridge handbook of consciousness. Cambridge University Press, New York, NY, US, 289–325. https://doi.org/10.1017/CBO9780511816789.012
  94. Asher Koriat and Robert A. Bjork. 2005. Illusions of Competence in Monitoring One’s Knowledge During Study. Journal of Experimental Psychology: Learning, Memory, and Cognition 31, 2 (2005), 187–194. https://doi.org/10.1037/0278-7393.31.2.187 Place: US Publisher: American Psychological Association.
  95. The intricate relationships between monitoring and control in metacognition: Lessons for the cause-and-effect relation between subjective experience and behavior. Journal of Experimental Psychology: General 135, 1 (Feb. 2006), 36–69. https://doi.org/10.1037/0096-3445.135.1.36
  96. Bracha Kramarski and Zemira R. Mevarech. 2003. Enhancing Mathematical Reasoning in the Classroom: The Effects of Cooperative Learning and Metacognitive Training. American Educational Research Journal 40, 1 (2003), 281–310. https://www.jstor.org/stable/3699433 Publisher: [American Educational Research Association, Sage Publications, Inc.].
  97. Tell me more? the effects of mental model soundness on personalizing an intelligent agent. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/2207676.2207678
  98. Steinar Kvale. 1994. InterViews: An introduction to qualitative research interviewing. Sage Publications, Inc, Thousand Oaks, CA, US. Pages: xvii, 326.
  99. Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’23). Association for Computing Machinery, New York, NY, USA, 1369–1385. https://doi.org/10.1145/3593013.3594087
  100. Rethinking Explainability as a Dialogue: A Practitioner’s Perspective. https://doi.org/10.48550/arXiv.2202.01875 arXiv:2202.01875 [cs].
  101. What do we want from Explainable Artificial Intelligence (XAI)? – A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research. Artificial Intelligence 296 (July 2021), 103473. https://doi.org/10.1016/j.artint.2021.103473
  102. John D. Lee and Neville Moray. 1994. Trust, self-confidence, and operators’ adaptation to automation. International Journal of Human-Computer Studies 40, 1 (Jan. 1994), 153–184. https://doi.org/10.1006/ijhc.1994.1007
  103. Understanding the Usability of AI Programming Assistants. https://doi.org/10.48550/arXiv.2303.17125 arXiv:2303.17125 [cs].
  104. Q. Vera Liao and Jennifer Wortman Vaughan. 2023. AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap. https://doi.org/10.48550/arXiv.2306.01941 arXiv:2306.01941 [cs].
  105. “What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–31. https://doi.org/10.1145/3544548.3580817
  106. Jennifer A. Livingston. 2003. Metacognition: An Overview. Technical Report. https://eric.ed.gov/?id=ED474273 ERIC Number: ED474273.
  107. Metacognition and Self-Regulation in Programming Education: Theories and Exemplars of Use. ACM Transactions on Computing Education 22, 4 (Sept. 2022), 39:1–39:31. https://doi.org/10.1145/3487050
  108. Zhuoran Lu and Ming Yin. 2021. Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3411764.3445562
  109. Students’ Metacognition and Cognitive Style and Their Effect on Cognitive Load and Learning Achievement. Journal of Educational Technology & Society 20, 3 (2017), 145–157. https://www.jstor.org/stable/26196126 Publisher: International Forum of Educational Technology & Society.
  110. Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–19. https://doi.org/10.1145/3544548.3581058
  111. Metacognition and action: a new pathway to understanding social and cognitive aspects of expertise in sport. Frontiers in Psychology 5 (2014). https://www.frontiersin.org/articles/10.3389/fpsyg.2014.01155
  112. Thomas P. Mackey and Trudi E. Jacobson. 2017. Reframing Information Literacy as a Metaliteracy | Mackey | College & Research Libraries. (April 2017). https://doi.org/10.5860/crl-76r1
  113. Prompt Middleware: Mapping Prompts for Large Language Models to UI Affordances. https://doi.org/10.48550/arXiv.2307.01142 arXiv:2307.01142 [cs].
  114. P. Madhavan and D. A. Wiegmann. 2007. Similarities and differences between human–human and human–automation trust: an integrative review. Theoretical Issues in Ergonomics Science 8, 4 (July 2007), 277–301. https://doi.org/10.1080/14639220500337708 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/14639220500337708.
  115. Brian Maniscalco and Hakwan Lau. 2014. Signal detection theory analysis of type 1 and type 2 data: Meta-d’, response-specific meta-d’, and the unequal variance SDT model. In The cognitive neuroscience of metacognition. Springer-Verlag Publishing, New York, NY, US, 25–66. https://doi.org/10.1007/978-3-642-45190-4_3
  116. Gennie Mansi and Mark Riedl. 2023. Why Don’t You Do Something About It? Outlining Connections between AI Explanations and User Actions. http://arxiv.org/abs/2305.06297 arXiv:2305.06297 [cs].
  117. Matthew M. Martin and Rebecca B. Rubin. 1995. A New Measure of Cognitive Flexibility. Psychological Reports 76, 2 (April 1995), 623–626. https://doi.org/10.2466/pr0.1995.76.2.623 Publisher: SAGE Publications Inc.
  118. Towards a common conceptual space for metacognition in perception and memory. Nature Reviews Psychology (Nov. 2023), 1–16. https://doi.org/10.1038/s44159-023-00245-1 Publisher: Nature Publishing Group.
  119. On the Design of AI-powered Code Assistants for Notebooks. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3544548.3580940
  120. Lynn Meltzer. 2014. Teaching Executive Functioning Processes: Promoting Metacognition, Strategy Use, and Effort. In Handbook of Executive Functioning, Sam Goldstein and Jack A. Naglieri (Eds.). Springer, New York, NY, 445–473. https://doi.org/10.1007/978-1-4614-8106-5_25
  121. Zemira R. Mevarech and Chagit Amrany. 2008. Immediate and delayed effects of meta-cognitive instruction on regulation of cognition and mathematics achievement. Metacognition and Learning 3, 2 (Aug. 2008), 147–157. https://doi.org/10.1007/s11409-008-9023-3
  122. Enabling Good Work Habits in Software Developers through Reflective Goal-Setting. IEEE Transactions on Software Engineering 47, 9 (Sept. 2021), 1872–1885. https://doi.org/10.1109/TSE.2019.2938525 Conference Name: IEEE Transactions on Software Engineering.
  123. Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence 267 (Feb. 2019), 1–38. https://doi.org/10.1016/j.artint.2018.07.007
  124. Stephen Monsell. 2003. Task switching. Trends in Cognitive Sciences 7, 3 (March 2003), 134–140. https://doi.org/10.1016/S1364-6613(03)00028-7
  125. J. Moon. 2000. Learning Journals | A Handbook for Reflective Practice and Professiona. https://www.taylorfrancis.com/books/mono/10.4324/9780203969212/learning-journals-jennifer-moon
  126. Daniel Muijs and Christian Bokhove. 2020. Metacognition and Self-Regulation: Evidence Review. Technical Report. Education Endowment Foundation. https://eric.ed.gov/?id=ED612286 Publication Title: Education Endowment Foundation ERIC Number: ED612286.
  127. T. O. Nelson. 1984. A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychological Bulletin 95, 1 (Jan. 1984), 109–133.
  128. Thomas O. Nelson. 1990. Metamemory: A Theoretical Framework and New Findings. In Psychology of Learning and Motivation, Gordon H. Bower (Ed.). Vol. 26. Academic Press, 125–173. https://doi.org/10.1016/S0079-7421(08)60053-5
  129. Thomas O. Nelson and John Dunlosky. 1991. When People’s Judgments of Learning (JOLs) are Extremely Accurate at Predicting Subsequent Recall: The “Delayed-JOL Effect”. Psychological Science 2, 4 (July 1991), 267–271. https://doi.org/10.1111/j.1467-9280.1991.tb00147.x Publisher: SAGE Publications Inc.
  130. Ikujiro Nonaka and Ryoko Toyama. 2003. The knowledge-creating theory revisited: knowledge creation as a synthesizing process. Knowledge Management Research & Practice 1, 1 (July 2003), 2–10. https://doi.org/10.1057/palgrave.kmrp.8500001 Publisher: Taylor & Francis _eprint: https://doi.org/10.1057/palgrave.kmrp.8500001.
  131. Metacognition in Psychology. Review of General Psychology 23, 4 (Dec. 2019), 403–424. https://doi.org/10.1177/1089268019883821 Publisher: SAGE Publications Inc.
  132. Shakked Noy and Whitney Zhang. 2023. Experimental evidence on the productivity effects of generative artificial intelligence. Science 381, 6654 (July 2023), 187–192. https://doi.org/10.1126/science.adh2586 Publisher: American Association for the Advancement of Science.
  133. Expertise reversal effects in writing-to-learn. Instructional Science 38, 3 (May 2010), 237–258. https://doi.org/10.1007/s11251-009-9106-9
  134. The Self-Regulation-View in Writing-to-Learn: Using Journal Writing to Optimize Cognitive Load in Self-Regulated Learning. Educational Psychology Review 32, 4 (Dec. 2020), 1089–1126. https://doi.org/10.1007/s10648-020-09541-1
  135. OpenAI. 2023. Customizing GPT-3 for your application. https://openai.com/blog/customizing-gpt-3
  136. LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code Generation. http://arxiv.org/abs/2308.02828 arXiv:2308.02828 [cs].
  137. Annemarie Sullivan Palincsar and Ann L. Brown. 1984. Reciprocal Teaching of Comprehension-Fostering and Comprehension-Monitoring Activities. Cognition and Instruction 1, 2 (1984), 117–175. https://www.jstor.org/stable/3233567 Publisher: Taylor & Francis, Ltd..
  138. A Slow Algorithm Improves Users’ Assessments of the Algorithm’s Accuracy. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (Nov. 2019), 102:1–102:15. https://doi.org/10.1145/3359204
  139. Do Users Write More Insecure Code with AI Assistants? https://doi.org/10.48550/arXiv.2211.03622 arXiv:2211.03622 [cs].
  140. Talking about Teaching Self-Regulated Learning: Scaffolding Student Teachers’ Development and Use of Practices that Promote Self-Regulated Learning. International Journal of Educational Research 47, 2 (2008), 97–108. https://doi.org/10.1016/j.ijer.2007.11.010 Publisher: Elsevier ERIC Number: EJ798056.
  141. Peter Pirolli and Stuart Card. 2005. The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis. In Proceedings of international conference on intelligence analysis, Vol. 5. 2–4.
  142. What Do We Think We Think We Are Doing?: Metacognition and Self-Regulation in Programming. In Proceedings of the 2020 ACM Conference on International Computing Education Research. ACM, Virtual Event New Zealand, 2–13. https://doi.org/10.1145/3372782.3406263
  143. ”It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers. https://doi.org/10.48550/arXiv.2304.02491 arXiv:2304.02491 [cs].
  144. David D. Preiss. 2022. Metacognition, Mind Wandering, and Cognitive Flexibility: Understanding Creativity. Journal of Intelligence 10, 3 (Sept. 2022), 69. https://doi.org/10.3390/jintelligence10030069 Number: 3 Publisher: Multidisciplinary Digital Publishing Institute.
  145. Dobromir Rahnev. 2023. Measuring metacognition: A comprehensive assessment of current methods. https://doi.org/10.31234/osf.io/waz9h
  146. Supporting Human-AI Collaboration in Auditing LLMs with LLMs. https://doi.org/10.48550/arXiv.2304.09991 arXiv:2304.09991 [cs].
  147. Leon Reicherts and Yvonne Rogers. 2020. Do Make me Think! How CUIs Can Support Cognitive Processes. In Proceedings of the 2nd Conference on Conversational User Interfaces (CUI ’20). Association for Computing Machinery, New York, NY, USA, 1–4. https://doi.org/10.1145/3405755.3406157
  148. Evan F. Risko and Sam J. Gilbert. 2016. Cognitive Offloading. Trends in Cognitive Sciences 20, 9 (Sept. 2016), 676–688. https://doi.org/10.1016/j.tics.2016.07.002
  149. Evan F. Risko and Megan O. Kelly. 2023. Thinking in the digital age: Everyday cognition and the dawn of a new age of metacognition research. Applied Cognitive Psychology 37, 4 (2023), 785–788. https://doi.org/10.1002/acp.4102 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/acp.4102.
  150. The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development. In Proceedings of the 28th International Conference on Intelligent User Interfaces (IUI ’23). Association for Computing Machinery, New York, NY, USA, 491–514. https://doi.org/10.1145/3581641.3584037
  151. Human Metacognition Across Domains: Insights from Individual Differences and Neuroimaging. Personality Neuroscience 1 (2018), e17. https://doi.org/10.1017/pen.2018.16
  152. Martin Ruskov. 2023. Grimm in Wonderland: Prompt Engineering with Midjourney to Illustrate Fairytales. https://doi.org/10.48550/arXiv.2302.08961 arXiv:2302.08961 [cs].
  153. Nikita A. Salovich and David N. Rapp. 2021. Misinformed and unaware? Metacognition and the influence of inaccurate information. Journal of Experimental Psychology: Learning, Memory, and Cognition 47, 4 (2021), 608–624. https://doi.org/10.1037/xlm0000977 Place: US Publisher: American Psychological Association.
  154. Advait Sarkar. 2023a. Exploring Perspectives on the Impact of Artificial Intelligence on the Creativity of Knowledge Work: Beyond Mechanised Plagiarism and Stochastic Parrots. In Proceedings of the 2nd Annual Meeting of the Symposium on Human-Computer Interaction for Work (CHIWORK ’23). Association for Computing Machinery, New York, NY, USA, 1–17. https://doi.org/10.1145/3596671.3597650
  155. Advait Sarkar. 2023b. Should Computers Be Easy To Use? Questioning the Doctrine of Simplicity in User Interface Design. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA ’23). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3544549.3582741
  156. What is it like to program with artificial intelligence? https://doi.org/10.48550/arXiv.2208.06213 arXiv:2208.06213 [cs].
  157. Your Prompt is My Command: On Assessing the Human-Centred Generality of Multimodal Models. Journal of Artificial Intelligence Research 77 (June 2023), 377–394. https://doi.org/10.1613/jair.1.14157
  158. Dogmatism manifests in lowered information search under uncertainty. Proceedings of the National Academy of Sciences 117, 49 (Dec. 2020), 31527–31534. https://doi.org/10.1073/pnas.2009641117 Publisher: Proceedings of the National Academy of Sciences.
  159. Dale H. Schunk and Peggy A. Ertmer. 2000. Self-regulation and academic learning: Self-efficacy enhancing interventions. In Handbook of self-regulation. Academic Press, San Diego, CA, US, 631–649. https://doi.org/10.1016/B978-012109890-2/50048-2
  160. Rolf Schwonke. 2015. Metacognitive Load – Useful, or Extraneous Concept? Metacognitive and Self-Regulatory Demands in Computer-Based Learning. Journal of Educational Technology & Society 18, 4 (2015), 172–184. https://www.jstor.org/stable/jeductechsoci.18.4.172 Publisher: International Forum of Educational Technology & Society.
  161. Ava Scott and Sam Gilbert. 2023. Metacognition guides intention offloading and fulfilment of real-world plans. https://doi.org/10.31234/osf.io/y46mq
  162. Designing Representations and Support for Metacognition in the Generalized Intelligent Framework for Tutoring. In Foundations of Augmented Cognition (Lecture Notes in Computer Science), Dylan D. Schmorrow and Cali M. Fidopiastis (Eds.). Springer International Publishing, Cham, 663–674. https://doi.org/10.1007/978-3-319-20816-9_63
  163. Tina Seufert. 2018. The interplay between self-regulation in learning and cognitive load. Educational Research Review 24 (June 2018), 116–129. https://doi.org/10.1016/j.edurev.2018.03.004
  164. Investigating the Impact of Backward Strategy Learning in a Logic Tutor: Aiding Subgoal Learning Towards Improved Problem Solving. International Journal of Artificial Intelligence in Education (Aug. 2023). https://doi.org/10.1007/s40593-023-00338-1
  165. Experts in the Shadow of Algorithmic Systems: Exploring Intelligibility in a Decision-Making Context. In Companion Publication of the 2020 ACM Designing Interactive Systems Conference (DIS’ 20 Companion). Association for Computing Machinery, New York, NY, USA, 263–268. https://doi.org/10.1145/3393914.3395862
  166. Ironies of Generative AI: Understanding and mitigating productivity loss in human-AI interactions. https://doi.org/10.48550/arXiv.2402.11364 arXiv:2402.11364 [cs].
  167. Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experiment. http://arxiv.org/abs/2307.03744 arXiv:2307.03744 [cs].
  168. Cognitive Flexibility, Constructivism, and Hypertext: Random Access Instruction for Advanced Knowledge Acquisition in Ill-Structured Domains. Educational Technology 31, 5 (1991), 24–33. https://www.jstor.org/stable/44427517 Publisher: Educational Technology Publications, Inc..
  169. GridBook: Natural Language Formulas for the Spreadsheet Grid. In 27th International Conference on Intelligent User Interfaces (IUI ’22). Association for Computing Machinery, New York, NY, USA, 345–368. https://doi.org/10.1145/3490099.3511161
  170. Keith E. Stanovich and Richard F. West. 2000. Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences 23, 5 (2000), 645–665. https://doi.org/10.1017/S0140525X00003435 Place: United Kingdom Publisher: Cambridge University Press.
  171. Mark Steyvers and Aakriti Kumar. 2023. Three Challenges for AI-Assisted Decision-Making. Perspectives on Psychological Science: A Journal of the Association for Psychological Science (July 2023), 17456916231181102. https://doi.org/10.1177/17456916231181102
  172. Sean M. Stone and Benjamin C. Storm. 2021. Search fluency as a misleading measure of memory. Journal of Experimental Psychology: Learning, Memory, and Cognition 47, 1 (2021), 53–64. https://doi.org/10.1037/xlm0000806 Place: US Publisher: American Psychological Association.
  173. Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models. https://doi.org/10.48550/arXiv.2305.11483 arXiv:2305.11483 [cs].
  174. Investigating Explainability of Generative AI for Code through Scenario-based Design. In 27th International Conference on Intelligent User Interfaces (IUI ’22). Association for Computing Machinery, New York, NY, USA, 212–228. https://doi.org/10.1145/3490099.3511119
  175. Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3411764.3445088
  176. Adam Svendsen and Bruce Garvey. 2023. An Outline for an Interrogative/Prompt Library to help improve output quality from Generative-AI Datasets. https://doi.org/10.2139/ssrn.4495319
  177. John Sweller. 1988. Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science 12, 2 (1988), 257–285. https://doi.org/10.1207/s15516709cog1202_4 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1207/s15516709cog1202_4.
  178. Cognitive Architecture and Instructional Design. Educational Psychology Review 10, 3 (Sept. 1998), 251–296. https://doi.org/10.1023/A:1022193728205
  179. What the DAAM: Interpreting Stable Diffusion Using Cross Attention. https://doi.org/10.48550/arXiv.2210.04885 arXiv:2210.04885 [cs].
  180. Kimberly D. Tanner. 2012. Promoting Student Metacognition. CBE—Life Sciences Education 11, 2 (June 2012), 113–120. https://doi.org/10.1187/cbe.12-03-0033 Publisher: American Society for Cell Biology (lse).
  181. Pina Tarricone. 2011. The Taxonomy of Metacognition. Psychology Press. Google-Books-ID: c1p6AgAAQBAJ.
  182. AI-Assisted Decision-making: a Cognitive Modeling Approach to Infer Latent Reliance Strategies. Computational Brain & Behavior 5, 4 (Dec. 2022), 491–508. https://doi.org/10.1007/s42113-022-00157-y
  183. Valerie Thompson and Kinga Morsanyi. 2012. Analytic thinking: do you feel like it? Mind & Society 11, 1 (June 2012), 93–105. https://doi.org/10.1007/s11299-012-0100-6
  184. The role of answer fluency and perceptual fluency as metacognitive cues for initiating analytic thinking. Cognition 128, 2 (Aug. 2013), 237–251. https://doi.org/10.1016/j.cognition.2012.09.012
  185. Sascha Topolinski and Rolf Reber. 2010. Immediate truth – Temporal contiguity between a cognitive problem and its solution determines experienced veracity of the solution. Cognition 114, 1 (Jan. 2010), 117–122. https://doi.org/10.1016/j.cognition.2009.09.009
  186. Clustering and switching as two components of verbal fluency: evidence from younger and older healthy adults. Neuropsychology 11, 1 (Jan. 1997), 138–146. https://doi.org/10.1037//0894-4105.11.1.138
  187. Christian Unkelbach and Rainer Greifeneder. 2013. A general model of fluency effects in judgment and decision making. In The experience of thinking: How the fluency of mental processes influences cognition and behaviour. Psychology Press, New York, NY, US, 11–32.
  188. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, 1–7. https://doi.org/10.1145/3491101.3519665
  189. Martin Valcke. 2002. Cognitive load: updating the theory? Learning and Instruction (2002).
  190. Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions. https://doi.org/10.48550/arXiv.2302.07248 arXiv:2302.07248 [cs].
  191. Crossing the bridge over Norman’s Gulf of Execution: revealing feedforward’s true identity. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). Association for Computing Machinery, New York, NY, USA, 1931–1940. https://doi.org/10.1145/2470654.2466255
  192. Designing Theory-Driven User-Centric Explainable AI. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3290605.3300831
  193. Examining students’ cognitive load in the context of self-regulated learning with an intelligent tutoring system. Education and Information Technologies 28, 5 (May 2023), 5697–5715. https://doi.org/10.1007/s10639-022-11357-1
  194. Cognitive load patterns affect temporal dynamics of self-regulated learning behaviors, metacognitive judgments, and learning achievements. Computers & Education 207 (Dec. 2023), 104924. https://doi.org/10.1016/j.compedu.2023.104924
  195. Patrick P. Weis and Eva Wiese. 2022. Know Your Cognitive Environment! Mental Models as Crucial Determinant of Offloading Preferences. Human Factors 64, 3 (May 2022), 499–513. https://doi.org/10.1177/0018720820956861 Publisher: SAGE Publications Inc.
  196. Mark Weiser. 1994. Creating the invisible interface: (invited talk). In Proceedings of the 7th annual ACM symposium on User interface software and technology (UIST ’94). Association for Computing Machinery, New York, NY, USA, 1. https://doi.org/10.1145/192426.192428
  197. Toward General Design Principles for Generative AI Applications. https://doi.org/10.48550/arXiv.2301.05578 arXiv:2301.05578 [cs].
  198. Perfection Not Required? Human-AI Partnerships in Code Translation. In 26th International Conference on Intelligent User Interfaces (IUI ’21). Association for Computing Machinery, New York, NY, USA, 402–412. https://doi.org/10.1145/3397481.3450656
  199. Better Together? An Evaluation of AI-Supported Code Translation. In 27th International Conference on Intelligent User Interfaces (IUI ’22). Association for Computing Machinery, New York, NY, USA, 369–391. https://doi.org/10.1145/3490099.3511157
  200. Adrian Wells. 2009. Metacognitive therapy for anxiety and depression. Guilford Press, New York, NY, US. Pages: xvii, 316.
  201. Improving metacomprehension accuracy in an undergraduate course context. Journal of Experimental Psychology: Applied 22, 4 (Dec. 2016), 393–405. https://doi.org/10.1037/xap0000096
  202. Philip H. Winne and Nancy E. Perry. 2000. Chapter 16 - Measuring Self-Regulated Learning. In Handbook of Self-Regulation, Monique Boekaerts, Paul R. Pintrich, and Moshe Zeidner (Eds.). Academic Press, San Diego, 531–566. https://doi.org/10.1016/B978-012109890-2/50045-7
  203. PromptChainer: Chaining Large Language Model Prompts through Visual Programming. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3491101.3519729
  204. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–22. https://doi.org/10.1145/3491102.3517582
  205. In-IDE Code Generation from Natural Language: Promise and Challenges. ACM Transactions on Software Engineering and Methodology 31, 2 (March 2022), 29:1–29:47. https://doi.org/10.1145/3487569
  206. Nick Yeung and Christopher Summerfield. 2012. Metacognition in human decision-making: confidence and error monitoring. Philosophical Transactions of the Royal Society B: Biological Sciences 367, 1594 (May 2012), 1310–1321. https://doi.org/10.1098/rstb.2011.0416
  207. Wordcraft: Story Writing With Large Language Models. In 27th International Conference on Intelligent User Interfaces (IUI ’22). Association for Computing Machinery, New York, NY, USA, 841–852. https://doi.org/10.1145/3490099.3511105
  208. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, Hamburg Germany, 1–21. https://doi.org/10.1145/3544548.3581388
  209. Understanding Circumstances for Desirable Proactive Behaviour of Voice Assistants: The Proactivity Dilemma. In Proceedings of the 4th Conference on Conversational User Interfaces (CUI ’22). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3543829.3543834
  210. The effects of group metacognitive scaffolding on group metacognitive behaviors, group performance, and cognitive load in computer-supported collaborative learning. The Internet and Higher Education 42 (July 2019), 13–24. https://doi.org/10.1016/j.iheduc.2019.03.002
  211. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming (MAPS 2022). Association for Computing Machinery, New York, NY, USA, 21–29. https://doi.org/10.1145/3520312.3534864
  212. Barry J. Zimmerman. 2001. Theories of self-regulated learning and academic achievement: An overview and analysis. In Self-regulated learning and academic achievement: Theoretical perspectives, 2nd ed. Lawrence Erlbaum Associates Publishers, Mahwah, NJ, US, 1–37.
  213. Barry J. Zimmerman and Adam R. Moylan. 2009. Self-regulation: Where metacognition and motivation intersect. In Handbook of metacognition in education. Routledge/Taylor & Francis Group, New York, NY, US, 299–315.
  214. Anat Zohar and Sarit Barzilai. 2013. A review of research on metacognition in science education: current and future directions. Studies in Science Education 49, 2 (Sept. 2013), 121–169. https://doi.org/10.1080/03057267.2013.847261 Publisher: Routledge _eprint: https://doi.org/10.1080/03057267.2013.847261.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Lev Tankelevitch (10 papers)
  2. Viktor Kewenig (4 papers)
  3. Auste Simkute (5 papers)
  4. Ava Elizabeth Scott (5 papers)
  5. Advait Sarkar (25 papers)
  6. Abigail Sellen (5 papers)
  7. Sean Rintel (14 papers)
Citations (34)
View on Semantic Scholar