- The paper synthesizes current evidence on GenAI’s risks—such as reduced critical thinking and creativity—and its potential to enhance human cognition.
- It reviews multidisciplinary design strategies, empirical studies, and ethical considerations across learning, creativity, and workflow domains.
- The study demonstrates that balanced integration of process-oriented AI support with human expertise can foster critical reflection and improve cognitive outcomes.
Understanding, Protecting, and Augmenting Human Cognition with Generative AI: Synthesis and Implications
Introduction
This synthesis paper from the CHI 2025 Tools for Thought Workshop provides a comprehensive overview of the current research landscape on the cognitive impacts of Generative AI (GenAI). It addresses both the risks and opportunities posed by GenAI for human cognition, focusing on critical thinking, learning, creativity, workflows, expertise, and human values. The paper also surveys design strategies for augmenting cognition, interaction paradigms, and the need for rigorous theory and measurement in this domain.
Cognitive Risks and Protective Strategies
Critical Thinking
GenAI systems, by automating knowledge production and providing polished, coherent outputs, can discourage active engagement and critical evaluation. Empirical studies show a shift from active information seeking to passive consumption, with users often over-relying on AI-generated information, especially when lacking domain confidence or when trusting AI outputs. This effect is exacerbated by the sycophantic and homogenizing tendencies of GenAI, which can create an illusion of comprehensive understanding and reduce the necessary state of perplexity for reflective thinking (Dewey, 1910). Notably, biased AI writing assistants have been shown to shift user attitudes and homogenize writing styles toward Western norms, diminishing cultural nuances and diversity.
Learning
Novices and students are particularly vulnerable to the negative impacts of GenAI, as they often lack the metacognitive strategies and domain schemas required for effective prompting and evaluation. Studies in AI-assisted code debugging and reading reveal that students with prior understanding use GenAI more intentionally and benefit more, while underprepared students exhibit over-reliance and reduced critical thinking, resulting in lower learning outcomes. Longitudinal data indicate a trend toward passive engagement and cognitive offloading, impeding the development of higher-order thinking skills.
Creativity
GenAI's ability to rapidly generate high-fidelity outputs risks inducing design fixation and homogenizing creative work. However, when used as instruments of inquiry, GenAI can facilitate reflection-in-action and support creative exploration, provided users are aware of its limitations and intentionally leverage its affordances. The timing of GenAI integration in workflows is critical; early introduction can stifle ideation, while delayed use can augment creativity. The shift in creative workflows also raises questions about the preservation of flow states and the integration of GenAI into established practices.
Workflows, Expertise, and Human Values
GenAI-driven automation prompts a reallocation of cognitive resources, raising questions about cognitive miserliness and satisficing. Experts tend to offload routine tasks to AI but retain control over complex analysis, reflecting concerns about accuracy, workflow compatibility, and the preservation of deep work. The use of GenAI also impacts intrinsic motivation, well-being, and professional identity, with excessive automation threatening autonomy, competence, and relatedness. The paper highlights the need to protect dialectical activities—tasks valued for their intrinsic nature—and to address emerging issues of emotional vulnerability and dependency in GenAI interactions.
Cognitive Augmentation: Design Approaches
Provocation and Challenge
Process-oriented support systems that provoke reflection, challenge assumptions, and scaffold meta-decision making are shown to enhance critical thinking and intent formulation. Designs such as metacognitive support agents and ignorant co-learners foster uncertainty and dissonance, compelling users to think reflexively. However, motivating engagement with such systems requires clear pedagogical goals and careful calibration of friction and challenge.
Scaffolding and Structuring
AI-driven scaffolding can guide users through complex tasks by surfacing structure, organizing stimuli, and supporting schema induction. Human-AI complementary workflows, such as iterative abstraction and analogical cue generation, enable users to reason forward and maintain agency. Over-scaffolding remains a risk, potentially reducing necessary cognitive effort and impeding skill development.
GenAI can augment cognition by dynamically transforming information across modalities (e.g., text, speech, visuals) and levels of formality. Such transformations expand conceptual environments and support manipulation of knowledge representations, but require careful design to balance abstraction, familiarity, and cognitive load.
Augmenting System 1 Processes
Beyond deliberate reasoning, GenAI can be leveraged to augment intuitive, emotional, and motivational processes. Approaches such as amplifying hallucinations for creative risk-taking and personalizing learning experiences via emotional pathways demonstrate potential for enhancing engagement and creativity. Ethical considerations around agency and emotional support are paramount.
Interaction Paradigms
Emerging interaction paradigms, including direct manipulation, feedforward design, and full-duplex communication, offer new affordances for expressing intent and managing cognitive load. Direct manipulation can reduce prompt formulation effort but may also diminish opportunities for task decomposition and reflection. The anthropomimetic design of proactive agents introduces trade-offs in conversational flow, over-reliance, and emotional impact.
Workflow Augmentation
Activity-centric computing, dynamic interface generation, and collaborative AI agents represent a shift toward workflow-level augmentation. Integrating GenAI into collaborative and distributed workflows remains challenging, with issues of context inference, goal alignment, and group dynamics. Group interactions with AI can mitigate over-reliance and foster boundary regulation, but may also exacerbate social loafing and groupthink. The rise of agentic AI necessitates new models for managing and verifying complex, multi-agent workflows.
User and Workflow Characterization
A nuanced understanding of user populations, workflows, and expertise is essential for designing effective GenAI systems. Heavy users of LLMs exhibit distinct interaction patterns, integrating AI into both rational and intuitive decision-making. Expertise in domain, AI, and management must be considered in system design and evaluation.
Theoretical Foundations
Existing theories from cognitive science, education, and pragmatism provide valuable lenses for understanding GenAI's impact, but may require extension to address compound human-AI biases and interactionist perspectives. Theories-of-change frameworks can inform design briefs and guide the development of psychological interventions via GenAI.
Measurement and Evaluation
Reliable measurement of constructs such as critical thinking, agency, and creativity is a major challenge, given their introspective and subjective nature. New frameworks for analyzing interaction patterns, such as cognitive activity and engagement modes, are needed for open-ended tasks. Longitudinal studies are critical for assessing the long-term impacts of GenAI on cognition and workflows.
Implications and Future Directions
The synthesis underscores the need for multidisciplinary collaboration, rigorous theory development, and methodological innovation in understanding and designing GenAI systems that protect and augment human cognition. Key implications include:
- Designing for process-oriented support to foster agency, critical thinking, and skill development.
- Balancing automation and augmentation to preserve deep work, intrinsic motivation, and expertise.
- Developing interaction paradigms that support intent expression, reflection, and workflow integration.
- Advancing measurement frameworks for evaluating cognitive impacts in open-ended, longitudinal contexts.
- Addressing ethical and emotional dimensions of GenAI use, particularly in domains involving vulnerability and identity.
Conclusion
GenAI presents a complex landscape of cognitive risks and opportunities, necessitating a shift from task-centric automation to intentional augmentation of human thought. The research and design space is vast, requiring focused efforts, cross-disciplinary engagement, and ongoing attention to the protection and transformation of core aspects of human cognition. Future work should prioritize the development of systems that not only automate but also enrich and expand the ways in which humans think, learn, and create.
