Human-AI Experience in Integrated Development Environments: A Systematic Literature Review

Abstract: The integration of AI into Integrated Development Environments (IDEs) is reshaping software development, fundamentally altering how developers interact with their tools. This shift marks the emergence of Human-AI Experience in Integrated Development Environment (in-IDE HAX), a field that explores the evolving dynamics of Human-Computer Interaction in AI-assisted coding environments. Despite rapid adoption, research on in-IDE HAX remains fragmented, which highlights the need for a unified overview of current practices, challenges, and opportunities. To provide a structured overview of existing research, we conduct a systematic literature review of 90 studies, summarizing current findings and outlining areas for further investigation. We organize key insights from reviewed studies into three aspects: Impact, Design, and Quality of AI-based systems inside IDEs. Impact findings show that AI-assisted coding enhances developer productivity but also introduces challenges, such as verification overhead and over-reliance. Design studies show that effective interfaces surface context, provide explanations and transparency of suggestion, and support user control. Quality studies document risks in correctness, maintainability, and security. For future research, priorities include productivity studies, design of assistance, and audit of AI-generated code. The agenda calls for larger and longer evaluations, stronger audit and verification assets, broader coverage across the software life cycle, and adaptive assistance under user control.

• The paper's main contribution is synthesizing 89 studies to reveal key practices and challenges in integrating AI into IDEs.
• It employs qualitative and experimental methodologies, including interviews and focus groups, to analyze human-AI interactions.
• Findings show AI tools can boost productivity (up to 55.8% faster tasks) while introducing risks like automation bias and verification overhead.

Human-AI Experience in Integrated Development Environments: A Systematic Literature Review

Abstract

This essay explores the systematic literature review conducted on the integration of AI into Integrated Development Environments (IDEs), an area known as Human-AI Experience (HAX) within IDEs or in-IDE HAX. The research undertakes a structured examination of 89 studies to provide a comprehensive overview of current practices, challenges, and opportunities in AI-enhanced software development. Key findings demonstrate that while AI offers productivity enhancements, it introduces challenges such as verification overhead and automation bias. Insights into two dominant interaction paradigms—autocompletion and conversational agents—are explored along with the implications on developer productivity and code quality. Furthermore, the essay outlines future research directions crucial for advancing HAX research.

Introduction

The integration of AI into Integrated Development Environments (IDEs) has rapidly evolved, reshaping the dynamics of software development. With a substantial increase in the adoption of AI tools, this systematic literature review addresses the fragmented nature of current research within the Human-AI Experience (HAX) in IDEs, or in-IDE HAX. Building upon previous exploratory work, this review synthesizes existing literature to map the current knowledge landscape and identify gaps for further inquiry.

Figure 1: Flow diagram of the study.

Methodology

Adopting the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) framework, the review identifies 254 studies, of which 89 are included in the final analysis. The review aims to provide an overview of in-IDE HAX research, categorizing studies by methodological approaches, study contexts, and key findings. Research questions focus on elucidating the key findings, extensively studied aspects, and potential directions for future research in in-IDE HAX.

Key Findings

Methodologies

The review encompasses various methodological approaches, predominantly qualitative and experimental. Among the 89 studies, qualitative methods are prevalent, including semi-structured interviews and focus groups, alongside experimental designs that often utilize mixed methods. This diversity highlights an evolving research landscape exploring the intricate dynamics between developers and AI-assisted IDEs.

Design Paradigms

Two primary paradigms for AI-assisted interaction within IDEs are identified: autocompletion-based assistance and conversational agents. Autocompletion minimizes cognitive load, supporting rapid decision-making, while conversational interactions enable complex problem-solving. Emerging hybrid models integrate both paradigms, suggesting a balanced approach for enhancing developer productivity through real-time, context-aware support.

Figure 2: Paper IDs by Context.

Impact on Productivity and Challenges

AI integration in IDEs leads to mixed outcomes. On one hand, developers experience significant productivity gains, with studies reporting up to 55.8% faster task completion using tools like GitHub Copilot. On the other hand, challenges arise, including verification overhead and automation bias, necessitating thorough validation of AI-generated outputs to maintain code quality. Novice developers are particularly susceptible to over-reliance, underscoring the need for structured prompting and adaptive guidance.

Emerging Areas and Opportunities

The review identifies underexplored areas such as AI assistance in requirements engineering, testing, and deployment. Moreover, the need for personalization strategies, AI governance frameworks, and ethical considerations persists. The examination of longitudinal effects and impacts on skill retention is emphasized as a key prospective research domain.

Figure 3: Paper IDs by Context and SDLC stage.

Implications and Future Research

Key contributions of the review include the synthesis of existing research into core HAX dimensions—design, impact, and quality. The concentrated research on specific tools like GitHub Copilot suggests a potential overgeneralization risk, calling for broader investigations into diverse AI-powered coding assistants. Future research directions focus on addressing personalization, ethical considerations, and longitudinal studies, which are pivotal for advancing responsible and sustainable AI integration in software development.

Conclusion

The systematic review establishes a foundational understanding of current practices and challenges in in-IDE HAX, providing valuable insights for both academia and industry. By highlighting research gaps and proposing avenues for future inquiry, the review aims to guide ongoing efforts to effectively integrate AI tools into development environments, ultimately enhancing software engineering practices and developer experiences.