Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents

Abstract: Autonomous software agents hold promise to increase developer productivity but make mistakes and exhibit novel failure modes, making human oversight central to successful human-agent collaboration. Existing research on agent oversight is largely conceptual; normative frameworks exist, but how users actually oversee agents is less known. In this paper, we bridge this gap by providing early empirical anchors for the theoretical discourse on agent oversight. Drawing on interviews with 17 experienced developers, we conduct an exploratory inquiry examining what forms of emergent oversight work developers perform, when, and how. We also document the oversight challenges developers face and the strategies they have started using to address them. We found at least four forms of emergent oversight work: a priori control, co-planning, real-time monitoring, and post hoc review. We show that oversight work is not only reactive and retrospective, as portrayed in existing research, but also preventative and proactive. We describe situated oversight challenges (e.g., difficulty reviewing agent-generated code) and outline heuristics developers adopt to address such challenges (e.g., using test results as guarantees for code correctness). We conclude with high-level takeaways, future research directions, implications for the human-centered design of software agents and for software engineering practice, and limitations of our research.

• The paper provides a qualitative empirical analysis of oversight practices in agentic software development, identifying four distinct oversight forms.
• It employs in-depth interviews with 17 experienced developers to uncover challenges in configuring, monitoring, and reviewing autonomous AI agents.
• The study reveals actionable heuristics and implications for tool design, regulatory compliance, and the transformation of professional skills in software engineering.

Human Oversight of Agentic Systems in Software Development: An Empirical Analysis

Introduction and Motivation

Agentic systems—autonomously acting AI agents—have been rapidly incorporated into software engineering workflows, particularly for code generation, debugging, code refactoring, and broader development tasks. However, these systems expose new risks, including erroneous outputs, novel failure modes, context misalignment, propagation of silent errors, and significant cybersecurity threats. In light of regulatory trends, such as the EU AI Act, human oversight is increasingly seen as both a practical necessity and a formal requirement for organizations deploying AI agents in software production.

Existing work on agent oversight largely remains theoretical or is anchored in controlled technical settings. This paper ("Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents" (2606.05391)) presents a qualitative, empirical analysis of how experienced software engineers oversee agentic systems in real-world contexts, what challenges emerge in the process, and which practical heuristics they adopt to bridge the gap between theory and practice.

Methodology

The study is based on in-depth interviews with 17 experienced developers, primarily situated in high-resource organizational settings. The participants cover a diverse set of software engineering roles and domains, but most are from a single large technology enterprise, which is acknowledged as a sampling limitation. Through open-ended and semi-structured interviews, the authors applied inductive qualitative coding to identify the core forms of oversight work as well as situated developer strategies.

Participants reported the use of agents for a spectrum of software engineering activities, including drafting specifications, end-to-end feature implementation, debugging, log analysis, code review, and automated documentation.

Figure 1: Participants' description of scenarios in which they use AI agents for software development illustrates the breadth of agent integration across the SDLC.

Emergent Forms of Oversight Work

The authors identify four key, temporally distinct forms of oversight work in agentic software development:

1. A Priori Control: Preventative measures and configurations before task delegation. This includes specifying constraints, customizing agent instructions, configuring agent autonomy, providing sandboxed testing environments, and preemptively limiting agent actions such as file deletion or tool invocation. A central challenge arises from the perceived black-box nature of agentic systems, limited visibility into agent inner mechanisms, and uncertainty about data handling.

2. Co-Planning: Proactive, collaborative planning between developer and agent. Typical co-planning activities entail iterative prompting, joint decomposition of high-level tasks into smaller, well-bounded units, seeding partial solutions, and spec-driven development. Challenges include articulating sufficiently detailed yet efficient instructions and translating implicit requirements into explicit natural language directives, often complicated by the agent's inability to infer unstated assumptions or operational context.

3. Real-Time Monitoring: In-task observation and, if necessary, intervention. This is the least frequently practiced form, largely because agent execution windows are short and tasks are intentionally decomposed. When monitoring does occur, it is generally lightweight (e.g., spot-checking agent logic traces or aborting non-productive executions), with developers highlighting poor reliability of agent-generated reasoning traces and subtle errors often evading detection.

4. Post Hoc Review: Retrospective evaluation and correction of agent outputs. This dominates current oversight practices and includes code diff review, test suite verification, use of LLM-as-judge, multi-agent cross-checking, and selective deep dives into reasoning or structural artifacts. Key difficulties stem from cognitive distance—the developer's lack of authorship over the generated code—resulting in lower comprehensibility and increased burden for iterative corrections.

Heuristics and Coping Mechanisms

Developers, confronted with the unscalability of exhaustive oversight and the complexity of agentic systems, have pragmatically shifted from idealized thoroughness to "good-enough" oversight, employing several functional heuristics:

• Plan Fidelity Proxy: Treating an agent's up-front plan as an accurate stand-in for its actual working, enabling plan review to substitute time-intensive code review.
• Test Suite Reliance: Using passing test cases as a sufficient proxy for correctness, especially when tests are comprehensive and closely aligned with the intended specification.
• Eyeballing and Spot-Checking: Employing selective, high-level review tactics—scanning method signatures, reviewing change summaries, and accepting visual or structural cues—to prioritize efficiency over depth.
• Epistemic Deference: Trusting agent outputs in unfamiliar domains or with novel libraries, and cross-validating with outputs from multiple agents as "social proof".

These heuristics are not arbitrary shortcuts but rather contextually rational adaptations to the scale, opacity, and velocity of agentic software workflows.

Implications and Theoretical Contributions

Rethinking the Scope of Oversight

A key theoretical contribution is the expansion of oversight beyond reactive verification and run-time monitoring. The authors compellingly demonstrate that oversight also comprises prior control and proactive collaborative planning, thus significantly broadening its operational window in the development lifecycle. The distinction between anticipatory (preventative and proactive) and responsive (reactive and evaluative) oversight is critical for both HCI researchers and systems designers.

Practical and Sociotechnical Implications

For system designers: The findings underscore the importance of exposing effective agent configuration mechanisms, facilitating more actionable inspection interfaces (e.g., real-time visualizations [xie2024waitgpt]), and providing advanced, context-aware prompt engineering support.

For practitioners: There is an emergent necessity for developers to shift professional skills from direct software craftsmanship to architectural assessment, oversight, and specification-driven development. This carries risks of eroding code authorship and increasing reliance on automation, emphasizing the need for active maintenance of technical know-how and "oversight literacy".

For organizations: The observed tension between regulatory mandates for meaningful human control and the efficiency-oriented heuristics adopted in practice poses governance and compliance challenges. Organizations must invest systematically in tooling, training, and workflow redesign to reconcile these competing demands, particularly in high-stakes or regulated environments.

Research Directions

There are several promising avenues for future empirical and technical research:

• Oversight in Complex, Long-Running Scenarios: The present focus on atomic, short-lived agent tasks must be expanded to cover more sophisticated agent assignments and emergent workflows.
• Tooling for Transparency and Intelligibility: New designs are needed to reduce the cognitive gap between agent execution and developer comprehension, including reversible rationales, granular time-stamped action logs, and actionable summary artifacts.
• Skill Transformation and Professional Development: Further research is warranted on the longitudinal impact of agent delegation on individual developer skillsets and identity, as well as on organizational adaptation to shifting oversight roles [daly2025_managementskillagents, weidmann2025_managementleadership].

Conclusion

This study offers an empirically grounded, fine-grained analysis of agentic oversight in professional software development, documenting a clear evolution in oversight work toward proactive and preventative forms, and the systematic application of satisficing heuristics in the face of scale and uncertainty. The research advances both practical understanding and theoretical conceptualization of oversight in sociotechnical systems, highlighting actionable interventions for tool builders, individual practitioners, and engineering organizations. Future progress in agentic software engineering will require a continued, human-centered focus on the nuanced realities of oversight, integrating empirical insights with evolving design and regulatory frameworks.

Figure 1: Participants' description of scenarios in which they use AI agents for software development depicts agent integration across design, implementation, verification, and maintenance phases.