Supervisory Engineering Work

• Supervisory engineering work is defined as the focused effort to direct, evaluate, and correct AI-generated code during software development.
• The concept emerges from mixed-methods research that highlights a shift from traditional code writing to a creation-to-verification balance, with measurable changes in task allocation.
• It involves practical actions such as prompt crafting, output assessment, and error correction, underlining its influence on developer productivity and experience.

Supervisory engineering work is a proposed category of software engineering effort associated with the adoption of AI coding assistants, defined as the work of “directing, evaluation, and correction of AI output”. In the most explicit formulation, it encompasses “specifying intent, crafting prompts, providing context, and iterating when output misses the mark,” “reading AI output and deciding what to accept, modify, or reject,” and “fixing errors, integrating output into existing code, and maintaining consistency” (Vella et al., 22 May 2026). The term was introduced to explain an observed reallocation of effort: developers reported spending less time on many traditional development tasks, especially writing code, yet the corresponding increase in conventional categories such as testing or review was too modest to fully account for where effort had moved (Vella et al., 22 May 2026).

1. Definition and conceptual boundaries

Supervisory engineering work is narrower than “all work with AI” and refers specifically to the engineering effort required to manage AI-generated output during software development. Its three canonical components are directing, evaluating, and correcting. Directing includes specifying intent, crafting prompts, supplying context, and iterating when output is off-target. Evaluating includes reading generated code, checking whether it is correct, maintainable, secure, and relevant, and deciding what to accept, reject, or revise. Correcting includes fixing hallucinations, adapting output to project constraints, integrating generated code into an existing codebase, and preserving consistency with architecture and team conventions (Vella et al., 22 May 2026).

The concept is presented as distinct from, but adjacent to, established software-development categories. It differs from traditional coding because the engineer increasingly elicits, shapes, selects, and repairs code rather than directly producing every line. It differs from debugging because it begins before one’s own code fails and includes judging whether generated output should enter the codebase at all. It differs from code review because it includes prompt iteration, context provisioning, and real-time triage during generation, not only retrospective inspection. It differs from testing because many verification acts around AI output are verification-like without necessarily being recognized by practitioners as “testing” in the conventional sense. The paper therefore treats supervisory engineering work as a category that “does not map cleanly onto traditional software development task categories” (Vella et al., 22 May 2026).

A plausible implication is that the term names a shift in the locus of engineering agency rather than a simple reduction in labor. The engineer remains responsible for acceptance, rejection, integration, and judgment, but the object of work increasingly becomes generated output rather than only self-authored code.

2. Empirical basis and methodological grounding

The concept is grounded in a longitudinal mixed-methods study of professional software engineers using AI coding assistants. Two questionnaires were administered six months apart, in October 2024 (Q1) and April 2025 (Q2) through Qualtrics using a convergent parallel mixed-methods design. The study recorded 224 responses at Q1, reduced to a final Q1 eligible sample: 158 after exclusions; 111 responses at Q2, reduced to a final Q2 eligible sample: 101; and a matched longitudinal cohort: 95 participants who completed both waves (Vella et al., 22 May 2026).

For some longitudinal analyses, item-level missingness produced slightly smaller matched samples: n = 88 for task-focus tests, n = 94 for developer-experience tests, and n = 95 for productivity tests. Attrition analysis used Fisher’s exact tests across gender, experience, role, language, company size, and country, with no significant differences (all $p > 0.05$) and small effect sizes ($V \le 0.23$), which the authors interpret as evidence that attrition was not systematic on measured demographics (Vella et al., 22 May 2026).

Quantitatively, the study used descriptive statistics, one-sample and paired Wilcoxon signed-rank tests, rank-biserial correlation $r_{rb}$, Spearman correlations, and Holm-Bonferroni corrections. Qualitatively, it used reflexive thematic analysis following Braun and Clarke, with inductive coding in NVivo, reflexive memoing, counter-example seeking, and iterative discussion. Supervisory engineering work was not measured through a single direct questionnaire item. It emerged as an interpretive construct from convergent quantitative and qualitative patterns, particularly from the gap between reduced time spent on conventional creation tasks and only modest increases in named verification tasks (Vella et al., 22 May 2026).

This methodological basis is important because the construct is empirical but not yet psychometrically stabilized. The paper explicitly treats it as a proposed category requiring further validation rather than as a settled taxonomy.

3. Task reallocation and the creation-to-verification shift

The principal quantitative context for supervisory engineering work is a broader creation-to-verification shift. The study examined six task categories: designing, writing code, refactoring, reviewing code, testing, debugging. By Q2, 82% reported spending less time on writing code. Mean perceived time allocation for writing code was Q1 $M = 2.10$ and Q2 $M = 1.93$ on a 1–5 Likert scale where 3 is neutral, and only 2% reported spending more time writing code at Q2 (Vella et al., 22 May 2026).

Other tasks were affected differently. Refactoring and testing were also below neutral; designing and debugging were near neutral; and reviewing code was the only task above neutral at both time points, with Q1 3.03 and Q2 3.15. In matched longitudinal analyses, individual task changes did not remain statistically significant after Holm-Bonferroni correction, but directional effects were notable: writing code showed $\Delta M = -0.18$, $r_{rb} = -0.35$, $p_{adj}=0.263$; testing showed $\Delta M = +0.25$, $r_{rb}=0.29$, $V \le 0.23$0; reviewing showed $V \le 0.23$1, $V \le 0.23$2, $V \le 0.23$3 (Vella et al., 22 May 2026).

The authors then grouped tasks theoretically using the Vee model into creation tasks—designing, writing, refactoring—and verification tasks—reviewing, testing, debugging. These were explicitly described as theoretical distinctions rather than psychometric scales, with low internal consistency: creation $V \le 0.23$4–$V \le 0.23$5 and verification $V \le 0.23$6–$V \le 0.23$7. Even so, the balance between the two shifted significantly: creation changed from $V \le 0.23$8, verification from $V \le 0.23$9, and the balance (Verification - Creation) moved from $r_{rb}$0 with $r_{rb}$1, $r_{rb}$2, and $r_{rb}$3 (Vella et al., 22 May 2026).

Supervisory engineering work is proposed to account for the fact that this reallocation is not exhausted by classical verification labels. The paper argues that engineers often do not classify activities such as deciding whether to accept a suggestion, scanning generated output for errors, or checking whether generated code integrates correctly as “testing” or “code review” in the traditional sense. This suggests that part of the reallocated effort appears in a verification-like but taxonomically under-specified form (Vella et al., 22 May 2026).

4. Internal structure: directing, evaluating, and correcting

The qualitative themes give the concept its operational content. In Theme 3 of RQ1, “From doing to supervising AI-generated code,” participants described a shift from directly producing code to supervising AI-generated output. The paper notes that although some participants characterized this as a move toward “design” or “higher-level thinking,” their accounts more consistently described supervision-oriented activities: directing the assistant, evaluating its suggestions, and deciding what to accept, modify, or discard (Vella et al., 22 May 2026).

Directing is the most upstream component. It includes intent specification, prompt construction, context provision, decomposition of requests, and iterative steering when the model’s output is misaligned. Participant reports such as “Sometimes, I'll not be happy with the original reference, and will ‘coach’ the AI to produce something that works better for my use-case” illustrate that prompting is not incidental but an active engineering act (Vella et al., 22 May 2026).

Evaluating is the central verification layer. It includes reading generated code, checking whether it is correct, maintainable, secure, and relevant, and judging whether the model has “lost the thread.” The study highlights an increase in precisely these activities, even where conventional review or testing metrics show only modest change. One participant described the added burden as increased time spent on “‘is this code correct’, ‘is it maintainable’, ‘is it secure’ and ‘is this just probabilistic hallucination’” (Vella et al., 22 May 2026).

Correcting is the downstream integration layer. It includes fixing hallucinations, repairing errors, adapting code to local conventions and architecture, and maintaining consistency with the surrounding codebase. It is not limited to defect removal. It also includes codebase alignment and constraint satisfaction at the project level. The paper therefore treats correction as broader than debugging in the ordinary sense (Vella et al., 22 May 2026).

This tripartite decomposition also resonates with adjacent engineering practice outside software coding. In DUCTILE, an LLM agent performs adaptive orchestration while deterministic tools perform verified engineering computation, and engineers retain “supervision and judgment” through plan review, execution oversight, output validation, and final approval (Pradas-Gomez et al., 10 Mar 2026). This suggests a broader family resemblance between supervisory engineering work around coding assistants and human-supervised agentic engineering systems: the human role shifts toward framing, monitoring, validation, and sign-off rather than disappearing.

5. Productivity, developer experience, and the supervisory burden

A central finding is the productivity-experience paradox. Perceived productivity remained high and stable: Q1 $r_{rb}$4, $r_{rb}$5, 84% reported improvement; Q2 $r_{rb}$6, $r_{rb}$7, 84% reported improvement. In the matched sample ($r_{rb}$8), productivity change was $r_{rb}$9, $M = 2.10$0, $M = 2.10$1, $M = 2.10$2, with 77% giving identical productivity ratings over time, 14% decreasing, 9% increasing, and no one transitioning to negative productivity perceptions (Vella et al., 22 May 2026).

At the same time, developer experience worsened for a substantial subset. Using the DevEx dimensions of feedback loops, cognitive load, and flow state, the matched cohort ($M = 2.10$3) showed feedback loops $M = 2.10$4, $M = 2.10$5, $M = 2.10$6, $M = 2.10$7; cognitive load $M = 2.10$8, $M = 2.10$9, $M = 1.93$0, $M = 1.93$1; and flow state $M = 1.93$2, $M = 1.93$3, $M = 1.93$4, $M = 1.93$5. The Negative cohort—defined as any dimension rated 1–2—nearly doubled from 14% to 27%, while the Positive cohort—all three dimensions rated 4–5—had only 37% retention (Vella et al., 22 May 2026).

The paper explicitly connects this deterioration to supervisory demands. Flow can erode because each suggestion introduces a cycle of prompting, waiting, reviewing, accepting or rejecting, and iterating. Cognitive load may not disappear but shift from manual production to interpretive burden, prompt refinement, monitoring for hallucination, and integration judgment. Feedback loops improve because answers, examples, and candidate code arrive quickly, but that speed does not remove the need for vigilance (Vella et al., 22 May 2026).

This suggests that supervisory engineering work is not “freed-up time” but differently allocated effort. The paper states exactly that for individual engineers: this is “not freed-up time but differently allocated effort.” A plausible implication is that the concept helps explain why throughput can improve even while the subjective quality of work degrades for some practitioners: rapid generation and rapid feedback coexist with persistent trust calibration and correction overhead.

6. Antecedents, adjacent patterns, and open questions

The term is new, but the underlying concern with supervision, judgment, and management burden has antecedents. Earlier work on software work conditions emphasized that supervisory behavior can shape the social and organizational environment in which software engineering is performed, including excessive hours, short-termism, and weak project management (Leipzig, 2013). That literature did not use the term “supervisory engineering work” in the AI-assistant sense, but it indicates that supervision has long been constitutive of software work rather than external to it.

In contemporary engineering automation, DUCTILE offers a closely related but distinct pattern. There, the agent performs adaptive orchestration, deterministic tools perform the substantive engineering computations, and the engineer remains responsible for plan approval, process supervision, output validation, and final judgment. The paper emphasizes inspectability, logging, and repeated-run evaluation, and it warns of “the tension between removing mundane tasks and creating an exhausting supervisory role” (Pradas-Gomez et al., 10 Mar 2026). This suggests that supervisory engineering work is not only a software-coding phenomenon but may generalize to a broader class of AI-mediated engineering practices in which humans supervise probabilistic orchestration layered over trusted tools.

The concept remains open in several respects. The originating study explicitly frames it as a proposal and asks whether it is “a genuinely new SDLC activity distinct from traditional categories like designing, coding, testing, and reviewing” (Vella et al., 22 May 2026). An alternative interpretation, also acknowledged in the paper, is that supervisory engineering work may be verification work that practitioners do not label as testing or review rather than a fully separate category. The evidence is based on perceptions rather than objective traces, the sample includes only continuing users of AI coding assistants, and the tool landscape changed rapidly during the study, with 37 tools mentioned and 82% of matched participants changing their tool combinations while the mean number of tools used rose from 1.9 to 2.9 (Vella et al., 22 May 2026).

The present state of the concept is therefore that of an empirically motivated interpretive category. It names an increasingly visible layer of engineering effort—direction, evaluation, and correction of AI output—while leaving open whether software-engineering taxonomies, team processes, and educational models should ultimately treat it as a separate activity or as a transformed form of verification and coordination (Vella et al., 22 May 2026).