How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study

Abstract: The rise of Software Engineering (SE) agents, i.e., LLM-based agents that can understand large codebases and carry out engineering tasks with limited human intervention, has been marked by rapid advances and adoption, but little is known about how developers build these systems in practice: existing studies mine repositories or examine deployment, but few investigate how SE agents are constructed. Through semi-structured interviews with 20 practitioners from 12 organizations and an online survey of 80 practitioners, this paper is the first to study how SE processes are changing in the development of SE agents and what challenges developers face. We find that as implementation becomes cheaper, bottlenecks shift rather than disappear: long-standing non-coding work such as requirements, coordination, review, and deployment becomes more visible, while reviewing and evaluating agent output becomes new and central. We characterize a seven-stage workflow and a shift toward evaluation-driven development, in which evaluation steers iteration and specifications become versioned artifacts read by both humans and agents. We further identify six challenges that teams face, together with the practices they adopt to address them, including unreliable evaluation signals, comprehension debt as code outpaces understanding, and behavioral changes introduced by provider-side model updates.

• The paper identifies a seven-stage SE agent development workflow that shifts emphasis from manual coding to evaluation-driven scaffolding and orchestration.
• The study employs a sequential mixed-methods approach—20 interviews and an 80-participant survey—to derive both inductive theory and deductive validation.
• Practitioners face challenges such as unreliable evaluation signals, provider-induced instability, and accumulating comprehension debt impacting maintainability.

Empirical Analysis of SE Agent Construction: A Mixed-Methods Perspective

Methodology and Study Design

The authors present an exploratory sequential mixed-methods analysis, operationalized through a rigorous sequence of semi-structured interviews (n=20, covering 12 organizations) and a broad-coverage validation survey ($n=80$) focused on the construction of software engineering (SE) agents—advanced LLM-driven systems automating SE tasks including code generation, debugging, review, and repository-level maintenance (2607.10856). The participant cohort skews toward R&D-intensive technology organizations at the frontier of agent adoption, encompassing both infrastructure engineers and practitioners building both general-purpose and domain/task-specialized SE agents.

The mixed-methods regime enables detailed inductive theory-building from interviews, iteratively refined via hybrid card-sorting analysis and member checking, with subsequent deductive validation against a larger practitioner base. This structure ensures both conceptual richness and broad generalizability.

Figure 1: Process overview—sequential design combines qualitative interviewing, thematic analysis, and validation via broad-scale survey.

Emergent Seven-Stage Agent Development Workflow

Empirical findings reveal a distinct seven-stage development workflow for SE agents, distinguished from classical software life-cycle models:

1. Requirements: Dual-audience specification targeting both human stakeholders and machine-readability for agents and automation.
2. Evaluation: Early, pervasive evaluation infrastructure as steering and selection mechanism, distinct from classic post-hoc verification.
3. Data: Contingent data curation for model adaptation—sometimes omitted for prompt-centric agents, but central for fine-tuning.
4. Construction: "Cheapest-first" escalation from prompt engineering to parameter-efficient fine-tuning or full SFT, stitched together by an extensible scaffolding/harness controlling context, memory, and tool-use.
5. Testing and Deployment: Combination of traditional correctness testing with system-level, task-oriented evaluation benchmarks.
6. Human Feedback: Instrumented deployment and continuous collection of operational, behavioral, and user-in-the-loop feedback.
7. Adaptive Maintenance: Ongoing evolution as upstream foundation models, APIs, or execution platforms are updated outside organizational control.

   Figure 2: Agent development workflow—depicts iterative, feedback-driven loop centering evaluation and scaffolding.

Notably, this workflow reflects a marked shift: implementation cost is dramatically reduced, abstracted away via agentic coding, and bottlenecks are displaced toward requirements engineering, orchestrating harnesses, evaluation, and review.

Process Reconfiguration: Evaluation-Driven, Spec-Centric, and Role-Agnostic

The study surfaces several process shifts driving SE agent development away from classical SE:

• Implementation at Higher Abstraction: Agentic coding automates code production, shifting expertise from micro-level coding toward system/harness design, context curation, and orchestrating tool interfaces. Empirically, practitioners no longer “write code” but instead review, refactor, and manage thousands of lines generated per iteration.
• Effort Unmasking and Creation: Traditional non-coding activities—requirements scoping, artifact curation, coordination—emerge as new bottlenecks. Simultaneously, new burdens around agent output review, provenance validation, and regeneration accumulate, displacing manual coding as the core effort.
• Evaluation-Driven Development: Evaluation becomes the central control surface and iterative feedback signal, co-evolving with requirements and used as the primary decision lever for progression, regression, and architecture evolution. Specification and benchmarks become first-class, version-controlled assets.
• Blurring Role Boundaries: Explosion in abstraction and automation leads to role fusion—engineers, researchers, and product managers frequently traverse traditional specialization as agentic scaffolds collapse the stack and automate previously siloed responsibilities.
• Executable Specifications as Core Artifact: Agent proficiency is increasingly determined by maintained, versioned prompts, step-definitions, and harness logic—these durable assets are engineered/tested/reviewed in parallel with code.

Central Challenges of the Agentic Paradigm

Practitioner interviews and survey ratings (strong consensus across most items, avg. agreement >80%) converge on six primary development challenges:

• Unreliable Evaluation Signals: Classic test-oracle issues are exacerbated—benchmarks quickly overfit, outcome metrics are gamed, and agentic runs are highly non-deterministic due to stochasticity, model evolution, and latent context/hardware/environmental variance. Teams respond with multi-layered, evolutionary, production-aligned evaluation stacks, but ground-truthing and cost control remain open.
• Provider-Induced Instability: "Change nothing, change everything": Model vendor updates can unpredictably shift agent performance, invalidate heuristics, and convert best-practice scaffolds into anti-patterns. Separation of enduring vs. workaround scaffolding is both necessary and difficult.
• Safety–Performance Trade-Offs: There is empirical evidence that teams knowingly accept safety regressions when safeguard mechanisms significantly degrade throughput or usability. Defensive measures are shifting beneath the prompt/harness layer into architectural sandboxes and privilege models.
• Tacit Contextual Knowledge: Despite sophisticated context retrieval and code ingestion, much project/domain knowledge remains unwritten and thus inaccessible to agents—unlocked only through human escalation. Explicit skill/rule encoding and progressive, dynamic context injection are preferred mitigations, but fundamental limitations rooted in tacit knowledge persist.
• Comprehension Debt Accumulation: Teams face a growing backlog of agent-generated code that exceeds review/understanding capacity and is less structurally rationalized than human code. Maintenance devolves to re-generation (regenerative software pattern), and understanding is shifted downstream.

  Figure 3: Comprehension debt formation and mitigation—code is accepted before understanding, leading to deferred review and increasing AI dependence or artifact regenerability.

• Productivity Metrics Obsolescence: Classical process metrics (e.g., LOC, PR count) decouple from delivered value as agent output volume explodes but the business value of voluminous code remains ambiguous or negative due to downstream review/maintenance costs.

Theoretical and Practical Implications

Theoretical

The paradigm outlined demonstrates that agentic SE development is not a process free-for-all, but an iterative, tightly evaluation-driven loop transforming the nature of process and artifact centricity. This shift away from deterministic, code-centered procedures toward dynamic, prompt- and evaluation-driven engineering raises foundational questions about the future roles of maintainability, explainability, and process governance.

The implication is that process models rooted in waterfall, spiral, or even agile/DevOps must adapt: agentic iteration generates new forms of debt (comprehension, regenerability) and new process anchors in harnesses and executable specifications. Research into process-aware orchestration, real-time evaluation signal evolution, and the impact of provider instability on socio-technical practices is now critical.

Practical

For practitioners, the findings suggest:

• Invest in Durable Scaffolding: Prefer modular, maintainable harnesses and clear distinction between temporary model-compensating logic and long-lived orchestration mechanisms.
• Prioritize Multi-layered Evaluation Protocols: Avoid overreliance on static benchmarks; blend production-aligned monitoring, multi-granular checks, and diversified success metrics.
• Expect Regenerability Over Artifact Preservation: Given rapid code drift and comprehension debt, invest in testing, spec, and infrastructure that enables quick re-generation, not just code-base curation.
• Accept Role Fluidity and Dynamic Specialization: Organizational structures and processes must support cross-specialization and rapid iteration, with individuals responsible for end-to-end pipeline slices.

Future Directions

This study highlights outstanding needs in automating review and comprehension support, dynamically robust evaluation protocol design, and in explicit handling of process risk induced by upstream vendor changes. Opportunities exist for agentic co-pilots specialized in review/maintenance, formalization of EDD/AgentOps protocols, and more robust validation/recoverability mechanisms.

A promising direction is meta-agentic design: using SE agents themselves for code review, maintenance, and even process adaptation—creating self-improving, evaluation-steering development ecosystems.

Conclusion

The paper establishes that construction of SE agents profoundly reconfigures software engineering practice: implementation is cheap, but the locus of engineering effort moves up the abstraction ladder to evaluation, coordination, specification, and maintainability. Classic process models yield to highly iterative, evaluation- and spec-centered loops, but new empirical and theoretical challenges, particularly in evaluation and maintainability, require urgent further research to ensure agentic engineering remains tractable and robust at scale.

Citation: "How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study" (2607.10856)