- The paper introduces ADEMM, an adaptive monitoring method that combines recurring surveys, interviews, and problem-owner review to assess developer efficiency without direct access to work environments.
- The study followed 27 developers across 12 survey cycles and 18 interviews, evolving the instrument from 49 Likert items to targeted binary measures while adding questions about emerging Generative AI-related friction.
- The method produced actionable insights, including a new requirements-analysis course, but its findings remain limited by self-reported data, non-response risk, single-site validation, and the absence of a controlled comparison with a static instrument.
Motivation and problem statement
The paper addresses a structural gap in the empirical literature on developer productivity and efficiency: most longitudinal studies assume that the monitoring organization either employs the observed developers or has access to their internal infrastructure. This assumption fails for consulting firms, professional education providers, bootcamps, and staffing platforms, whose business model is precisely to support professionals they do not employ and whose work environments they cannot observe. The authors further argue that existing longitudinal instruments (e.g., the fixed experience-sampling instruments of Kuutila et al., Russo et al., and Meyer et al.) are defined a priori and never revised, so factors that emerge during collection — such as friction introduced by Generative AI tools — systematically fall outside observation until a new study begins. The research question is: How can developer efficiency be monitored longitudinally and adaptively without direct access to the work environment?
The paper is careful about terminology: it distinguishes efficiency (the cost angle — time, effort, friction required to achieve an outcome, consistent with the SPACE efficiency/flow dimension and flow-efficiency metrics in the Lean literature) from productivity (the achieved outcome), noting that the empirical literature it builds upon mostly measures productivity. ADEMM deliberately targets the cost and friction dimension rather than aggregate delivery.
Research design
The study follows Design Science Research (Peffers et al.) combined with Action Design Research, treating ADEMM itself as the artifact and the concrete surveys/interview scripts as situated instantiations. It was conducted between February and May 2026 within the Dev Eficiente consulting practice in Brazil, with 27 developers (12 senior, 10 mid-level, 5 junior) employed at distinct companies. Data collection comprised twelve survey cycles over roughly four months — weekly for weeks 1–7, biweekly for weeks 8–12 — with respondent counts ranging from 13 to 25 per cycle, plus 18 semi-structured interviews in two qualitative rounds. Quantitative analysis used descriptive statistics and mixed-effects models; qualitative data were analyzed with deductive and inductive thematic analysis (236 codes consolidated into seven themes, including dependencies/communication, planning/prioritization, technical knowledge, and AI use as both support and friction). A problem owner — the lead of the consulting practice — participated in all evaluation meetings and held authority over item retention, removal, and inclusion.
The artifact and its evolution
ADEMM comprises three integrated components: a recurring quantitative channel (closed binary items plus two open-ended questions), a qualitative channel (semi-structured interviews) that feeds instrument redesign, and a joint review process with the problem owner applying explicit evolution criteria. Item removal requires two converging signals: low response variance across the iteration and the problem owner's judgment that the item is not actionable. Inclusion follows recurring qualitative patterns. The instrument went through five versions: Survey I (49 Likert items in twelve factor groups) was judged too long for weekly cadence; Survey II reduced it to 19 binary items with open questions; Survey III removed two low-variance, low-usefulness items (work process; psychological safety); Survey IV was deliberately unchanged, preserving comparability; and Survey V removed four further low-variance items and added three items on AI use, responding to an emergent interview theme. A work diary tested in Iterations 2–3 received a single response in total and was discontinued — a data-driven negative result the authors document explicitly.
The evaluation against four criteria (emerging-factor identification, adaptability, practical usefulness, cross-iteration comparability) produced one concrete organizational outcome: a new course on requirements analysis was created mid-study in response to two persistent findings — dependence on other areas and gaps in business/requirements understanding. This demonstrates that the method generated actionable knowledge during, not after, execution.
Design principles and transferability
The iterative history was formalized into three design principles expressed in CIMO (context–intervention–mechanism–outcome) format: DP1, joint prioritization with the problem owner guided by actionability; DP2, combination of closed items and a qualitative channel per iteration; DP3, removal by low variance and inclusion by emerging qualitative signal. ADEMM is then abstracted into an eight-step, domain-independent process that operationalizes the six DSR activities, with explicit contextual variation points (cadence, item format, interview substitutes, formalization format).
The paper's contribution relative to a generic DSR/ADR application is stated precisely: DSR prescribes evaluation cycles but not item-pruning criteria, cadence, non-response handling (the authors adopt a fixed-denominator convention, N=27, noting it yields conservative percentages in low-response weeks), or the format of design knowledge. ADEMM resolves these five decision points with tested, documented conventions. A hypothetical second instantiation — monitoring freelance product designers — is used to argue that what transfers is the process logic, not instrument content; the specific cadence, binary format, and iteration count are local calibration decisions.
Limitations
The authors are candid that the design principles remain nascent design theory: no second instantiation in another domain has been conducted. Several weaknesses of the method itself are identified. The low-variance pruning criterion favors frequency over severity, so rare-but-severe factors could be discarded, and it cannot distinguish irrelevant from seasonal factors whose relevance temporarily disappears; the authors suggest a "hibernation" mechanism as a future extension. Actionability-based prioritization systematically deprioritizes structural barriers outside the organization's control — the external-validation item was removed in Iteration 5 on exactly these grounds. The diary's low adherence was confounded by the absence of reminders or incentives, so self-report instruments were not conclusively ruled out. Interview timing was not precisely mapped to collection weeks, precluding fine-grained quantitative cross-referencing.
On the validity side, the study measures self-reported perceptions, not objective output; there is no control group, no counterfactual, and no comparison against a static instrument, so the claim that adaptation improves the instrument is supported logically rather than experimentally. Non-response bias is a live concern, particularly in Week 9, where sharp drops in monitored factors coincided with low adherence. External validity is limited to a single Brazilian program; self-evaluation bias and artifact–organization co-evolution further complicate attribution of the positive evaluation to the method itself.
Conclusion
ADEMM demonstrates that longitudinal, adaptive monitoring of developer efficiency is feasible without employment ties or infrastructure access, by structuring reliance on developer self-report through explicit pruning criteria, a qualitative channel that surfaces unanticipated dimensions, and problem-owner review anchored to organizational actionability. The three open questions the paper itself flags are concrete: replication in a non-software domain to move the principles beyond nascent design theory; a pruning criterion augmented with severity indicators; and a controlled comparison against a fixed-instrument alternative to test whether adaptation actually outperforms stability.