GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis

Abstract: Does GitHub Copilot (GHCP) make engineers more productive, or do the engineers who use it more differ from those who use it less? And even within a single engineer, are GHCP-heavy weeks just busy weeks in which more of everything gets done? We study these questions using 43 weeks of data from 16,223 software engineers across Microsoft's Cloud+AI organization. Engineer fixed effects address the first concern by comparing each engineer against themselves rather than against other engineers, eliminating time-invariant differences in skill, role, and team. Active coding time and browser time then enter a Poisson Pseudo-Maximum Likelihood model with two-way fixed effects to address the harder, within-engineer confound: that GHCP-heavy weeks coincide with high-effort weeks. This defines our estimand as an efficiency effect: more pull requests completed at equivalent levels of coding time. Engineers are estimated to complete 40.5% more PRs in their highest GHCP usage weeks relative to their zero-usage weeks, holding measured development effort constant. The gradient is monotonic with diminishing returns at high intensity. Seven robustness and falsification tests target the remaining plausible alternative explanations (non-coding AI engagement, team-level shocks, within-week task reallocation, cross-week contamination, PR slicing into smaller units, shifts toward easier task types, and sensitivity to how the treatment is operationalized). Under an explicitly stated conditional-independence assumption, the within-engineer design estimates a tool-specific efficiency effect that is consistent with all seven robustness tests.

• The paper quantifies the dose-response effect of GitHub Copilot on productivity by analyzing over 16,000 engineers over 43 weeks.
• It employs a Poisson Pseudo-Maximum Likelihood model with fixed effects to reveal a 40.5% increase in pull requests during high usage weeks.
• Robust falsification tests confirm a monotonic, diminishing returns gradient, underscoring Copilot’s role in enhancing developers' efficiency.

Observational Dose-Response Analysis of GitHub Copilot and Developer Productivity

Study Objectives and Rationale

This paper investigates the causal relationship between GitHub Copilot (GHCP) usage intensity and developer productivity, specifically measured by completed pull requests (PRs) per week. The study addresses two primary confounds: (1) between-engineer selection bias and (2) within-engineer, time-varying confounding, both of which challenge the validity of non-experimental productivity measurements. By employing fixed effects at the engineer and week levels and controlling for coding and browser time, the authors attempt to disentangle Copilot's efficiency effect from extraneous correlates of productivity.

Methodological Framework

A Poisson Pseudo-Maximum Likelihood (PPML) estimator with two-way fixed effects is used on a panel dataset of 16,223 software engineers at Microsoft, covering 43 weeks and 413,732 engineer-week observations. Copilot usage is operationalized as interaction depth, split into four categories: Zero, Low, Moderate, and High. Coding and browser time serve as proxies for effort controls. The study defines the estimand as the "efficiency effect"—the increased PR output per unit of coding time attributable to GHCP, holding effort constant.

The model specification relaxes the proportionality assumption between coding hours and PRs, allowing for diminishing returns to effort, as demonstrated visually by the flattening slope in average PRs versus active coding hours.

Figure 1: Average PRs vs.\ active coding hours, with proportional reference line.

Within-Engineer Variation and Data Distribution

The analysis leverages within-engineer, week-to-week variation in Copilot intensity rather than cross-sectional comparisons, thus eliminating time-invariant confounds such as ability, role, and team allocation. The data evidences substantial intra-individual variation in both Copilot usage and PR output across weeks.

Figure 2: Weekly GHCP usage trajectories for three sampled engineers. Dot size = PRs completed; dot color = coding hours.

Dose-Response Findings

Copilot usage depth exhibits a strong monotonic association with PR throughput, producing a 41% increase in completed PRs in the highest-usage weeks relative to zero-usage weeks, controlling for coding and browser time. The dose-response relationship demonstrates diminishing returns at elevated usage levels—21% (Low), 39% (Moderate), and 41% (High).

Figure 3: Usage depth dose-response.

Robustness Analysis

Seven falsification and robustness tests address potential confounds:

• Placebo treatment: Substituting non-coding M365 Copilot usage shows no significant association with PR output, excluding generic AI engagement as the driver.

  Figure 4: Placebo treatment: non-coding M365 Copilot breadth vs.\ GHCP interaction depth as the treatment variable.

• Placebo outcome: GHCP usage does not predict teammates' PRs, excluding team-level shocks.

  Figure 5: Placebo outcome: own PRs vs.\ teammates' PRs by GHCP usage depth.

• Task-mix test: Weeks of high GHCP usage increase both authored PRs and reviews given, rebutting a within-week reallocation explanation.

  Figure 6: Task-mix test: GHCP gradient on PRs authored vs.\ reviews given, holding the specification fixed.

• Timing tests: Lagged and lead-week Copilot usage coefficients are negligible, supporting contemporaneous association and ruling out contamination from persistent productive states.

  Figure 7: Timing tests: current ($t$), lagged ($t{-}1$), and leading ($t{+}1$) GHCP usage coefficients.

• PR size decomposition: The dose-response is steepest for large PRs (7+ files), undermining the "PR slicing" artifact.

  Figure 8: PR size decomposition: dose-response by files touched (ADO subsample).

• PR-type decomposition: Excluding configuration-only/documentation PRs strengthens the gradient, ruling out easier task mix as confound.

  Figure 9: PR-type decomposition: dose-response for config/\allowbreak documentation-only PRs vs.\ all other PRs (ADO subsample).

• Alternative operationalization: Using usage breadth (days with Copilot activity) as the treatment yields a qualitatively identical monotonic gradient.

  Figure 10: Alternative treatment operationalization: usage breadth dose-response.

Limitations and Assumptions

The primary limitation is the reliance on an untestable conditional independence assumption: after accounting for fixed effects and effort, GHCP intensity must not be jointly determined by unobserved factors affecting PR output. Remaining threats include unmeasured motivational states or qualitative task difficulty. The outcome (completed PRs) captures only activity and efficiency, lacking dimensions such as code quality and communication. Conditioning on effort may understate the total effect by absorbing Copilot's time-savings impact.

Implications and Future Directions

The findings implicate Copilot as a tool that enhances engineering efficiency, raising output per coding hour, with diminishing marginal utility at high usage intensity. From a practical perspective, this supports Copilot's continued deployment and integration as a means to augment throughput without altering team composition or skill levels. Theoretically, this provides empirical evidence for the productivity-enhancing capabilities of LLM-based coding assistants in real production settings, and narrows but does not fully resolve non-experimental causal identification. Future research may explore alternative outcome metrics, longitudinal effects beyond immediate throughput, and integration with quality and collaborative measures, as well as attempts at natural experimentation if feasible.

Conclusion

The study presents credible evidence that, controlling for effort, engineers complete up to 40% more PRs in their highest intensity Copilot usage weeks. This monotonic, saturating association persists through a rigorous falsification battery, ruling out generic AI engagement, task-mix reallocation, team-level shocks, and measurement artifacts. While ultimate causal certainty cannot be achieved absent experimentation, the methodology narrows the set of plausible confounds and provides actionable insight into Copilot's role in augmenting developer productivity within organizational settings (2606.00438).