Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

Abstract: Organizations rolling out agentic command line tools like Anthropic's Claude Code and GitHub's Copilot CLI need to know who will try them, who will keep using them, and whether the tools produce enough output to justify their cost. At organizational scale, token spend can run into millions of dollars annually, so misreading adoption, retention, or impact can make a rollout expensive without changing engineering velocity. Studying tens of thousands of engineers at Microsoft over its early-2026 rollout, we find that first use spread primarily through social networks, retention was associated more with engineers' coding activity than with demographics, and adopters merged roughly 24% more pull requests than they would have otherwise. We use merged pull requests as our proxy for output -- acknowledging that a merged PR is not the same as the value it delivers -- and the lift persists across our four-month window. These results suggest that CLI coding agents are neither uniformly adopted nor mere novelty effects and that organizations should treat visible peer use as central to rollout strategy.

• The paper demonstrates that social exposure, especially skip-level peer usage, increases CLI AI tool adoption by up to 216%.
• It employs discrete-time logistic regression and fixed-effect Poisson models on telemetry data to study adoption, retention, and productivity outcomes.
• Persistent throughput gains, including a 24% lift in merged pull requests, suggest significant enterprise ROI for agentic CLI tools.

Adoption and Impact of Command-Line AI Coding Agents in a Large Enterprise Rollout

Introduction and Context

This paper presents an organization-scale empirical analysis of the adoption patterns and productivity impact of agentic command-line AI coding tools—specifically, Anthropic's Claude Code and GitHub Copilot CLI—during Microsoft's early 2026 rollout. The motivation is rooted in the opaque return on investment for large-scale deployments of LLM-driven developer agents, given escalating token costs and uncertain organizational impact. The authors leverage a unique telemetry dataset involving tens of thousands of Microsoft engineers, leveraging both precise tool usage logs and human resource data to interrogate the social, behavioral, and demographic factors driving CLI tool uptake, retention, and downstream effects on code contributions.

Methodology

Adoption Study

The adoption analysis focuses on Copilot CLI, the tool with a well-defined rollout population, splitting adoption into two canonical phases: first use (initiation) and short-term retention. The initial-use model implements discrete-time logistic regression over engineer-week panels, employing week and division fixed effects, clustered standard errors, and five predictor sets (career stage, tenure, baseline PR activity, prior Copilot IDE use, social exposure). Retention is modeled as sustained usage within 14 days after first exposure.

Outcomes Study

To measure productivity lift, the outcomes component utilizes both a synthetic-control (BSTS-CausalImpact) approach and a within-person panel construction. The primary outcome is PR merges normalized over 28 days, controlling for seasonality, individual fixed effects, and team-level variation. Tool comparisons (Claude Code vs. Copilot CLI) and effect moderators (career stage, tenure) are rigorously probed using fixed-effect Poisson generalized linear models.

Adoption Dynamics: Social Contagion and Behavioral Signals

The analysis demonstrates that initial tool adoption is not homogeneous but highly modulated by social exposure. Skip-level peer usage—engineers working two reporting levels above or laterally—emerges as the dominant predictor, increasing first-use odds by up to 216% when over a quarter of skip-level peers adopt. Reviewer-peer and direct manager adoption also exhibit strong positive associations.

Interestingly, prior IDE-based Copilot use predicts higher rates of initial CLI try-out but lower retention, suggesting that engineers already satisfied with IDE-based assistance either revert or treat CLI tools as supplementary. Baseline PR activity also predicts both increased adoption and retention, especially among the most productive contributors.

Demographic moderators such as career stage and tenure exhibit negligible effects relative to social-behavioral predictors. Only the highest and lowest ends of the IC/manager ladders show minor gradients in adoption/retention. This aligns with self-reported qualitative feedback emphasizing that senior engineers can better decompose tasks and parallelize streams using agentic tools, while junior engineers may lack sufficient metacognitive skills to validate or steer agent output.

Impact on Productivity: Persistent Throughput Gains

The organizational adoption of agentic CLI tools correlates with a statistically robust +24% lift in merged pull requests (95% CI: +14.5% to +33.7%), with no significant fade-out over at least four months post-adoption. This contrasts with prior literature on IDE-based AI tools, where productivity surges often attenuate after a few months [he2026cursor]. The monotonic, dose-responsive relation between weekly tool usage and PR merges peaks at +50% in five-use-day weeks, indicating that frequent users disproportionately accrue throughput gains.

When contrasting tools, Copilot CLI delivers a +24.9% lift in PR merges versus +11.4% for Claude Code, a more than 2$\times$ differential (p < 0.0001), despite external developer surveys initially rating Claude Code as more favorable for agentic work. The paper posits plausible explanations: task mix divergence or deeper integration/alignment of Copilot CLI with Microsoft's engineering infrastructure.

Moderation analysis reveals a "C"-shaped tenure effect—largest benefits accrue to the most and least tenured engineers (albeit small-$n$ caveats and possible onboarding conflation). Mid-level contributors see weaker but still-material gains. For career stage, both junior ICs and the most senior managers exhibit higher delta-PR throughputs compared to the modal IC4 reference.

Implications and Theoretical Consequences

These findings challenge the presumption of uniform or novelty-driven adoption in enterprise developer tools. Social network effects predominate, underlining the necessity for visible peer usage and active community reinforcement during tool rollout. The high persistence of throughput lift—contradicting fadeout trends observed in IDE-based tools—suggests a genuine transformation in developer workflows, with emergent behaviors such as parallelized task streams and increased willingness to tackle deferred or complex work.

From an organizational perspective, the magnitude of output gains (24%+) implies that token expenditure for agentic CLI tools is at least partially justified by raw throughput increases. However, the authors are explicit in noting the unresolved question of whether these throughput gains translate into commensurate increases in real business value—that is, the relationship between merged PRs and quality-adjusted, shipped product remains an open empirical question.

Limitations

Confounds such as peer homophily, policy-driven migration, and PR quantity vs. quality tradeoffs are acknowledged and partially mitigated through design (fixed effects, placebo checks, etc.), but definitively isolating causal impact remains challenging. The Microsoft context provides generalizability across a wide range of development types but may introduce integration and preferential-usage artifacts not present in less instrumented organizations.

Future Directions

The authors call for a research agenda centered on higher-fidelity measures of software value and quality in the wake of AI augmentation, including long-horizon, team-level, and quality-centric outcomes. They also note organizational design challenges in managing token economies at scale and structuring socially optimal adoption networks.

Conclusion

This study provides one of the most methodologically robust organizational analyses to date on agentic command-line AI coding agents. It presents direct evidence that adoption spreads predominantly along social ties and that agentic CLI usage yields persistent, substantial gains in code contribution velocity. While these results support organizational investment in agentic developer AI, they also crystallize key open questions—especially around quality metrics and long-term developer workflow evolution—that remain critical targets for the AI-in-software-engineering research community.

---

Reference:

"Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI" (2607.01418)