From Conversation to Contribution: Characterizing Coding Agent in Open-Source Software

Abstract: AI coding assistants such as GitHub Copilot and Cursor have evolved from code-suggestion tools into conversational collaborators, enabling vibe-coding workflows in which developers guide AI-generated code through natural-language dialogue. Although researchers have increasingly recognized the importance of AI coding agents and begun examining their impact on open-source development, a comprehensive understanding of how developers' chat-based interactions with AI relate to subsequent open-source development and collaboration remains limited. This hinders efforts to effectively design, evaluate, and govern AI-assisted open-source software development. To address this gap, we collected 13,360 AI conversation sessions comprising 79,172 user messages from 1,356 OSS repositories, linked them to repository development histories, and complemented this analysis with a targeted developer survey. We find heavier AI use in smaller, less mature, and less collaborative repositories. After AI adoption, projects tended to show more active contributors and lower contributor concentration (p < .001), although communication remained highly concentrated. Code Writing was the dominant chat purpose, and nearly all AI chat sessions were followed by subsequent commits. We find no broad deterioration in code-quality signals or pull request merging rates. However, developers perceive others' AI-generated code as harder to maintain than their own (p = .029) and view AI as lowering barriers to OSS contribution. While most developers (68%) are willing to share their chat, concerns remain around appearing incompetent, increasing reviewer burden, and exposing ideas to competitors. These findings provide a large-scale empirical characterization of AI-assisted OSS contribution and offer practical insights for designing and governing responsible vibe-coding practices in open-source development.

• The paper provides a mixed-methods analysis of 12,108 AI chat sessions across 1,240 repositories, using LLM-labeled intents, GitHub histories, interrupted time-series models, and a developer survey to characterize coding-agent use.
• The paper finds that AI use is concentrated in smaller projects, peaks early, and declines from 32.8% of post-adoption commits in month one to 4.4% after twelve months, while 98.9% of chats precede commits.
• The paper reports no broad deterioration in observable quality signals, but finds fewer issues, slower issue resolution, more pull-request activity, persistent communication concentration, and strong developer support for disclosure guidelines.

Overview and motivation

This paper presents a large-scale mixed-methods characterization of "vibe coding" — intent-driven, conversational AI-assisted development — in open-source software (OSS). The authors link developers' IDE-based chat sessions with AI coding assistants (Cursor, GitHub Copilot, Claude Code) to subsequent repository activity, addressing a gap left by prior work that examined only observable outcomes such as commits, pull requests, and code-quality signals (He et al., 6 Nov 2025, Agarwal et al., 20 Jan 2026, Robbes et al., 26 Jan 2026). The study is organized around three research questions: how OSS repositories use AI coding assistants, how repository dynamics change after adoption, and how developers perceive AI use in OSS.

The dataset comprises 13,360 AI chat sessions containing 79,172 user messages from 1,356 repositories, collected via SpecStory chat-history artifacts located through GitHub Code Search. After filtering inaccessible and trivial projects, 1,240 repositories with 12,108 sessions remained, spanning September 2024 to March 2026, alongside full GitHub histories (657,971 commits, 9,510 PRs, 12,747 issues, 120,489 CI records). A developer survey (25 valid responses, 4.2% response rate) supplements the repository-level analysis.

Methodology

The study combines three data sources. Chat-purpose labels were assigned using a seven-category behavioral-intent taxonomy (Code Writing, Failure Reporting, Delegation, Inquiry, Context Specification, Workflow Control, Validation) via an LLM classifier achieving macro-F1 of 0.83 against human annotations. Project-type labels achieved 90.5% accuracy on manual verification. Repository dynamics were analyzed relative to each repository's first observed AI-chat timestamp, excluding the adoption month; because chat histories can predate GitHub publication, only the 608 repositories (49.0%) whose first observed chat followed publication form the main cohort, validated on a more mature subset of 114 repositories.

Temporal trends were modeled with interrupted time-series (ITS) specifications including repository fixed effects, repository age, and clustered standard errors. Before–after comparisons used paired Wilcoxon signed-rank tests, with Benjamini–Hochberg FDR correction within test families. Concentration was quantified using top-contributor shares and Herfindahl–Hirschman Indices over contributors, commenters, reviewers, files, and modules.

How repositories use AI coding assistants

AI-related commits accounted for 44.3% of post-adoption commits on average (median 34.3%), but usage decayed rapidly: from 32.8% of post-adoption commits in the first month to 16.1% at six months and 4.4% at twelve months (Spearman $\rho = -0.940$, $p < .001$; binomial regression OR = 0.762 per month). This indicates AI assistance is front-loaded around adoption and progressively phased out as repositories mature. In multivariable models, only repository size (OR = 0.834) and contributor count (OR = 0.531) remained significant negative predictors of AI-use intensity; age, language, and domain showed no robust associations. The implication is that vibe coding is concentrated in smaller, less collaborative projects rather than uniformly distributed across OSS.

Chat purposes were dominated by Code Writing (dominant purpose in 53.9% of repositories), followed by Delegation (15.5%), Failure Reporting (12.7%), and Inquiry (8.9%). Notably, 98.9% of chat sessions were followed by subsequent commits, and 96.1% by commits touching development files — evidence that these conversations are tightly coupled to contribution activity. However, chat purpose did not statistically predict the file-type composition or overall size of subsequent commits after accounting for repository-level clustering; only source-code churn differed significantly across categories ($p=.001$, $q=.007$). Changes were broad at the file level (mean 25.7 files per AI-related commit) but concentrated at the module level (top module: 62.2% of changes on average).

Post-adoption repository dynamics

Commit activity declined after adoption: mean commits per repository-month fell from 19.2 to 7.7 ($p<.001$), and ITS models showed a short-lived post-adoption burst followed by a non-significant negative trend in the main cohort, with a significant decline in the validation cohort. This burst-then-decline pattern matches prior findings on Cursor adoption (He et al., 6 Nov 2025), lending external consistency.

Contrary to concerns raised in prior work about quality degradation, the paper finds no broad deterioration in observable code-quality signals. Bug/fix commit share rose from 9.6% to 16.9% ($p<.001$), but raw bug/fix commit counts did not increase, and ITS models showed no robust growth in defect-related activity — the share increase reflects reduced total commit volume rather than more defects. Test-touching commit share, CI failure rates, and CI success rates were essentially stable. This contradicts the persistent complexity increases reported for Cursor and autonomous agents elsewhere (He et al., 6 Nov 2025, Agarwal et al., 20 Jan 2026), though the discrepancy may stem from different cohorts, tools, or outcome measures.

Issue and PR activity shifted compositionally rather than expanding: issues became rarer (share of opened items fell from 29.6% to 19.2%, $p=.001$) but took substantially longer to resolve (mean resolution time rose from 10.5 to 29.4 days, $p=.002$), while PR openings increased and PR merge rates did not decline. Collaboration patterns showed modest broadening — active contributors increased from 1.4 to 1.7 per repository ($p<.001$) and contributor concentration decreased — but this pattern did not replicate in the validation cohort, where active-contributor share actually declined, so the authors caution it may reflect cohort composition. Communication and review remained highly concentrated throughout, with no significant change in any concentration measure.

Developer perceptions

Survey respondents viewed AI as lowering barriers to OSS contribution: 76% agreed AI assistance lowers contribution barriers, 80% said it makes a first contribution more comfortable, and the contribution-access construct scored above neutral (mean 3.8, $p=.002$). A notable asymmetry emerged: concern about others' AI-generated code was significantly above neutral (mean 3.9, $p < .001$0), while concern about one's own was not (mean 3.1), and respondents rated others' AI-generated code as imposing greater maintenance burden than their own ($p < .001$1). Respondents also strongly endorsed clearer disclosure guidelines (mean 4.2, $p < .001$2).

While 68% were willing to share their chat histories, non-willing respondents cited reputational risks (appearing incompetent), practical risks (reviewer burden), and strategic risks (exposing ideas to competitors or AI vendors). Respondents drew clear boundaries on appropriate use: most approved of AI for exploring solutions (92%), small bug fixes (84%), documentation (84%), and small features (80%), but opposed it for security-related changes (48%), large feature development (44%), and performance-critical changes (36%).

Limitations

The authors are explicit that the design supports associations, not causal claims: repositories may adopt AI while undergoing other concurrent changes that independently affect the observed outcomes. Generalizability is constrained by sample composition — roughly 90% of repositories were created in 2025 or later and are small or solo-maintained — although validation-cohort analyses suggest main patterns are not artifacts of this skew. The survey's 25 responses cannot represent the broader OSS population, though triangulation with repository data reduces reliance on it. Construct validity depends on LLM-based labeling (F1 = 0.83 for chat purposes) and proxy metrics (commits, test-file touches, CI outcomes) that do not directly measure correctness or maintainability. Findings are limited to public OSS during a period of rapid tooling evolution.

Conclusion

This paper provides one of the first large-scale linkages between developers' private AI conversations and subsequent public OSS development activity. Its central empirical findings — early-concentrated and decaying AI use, dominance of Code Writing, near-universal coupling of chats to commits, absence of observable quality deterioration, compositional shifts toward PR-based work with slower issue resolution, and persistent communication concentration despite broader contributor participation — collectively temper both optimistic productivity narratives and alarmist quality narratives. The results motivate lightweight transparency mechanisms (structured AI-use summaries in commits or PRs, repository-level AI-usage metadata) and leave open specific questions: whether AI adoption causally reshapes collaboration toward post-hoc review, how adoption differs between bootstrapped and mature projects, and whether disclosure norms can be designed that reduce stigma without imposing reviewer burden.