Beyond Simpson's Paradox: A Cascade of Confounders in AI Agent Pull-Request Co-Authorship

Abstract: Pooled across five AI coding agents, pull requests (PRs) with a human Co-Authored-By trailer merge less often than purely-autonomous ones (53.8% vs. 79.8%) -- yet this aggregate finding is a textbook Simpson's Paradox. Stratifying 33,596 PRs from the AIDev dataset by agent identity reverses the conclusion: Copilot and Devin show large positive within-agent gaps (+41.2 and +33.5 pp, both p<0.001), while Cursor, Claude Code, and Codex show small effects whose cross-sectional 95% CIs span zero. The paradox is driven entirely by agent composition: Codex, which dominates 64.9% of the dataset, achieves high merge rates while rarely using co-authorship. But Simpson's Paradox is only the first layer of a cascade of confounders: within-repo controls eliminate Devin's gap (+33.5 to +1.6 pp, p=0.73); a commit-count control further halves Copilot's within-repo gap (+36.2 to +24.4 pp); restricted to multi-commit PRs, the Copilot within-repo effect dissolves to +4.8 pp (p=0.59). No agent retains a clear co-authorship effect once both repository selection and PR structure are controlled. Our findings caution against reporting agent-pooled statistics without stratification and demonstrate that cross-sectional co-authorship associations are largely selection and PR-structure artefacts rather than evidence of a causal benefit.

• The paper demonstrates that aggregate pull request statistics are misleading due to Simpson's Paradox, necessitating agent-stratified analysis.
• Rigorous controls using repository and commit count adjustments show that the apparent co-authorship benefits largely vanish.
• The study underscores that markers of human involvement primarily reflect workflow quality rather than providing direct causal benefits to merge outcomes.

Cascade Confounders in AI Agent Pull-Request Co-Authorship: Beyond Simpson's Paradox

Introduction

This work interrogates the relationship between human co-authorship, as operationalized by the GitHub "Co-Authored-By" trailer, and pull request (PR) merge outcomes for AI coding agents. Utilizing 33,596 PRs from the AIDev dataset, submitted by five distinct AI agents (Codex, Copilot, Devin, Cursor, Claude Code), the authors systematically reveal layers of confounding impacting naïve aggregate analysis—most notably a classic Simpson's Paradox—before dissecting further sources of structural and selection bias through rigorous stratification and modeling.

Simpson's Paradox and Agent-Stratified Analysis

Initial pooled statistics suggest a substantial negative association: collaborative (Co-Authored-By) PRs merge at 53.8%, whereas purely autonomous agent PRs merge at 79.8%—a -26.0 percentage-point (pp) difference, highly significant ($\chi^2 = 2424$, $p \sim 0$). However, stratification by agent identity reverses this conclusion for four of five agents, with Copilot and Devin showing large positive within-agent co-authorship effects (+41.2 and +33.5 pp, $p<0.001$), and Codex, Cursor, Claude Code exhibiting gaps whose 95% CIs cross zero.

This reversal is wholly attributable to agent composition: Codex, which accounts for 64.9% of PRs, achieves high merge rates (82.6%) and rarely uses Co-Authored-By (1.2%). Conversely, Copilot and Devin—low merge rates, high co-authorship rates—depress the pooled collaborative merge rate. The single-agent dominance in aggregate statistics demonstrates that reporting pooled merge rates without agent stratification is fundamentally misleading.

Cascade of Confounders: Repository and PR Structure

The agent-stratified Simpson's Paradox is itself only the first confounding layer. Further controls—restricting to within-repository comparisons, adjusting for PR commit count—substantially attenuate observed co-authorship effects. For example, Devin's within-agent gap collapses from +33.5 pp to +1.6 pp ($p=0.73$), while Copilot's effect drops from +36.2 pp (within-repo, $p<0.001$) to +24.4 pp (commit-controlled), dissipating to +4.8 pp ($p=0.59$) for multi-commit PRs. No agent retains a statistically significant co-authorship effect when both repository selection and PR structure are controlled.

The primary Copilot signal reflects structural workflow correlates—pure-autonomous PRs are predominantly single-commit drafts (54.5% marked [WIP]); thus, co-authorship effects primarily capture PR maturation and curation rather than causal benefit of human involvement.

Collaboration Modes: Author/Committer Attribution

A more granular classification based on author and committer yields three dominant modes:

• Fully Autonomous: Bot authored and bot committed; merge rate 47.6%
• Agent Draft: Bot authored, human committed; merge rate 64.3%
• Human Both: Human authored and committed; merge rate 82.0%

This monotonicity persists post-commit-count control, indicating the author/committer signal is robust to PR structural confounders. The intermediate agent-draft category (64.3%) illustrates the compensatory role of human sign-off on agent-originated content.

Multi-Agent Repository Adoption

A difference-in-differences regression of weekly merge rates (233 multi-agent repositories) reveals that adoption of a second agent is associated with a significant decline in merge rates (-12.1 pp, SE = 3.6, $p<0.001$, 95% CI [-19.2, -5.0] pp). Whether this reflects lower quality contributions from the second agent or disruption to the incumbent agent's workflow remains unresolved.

Discussion

Measurement and Causal Ambiguity

The use of Co-Authored-By as a proxy for collaboration is agent-inserted and non-standardized; its conservative application for some agents (Copilot, Devin, Cursor) limits generalizability and introduces measurement bias. The observational nature of the study precludes causal inference; selection effects likely dominate—developers may selectively co-author agent PRs perceived as worthwhile, with apparent co-authorship effects predominantly marking quality and curation rather than creating it.

Practical and Theoretical Implications

Aggregate statistics for pooled multi-agent datasets are dominated by agent composition and structural workflow; agent-stratified, repository-controlled, and PR-structure-controlled analyses are required to avoid misleading conclusions. The graded benefit of collaboration shown in author/committer classification, in contrast to the vanishing co-authorship effect under structural controls, counsels caution for practitioners and empirical researchers in both deployment and study of agentic workflows.

This work motivates a taxonomy of human involvement (fully autonomous $\rightarrow$ agent draft $\rightarrow$ co-authored $\rightarrow$ human both), with recommendations nuanced by agent identity and workflow structure, particularly for Copilot whose apparent co-authorship benefit is nearly fully absorbed by structural PR controls.

Future Directions

The results underline the importance of properly controlled, agent-stratified, and workflow-sensitive empirical studies for evaluating human-AI collaboration in software engineering. Future work should pursue randomized interventions, finer workflow instrumentation, and broader cross-agent generalization, incorporating PR size, task type, reviewer experience, and survivorship bias for further resolution.

Conclusion

The paper establishes that apparent aggregate effects of human co-authorship on AI agent PR merge rates are confounded by agent composition and PR structural artefacts. True within-agent analyses reveal reversals characteristic of Simpson's Paradox, but these effects vanish under repository and PR-structure controls. The practical impact is a caution against reporting pooled statistics, and a theoretical implication that surface-level co-authorship associations primarily indicate workflow and selection phenomena, not causal benefit.

For empirical AI-SE research and agentic tool practitioners, rigorous stratification and workflow control are imperative. Co-authorship markers function more as proxies for PR quality and process than as evidence for causal human value-add in agent-submitted PRs.