Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code

Abstract: As AI coding agents (e.g., GitHub Copilot, Devin, OpenAI Codex, Cursor) submit pull requests to open-source repositories at scale, a key question arises: do human reviewers gradually lower their scrutiny for AI-generated code over time? We conduct a longitudinal within-reviewer analysis using the AIDev dataset, studying 400 repeat reviewers who collectively submitted 11,429 reviews over a seven-month observation period. Comparing each reviewer's early and late review episodes, we observe a population-level shift in approval rate from 30.1% to 36.8% (Wilcoxon signed-rank p < 10^{-6} on paired shifts). Pooled by within-reviewer experience decile, the cumulative gap reaches +14.5 pp from first to tenth decile. This shift is experience-driven (persists after controlling for calendar time), agent-specific (human PR approval rates decline over the same period), and not explained by PR difficulty (median PR size is flat). However, review latency increases rather than decreases (+3.5x), while inline comment volume decreases (-22%, p=0.0014), suggesting reviewers spend more time in queue but less time actively inspecting code. The combination of rising approval, declining comment effort, and increasing queue time is most consistent with reflexive habituation under growing workload rather than rational trust calibration alone.

• The paper finds that human reviewers’ approval rates for AI agent code significantly increased by 14.5 percentage points over time, evidencing habituation.
• It uses a within-reviewer longitudinal split and controls for PR size and agent specificity to isolate trends unique to AI-generated code.
• The study highlights that reduced inline comments and increased review latency are linked to diminished scrutiny, underscoring risks of prolonged exposure.

Longitudinal Habituation in Human Review of AI Agent Code

Overview

The paper "Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code" (2606.22721) presents a rigorous longitudinal analysis of human reviewer behavior in the approval process for AI-generated code submissions (pull requests, PRs) on open-source repositories. Using the AIDev dataset (11,429 reviews across 400 repeat reviewers, spanning 207 days), the authors characterize intra-reviewer trends, uncovering a systematic increase in approval rates and reduction in active scrutiny for AI agent PRs over time. Contrasts to human-authored PRs and detailed analysis of review comment behavior provide evidence that observed behavioral shifts are agent-specific and consistent with reflexive habituation, rather than purely rational trust calibration.

Methodological Design

The study deploys a within-reviewer longitudinal split, sorting each reviewer's agent PR reviews chronologically and comparing behavioral metrics between early and late periods. Approval rate shifts are measured across deciles of reviewer experience, ensuring that findings are not artifacts of reviewer composition or calendar effects. Analyses control for PR difficulty via PR size, and for agent specificity by examining overlapping reviewer cohorts handling both human and agent PRs. Inline comment volume and review latency function as proxies for inspection effort versus queue time, dissecting the nature of changing reviewer attention.

Empirical Findings

Main Approval Shift

A repeated, statistically significant increase is seen in approval rates for agent PRs, rising from 27.9% in early reviews to 42.4% in the latest decile—a cumulative shift of +14.5 percentage points. Contrarily, change-request rates decline concurrently. This shift persists after adjusting for calendar effects, and is not explained by fluctuations in PR size, indicating it is truly reviewer-experience driven.

Figure 1: Comparative monthly trends in approval rates for agent-generated versus human-generated PRs, with agent PR approval rising while human PR approval declines.

Review Effort and Latency

Analysis of inline comments reveals a 22% reduction in comment count and a 28% reduction in comment word count per review, with a strong negative correlation (Spearman $\rho=-0.556$) between increased approval rates and comment decline. Review latency increases by a factor of 3.5, indicating reviewers devote longer queue time prior to submission, even as active inspection depth falls.

Agent vs. Human PR Control

Monthly calendar-based comparison demonstrates agent-specific behavioral adaptation: approval rates for agent PRs increase substantially, while those for human PRs decline in the same repositories and reviewer pool. By the end of the observation window, agent PRs are approved >10pp more often than human PRs, evidencing differential trust.

Cross-Agent Consistency

Within reviewers handling PRs from multiple AI agents, approval-rate shifts are strongly correlated across agents (e.g., Copilot, Devin, Codex), implicating reviewer-general adaptation as opposed to agent-improved code quality. However, smaller reviewer populations for certain agents (e.g. Cursor, Codex) limit power for firm agent-specific conclusions.

Interpretation and Implications

The joint pattern of rising approval rates, declining inspection effort, and increasing review latency is most consistent with reflexive habituation—a behavioral shift driven by repetitive exposure and positive prior outcomes, as opposed to deliberate trust recalibration. Teams deploying human oversight for agentic software development must recognize the threat: reviewers become less vigilant with prolonged exposure, underscoring the need for active mitigation.

Proposed practical interventions include:

• Rotation policies to limit prolonged reviewer-exposure to agent PRs.
• Streak audits targeting long consecutive approval runs for secondary inspection.
• Trend dashboards tracking reviewer trajectories and defect outcomes.

These results extend previous findings on automation-induced complacency in software engineering, security, and aviation processes, highlighting specific vulnerabilities in the oversight layer for AI agent code integration.

Theoretical and Future Directions

Empirical behavioral adaptation among human reviewers is a critical variable for future AI safety and SE workflows, challenging the assumption that human review functions as a consistent gate on agentic output. There are opportunities for developing algorithmic reviewer assignment models that optimize for vigilance, and for empirical integration of behavioral signals in automated review pipelines. Further research should address causal characterization leveraging more granular agent code quality measures and extend to enterprise repositories and longer time frames.

Conclusion

The analysis demonstrates that human reviewers systematically become more approving and less scrupulous when reviewing AI agent-generated code over time, independent of calendar effects or PR difficulty. The behavioral shift is agent-specific and correlates with reduced inspection effort, implying habituation rather than rational trust updating. These findings have direct implications for quality control in agentic software engineering, mandating counterbalancing oversight mechanisms to sustain rigorous review standards.