- The paper analyzes 94,813 development events from 557 agent sessions and 33,097 pull requests, using path classification, transcript extraction, clustered statistics, and repository-level artefact analysis.
- The study finds agent instructions and working notes account for 60.5% of documentation interactions, while API references represent 1.3% and troubleshooting guides 0.4%, indicating that documentation priorities should shift toward agent-facing files.
- Agents initiate 70.2% of documentation interactions, documentation appears in 41.5% of pull requests, and code usually precedes documentation, but the traces show no consistent documentation-to-validation sequence and leave read-to-code effects unresolved.
Overview and motivation
This paper by Gao and Chen (Peking University) reports a behaviour-grounded empirical study of how autonomous coding agents interact with technical documentation. Its premise is that documentation research has historically presupposed a human reader, while an increasing share of software changes is now authored by agents; consequently, guidance on "agent-friendly" documentation (clear headings, runnable examples, LLMs.txt) rests on intuition rather than observation. The authors invert the usual direction: instead of proposing documentation qualities and testing whether agents benefit, they observe what agents actually do with documentation in real development traces and derive design implications from those observations.
The study addresses three research questions: RQ1 — which document types, task stages, and interaction behaviours characterise agent documentation work; RQ2 — what precedes and follows documentation interaction, including triggers and downstream actions; RQ3 — whether the code–documentation loop is bidirectional, i.e., whether agents both consume and produce documentation and in what temporal order relative to code.
Data and method
The analysis combines two complementary public datasets that are deliberately never pooled. SWE-chat (Baumann et al., 22 Apr 2026) provides process evidence: 557 parseable sessions (sampled from 5,850 via stratification by agent and session length) yielding 94,813 development events, of which 3,033 are documentation interactions. AIDev (Li et al., 9 Feb 2026) provides artefact-level evidence: 33,097 agentic pull requests from the curated subset, with 690,260 file-level change records after removing null filenames and vendored paths.
Documentation identification uses a two-tier classifier over repository-relative paths. Tier 1 applies deterministic filename/path rules assigning one of 15 document types plus orthogonal flags (machine_readable, vendored). Tier 2 resolves ambiguous paths with a LLM: 527 distinct paths were classified, with 500 LLM-labelled (covering 98.4% of ambiguous events) and 27 assigned by keyword fallback. This tier surfaced the previously uncategorised agent_working_note class. Notably, the authors removed a "purpose" dimension from their coding scheme because intent cannot be recovered from tool-call logs without unfalsifiable inference.
Event extraction handles four incompatible transcript formats through format-specific extractors emitting a common schema over a 20-symbol event alphabet. Two extraction defects materially affected results and are reported candidly: failure to parse shell-embedded apply_patch paths (which would have registered zero documentation events for one agent family) and mishandling of non-string tool output (which excluded nine sessions and 4,358 events). Extracted tool-event counts matched SWE-chat's own tool_call_count exactly in five of six spot-checked sessions.
Statistical treatment is careful about non-independence. Because events nest within sessions and PRs within repositories (the largest AIDev repository contributes 8,911 PRs), primary uncertainty estimates use cluster bootstraps (2,000 resamples; sessions for SWE-chat, repositories for AIDev). Clustering widens intervals substantially — up to 14× on the AIDev side relative to Wilson intervals, which are retained only as independence-assuming references. For RQ2 action windows, a logistic GEE (exchangeable correlation, clustered by session) adjusts for development stage, within-session position, session length, and agent family. Headline proportions are additionally reported under three weighting schemes to correct deliberate oversampling of minority agents.
RQ1: What agents do with documentation
Documentation interaction occurred in 56.7% of sessions (cluster CI 52.6–60.5%), distributed across the whole trajectory rather than concentrated at task start. The central result is the document-type distribution:
| Document type |
Events |
Share |
| Agent instructions (AGENTS.md, CLAUDE.md, etc.) |
1,074 |
35.4% |
| Agent working notes (plans, thoughts/, logs) |
760 |
25.1% |
| Task / requirements |
301 |
9.9% |
| Configuration |
205 |
6.8% |
| README |
197 |
6.5% |
| Architecture / ADR |
120 |
4.0% |
| API reference |
40 |
1.3% |
| Troubleshooting |
11 |
0.4% |
Agent-facing artefacts account for 60.5% of all documentation interactions (session-cluster CI 53.9–66.5%; 55.1% under agent reweighting), whereas the nine genres constituting classical technical documentation account for 10.6%, API references for 1.3%, and troubleshooting documentation for 0.4%. The boundary around "classical" documentation is contestable — adding README, configuration, and requirements raises the share to 33.8% — so the authors restrict strong claims to the extreme categories. The dominance holds on both sides of the read/write split: 57.4% of consultation and 63.7% of production target agent-facing documents.
Interaction behaviours show five attested types: Read (1,328), Edit (1,007), Create (394), Search (282), Discover (5). Three types from the initial scheme — Compare, Follow-reference, Verify — are entirely unattested as tool-call behaviour and were removed rather than reported as rare. Production occurs at 0.87× the rate of consultation. Documentation reads occur in runs (P(read doc∣read doc)=0.270) or lead to reasoning (0.245); the assumed read-then-code pattern is nearly absent at the adjacent level: P(edit code∣read doc)=0.002, representing three occurrences among 1,328 reads.
RQ2: Triggers and downstream actions
Consultation is overwhelmingly self-initiated: 70.2% of all interactions (CI 66.7–73.3%) arise from agent initiative or implementation need, versus 7.5% failure-driven (tool failures dominate; test failures contribute only 7 events corpus-wide). Self-initiated interactions outnumber failure-driven ones by 9.3×. This contradicts the framing of documentation as a resource agents turn to when stuck.
The downstream-action analysis yields the paper's most statistically nuanced result. Within three events of consultation, running tests (lift 0.23, cluster CI 0.08–0.45; adjusted OR 0.39 [0.25, 0.60]) and building (lift 0.15; OR 0.25 [0.14, 0.44]) are less frequent than baseline, robust across both unadjusted and stage-adjusted analyses. Conversely, the two authoring outcomes are specification-dependent: documentation creation is elevated unadjusted (lift 1.67, CI 1.14–2.31) but its adjusted interval includes unity (OR 1.41 [0.98, 2.02]), while code editing shows no unadjusted elevation (lift 1.05) yet is elevated after adjustment (OR 1.33 [1.09, 1.62]). The authors attribute this discrepancy to stage confounding — consultation concentrates in particular trajectory phases — and treat reduced test/build activity as the finding while leaving any consultation-to-authoring coupling explicitly unresolved.
In failure recovery, reading documentation is the first move in only 109 of 2,034 episodes (5.4%). Documentation-based recovery has the highest point-estimate resolution rate (7/11 = 63.6%), but with 11 observable outcomes the interval (35.4–84.8%) overlaps every alternative strategy; the authors explicitly decline to rank recovery strategies on this basis.
RQ3: Consumption, production, and ordering
Both consumption and production are substantial. Within sessions, 58.2% of documentation-active sessions involve both reading and writing. Across pull requests, 41.5% of agentic PRs change documentation (repository-cluster CI 35.8–45.4%), code–documentation co-change occurs in 32.0%, and 9.6% change documentation only. The temporal direction is asymmetric: among multi-commit PRs where order is observable, code is touched first 4.7× more often than documentation; restricting to different-commit cases, code comes first in 82.5% (CI 78.7–86.0%). Documentation trails code, reproducing the asymmetry long documented in human comment-maintenance literature. Merge rates are not distinguishable once clustering is applied (81.1% vs. 75.0%, overlapping intervals).
A notable secondary loop: agents edit their own instruction files — AGENTS.md appears among the most-changed documentation files (692 PRs), along with CLAUDE.md (362) and copilot-instructions.md (287) — closing a feedback path from agent output back to agent input that existing documentation models do not capture.
The two-lobed cycle model
The authors began with a linear information-seeking journey (Discover → Retrieve → Interpret → Apply → Validate → Update) adapted from human-developer accounts. The traces do not support it. Two stages are entirely unattested: Validate (consultation followed by a test/build run) and Escalate have zero events under the operational definitions. Apply is weakly attested (75 events). Contribute/Update, the terminal stage in the linear model, is the largest category (1,401 events), exceeding Retrieve (1,344).
The revised descriptive model is a two-lobed cycle: a recurrent consultation lobe whose strongest transitions are back into itself (0.270) and into reasoning (0.245), loosely coupled to a largely independent production lobe. Neither lobe-to-lobe connection is consistent across statistical specifications, and no observed edge runs into validation. The authors offer two candidate mechanisms — externalised reasoning due to bounded context windows (documentation as working memory) versus reliance on the test suite as a cheaper oracle than prose — and note that distinguishing them requires future work. In either case, prose was not observed functioning as a specification.
Implications for documentation design
The paper separates supported implications from commonly asserted ones its data do not support. Supported: instruction files are the highest-frequency documentation surface (~27× more interactions than API references), suggesting prioritisation of their correctness; local retrievability matters given that reads cluster in runs while link-following is unattested; agent-authored notes constitute a new maintenance surface invisible to existing hygiene tooling; executable artefacts (doctests, schemas, runnable examples) are a hypothesis-worthy route to making validation observable; troubleshooting guides are marginal to agent recovery behaviour.
Explicitly unsupported: actionability presumes a read→act coupling that the data leave unresolved (adjacent probability 0.002; lift 1.05 vs. adjusted OR 1.33); verifiability describes zero observed behaviour; documentation as primary failure-recovery resource is contradicted by both trigger distribution and recovery analysis; and no recovery-strategy ranking is warranted at n=11. For tool builders, the paper warns that shell-centric agents hide file access inside command strings (so tool-name-based analyses systematically undercount) and that pre-2024 file taxonomies lack categories for agent instructions and working notes, relegating the majority of agent documentation interaction to a residual bucket.
Limitations and open questions
The paper is unusually explicit about threats. On construct validity, path-based identification misses docstrings and inline comments, so absolute rates are lower bounds; purpose is unmeasured by design; and critically, the Tier-2 labels underlying the 25.1% working-notes share rest on unvalidated LLM classification — no inter-rater reliability statistic has been computed, and dual human coding of a subsample is identified as the necessary next step. On internal validity, the stage heuristic is sticky (inflating debugging shares), outcome detection relies on regular expressions with unquantifiable selection effects, transitions are first-order (longer-range influence mediated through reasoning would be undetected), and trigger assignment uses a fixed four-event lookback that biases toward the largest category. On external validity, SWE-chat is opt-in telemetry with 87% of the corpus from a single agent family, AIDev over-represents early-adopter public repositories, neither generalises to private codebases, and the corpus is a snapshot of conventions roughly two years old — the authors expect the existence and prominence of agent-facing categories to persist but not the exact 60.5% share. No multiple-comparison correction is applied across strata. Open questions include whether documentation-based recovery is genuinely more effective, whether the consultation→code association survives better-controlled designs, and whether context-window constraints or oracle economics explain the absence of prose-based validation.
Conclusion
Across 557 real agentic sessions and 33,097 agentic pull requests, this study establishes that agentic development has produced a documentation genre that existing research has not studied: agent instruction files and working notes, jointly accounting for 60.5% of observed documentation interaction and written nearly as often as they are read, while API references (1.3%) and troubleshooting guides (0.4%) — the targets of most current tooling — are marginal. Three widely assumed mechanisms fail to receive consistent behavioural support: the read→code coupling remains unresolved across specifications, documentation rarely initiates failure recovery (5.4% of episodes), and no documentation-based validation sequence was ever observed. The practical implication is a reallocation of documentation effort toward instruction files; the research implication is that documentation now serves two audiences with measurably different behaviours, and the newer audience writes almost as much as it reads.
