Establish whether documentation-based recovery is more effective than alternative recovery strategies

Establish whether using documentation as the first recovery action after a tool error, test failure, or build failure resolves agent failures more effectively than reading code, retrying the failed action, editing directly, or other recovery strategies.

Background

Among 2,034 observed failure episodes, documentation was the first recovery action in only 109 cases, and an observable resolution outcome was available for just 11 of those cases. Although documentation-based recovery had the highest point estimate for resolution, its confidence interval was wide and overlapped those of the other strategies.

The authors therefore leave unresolved whether documentation-based recovery has a genuine effectiveness advantage. A larger dataset with substantially more observable recovery outcomes is needed to compare the strategies reliably.

References

Documentation-based recovery has the highest point estimate of the resolution rate (7/11 = 63.6\%), but its 95\% interval is 35.4--84.8\% and overlaps every other strategy. We therefore report this as suggestive and explicitly not a finding: 11 episodes with an observable outcome cannot support a ranking of recovery strategies. Establishing whether documentation-based recovery is genuinely more effective is therefore an important motivation for a larger sample (Section~\ref{sec:threats}).

— From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation  (2608.20195 - Gao et al., 20 Aug 2026) in Section 4.3, “Failure recovery”