Attribute keyboard-detection differences across assessment configurations

Determine how much of the difference in recovery of keyboard-related accessibility findings between interactive worker agents and noninteractive vision-language models is attributable to browser interaction, instructions, tools, and the evidence extracted by each procedure.

Background

The evaluation reports that worker agents recover more keyboard-related reference cases than the noninteractive vision-LLM configurations, but the tested configurations differ in several respects at once. In particular, they vary in interaction capabilities, instructions, available tools, and the evidence supplied to the model.

Because these factors are confounded in the archival comparison, the observed performance difference cannot be assigned specifically to browser interaction. The authors explicitly leave unresolved how much each factor contributes, making controlled experimentation with separately varied components necessary.