- The paper introduces 27 procedurally generated CAPTCHA families targeting five human–agent gaps: scene inference, temporal integration, counting, latent-state tracking, and perception-to-action alignment.
- The benchmark shows a 98.8% human pass rate versus 0.9–5.9% for six frontier GUI-agent systems, with several CAPTCHA families producing zero successful agent attempts.
- The results indicate that more reasoning and higher spending provide little benefit because current failures arise mainly from interactive grounding and precise browser actions, though durability against adaptive agents remains uncertain.
Motivation and threat model
The paper addresses the collapse of the "bot-hard" assumption that underpins CAPTCHA-based web security. The authors trace three prior arms-race cycles—distorted text versus OCR, image grids versus object recognition, and logic puzzles versus multimodal LLMs (MLLMs)—and document that each defense was eventually neutralized. Their empirical motivation is stark: on existing deployed CAPTCHA systems, frontier models such as Gemini3-Pro and GPT-5.2 reach 100% Pass@1 on traditional logic puzzles, and 80–90% on complex reasoning tasks like Bingo when driven by computer-use agents such as Claude-Cowork-Opus4.5. The authors argue that a machine pass rate of even 50% constitutes a critical security failure, since a single successful breach can expose sensitive data.
The qualitative analysis of agent trajectories explains why current CAPTCHAs fail: they reduce to short, decomposable workflows of locally verifiable micro-decisions followed by straightforward UI operations—a regime where tool-using agents are highly reliable. A correlation analysis between per-family Pass@1 and logged trajectory metrics confirms this structurally: for current CAPTCHAs, success correlates non-trivially with interaction length and negatively with reasoning-token expenditure, whereas for Next-Gen families these correlations vanish to near zero. This motivates the paper's central design thesis: rather than making decomposable puzzles harder, defenses should target the persistent human–agent gap in interactive perception, memory, decision-making, and action.
The authors model GUI-agent CAPTCHA solving as an extended POMDP W=(S,O,X,Aweb,Athink,Z,Tenv,U,R,κ), in which the agent receives partial observations ot=(It,Dt,metat)—a screenshot, a filtered DOM-derived interaction view, and metadata—and maintains an internal workspace updated by xt+1=U(xt,ot,atthink). This formulation organizes five targeted cognitive-gap categories:
- G1 — Scene-structure inference: brittleness in extracting task-relevant structure from observations (depth cues, reflections, shadow geometry).
- G2 — Temporal integration: decisive information revealed only through multi-step interaction (motion cues, sequential reveals).
- G3 — Numerosity and discrete invariants: verifier reward highly sensitive to small perceptual misreadings of counts or parity.
- G4 — Latent-state tracking: carrying intermediate variables across steps under working-memory constraints.
- G5 — Perception-to-action alignment: reliably translating correct internal decisions into precise browser primitives such as drag-and-drop.
The resulting system comprises 27 newly designed CAPTCHA families (e.g., Mirror, Structure From Motion, Hole Counting, Box Folding, Static/Dynamic Jigsaw, Red Dot), of which 25 are fully procedurally generated with rule-defined ground truth encoded at generation time—"correctness by construction" rather than external checking. For backend-supported types, instance generation is effectively unbounded, resisting memorization and template matching. Verification is server-side with state-based predicates, action-consistency checks, and anti-replay nonces with short TTLs.
Data curation pipeline
The pipeline filters candidate families using a lightweight stress test: a family is retained only if Gemini 3 Flash achieves below 30% Pass@1 on a 20-instance pilot set while humans exceed 90% success on a sampled subset. This yields the 27 retained families and two benchmark tiers: a main set of 519 puzzles and a budget-friendly subset of 135 (5 per family). Two families rely on curated human-crafted instances; the rest are script-generated. The benchmark is explicitly positioned as a snapshot of a broader deployable defense system rather than the primary artifact.
Main results
Under a fixed Browser-Use evaluation protocol on live web episodes (Playwright, visible mode, state reset per puzzle), the human–agent separation is large:
| Backbone |
Avg Pass@1 (%) |
Total cost ($)</th>
</tr>
</thead><tbody><tr>
<td>Human</td>
<td>98.8</td>
<td>–</td>
</tr>
<tr>
<td>GPT-5.2-xHigh</td>
<td>5.9</td>
<td>3122.3</td>
</tr>
<tr>
<td>Gemini-3-Flash-High</td>
<td>3.2</td>
<td>6.5</td>
</tr>
<tr>
<td>Claude-Opus4.5-Ext-ThinkingHigh</td>
<td>3.0</td>
<td>224.0</td>
</tr>
<tr>
<td>Gemini-3-Pro-High</td>
<td>1.3</td>
<td>50.1</td>
</tr>
<tr>
<td>Doubao-Seed-1.8-Thinking-HighEffort</td>
<td>1.3</td>
<td>26.6</td>
</tr>
<tr>
<td>Qwen3-VL-Plus-ThinkingHigh</td>
<td>0.9</td>
<td>28.3</td>
</tr>
</tbody></table></div>
<p>Several families (e.g., 3D_Viewpoint, Dynamic_Jigsaw, Hole_Counting, Subway_Paths, Temporal_Object_Continuity, Trajectory_Recovery, Illusory_Ribbons, Multi_Script, Spooky_Text/Shape_Grid/Jigsaw) yield 0% across all backbones. The strongest result against any family is Claude-Opus4.5's 60% on Static_Jigsaw, indicating that drag-and-drop execution remains partially solvable by the most capable agentic stack—a residual weakness worth noting.</p>
<p>Three ablations sharpen the interpretation. First, <strong>reasoning helps only marginally</strong>: raising Gemini-3-Flash's <a href="https://www.emergentmind.com/topics/thinking-mode" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">thinking mode</a> from Minimal to High lifts Pass@1 from 0.6% to just 3.2%, despite reasoning-token usage growing from zero to 18.5K tokens—evidence that failures stem from interactive perception–action bottlenecks rather than insufficient deliberation. Second, <strong>cost does not buy success</strong>: GPT-5.2-xHigh spent roughly $3,122 for 5.9% Pass@1, breaking the compute–accuracy scaling typically observed on static benchmarks; high-reasoning models also require 16–77 minutes per puzzle, imposing a temporal asymmetry relative to the human average of 31 seconds. Third, framework choice matters but does not close the gap: on the 135-puzzle subset with Claude-Opus4.5 held fixed, CrewAI scores 0.00%, Browser-Use 1.48%, and Claude Cowork 4.44%.
Limitations and open questions
The paper concedes several constraints. Evaluation of the two most expensive backbones (GPT-5.2-xHigh, Claude-Opus4.5) was restricted to the 135-puzzle subset due to API latency and cost, so their full-set averages are not directly comparable to the other four backbones evaluated on all 519 puzzles. The human study is small-scale, and the authors acknowledge that accessibility—alternative modalities, localization, adaptive difficulty—requires further attention before deployment. The defense's durability is inherently contingent: it exploits gaps in current agents, and the near-zero trajectory correlations do not preclude future agents with stronger grounding and manipulation capabilities from closing them. Whether procedural diversity and server-side verification suffice against adaptive adversaries trained specifically on these families remains an open question, as does the scalability of the human-friendliness guarantee as difficulty knobs are tuned adversarially.
Conclusion
This paper documents empirically that reasoning-heavy MLLMs driving GUI agents have collapsed the security margin of existing CAPTCHA systems, and proposes a generative, automatically verifiable defense framework built around five categories of human–agent cognitive gap. The headline result—a 98.8% human pass rate against a maximum of 5.9% for the best frontier backbone, achieved at prohibitive attacker cost and latency—demonstrates that interactive grounding and action-execution bottlenecks currently provide a more robust discrimination boundary than static perceptual or logical difficulty. The released platform and benchmark offer a standardized means of tracking whether this margin persists as agent capabilities advance.
Paper to Video (Beta)
No one has generated a video about this paper yet.
Whiteboard
No one has generated a whiteboard explanation for this paper yet.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Open Problems
We haven't generated a list of open problems mentioned in this paper yet.
|