An Empirical Study & Evaluation of Modern CAPTCHAs (2307.12108v1)

Abstract: For nearly two decades, CAPTCHAs have been widely used as a means of protection against bots. Throughout the years, as their use grew, techniques to defeat or bypass CAPTCHAs have continued to improve. Meanwhile, CAPTCHAs have also evolved in terms of sophistication and diversity, becoming increasingly difficult to solve for both bots (machines) and humans. Given this long-standing and still-ongoing arms race, it is critical to investigate how long it takes legitimate users to solve modern CAPTCHAs, and how they are perceived by those users. In this work, we explore CAPTCHAs in the wild by evaluating users' solving performance and perceptions of unmodified currently-deployed CAPTCHAs. We obtain this data through manual inspection of popular websites and user studies in which 1,400 participants collectively solved 14,000 CAPTCHAs. Results show significant differences between the most popular types of CAPTCHAs: surprisingly, solving time and user perception are not always correlated. We performed a comparative study to investigate the effect of experimental context -- specifically the difference between solving CAPTCHAs directly versus solving them as part of a more natural task, such as account creation. Whilst there were several potential confounding factors, our results show that experimental context could have an impact on this task, and must be taken into account in future CAPTCHA studies. Finally, we investigate CAPTCHA-induced user task abandonment by analyzing participants who start and do not complete the task.

• The paper demonstrates significant solving time variability, with reCAPTCHA's click-based variant being the fastest and game-based CAPTCHAs requiring considerably longer times.
• The paper finds that user preferences extend beyond speed, as game-based CAPTCHAs remain popular despite their longer completion times.
• The paper reveals that experimental context markedly influences solving times and abandonment rates, emphasizing the need for realistic usability study designs.

An Empirical Study on the Evaluation of Modern CAPTCHAs

CAPTCHAs are an ubiquitous security tool used to distinguish between human users and automated bots on the internet. The paper "An Empirical Study on the Evaluation of Modern CAPTCHAs" explores the practicality and user experience of various CAPTCHA types currently deployed across popular websites. This paper builds on previous research to extend the understanding of CAPTCHA solving times, user preferences, and factors influencing user engagement.

To gather data, the authors conducted a multi-phase paper encompassing both website inspections and user-based evaluations. These evaluations included a manual inspection of 200 popular websites to identify the types of CAPTCHAs in use, followed by a user paper involving 1,000 participants recruited via Amazon Mechanical Turk (MTurk). Participants in the user paper were asked to solve 10 different CAPTCHA types to gauge solving times, preferences, and the contextual influence on user performance.

Key Findings

Solving Time Variability

The paper reaffirms previous findings on the substantial variability in solving times across different CAPTCHA types. The average solving time for distorted text CAPTCHAs ranged from approximately 9 to 15 seconds. ReCAPTCHA's click-based variant yielded the fastest median solving time at 3.7 seconds, while more complex game-based CAPTCHAs like Arkose Labs' puzzles took significantly longer to complete, often exceeding 18-42 seconds.

User Preferences

A critical insight from the research is the disjunction between solving times and user preferences. Although participants generally preferred CAPTCHAs that required less time to solve, game-based CAPTCHAs, despite having higher median solving times, were surprisingly well-received. This indicates that user preference is influenced by factors beyond just the time required to solve a CAPTCHA.

Impact of Experimental Context

One novel aspect of this paper is the examination of the impact of experimental context on CAPTCHA solving times. Participants in a contextualized setting—where solving a CAPTCHA was part of a broader activity like creating an account—showed increased solving times compared to those directly tasked with solving CAPTCHAs. This context-based difference was statistically significant, with up to a 57.5% increase in solving times noted for reCAPTCHA (easy click).

Demographic Influences

The analysis also confirmed previously reported demographic trends. Older participants exhibited higher solving times across all CAPTCHA types, consistent with prior research. However, contrary to expectations set by previous studies, the self-reported educational level did not correlate significantly with CAPTCHA solving times.

CAPTCHA-Induced Abandonment

A supplementary investigation into user abandonment revealed that the context and compensation levels considerably influence dropout rates. The contextualized settings had a 120% higher abandonment rate compared to direct tasks, emphasizing the impact of user perception and task framing on engagement.

Practical and Theoretical Implications

The implications of these findings are multifaceted:

• Website Security: The research supports the notion that the choice of CAPTCHA type can profoundly impact user experience and engagement. Faster, user-preferred CAPTCHA types like reCAPTCHA's click-based variant can enhance the user experience without compromising security.
• User-Centric Design: The preference for game-based CAPTCHAs, despite their longer solving times, suggests incorporating engaging elements can mitigate user frustration and abandonment. Balancing security requirements with user enjoyment is crucial.
• Experimental Design in Usability Studies: The pronounced impact of experimental context on solving times provides a critical guideline for designing future user studies. Researchers must ensure that paper contexts closely mimic real-world scenarios to obtain ecologically valid results.

Future Directions

The paper opens several avenues for future research. Detailed controlled studies could explore fine-grained performance metrics and user behaviors not captured in this broad analysis. Furthermore, there is an evident need for understanding the specific features of game-based and behavior-based CAPTCHAs that contribute to their higher user preference. Investigating CAPTCHA-induced abandonment across diverse web contexts and compensation models could also offer insights to further refine usability and security trade-offs.

In conclusion, this empirical investigation provides a comprehensive assessment of modern CAPTCHAs, contributing valuable insights into their usability, user preferences, and performance under varying contexts. This paper aids in bridging the gap between theoretical CAPTCHA designs and their practical implications for user experience and internet security.