On the Internet, Nobody Knows You're an LLM Bot: Unmasking Web Agents with Multi-Layer Fingerprinting

Abstract: Since 2023, a new class of bots has emerged: Web Agents. They can automate complex tasks on the Web, going beyond traditional browser automation tools such as Selenium, Puppeteer, or Playwright. Leveraging LLMs, these agents are capable of solving anti-bot mechanisms, mimicking human behavior, and, in some cases, operating directly from the local machine of the user configuring them. As a result, it is becoming increasingly difficult for website administrators to detect and block these LLM-based bots. Modern Web Agents commonly integrate stealth and anti-detection techniques, while numerous proprietary and open-source anti-bot mechanisms have emerged recently, specifically to block them. However, despite their growing prevalence, there is little evaluation of the effectiveness of state-of-the-art anti-bot mechanisms against these LLM-based bots and their stealth capabilities. Likewise, no prior work has comprehensively studied how to characterize and distinguish Web Agents deployed either in the cloud or locally. This paper addresses these open questions by deploying multiple honeysites protected by one or more anti-bot mechanisms (e.g., robots.txt, CAPTCHAs, proof-of-work, and Cloudflare's free proprietary solutions). We integrated network-, HTTP-, and browser-level fingerprinting techniques, and prompted six LLM-based Web Agents to visit the deployed honeysites. Our analysis reveals three main findings: (i) some Web Agents were able to bypass all evaluated anti-bot mechanisms; (ii) all evaluated Web Agents can be distinguished both from humans and from one another using multi-layer fingerprinting techniques across network, HTTP and browser layers; (iii) stealth and anti-detection mechanisms often increase detectability rather than decrease it.