GPTBot: OpenAI Web Crawler for AI Training

• GPTBot is OpenAI’s dedicated web crawler for AI training, identified by its unique user-agent in robots.txt files.
• Empirical studies reveal that GPTBot restrictions vary by website popularity, content type, and political leaning, affecting data representativeness.
• Blocking GPTBot influences downstream model training, urging researchers to adjust sampling strategies for more balanced corpora.

Searching arXiv for papers specifically about GPTBot as OpenAI’s crawler and closely related context. GPTBot is OpenAI’s dedicated web crawler for AI training, identified in practice through the GPTBot user-agent in websites’ robots.txt files. In the literature provided here, it is distinguished from other OpenAI agents with different functions, especially ChatGPT-User and OAI-SearchBot, and is analyzed primarily as a mechanism by which websites can allow or deny model-training access under the Robots Exclusion Protocol (Bouchaud et al., 10 Oct 2025). More broadly, GPT-based bots are framed elsewhere as transformer-based conversational systems fine-tuned using supervised machine learning and reinforcement learning techniques, but GPTBot in the strict sense is not a conversational interface: it is a crawler whose significance lies in how access restrictions may shape AI training corpora and, by extension, downstream model behavior (Bahrini et al., 2023).

1. Definition and scope

GPTBot is treated in the cited measurement literature as OpenAI’s crawler for “generative AI training,” operationalized entirely through the GPTBot user-agent string as it appears in robots.txt rules (Bouchaud et al., 10 Oct 2025). This distinguishes it from ChatGPT-User, which is framed as a user-initiated crawler retrieving content in response to specific user queries, and from OAI-SearchBot, which is framed as an AI search crawler (Bouchaud et al., 10 Oct 2025). The distinction is central because the meaning of a block against GPTBot is narrower than a generic block against “OpenAI”: it targets a specific acquisition channel for training data rather than all OpenAI-mediated access.

This narrow definition matters because discussions of “GPT bots” often conflate several different entities. In one sense, GPT-based bots are conversational systems built on transformer architecture and trained on massive text corpora, then fine-tuned using supervised and reinforcement learning techniques (Bahrini et al., 2023). In another, custom GPTs in the GPT Store are user-configured agents combining prompt instructions, browsing, tools, and external actions (Zhao et al., 2024). GPTBot belongs to neither category. It is a crawler-specific infrastructure component whose relevance is tied to data collection, access governance, and dataset composition (Bouchaud et al., 10 Oct 2025).

A practical implication is that a site’s treatment of GPTBot cannot be read as a complete statement about its treatment of all OpenAI systems. The 2025 crawler-restrictions study explicitly notes that blocking GPTBot does not necessarily mean exclusion from OpenAI training data, because content could still enter through other channels such as ChatGPT-User, licensed datasets, search products, or other curation pipelines (Bouchaud et al., 10 Oct 2025). This suggests that GPTBot is best understood as one data-ingestion pathway within a broader ecosystem rather than as a complete proxy for model training inputs.

2. Protocol basis and measurement

The empirical study of GPTBot is grounded in the Robots Exclusion Protocol. Restriction measurement is based on robots.txt directives at the root of a domain or subdomain, with GPTBot restrictions inferred either from explicit User-Agent: GPTBot rules or from catch-all rules that would also apply to it (Bouchaud et al., 10 Oct 2025). The same study reports that 64.1% of collected robots.txt files used only catch-all rules, which is important because many GPTBot restrictions are therefore indirect rather than crawler-specific (Bouchaud et al., 10 Oct 2025).

The paper’s unit of analysis is generally the fraction of websites whose robots.txt disallows GPTBot, usually at the domain root, rather than a URL-by-URL crawlability score (Bouchaud et al., 10 Oct 2025). It reports “fully blocked,” “disallowing,” or “restricting” GPTBot, but does not provide a formal parser specification, rule-precedence algorithm, or a separate taxonomy of full versus partial restrictions beyond those labels (Bouchaud et al., 10 Oct 2025). Accordingly, the available evidence is domain-level and descriptive rather than protocol-theoretic.

The website universe in that study is the one million most visited websites worldwide, drawn from Google’s Chrome User Experience Report (CrUX), supplemented by historical robots.txt data from CommonCrawl and by separate collections for annotated news outlets and domains with continuous ideology scores (Bouchaud et al., 10 Oct 2025). In August 2025, the CrUX top million is described as covering 95% of global Google Chrome traffic in page loads, which gives the GPTBot restriction statistics broad observational scope (Bouchaud et al., 10 Oct 2025).

3. Prevalence of restrictions

GPTBot became one of the most commonly named and blocked AI crawlers in robots.txt. Among sites with granular crawler-specific rules, GPTBot appears in 19.9% of robots.txt files, ahead of ClaudeBot at 14.7%, Google-Extended at 14.4%, Applebot-Extended at 10.2%, and meta-externalagent at 8.8% (Bouchaud et al., 10 Oct 2025). In the August/September 2025 snapshot of the top one million websites, 10.6% fully blocked GPTBot, compared with 9.1% for ClaudeBot, 8.9% for Google-Extended, 9.5% for CCBot, and 4.0% for GoogleBot (Bouchaud et al., 10 Oct 2025).

The same study shows that sites are more likely to block GPTBot than OpenAI’s user-initiated crawler. ChatGPT-User was blocked by only 6.5% of sites, which the authors interpret as evidence that website owners were more likely to deny OpenAI access for model training than for query-time retrieval (Bouchaud et al., 10 Oct 2025). This difference recurs in the news analysis as well, where training-related crawlers are blocked more often than user-facing crawlers (Bouchaud et al., 10 Oct 2025).

Historically, GPTBot blocking is presented as a post-2023 phenomenon. The paper states that GPTBot disallowance rises after OpenAI announced support for crawler controls in summer 2023, while GoogleBot’s blocking rate remains stable over 2023–2025 (Bouchaud et al., 10 Oct 2025). The published evidence is qualitative rather than a full numerical time series, but it supports the view that GPTBot restrictions are part of a recent reconfiguration of web access norms around AI training.

4. Variation by popularity, content type, and news quality

Restriction patterns for GPTBot are highly non-uniform. Across the top million websites, 10.6% block GPTBot, but among the top thousand this rises to 25.2%; the paper also reports 13.4% among the top 100K (Bouchaud et al., 10 Oct 2025). This means the most prominent sites are markedly more likely to deny access than the long tail of the web.

Content category also matters. For GPTBot, disallowance exceeds 14% in Entertainment and in News & Politics, falls to 9.0% for Education, and reaches only 4.0% for Shopping & Auctions (Bouchaud et al., 10 Oct 2025). In the abstract’s simplified formulation, 34.2% of news outlets disallow GPTBot, whereas only 4% of shopping websites do (Bouchaud et al., 10 Oct 2025).

The news-outlet analysis is especially consequential because it connects GPTBot restrictions to source quality. Using 3,668 outlets labeled by Media Bias/Fact Check, the study finds that 34.2% of media outlets disallowed GPTBot overall, but 55.4% of “high factual reporting” outlets blocked it, compared with 8.4% of “mixed factual reporting” outlets and 3.7% of “low factual reporting” outlets (Bouchaud et al., 10 Oct 2025). This suggests that GPTBot is disproportionately excluded from sources many readers would regard as high-value for factual training data.

Slice	GPTBot blocked
Top 1M websites	10.6%
Top 1K websites	25.2%
News outlets overall	34.2%
High factual reporting news	55.4%
Shopping & Auctions	4.0%

These asymmetries matter because they imply that “access to the web” is not equivalent to access to a representative sample of highly visible or highly factual content. The paper’s interpretation is that GPTBot can still reach most of the top million websites, but its exclusions are concentrated in parts of the web that are especially salient for factual and balanced training data (Bouchaud et al., 10 Oct 2025).

5. Political asymmetry and downstream implications

The most distinctive finding in the GPTBot literature summarized here is the political asymmetry of blocking. In the MBFC-annotated news sample, 58.0% of “neutral” outlets block GPTBot, compared with 19.6% of left-leaning outlets and only 4.1% of right-leaning outlets (Bouchaud et al., 10 Oct 2025). In the broader ideological dataset using Robertson et al.’s continuous audience-ideology scores, 18.5% of politically balanced domains block GPTBot, versus 7.3% of hyperpartisan left domains and 7.0% of hyperpartisan right domains (Bouchaud et al., 10 Oct 2025). The paper models this relationship with a quadratic logistic regression and shows it as a U-shaped curve (Bouchaud et al., 10 Oct 2025).

A temporal comparison reinforces the interpretation that this is not merely an old feature of web crawling. Before GPTBot existed, in September 2022, disallowance of CCBot did not differ meaningfully between balanced and skewed domains—2.6% versus 3.0% (Bouchaud et al., 10 Oct 2025). The later center-skewed blocking pattern is therefore presented as associated with the rise of AI-training-specific restrictions.

The same paper links these asymmetries to measured shifts in downstream corpora, while explicitly avoiding a clean causal claim. In FineWeb, the share of tokens from MBFC sources labeled “high factual reporting” fell from 0.46% to 0.26%, a 41.3% relative decrease, while hyperpartisan right content increased from 19.3% to 24.8% and hyperpartisan left content rose from 18.9% to 22.4% (Bouchaud et al., 10 Oct 2025). Elsewhere in the same discussion, the paper summarizes that between pre-2023 and post-2023 periods, representation of hyperpartisan domains increased by 54.6% on the left and 43.5% on the right, while balanced domains decreased by 11.3%, “without inferring causation” (Bouchaud et al., 10 Oct 2025).

This suggests that GPTBot restrictions are not merely an access-control curiosity. A plausible implication is that heterogeneous blocking patterns can alter the epistemic and ideological profile of web-derived training corpora, especially if model builders rely heavily on GPTBot-like crawling and do not actively compensate for selectivity (Bouchaud et al., 10 Oct 2025).

6. Relation to GPT-based systems and applications

Although GPTBot is a crawler rather than a conversational agent, its significance is inseparable from the broader GPT ecosystem. GPT-based systems are described in the literature as transformer-based LLMs trained on very large text corpora and adapted through supervised fine-tuning and reinforcement learning, with applications across business, education, science and technology, government and politics, healthcare and medicine, communication, arts and culture, and lifestyle (Bahrini et al., 2023). In software engineering, such systems can perform credibly on some language-heavy tasks while failing on deeper semantic or optimization-heavy tasks (Sridhara et al., 2023). In robotics, GPT-like systems are useful as planners or demonstration generators but are too unstable to sit directly in the execution loop (Jin et al., 2023). These accounts establish why training data composition matters: GPT-derived models are widely deployed, but their behavior is sensitive to corpus quality, domain coverage, and alignment procedures.

The crawler-restriction study therefore treats GPTBot as one important upstream mechanism shaping the data available for such models (Bouchaud et al., 10 Oct 2025). That role differs sharply from the GPT Store ecosystem, where “custom GPTs” are configured through prompt instructions, reference resources, browsing, data analysis, image generation, and external actions (Zhao et al., 2024). It also differs from security studies of custom GPT agents, which focus on expert prompts, chat history, tools, and uploaded knowledge as an attack surface for information leakage and tool misuse (Wu et al., 28 Nov 2025). GPTBot is neither a custom agent nor a tool-using assistant; its encyclopedic relevance lies in how its access is governed and what that implies for model training.

The distinction is also important for public discussion. The paper on custom GPT storefronts explicitly notes that it is not about OpenAI’s web crawler literally named GPTBot, but about GPT-based bots in the GPT Store ecosystem (Zhao et al., 2024). A careful treatment therefore reserves “GPTBot” for the crawler unless context clearly indicates a looser reference to GPT-based bots in general.

7. Limitations, misconceptions, and governance significance

A common misconception is that blocking GPTBot is equivalent to excluding a site from OpenAI training data. The available evidence does not support that stronger claim. The crawler-restriction paper emphasizes that robots.txt governs one data acquisition mechanism, not the totality of training inputs, and that content from a blocked site could still appear via licensed datasets, search products, other curation pipelines, or user-mediated channels such as ChatGPT-User (Bouchaud et al., 10 Oct 2025).

Another misconception is that robots.txt fully captures the legal or normative boundary of acceptable use. The same paper notes that robots.txt is only one layer of access control and cites prior work showing that if natural-language Terms of Service are considered, the share of restricted content in CommonCrawl-derived corpora can be far larger—45% in C4 versus only 5% when judged by robots.txt alone (Bouchaud et al., 10 Oct 2025). GPTBot blocking rates therefore likely understate the broader level of contested or restricted content.

The policy significance of GPTBot follows from this dual incompleteness. On one side, GPTBot restrictions are meaningful because the paper cites external work showing that OpenAI, Anthropic, Meta, and CommonCrawl respected robots.txt as stated (Bouchaud et al., 10 Oct 2025). On the other, those restrictions neither exhaust the relevant governance surface nor determine total training inclusion. The resulting picture is not one of simple exclusion but of selective friction.

The paper’s mitigation suggestions are correspondingly modest. Model builders should not rely on passive collection plus standard cleaning heuristics alone; they should actively measure the distribution of crawling allowances and adjust sampling or curation strategies to preserve representativeness (Bouchaud et al., 10 Oct 2025). This is less a solution than a design requirement. If GPTBot is systematically barred from highly factual, highly popular, and politically balanced sources, then any web-derived training corpus assembled through that pathway must be audited for compositional skew rather than treated as a neutral snapshot of the web (Bouchaud et al., 10 Oct 2025).