Palanor research · AI crawler survey · October 2026
One in five of the web’s top sites tells AI crawlers to keep out
In October 2026 Palanor read the robots.txt file of the 10,000 most-linked websites. Of the 7,742 we could read, 1,491 (19.3%) tell at least one named AI crawler to stay off the whole site. Among the top 1,000 the share is 23.8%.
Top 10,000
19.3%
block an AI crawler
Top 1,000
23.8%
block an AI crawler
Most blocked
CCBot
13.5% of sites
State a preference
0.3%
publish AI-use terms
By crawler
Common Crawl CCBot and OpenAI GPTBot are turned away most
Share of readable sites whose robots.txt disallows each crawler across the whole site. Blockers rarely stop at one: the median site that blocks any of these blocks 6 of the 8.
| Crawler | Bar | Share | Sites |
|---|---|---|---|
| Common Crawl CCBot | 13.5% | ||
| OpenAI GPTBot | 13.3% | ||
| ByteDance Bytespider | 13.0% | ||
| Anthropic ClaudeBot | 12.6% | ||
| Meta-ExternalAgent | 11.5% | ||
| Google-Extended | 10.0% | ||
| Applebot-Extended | 9.6% | ||
| PerplexityBot | 8.3% |
By rank
The top 1,000 block more often: 23.8%, against 16.9% for sites ranked 5,001–10,000
| Rank | Bar | Share | Sites |
|---|---|---|---|
| Top 100 | 21.6% | ||
| 101–1,000 | 24.0% | ||
| 1,001–5,000 | 20.8% | ||
| 5,001–10,000 | 16.9% |
By domain ending
.fr sites block most; universities and governments rarely do
Endings with at least 80 readable sites. .fr leads at 35.8%; .cn is lowest at 1.1%, with .edu at 4.5% and .gov at 6.2%.
| Ending | Bar | Share | Sites |
|---|---|---|---|
| .fr | 35.8% | ||
| .de | 25.4% | ||
| .com | 24.1% | ||
| .uk | 23.1% | ||
| .net | 15.9% | ||
| .org | 13.0% | ||
| .jp | 12.9% | ||
| .gov | 6.2% | ||
| .ru | 6.1% | ||
| .edu | 4.5% | ||
| .io | 4.2% | ||
| .cn | 1.1% |
Who
Some of the names
amazon.com, tiktok.com, yahoo.com, nytimes.com, cnn.com and bbc.co.uk all block GPTBot, ClaudeBot and Google-Extended in robots.txt (checked October 8, 2026). Sites change these files often; the download below has every site as read in October 2026.
Every month
Get the next reading, and the site list
The survey runs on the 2nd of every month. The Read, Palanor’s free Monday email, carries the new numbers when they land. Join and you can download all 10,000 sites with what each one blocks.
Method
How the survey works
Sites. The top 10,000 domains of the Majestic Million, a list ranked by the number of other sites linking in (Majestic, CC BY 3.0). Read in October 2026: 10,000 sites, 7,742 readable (77.4%), 735 refused the request and 1,523 could not be reached. A site with no robots.txt (470 of them) counts as readable and blocking nothing.
What counts as a block. A group in robots.txt that names the crawler and disallows the whole site (Disallow: /). A site that blocks every crawler with User-agent: * (186 sites) is not counted, because that is not a choice about AI. We track OpenAI GPTBot, Anthropic ClaudeBot, Google-Extended, Common Crawl CCBot, PerplexityBot, Applebot-Extended, ByteDance Bytespider, Meta-ExternalAgent; a dozen other AI agents (OAI-SearchBot, ChatGPT-User, cohere-ai, Amazonbot and others) count toward “at least one”.
Preferences. “Publish AI-use terms” means a Content-Usage line (the IETF AI preferences draft) or an RSL License line in robots.txt.
What it can’t see. robots.txt is a request, not a lock. Many sites block crawlers at their firewall or CDN instead, so the true share that keeps AI out is higher than this count. Each figure is also a series in the Palanor lattice, starting with Top Websites Blocking AI Crawlers.
Cite as: Palanor AI crawler survey, October 2026. palanor.com/research/ai-crawlers