One in five websites now blocks AI.

Every website can publish a small file, robots.txt, that tells automated crawlers what they may read. It's a request, not a lock, but the major AI companies say their crawlers honour it. That makes it the clearest public record of who wants their work kept away from AI.

In October we read that file for the top 10,000 sites on the web, ranked by the Majestic Million. 7,742 answered with a file we could read, or with no file at all, which means nothing is blocked. Of those, 19.3% block at least one named AI crawler across the whole site. Among the top 1,000 sites, it's 23.8%.

Who gets blocked

The most-blocked crawler is CCBot, from Common Crawl, the non-profit whose open archive went into the training data of most large language models: 13.5% of sites block it. OpenAI's GPTBot is close behind at 13.3%, then ByteDance's Bytespider (13.0%) and Anthropic's ClaudeBot (12.6%). PerplexityBot is the least blocked of the eight we track, at 8.3%.

Two of the eight aren't crawlers in the usual sense. Google-Extended (10.0%) and Applebot-Extended (9.6%) are switches: blocking them keeps a site out of Google's and Apple's AI work while still letting it appear in their search results. One site in ten has flipped them. Plenty of publishers want to stay findable without being trained on.

Who blocks

Bigger sites are likelier to block. 24.0% of sites ranked 101 to 1,000 do, against 16.9% of those ranked 5,001 to 10,000. The very top 100 sits at 21.6%.

Where a site is registered matters even more. 35.8% of .fr sites block an AI crawler, the highest of any domain ending with enough sites to measure. German (.de, 25.4%) and British (.uk, 23.1%) sites are also above the average, as is .com (24.1%).

At the other end, universities (.edu, 4.5%) and governments (.gov, 6.2%) almost never block. Their job is to be read, and most of what they publish is public by design. Chinese sites (.cn, 1.1%) barely block either.

What robots.txt doesn't show

This is a floor, not a full count. Many sites block AI in other ways: at their firewall, through services like Cloudflare, or behind logins and paywalls. robots.txt only shows what a site is willing to say in public. Another 186 sites block every crawler, AI or not, with a blanket rule; we don't count a blanket rule as blocking AI.

It also doesn't say what a site wants AI to do with its content. Two newer standards let a site say that in a machine-readable way, Content-Usage and the Really Simple Licensing tag. Fewer than one site in 300 uses either. For now, the web's answer to AI is mostly yes or no.

Why it matters

The open web is closing, one door at a time.

Every AI answer is built from what crawlers could read. If the sites with the best-edited, most current writing keep opting out, models learn more from what's left. That's not a prediction about quality. It's a question worth measuring, so we'll re-run this survey on the 2nd of every month and publish the trend.

For a business it's now a real decision with two sides. Blocking protects work you might license. Not blocking keeps you in the answers your customers increasingly read instead of your site. Either is defensible. Not deciding is the only bad choice.

The full results, including a table of every site we read, are at palanor.com/research/ai-crawlers. The headline figures are live signals: share blocking any AI crawler and among the top 1,000.

Method. A site counts as blocking when its robots.txt has a group naming the crawler that disallows the whole site. Sites that refused the request (401, 403, 429) or didn't answer are left out; sites with no file count as not blocking.