Top Websites Blocking AI Crawlers

## What this measures

The share of the 10,000 most-linked websites whose robots.txt file tells at least one named AI crawler (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and others) to stay off the whole site.

## Why it matters

It is the open web's answer to AI: how much of it is closing the door to AI training and AI search. A rising share means less freely crawlable material for AI companies and a stronger hand for publishers in licensing talks.

## How to read it

Read the trend. The per-crawler series show which companies the web is refusing most; the top-1,000 series shows whether the largest sites lead or lag.

## How it is counted

A site counts as blocking a crawler when its robots.txt names that crawler in a User-agent line and disallows the whole site ("/") for it, with no Allow for the root. A blanket rule that shuts out every crawler (User-agent: * / Disallow: /) is not counted, because it blocks search engines too and is a different decision; path-level rules don't count either. The denominator is every site whose robots.txt we could read plus sites with no robots.txt (by the robots standard, RFC 9309, a missing file allows everything). Sites that time out or refuse our request are left out; coverage is recorded with each reading.

## What it does not say

robots.txt is a request, not a wall: it records what a site asks of crawlers, not whether crawlers obey, and it says nothing about paid licensing deals or technical blocking at the network level. The sample is ranked by links, not traffic, and is weighted toward large, established sites. The survey begins in October 2026. ## Source

Palanor robots.txt survey. Once a month Palanor reads the public robots.txt file of each of the 10,000 most-linked websites in the Majestic Million (majestic.com, licensed CC BY 3.0), using its own crawler name (PalanorBot). Palanor computes and publishes the shares; the site list and the files are not republished.